Video Instance Segmentation via Multi-scale Spatio-temporal Split Attention Transformer
MS-STS Attention Transformer
Omkar Thawakar^1, Sanath Narayan^2, Jiale Cao^3 , Hisham Cholakkal^1,
Rao Muhammad Anwer^1, Muhammad Haris Khan^1, Salman Khan^1,
Michael Felsberg^4, Fahad Shahbaz Khan^1,4
O. Thawakar et al.
$^1$Mohamed bin Zayed University of Artificial Intelligence, UAE

$^2$Inception Institute of Artificial Intelligence, UAE

$^3$Tianjin University, China     $^4$Linköping University, Sweden

State-of-the-art transformer-based video instance segmentation (VIS) approaches typically utilize either single-scale spatio-temporal features or per-frame multi-scale features during the attention computations. We argue that such an attention computation ignores the multi-scale spatio-temporal feature relationships that are crucial to tackle target appearance deformations in videos. To address this issue, we propose a transformer-based VIS framework, named MS-STS VIS, that comprises a novel multi-scale spatio-temporal split (MS-STS) attention module in the encoder. The proposed MS-STS module effectively captures spatio-temporal feature relationships at multiple scales across frames in a video. We further introduce an attention block in the decoder to enhance the temporal consistency of the detected instances in different frames of a video. Moreover, an auxiliary discriminator is introduced during training to ensure better
foreground-background separability within the multi-scale spatio-temporal feature space. We conduct extensive experiments on two benchmarks: Youtube-VIS (2019 and 2021). Our MS-STS VIS achieves state-of-the-art performance on both benchmarks. When using the ResNet50 backbone, our MS-STS achieves a mask AP of 50.1%, outperforming the best reported results in literature by 2.7% and by 4.8% at higher overlap threshold of AP$_{\mathtt{75}}$, while being comparable in model size and speed on Youtube-VIS 2019 val. set. When using the Swin Transformer backbone, MS-STS VIS achieves mask AP of 61.0% on Youtube-VIS 2019 val. set. Our code and models are available at <https://github.com/OmkarThawakar/MSSTS-VIS>.

§ INTRODUCTION

Video instance segmentation (VIS) is a challenging computer vision problem with numerous real-world applications, including intelligent video analysis and autonomous driving. Given a video sequence, the task is to simultaneously segment and track all object instances from a set of semantic categories. The problem is particularly challenging since the target object needs to be accurately segmented and tracked despite appearance deformations due to several real-world issues such
as, target size variation, aspect-ratio change and fast motion.

axis lines = left,
ymin =35, ymax= 100,
xmin =30, xmax = 52,
xlabel= Accuracy (AP),
ylabel= Parameters (M),
(legend) at (axis description cs:0.95,0.55);

[only marks,
mark=otimes*, purple,
mark size=3.5pt
(31.1, 58.1);

[only marks,
mark=otimes*,  gray,
mark size=3.5pt
(30.6, 50.5);

[only marks,
mark=otimes*, orange,
mark size=3.5pt
(40.4, 69.0);

[only marks,
mark=otimes*, green,
mark size=3.5pt
(36.1, 40);

[only marks,
mark=otimes*, pink,
mark size=3.5pt
(34.8, 37.5);

[only marks,
mark=otimes*, violet,
mark size=3.5pt
(36.2, 57.2);

[only marks,
mark=otimes*, blue,
mark size=3.5pt
(42.8, 40);

[only marks,
mark=otimes*, cyan,
mark size=3.5pt
(47.4, 49.3);

[only marks,
mark=triangle*, magenta,
mark size=5pt
(50.1, 50.6);

[draw=none,fill=none, anchor= south east] at
Method     AP    AP$_{\mathtt{50}}$     AP$_{\mathtt{75}}$     Params(M)

<ref> MaskTrack-RCNN [6]      30.3     51.1     32.6     58.1

<ref> STEmSeg [1]      30.6     50.7      33.5     50.5

<ref> PCAN [11]      36.1     54.9      39.4     40.0

<ref> Cross-VIS [28]      34.8     56.8      38.0     37.5

<ref> VisTR [22]      36.2     59.8      36.9     57.2

<ref> Propose-Reduce [15]      40.4    63.0      43.8     69.0

<ref> IFC [8]      42.8     65.8      46.8     39.3

<ref> SeqFormer [24]      47.4     69.8     51.8     49.3

<ref> MS-STS VIS       50.1     73.2      56.6     50.6

On the left: Qualitative comparison of our MS-STS VIS with the baseline in the case of target appearance deformations on two example videos. For each method, four sample video frames are shown along the columns. The baseline struggles to accurately predict the rabbit class instance undergoing appearance deformations due to scale variation, aspect-ratio change and fast motion. As a result, the predicted mask quality is hampered (marked by white dotted region). Similarly, the mask quality is also deteriorated for the mouse class. Our MS-STS VIS addresses these issues by capturing multi-scale spatio-temporal feature relationships, leading to improved mask quality. Best viewed zoomed in. On the right: Accuracy (AP)  model size (params) comparison with existing methods using a single model and single-scale inference on YouTube-VIS 2019 val. set. We also report performance at AP$_{\mathtt{50}}$ and AP$_{\mathtt{75}}$. All methods here utilize the same ResNet50 backbone. Compared to the best reported results in literature [24], our MS-STS VIS achieves absolute gains of 2.7% and 4.8% in terms of overall mask AP and at higher overlap threshold of AP$_{\mathtt{75}}$, respectively, while being comparable in model size and speed ([24]: 11 FPS  Ours: 10 FPS).

Recently, transformers [20] have shown promising results on several vision tasks, including VIS [22, 24]. The recent transformer-based VIS approaches [22, 24] are built on DETR [4] and Deformable DETR [29] frameworks, utilizing an encoder-decoder architecture along with instance sequence matching and segmentation mechanisms to generate final video mask predictions. These approaches typically employ either single-scale spatio-temporal features [22] or per-frame multi-scale features [24] during attention computations at the encoder and decoder. However, such an attention computation ignores the multi-scale spatio-temporal feature relationships, which are crucial towards handling target appearance deformations due to real-world challenges such as, scale variation, change in aspect-ratio and fast motion in videos.

In this work, we investigate the problem of designing an attention mechanism, within the transformer-based VIS framework, to effectively capture multi-scale spatio-temporal feature relationships in a video.

With this aim, we introduce a multi-scale spatio-temporal attention mechanism, which learns to aggregate the necessary attentions performed along the spatial and temporal axes without losing crucial information related to target appearance deformations in both the spatial and temporal axes.

In addition to target appearance deformations, another major challenge in VIS is the accurate delineation of the target object in the presence of cluttered background. Surprisingly, existing transformer-based VIS approaches do not employ an explicit mechanism to enforce foreground-background (fg-bg) separability. Here, we introduce a loss formulation that improves fg-bg separability by emphasizing the fg regions in multi-scale spatio-temporal features while simultaneously suppressing the bg regions.

Contributions: We propose a transformer-based video instance segmentation framework, MS-STS VIS, with the following contributions.

* We propose a novel multi-scale spatio-temporal split (MS-STS) attention module in the transformer encoder for effectively capturing spatio-temporal feature relationships. Our MS-STS module first attends to features across frames at a given spatial scale via an intra-scale temporal attention block, and then progressively attends to neighboring spatial scales across frames via an inter-scale temporal attention block to obtain enriched feature representations. To further improve the video mask prediction, we introduce an attention block in the decoder that enhances the temporal consistency of the detected instances in different frames of a video.

* We introduce an auxiliary discriminator network during training to enhance the fg-bg separability within the multi-scale spatio-temporal feature space. Here, the discriminator network is trained to distinguish between ground-truth and predicted mask, while the encoder is learned to fool the discriminator by generating features that result in better mask predictions.

* Comprehensive experiments are performed on Youtube-VIS 2019 and 2021 datasets. Our proposed MS-STS VIS sets a new state-of-the-art on both datasets. When using the ResNet50 backbone, our MS-STS VIS outperforms all existing methods with an overall mask AP of 50.1% on Youtube-VIS 2019 val. set, while being on par in terms of model size and speed compared to the state-of-the-art method (see Fig. <ref> (right)). Specifically, the proposed MS-STS VIS achieves a significant performance improvement over the baseline, in case of target appearance deformations due to scale variation, aspect-ratio change and fast motion (see Fig. <ref> (left)).

§ BASELINE FRAMEWORK

We base our approach on the recently introduced SeqFormer [24]. SeqFormer is built on the Deformable DETR [29] framework, which comprises a CNN backbone followed by a transformer encoder-decoder with deformable attention. We choose SeqFormer as our base framework to demonstrate the impact of our proposed contributions on a strong baseline. Here, a video clip $\mathbf{x}$ consisting $T$ frames of spatial size $H^0\times W^0$ with a set of object instances is input to the backbone.  Latent feature maps for each frame are obtained from the backbone at $L$ multiple scales $2^{-l-2}$ with $1\leq l\leq L$ and passed through separate convolution filters for each scale. The output feature dimension for each of these convolution filters is set to $C$. The resulting multi-scale feature maps of each frame are then input to a transformer encoder comprising multi-scale deformable attention blocks. For each frame, the transformer encoder outputs multi-scale feature maps with the same size as its input. These encoder output features maps from each frame along with $n$ learnable instance query embeddings $\mathbf{I}^Q \in \mathbb{R}^C$ are then input to the transformer decoder comprising a series of self- and cross-attention blocks. The $n$ instance queries are further decomposed into $n$ box queries $\mathbf{B}^Q$ per-frame and are used to query the box features from the encoder feature maps of the corresponding frame. The learned box queries across $T$ frames are then aggregated temporally to obtain $n$ instance features $\mathbf{I}^O \in \mathbb{R}^C$. These instance features output by the decoder are then used for video instance mask prediction. We refer to [24] for additional details.

Limitation: As discussed above, the aforementioned SeqFormer framework independently utilizes per-frame multi-scale features during attention computations. As a result, it ignores the spatio-temporal feature relationships during attention computation that is crucial for the VIS problem. Different from SeqFormer that utilizes per-frame spatial features at multiple scales, our approach performs multi-scale spatio-temporal attention computation. Such a multi-scale spatio-temporal attention is especially desired in cases when the target object undergoes appearance deformations due to real-world challenges such as, scale variation, aspect-ratio change and fast motion in videos (see Fig. <ref> and <ref>). Furthermore, distinct from the baseline SeqFormer, our approach employs an explicit mechanism to ensure accurate delineation of foreground objects from the cluttered background by enhancing fg-bg separability.

§ METHOD

(a) Our MS-STS VIS architecture comprises a backbone, a transformer encoder-decoder and a instance matching and segmentation block. Here, our key contributions are: (i) a novel MS-STS attention module in the encoder to capture spatio-temporal feature relationships at multiple scales across frames, (ii) a temporal attention block in the decoder for enhancing the temporal consistency of the box queries and (iii) an adversarial loss for enhancing foreground-background (fg-bg) separability. Specifically, the MS-STS attention module (b) attends to the input backbone features by employing intra- and inter-scale temporal attentions to generate enriched features which are then fused with standard baseline features. While the intra-scale temporal attention block (c) enriches features across frames within a given scale, its inter-scale counterpart (d) progressively attends to multiple spatial scales across frames to obtain spatio-temporally enriched features. These enriched features are further improved using an adversarial loss (e) to enhance fg-bg separability. The resulting encoder features along with the temporally consistent instance features from the  decoder are used within the matching and segmentation block for the video instance mask prediction.

§.§ Overall Architecture

Fig. <ref>(a) shows the overall architecture of the proposed MS-STS VIS approach built on the baseline framework described above. Our MS-STS VIS comprises a backbone network, a transformer encoder-decoder and a sequence matching and segmentation block. The focus of our design is the introduction of a novel multi-scale spatio-temporal split (MS-STS) attention module (Fig. <ref>(b)) in the transformer encoder to effectively capture spatio-temporal feature relationships at multiple scales across frames in a video. The MS-STS attention module comprises intra- and inter-scale temporal attention blocks (Fig. <ref>(c) and (d)). The intra-scale block takes $C$-dimensional backbone features $\mathbf{F}$ as input and enriches features across frames in a video within a given scale, whereas the inter-scale block then progressively attends to multiple spatial scales across frames to generate spatio-temporally enriched features $C$-dimensional $\hat{\mathbf{Z}}$. These features $\hat{\mathbf{Z}}$ are fused in each encoder layer with the base features output by the standard deformable attention. While backbone features $\mathbf{F}$ are used as input to the first encoder layer, the subsequent layers utilize the outputs from the preceding layer as input. As a result, multi-scale spatio-temporally enriched features $C$-dimensional $\mathbf{E}$ are output by the encoder.

Afterwards, these enriched features $\mathbf{E}$ from the encoder are input to the transformer decoder. To achieve temporal consistency among box queries from different frames, we introduce a temporal attention block within the transformer decoder. Next, the encoder features $\mathbf{E}$ along with the instance features $\mathbf{I}^O$ (aggregated temporally attended box queries) from the decoder are utilized within the instance matching and segmentation block to obtain the video instance mask prediction. To further improve the predicted video instance mask quality, we introduce an adversarial loss  during training to enhance foreground-background (fg-bg) separability. The adversarial loss (Fig. <ref>(e)) strives to enhance the encoder features by discriminating between the predicted and ground-truth masks, utilizing the encoder features $\mathbf{E}$, the input frames $\mathbf{x}$ and the binary object mask $\mathbf{M}$. Next, we describe our MS-STS attention module-based encoder.

§.§ MS-STS Attention Module Based Encoder

Within the proposed MS-STS VIS framework, we introduce a novel multi-scale spatio-temporal split (MS-STS) attention module in the transformer encoder to effectively capture spatio-temporal feature relationships at multiple scales across frames. To this end, the MS-STS module takes the backbone features as input and produces multi-scale spatio-temporally enriched features, which are then fused with the standard features within the base framework. The MS-STS module (see Fig. <ref>(b)) comprises an intra-scale and an inter-scale attention block (see Fig. <ref>(c) and (d)) described next.

Intra-scale Temporal Attention Block:

Given the backbone features as input, our intra-scale block independently attends to each scale (spatial resolution) temporally across frames (see Fig. <ref>(c)). Let $\mathbf{z}_{s,t}^{l}$ be the feature at spatial scale $l$, position $s$ and frame $t$, where $s\in [0,S^l-1]$, $t\in [0,T-1]$, and $S^l=H^l \cdot W^l$. The intra-scale temporal attention block processes the features $\mathbf{Z}^{l} \in \mathbb{R}^{S^l \times T\times C}$ using intra-scale self-attention ($\text{SA}_{\text{intra}}$), layer normalization (LN), and MLP layers as:

\begin{equation}
\mathbf{Y}^{l} = \mathrm{SA_{intra}}(\mathrm{LN}(\mathbf{Z}^{l})) + \mathbf{Z}^{l},  \qquad
\mathbf{\tilde{Z}}^{l} = \mathrm{MLP}(\mathrm{LN}(\mathbf{Y}^{l})) + \mathbf{Y}^{l},
\end{equation}

\begin{equation}
\text{where} \quad \mathrm{SA_{intra}}(\mathrm{LN}(\mathbf{z}^{l}_{s,t})) =  \sum_{\acute{t}=0}^{T-1} \mathrm{Softmax}\left(\frac{\mathbf{q}^{l}_{s,t}\cdot\mathbf{k}^{l}_{s,\acute{t}}}{\sqrt{C}}\right)\mathbf{v}^{l}_{s,\acute{t}},
\end{equation}

where $\mathbf{q}^{l}_{s,t},\mathbf{k}^{l}_{s,t}, \mathbf{v}^{l}_{s,t} \in \mathbb{R}^{D}$ are the query, key, and value vectors obtained from $\mathbf{z}_{s,t}^{l}$ (after LN) using the embedding matrices $\mathbf{W}_{q}, \mathbf{W}_{k},\mathbf{W}_{v} \in \mathbb{R}^{C \times C}$. The intra-scale temporal attention operates on each spatial scale $l$ across frames and produces temporally relevant intermediate features $\{\mathbf{\tilde{Z}}^{l}\}_{l=1}^{L}$.

Inter-scale Temporal Attention Block:

The inter-scale temporal attention block takes the intermediate features output by the intra-scale block and aims to learn similarities between the spatio-temporal features across two neighbouring spatial scales (see Fig. <ref>(e)).
Let $\mathbf{\tilde{Z}}^{l} \in \mathbb{R}^{S^l \times T \times C}$ and $\mathbf{\tilde{Z}}^{l+1}$ be the intra-scale attended features at spatial scales $l$ and $l+1$.
To compute inter-scale temporal attention between the two neighbouring scale features, we first upsample the lower resolution features by $\times$2 using bilinear interpolation, concatenate along the feature dimensions and project it into $C$ dimensions using $\mathbf{W}_{p} \in \mathbb{R}^{2C \times C}$ as:

\begin{equation}
\mathbf{H}^{l} = \mathbf{W}_{p}(\mathrm{CONCAT}(\mathbf{\tilde{Z}}^{l}, \mathrm{UPSAMPLE}(\mathbf{\tilde{Z}}^{l+1})).
\end{equation}

We reshape $\mathbf{H}^{l} \in \mathbb{R}^{S^{l} \times T \times C}$ to $\mathbb{R}^{(S^{l} \cdot T) \times C}$ and then compute the joint spatio-temporal attention to obtain enriched features $\mathbf{\hat{Z}}^l$ given by

\begin{equation}
\mathbf{Y}^{l} = \mathrm{SA_{inter}}(\mathrm{LN}(\mathbf{H}^{l})) + \mathbf{H}^{l}, \qquad \mathbf{\hat{Z}}^{l} = \mathrm{MLP}(\mathrm{LN}(\mathbf{Y}^{l})) + \mathbf{Y}^{l},
\end{equation}

where the inter-scale self-attention ($\text{SA}_{\text{inter}}$) computation is given by

\begin{equation}
\mathrm{SA_{inter}} = \sum_{\acute{s}=0}^{S-1} \sum_{\acute{t}=0}^{T-1} \mathrm{Softmax}\left(\frac{\mathbf{q}^{l}_{s,t}\cdot\mathbf{k}^{l}_{\acute{s},\acute{t}}}{\sqrt{C}}\right)\mathbf{v}^{l}_{\acute{s},\acute{t}}.
\end{equation}

Example attention map visualizations obtained at the output of the baseline encoder and our MS-STS attention module-based encoder. Here, we show the comparison on example videos from Youtube-VIS 2019 val. set. The baseline struggles to accurately focus on the car instance undergoing significant scale variation, where the size becomes extremely small in the later frames. Similarly, it fails to accurately focus on the shark instance undergoing aspect-ratio change and the person instance partially visible only in the first two frames of the middle video. In the last video, the baseline inaccurately highlights the irrelevant object (in orange) occluding the target panda instance.
Our MS-STS attention module-based encoder, which strives to capture spatio-temporal feature relationships at multiple scales, successfully focuses on these challenging targets despite scale variations (car), aspect-ratio change (shark), partial visibility (person) and appearance deformations due to occlusion (panda). Best viewed zoomed in.

To summarize, our MS-STS attention module utilizes an intra-scale temporal attention block to first attend to features across frames at a given spatial scale. It then employs an inter-scale temporal attention block to progressively attend to neighboring spatial scales across frames for obtaining enriched feature representations  $\mathbf{\hat{Z}}^l$. The resulting features $\mathbf{\hat{Z}}^l$ are fused with the standard baseline features through a convolution operation. Finally, the encoder outputs enriched multi-scale spatio-temporal features $\mathbf{E}$ after processing the input features through $N_d$ attention layers. Fig. <ref> shows example attention maps obtained at the output of our MS-STS attention module-based encoder. The attention maps are computed as the average activation strength across the $C$ features at a particular spatial position for the largest scale considered, , $\nicefrac{H}{8} \times \nicefrac{W}{8}$. The attention maps are shown for example videos from Youtube-VIS 2019 val. set depicting target appearance deformations. We also compare our encoder output to that of the baseline encoder. Compared to the baseline, our MS-STS module-based encoder better focuses on the car instance (left video) undergoing scale variations, the shark instance (middle video) undergoing aspect-ratio change, the person instance (middle video) partially visible only in the first two frames and the panda instance (right video) exhibiting target deformations due to the irrelevant
object (in orange) occluding the target panda instance.

§.§ Enhancing Temporal Consistency in Decoder

As discussed earlier, the transformer decoder in the base framework comprises a series (layers) of alternating self- and cross-attention blocks, operating on the box queries $\mathbf{B}^Q$ of individual frames. Although the video-level instance features $\mathbf{I}^O$ are obtained through an aggregation of the corresponding frame-level box features, the temporal consistency of the detected instances is hampered likely due to the per-frame attention computation of the box queries. To alleviate this issue, we introduce a temporal attention block in between the self- and cross-attention blocks of a decoder layer. Our temporal attention block attends to the sequence of box queries $\{\mathbf{B}_i^Q\}_{i=1}^{T}$ from $T$ frames and produces temporally consistent queries at its output. Such a temporal attention between the per-frame box queries of an instance enables information flow between the probable box locations across frames, thereby improving the temporal consistency of the video mask predictions. The resulting temporally-attended box queries are then utilized to query the multi-scale spatio-temporally enriched features $\mathbf{E}$, output by the encoder, for computing the box features $\mathbf{B}^O$ in a per-frame manner. Afterwards, these box features are aggregated to obtain video-level instance features $\mathbf{I}^O$. Finally, the resulting instance features along with the box features and multi-scale spatio-temporally enriched features $\mathbf{E}$ are input to an instance matching and segmentation block for generating the video mask predictions.

§.§ Enhancing Foreground-Background Separability

Both our MS-STS attention in encoder and temporal attention in decoder promote multi-scale spatio-temporal feature enrichment and temporal consistency, respectively, across frames. This helps to address the critical issue of appearance deformations in the target object due to challenges such as, scale variation, aspect-ratio change and fast motion. In addition to addressing target appearance deformations across frames, another common challenge in the VIS problem is the accurate delineation of foreground objects from the cluttered background. To this end, we introduce an adversarial loss during training of our MS-STS VIS framework for enhancing foreground-background (fg-bg) separability. To the best of our knowledge, we are the first to explore an adversarial loss within a transformer-based VIS framework for enhancing fg-bg separability.

In our adversarial formulation, the objective is to discriminate between the ground-truth mask and the predicted mask output by our MS-STS VIS. With this objective, we introduce an auxiliary discriminator network during training. The discriminator takes the input frames along with the corresponding encoder features and binary masks as its input. Here, the binary mask $\mathbf{M}$ is obtained either from the ground-truth or predictions, such that all object instances (category-agnostic) within a frame are marked as foreground. While the discriminator $D$ attempts to distinguish between ground-truth and predicted binary masks ($\mathbf{M}_{gt}$ and $\mathbf{M}_{pred}$, respectively), the encoder learns to output enhanced features $\mathbf{E}$ such that the predicted masks $\mathbf{M}_{pred}$ are close to ground-truth $\mathbf{M}_{gt}$.

Let $\mathbf{F}_{gt} = \mathrm{CONCAT}(\mathbf{x},\mathbf{E},\mathbf{M}_{gt})$ and $\mathbf{F}_{pr} = \mathrm{CONCAT}(\mathbf{x},\mathbf{E},\mathbf{M}_{pred})$ denote the real and fake input, respectively, to the discriminator $D$. Similar to [9], the adversarial loss is then given by

\begin{equation}
\min_{Enc} \max_{D} \mathbb{E}[\log D(\mathbf{F}_{gt})] + \mathbb{E}[\log (1 - D(\mathbf{F}_{pr}))] + \lambda_1 \mathbb{E}[||D(\mathbf{F}_{gt})-D(\mathbf{F}_{pr})||_1].
\end{equation}

Since the mask prediction depends on the quality of the encoder features that are decoded by the queries, we treat our encoder $Enc$ as a generator in our adversarial formulation above. As a result, the encoder learns to better delineate foreground and background regions leading to improved video instance mask prediction. Note that the discriminator network is utilized only during  training.

§ EXPERIMENTS

§.§ Experimental Setup

The YouTube-VIS 2019 [26] dataset contains 2883 high-quality videos with 131$K$ annotated object instances belonging to 40 different categories.
The YouTube-VIS 2021 [25] dataset contains 3,859 high-quality videos with 232$K$ annotated object instances belonging to 40 different improved categories.
YouTube-VIS 2021[25] has a 40-category label set by merging eagle and owl into bird, ape into monkey, deleting hands, and adding flying disc, squirrel and whale, maintaining the same number of categories as YouTube-VIS 2019[26] set.

Evaluation Metrics: We follow the standard protocol, where the evaluation metrics, Average Precision (AP) and Average Recall (AR), are adapted from image instance segmentation with the video Intersection over Union (IoU) of the mask sequences as the threshold.

Implementation Details:
We choose ResNet-50 [7] as the default backbone, unless otherwise specified, for our MS-STS VIS framework. Outputs from conv${}_3$, conv${}_4$ and conv${}_5$ of the Resnet backbone are utilized to obtain multi-scale feature inputs to our transformer encoder-decoder, as in [29]. Both encoder and decoder layers are set to $N_d{=}6$. The feature dimension $C$ is set to 256, while the number of instance queries is set to 300 and length of video clip $T{=}5$, as in [24]. We use the AdamW optimizer with a base learning rate (LR) of 2$\times10^{-4}$, ($\beta_1$, $\beta_2) {=} (0.9, 0.999)$ and a weight decay of $10^{-4}$. LR of linear projections of deformable attention modules and the backbone are scaled by $0.1$. The model is first pretrained on COCO [16] for 24 epochs with a batch size of 2. Similar to [24], the pretrained weights are then used to train the model on Youtube-VIS and COCO dataset for 12 epochs with $T{=}5$ and batch size set to 2. The LR is scaled by a factor of $0.1$ at $4^{th}$ and $10^{th}$ epochs. The framework is trained on 8 Nvidia V100 GPUs using PyTorch-1.9 [18].

State-of-the-art comparison on YouTube-VIS 2019 $\mathtt{val}$ set. Our MR-STS VIS consistently outperforms the state-of-the-art results reported in literature. When using the ResNet-50 backbone, MS-STS VIS achieves overall mask AP score of 50.1% with an absolute gain of 2.7% over the best existing SeqFormer, while being comparable in terms of model size and speed (SeqFormer: 11 FPS  Ours: 10 FPS). Similarly, when using the ResNet-101 backbone, our MS-STS VIS achieves overall mask AP of 51.1%. Further, MS-STS VIS achieves the best accuracy reported on this dataset with a mask AP of 61.0% and outperforms SeqFormer with an absolute gain of 1.7%, using the same Swin-L backbone.

IoUTracker+ [26]     ICCV 2019     ResNet-$50$     -     23.6     39.2     25.5     26.2     30.9

OSMN [27]     CVPR 2018     ResNet-$50$     Two-Stage     27.5     45.1     29.1     28.6     33.1

DeepSORT [23]     ICIP 2017     ResNet-$50$     Two-stage     26.1     42.9     26.1     27.8     31.3

FEELVOS [21]     CVPR 2019     ResNet-$50$      Two-stage     26.9     42.0     29.7     29.9     33.4

SeqTracker [26]     ICCV 2019     ResNet-$50$     -     27.5     45.7     28.7     29.7     32.5

MaskTrack R-CNN [26]     ICCV 2019     ResNet-$50$     Two-stage     30.3     51.1     32.6     31.0     35.5

MaskProp [2]     CVPR 2020     ResNet-$50$     -     40.0     -     42.9     -     -

SipMask-VIS [3]     ECCV 2020     ResNet-$50$     One-stage     32.5     53.0     33.3     33.5     38.9

SipMask-VIS [3]     ECCV 2020     ResNet-$50$      One-stage     33.7     54.1     35.8     35.4     40.1

STEm-Seg [1]     ECCV 2020     ResNet-$50$      -     30.6     50.7     33.5     31.6     37.1

Johnander [10]     GCPR 2021     ResNet-$50$      -     35.3     -     -     -     -

CompFeat [5]     AAAI 2021     ResNet-$50$     -     35.3     56.0     38.6     33.1     40.3

CrossVIS[28]      ICCV 2021     ResNet-$50$      One-stage     36.3     56.8     38.9     35.6     40.7

PCAN [11]     NeurIPS 2021     ResNet-$50$     One-stage     36.1     54.9     39.4     36.3     41.6

VisTR [22]     CVPR 2021     ResNet-$50$     Transformer     35.6     56.8     37.0     35.2     40.2

SeqFormer [24]     Arxiv 2021     ResNet-$50$     Transformer     47.4     69.8     51.8     45.5     54.8

MS-STS VIS (Ours)          ResNet-$50$     Transformer      50.1     73.2     56.6     46.1     57.7

MaskTrack R-CNN [26]    ICCV 2019     ResNet-$101$     Two-stage     31.9     53.7     32.3     32.5     37.7

MaskProp [2]     CVPR 2020     ResNet-$101$      -     42.5     -     45.6     -     -

STEm-Seg [1]
ECCV, 2020
34.6     55.8     37.9     34.4     41.6

CrossVIS [28]     ICCV 2021     ResNet-$101$     One-stage     36.6     57.3     39.7     36.0     42.0

PCAN [11]     NeurIPS 2021     ResNet-$101$     One-stage     37.6     57.2     41.3     37.2     43.9

VisTR [22]     CVPR 2021     ResNet-$101$     Transformer     38.6     61.3     42.3     37.6     44.2

SeqFormer [24]     Arxiv 2021     ResNet-$101$     Transformer     49.0     71.1     55.7     46.8     56.9

MS-STS VIS (Ours)          ResNet-$101$     Transformer      51.1     73.2     59.0     48.3     58.7

SeqFormer [24]     Arxiv 2021     Swin-L     Transformer     59.3     82.1     66.6     51.7     64.4

MS-STS VIS (Ours)          Swin-L     Transformer      61.0     85.2     68.6     54.7     66.4

§.§ State-of-the-art Comparison

Tab. <ref> presents the state-of-the-art comparison on the YouTube-VIS 2019 val. set. When using the ResNet-50 backbone, the recent one-stage PCAN [11] and CrossVIS [28] approaches achieve an overall mask accuracy (AP) of 36.1% and 36.3%, respectively. With the same ResNet-50 backbone, the first transformer-based VIS approach, VisTR [22], built on DETR framework achieves an overall mask AP of 35.6%. Among existing methods, the recently introduced SeqFormer [24] based on Deformable DETR framework achieves the best overall accuracy with a mask AP of 47.4%. Our proposed MS-STS VIS approach outperforms SeqFormer [24] by achieving an overall mask AP of 50.1%, using the same ResNet-50 backbone. Specifically, our MS-STS VIS provides an absolute gain of 4.8% at a higher overlap threshold of AP$_{\mathtt{75}}$ over SeqFormer. Similarly, our MS-STS VIS consistently outperforms SeqFormer with an overall mask AP of 51.1%, when using the ResNet-101 backbone. Finally, when using the recent Swin Transformer backbone, the proposed MS-STS VIS achieves the best accuracy reported in literature with an overall mask AP of 61.0%.

State-of-the-art comparison on YouTube-VIS 2021 $\mathtt{val}$ set. All results are reported using the same ResNet-50 backbone. Our MS-STS VIS achieves state-of-the-art results with an overall mask AP of 42.2% and an absolute gain of 2.8% over the best existing SeqFormer at a higher overlap threshold of AP$_{\mathtt{75}}$.

MaskTrack R-CNN[25]     $28.6$     $48.9$     $29.6$     $26.5$     $33.8$

SipMask-VIS[3]     $31.7$     $52.5$     $34.0$     $30.8$     $37.8$

VisTR[22]     $31.8$     $51.7$     $34.5$     $29.7$     $36.9$

CrossVIS[28]     $34.2$     $54.4$     $37.9$     $30.4$     $38.2$

IFC[8]     $36.6$     $57.9$     $39.9$     $-$     $-$

SeqFormer[24]     $40.5$     $62.4$     $43.7$     $36.1$     $48.1$

MS-STS VIS (Ours)     42.2     63.7     46.5     41.7     51.1

Tab. <ref> reports the state-of-the-art comparison on the YouTube-VIS 2021 val. set. Among existing methods, CrossVIS [28] and IFC [8] achieve overall mask AP scores of 34.2% and 36.6%, respectively. SeqFormer [24] obtains an overall mask AP of 40.5%. Our MS-STS VIS sets a new state-of-the-art with an overall mask AP of 42.2%. Specifically, MS-STS VIS provides an absolute gain of 2.8% over SeqFormer at higher overlap threshold of AP$_{\mathtt{75}}$, when using the same ResNet-50 backbone.

On the left: Impact of our contributions when progressively integrating them into the baseline on the Youtube-VIS 2019 val. set. We observe a consistent performance improvement due to the integration of our proposed contributions. Our final MS-STS VIS (row 4) achieves an absolute gain of 3.7% over the baseline. On the right: Attribute-based performance comparison between the baseline and our  MS-STS VIS on the custom set comprising 706 videos in total. Here, we present the comparison on fast motion, target size change and aspect-ratio change attributes. Our MS-STS VIS achieves consistent improvement in performance over the baseline on all attributes. See Sec. <ref> for details.

Baseline     $46.4$

Baseline + MS-STS     $48.4$

Baseline + MS-STS + T-Dec     $49.1$

Baseline + MS-STS + T-Dec + FG-BG Loss     50.1

Attribute Type     #Videos     Baseline      MS-STS VIS

Custom Set     $ 706 $     $ 49.5 $     53.7

Fast Motion     $ 605 $     $ 48.9 $     54.2

Target Size Change     $ 706 $     $ 49.5 $     53.7

Aspect-Ratio Change     $ 687 $     $ 49.2 $     53.9

§.§ Ablation Study

Here, we first evaluate the merits of our three proposed contributions: MS-STS attention module-based encoder (Sec. <ref>), temporal attention in the decoder (Sec. <ref>) and the adversarial loss for enhancing fg-bag separability (Sec. <ref>). Tab. <ref> (left) shows the baseline comparison on the YouTube-VIS 2019 val. set. All results reported in Tab. <ref> (left) are obtained using the same ResNet50 backbone. As discussed earlier, our MS-STS VIS employs SeqFormer as its base framework.  We train the baseline SeqFormer (denoted here as Baseline) using the official implementation and achieve an overall mask AP score of 46.4%. The introduction of our MS-STS attention module-based encoder within the baseline (referred as Baseline + MS-STS) significantly improves the overall performance to 48.4% with an absolute gain of 2.0%. The overall performance is further improved to 49.1% with the integration of the temporal attention in the decoder (denoted as Baseline + MS-STS + T-Dec). Finally, the introduction of the adversarial loss during the training for enhancing the fg-bg separability provides an absolute gain of 1.0% (denoted as Baseline + MS-STS + T-Dec + FG-BG Loss). Our final MS-STS VIS achieves an absolute gain of 3.7% over the baseline.

We further analyze the performance of our method (MS-STS VIS) under three specific challenging scenarios: fast motion, target size change (scale variation) and aspect-ratio change. To this end, we classified the videos into three categories: (i) fast motion of the object, (ii) object size changes, and (iii) aspect-ratio changes. In particular, we follow the well-established VOT-2015 benchmark [12] to label a particular video from above categories as follows:

fast motion: if object center in current frame moves by at least 30% of its size in previous frame.
change in object size: if the ratio of the maximum size to the minimum size of an object in the video is greater than 1.5.
change in aspect-ratio: if the ratio of the maximum to the minimum aspect (width/height) of the bounding box enclosing an object in the video is greater than 1.5.

Based on the aforementioned criteria, we first select the newly added videos in Youtube-VIS 2021 training set while retaining the same set of classes as in Youtube-VIS 2019 dataset. We refer these selected videos as custom set and classify them into the aforementioned attributes. To evaluate the performance of our MS-STS VIS and the baseline, we use the models trained on Youtube-VIS 2019 training set with the same ResNet-50 backbone. Note that we ensure that there is no overlap between the videos in Youtube-VIS 2019 training set and our custom set (only comprising the newly added videos from the Youtube-VIS 2021 training set). Tab. <ref> (right) shows the comparison between the baseline SeqFormer and our MS-STS VIS. On the entire custom set, our MS-STS VIS obtains significantly improved performance over the baseline. Specifically, MS-STS VIS achieves absolute gains of 5.3%, 4.2% and 4.7% over the baseline on fast motion, target size change and aspect-ratio change attributes, respectively.

§.§ Qualitative Analysis

Fig. <ref> and <ref> show qualitative results obtained by our MS-STS VIS framework on example videos from the Youtube-VIS 2019 val. and 2021 val. sets, respectively. We observe our MS-STS VIS framework to obtain promising video mask prediction in various challenging scenarios involving target appearance deformations due to fast motion, aspect-ratio change and scale variation. , in Fig. <ref>, video masks are predicted accurately for hand in row 1 (scale change), eagle in row 5 (fast motion, aspect-ratio change, scale variation), panda in row 3 (aspect-ratio change), . Similarly, we observe promising video masks predictions for leopard in row 1 (fast motion), person in row 3 (scale variation), dog in row 2 (aspect-ratio change, scale variation) in Fig. <ref>. These results show the efficacy of our MS-STS VIS framework under different challenges for the task of video instance segmentation.

Qualitative results obtained by our MS-STS VIS framework on seven example videos in the Youtube-VIS 2019 val set. Our MS-STS VIS achieves promising video mask prediction in various challenging scenarios including, fast motion (eagle in row 5), scale change (hand in row 1), aspect-ratio change (panda in row 3, person in row 4), multiple instances of same class (tiger in row 7).

Qualitative results on seven example videos in the Youtube-VIS 2021 val set. Our MS-STS VIS achieves favorable video mask prediction in various scenarios involving target appearance deformations: fast motion (leopard in row 1), scale variation (person and seal in row 4), aspect-ratio change (dog in row 2, tiger in row 6).

§ RELATED WORK

Two-stage VIS: Several VIS methods [2, 26, 14] adapt the two-stage pipeline, such as Mask R-CNN [6] by introducing an additional tracking branch for target association. The work of [26] introduces MaskTrack R-CNN that jointly performs detection, segmentation and tracking. Bertasius  [2] utilize a branch in Mask R-CNN to propagate frame-level instance masks based on deformable convolutions from each frame to other video frames within a temporal neighborhood. The work of [14] introduces a modified variational autoencoder (VAE) on top of Mask R-CNN for instance-level video segmentation and tracking.

Single-stage VIS: Several works [3, 1, 11, 13, 17] adapt the one-stage pipeline, such as FCOS detector [19], where a linear combination of mask bases are directly predicted as final segmentation. SipMask [3] introduces a spatial information preservation module for real-time VIS. The work of [1] introduces an approach where a short 3D convolutional spatio-temporal volume is adopted to learn pixel-level embedding by posing segmentation as a bottom-up grouping. The work of [11] proposes to refine a space-time memory into a set of instance and frame-level prototypes, followed by an attention scheme.

Transformer-based VIS:
Wang  [22] introduce a transformer-based encoder-decoder architecture, named VisTR, that formulates VIS as a direct end-to-end parallel sequence prediction task.
In the encoder, VisTR utilizes a single-scale attention that computes similarities between all pairs of features from multiple spatial locations of a low-resolution feature map, across frames. The work of [8] introduces inter-frame communication transformers, where memory tokens are used to communicate between frames. The recent SeqFormer [24], built on Deformable DETR framework [29],  utilizes per-frame multi-scale features during attention computations. While demonstrating promising results, SeqFormer struggles in case of target deformations likely due to not explicitly capturing the spatio-temporal feature relationships during attention computation. To address these issues, we proposed a framework comprising an encoder that captures multi-scale spatio-temporal feature relationships. We also introduced an attention block in decoder to enhance temporal consistency of detected instance in different frames and an adversarial loss during training that ensures better fg-bg separability within  multi-scale spatio-temporal feature space.

§ CONCLUSIONS

We proposed a transformer-based video instance segmentation framework, named MS-STS VIS, which comprises a novel multi-scale spatio-temporal split attention (MS-STS) module to effectively capture spatio-temporal feature relationships at multiple scales across frames in a video.  We further introduced an auxiliary discriminator network during training that strives to enhance fg-bg separability within the multi-scale spatio-temporal feature space. Our MS-STS VIS  specifically tackles target appearance deformations due to real-world challenges such as, scale variation, aspect-ratio change and fast motion in videos. Our extensive experiments on two datasets reveal the benefits of the proposed contributions, achieving state-of-the-art performance on both benchmarks.

§ ACKNOWLEDGEMENTS

This work was partially supported by VR starting grant (2016-05543), the Wallenberg AI, Autonomous Systems and Software Program (WASP), by the Swedish Research Council through a grant for the project Algebraically Constrained Convolutional Networks for Sparse Image Data (2018-04673), and the strategic research environment ELLIIT, in addition to the compute support provided at the Swedish National Infrastructure for Computing (SNIC), partially funded by the Swedish Research Council through grant agreement no. 2018-05973.

[1]
Athar, A., Mahadevan, S., Osep, A., Leal-Taixé, L., Leibe, B.: Stem-seg:
Spatio-temporal embeddings for instance segmentation in videos. In: ECCV

[2]
Bertasius, G., Torresani, L.: Classifying, segmenting, and tracking object
instances in video with mask propagation. In: CVPR (2020)

[3]
Cao, J., Anwer, R.M., Cholakkal, H., Khan, F.S., Pang, Y., Shao, L.: Sipmask:
Spatial information preservation for fast image and video instance
segmentation. In: ECCV (2020)

[4]
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.:
End-to-end object detection with transformers. In: ECCV (2020)

[5]
Fu, Y., Yang, L., Liu, D., Huang, T.S., Shi, H.: Compfeat: Comprehensive
feature aggregation for video instance segmentation. AAAI  (2021)

[6]
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)

[7]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: CVPR (2016)

[8]
Hwang, S., Heo, M., Oh, S.W., Kim, S.J.: Video instance segmentation using
inter-frame communication transformers. In: NeurIPS (2021)

[9]
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with
conditional adversarial networks. In: CVPR (2017)

[10]
Johnander, J., Brissman, E., Danelljan, M., Felsberg, M.: Learning video
instance segmentation with recurrent graph neural networks. In: GCPI (2021)

[11]
Ke, L., Li, X., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F.: Prototypical
cross-attention networks for multiple object tracking and segmentation. In:
NeurIPS (2021)

[12]
Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Cehovin, L., Fernandez,
G., Vojir, T., Hager, G., Nebehay, G., Pflugfelder, R.: The visual object
tracking vot2015 challenge results. In: ICCV workshops (2015)

[13]
Li, M., Li, S., Li, L., Zhang, L.: Spatial feature calibration and temporal
fusion for effective one-stage video instance segmentation. In: CVPR (2021)

[14]
Lin, C., Hung, Y., Feris, R., He, L.: Video instance segmentation tracking with
a modified vae architecture. In: CVPR (2020)

[15]
Lin, H., Wu, R., Liu, S., Lu, J., Jia, J.: Video instance segmentation with a
propose-reduce paradigm. In: arXiv preprint arXiv:2103.13746 (2021)

[16]
Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D.,
Dollár, P., Zitnick, L.: Microsoft coco: Common objects in context. In:
ECCV (2014)

[17]
Liu, D., Cui, Y., Tan, W., Chen, Y.: Sg-net: Spatial granularity network for
one-stage video instance segmentation. In: CVPR (2021)

[18]
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen,
T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E.,
DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,
Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep
learning library. In: NeurIPS (2019),

[19]
Tian, Z., Shen, C., Chen, H., He, T.: FCOS: Fully convolutional one-stage
object detection. In: ICCV (2019)

[20]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.,
Kaiser, L., Polosukhin, I.: Pattention is all you need. In: NeurIPS (2017)

[21]
Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.:
Feelvos: Fast end-to-end embedding learning for video object segmentation.
In: CVPR (2019)

[22]
Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., Xia, H.: End-to-end
video instance segmentation with transformers. In: CVPR (2021)

[23]
Wojke, N., Bewley, A., Paulus, D.: Simple online and realtime tracking with a
deep association metric. In: ICIP (2017)

[24]
Wu, J., Jiang, Y., Zhang, W., Bai, X., Bai, S.: Seqformer: a frustratingly
simple model for video instance segmentation. In: arXiv preprint
arXiv:2112.08275 (2021)

[25]
Xu, N., Yang, L., Yang, J., Yue, D., Fan, Y., Liang, Y., Huang, T.S.:
Youtube-vis dataset 2021 version. <https://youtube-vos.org/dataset/vis>

[26]
Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: ICCV (2019)

[27]
Yang, L., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video
object segmentation via network modulation. In: CVPR (2018)

[28]
Yang, S., Fang, Y., Wang, X., Li, Y., Fang, C., Shan, Y., Feng, B., Liu, W.:
Crossover learning for fast online video instance segmentation. In: ICCV

[29]
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable
transformers for end-to-end object detection. In: ICLR (2021)


:
ML4H Extended Abstract 2020 2020 Machine Learning for Health (ML4H) 2020

# Exploring Gender Disparities in Time to Diagnosis

Tony Y. Sun<EMAIL_ADDRESS>
Oliver J. Bear Don’t Walk IV<EMAIL_ADDRESS>
Jennifer L. Chen<EMAIL_ADDRESS>
Harry Reyes Nieva<EMAIL_ADDRESS>
Noémie Elhadad<EMAIL_ADDRESS>
Columbia University

###### Abstract

Sex and gender-based healthcare disparities contribute to differences in
health outcomes. We focus on time to diagnosis (TTD) by conducting two large-
scale, complementary analyses among men and women across 29 phenotypes and
195K patients. We first find that women are consistently more likely to
experience a longer TTD than men, even when presenting with the same
conditions. We further explore how TTD disparities affect diagnostic
performance between genders, both across and persistent to time, by evaluating
gender-agnostic disease classifiers across increasing diagnostic information.
In both fairness analyses, the diagnostic process favors men over women,
contradicting the previous observation that women may demonstrate relevant
symptoms earlier than men. These analyses suggest that TTD is an important yet
complex aspect when studying gender disparities, and warrants further
investigation.

###### keywords:

gender disparities, time to diagnosis, electronic health record data

##  1 Introduction

Sex and gender111We recognize gender is a social construct while sex is a
biological variable (World Health Organization, 2002). We use the terms
“gender,” “men,” and “women” based on available labels. play significant roles
in the initial diagnosis and treatment of disease. An increasing body of
literature has documented numerous examples of biological sex differences in
disease presentation as well as gender disparities in care (Regitz-Zagrosek
and Seeland, 2013; Regitz-Zagrosek, 2012; Westergaard et al., 2019). For
example, recent research shows that women with acute myocardial infarction
present with different symptoms than men (i.e., sex difference). Healthcare
providers, which tend to use diagnostic criteria based on typical male
presentation, fail to recognize in time heart attacks in women (i.e., gender
bias). These two factors contribute to decreased survival outcomes for women
(Shaw et al., 2006; Mehta et al., 2016; Greenwood et al., 2018).

In the machine learning for healthcare community, there is growing interest in
creating methodology to identify health disparities as well as to mitigate
them (Chen et al., 2020). Here, we focus on time to diagnosis (TTD),
traditionally defined as the interval from first alert symptoms to diagnosis
of a disease in a patient (Launay et al., 2016), and explore potential TTD-
related gender disparities. Using the electronic health records of 195,000
patients (113K women and 81K men), and phenotype definitions for 29 conditions
(wherein each patient is assigned an official diagnosis time), we carry out
two complementary analyses for each of the 29 conditions: (1) we compare mean
TTD between men and women; and (2) given a gender-agnostic disease classifier,
we assess its gender fairness as presenting symptoms are accumulated over time
and holistically assess bias in a time-persistent manner (through mean squared
discrimination). This study is approved by our IRB.

##  2 Dataset

Using data from Columbia University Irving Medical Center, we identified
patients at least 13 years of age with at least 3 years of continuous
observation between January 2010 and January 2019. Using 29 established
phenotype definitions publicly available on the OHDSI platform, we constructed
29 corresponding cohorts (OHDSI, ). As such, patients in each cohort can be
aligned in time according to their official diagnosis time.

For each patient in a given cohort, we extracted their gender and all coded
condition occurrences observed within 3 years prior to phenotype diagnosis.
Along with each coded condition, we keep track of date of its first occurrence
in a patient longitudinal record. The final dataset comprises 113,885 women
(mean age, 55.7 years) and 81,742 men (mean age, 55.9 years) with 13,852
unique condition codes.

##  3 Time to Diagnosis among Women and Men

### Analysis.

Here we extend the traditional definition of TTD to include all presenting
conditions prior to a phenotype official diagnosis. We compute a given
condition’s TTD as mean time interval between the condition’s first occurrence
in a patient record and the phenotype diagnosis time (Fig.
LABEL:fig:ttd_analysis).

fig:ttd_analysis

Figure 1: TTD computation for each phenotype (e.g., Crohn’s disease).

### Results.

Women were consistently diagnosed later than men, when presenting with the
same conditions prior to a phenotype’s diagnosis. In $61.5\%$ of all
conditions for patients that go on to develop the same phenotype, women were
diagnosed later than men (mean difference of $14.8$ days with the same
presentation), when averaged across the 29 phenotypes. Additionally, 1,907
conditions ($13.76\%$) had 100+ day TTD difference across gender.

##  4 Time-Variant Model Fairness in Diagnosis Classification

### Analysis.

For each phenotype, we train a gender agnostic binary classifier that
diagnoses patients based on clinical observations documented up to 3 years
prior to the official disease diagnosis. As such, the trained classifier is
“fully knowledgeable” as it contains all observations up to time of diagnosis,
particularly the relevant symptoms (e.g., abdominal pain indicating Crohn’s
disease). (See Appendix A for details).

Given that women have on average longer TTD than men, we assess whether women
should be diagnosed earlier than men according to the trained classifier. To
test this hypothesis, we measure the classifiers’ performance on a cohort of
patients with decreasing levels of right censoring (see Fig.
LABEL:fig:bin_figure) up to 3 years prior to their diagnosis. In essence, this
testing procedure aims to mimic a provider diagnosing patients at increasing
time steps with knowledge limited to patient history accumulated thus far.

fig:bin_figure

Figure 2: Test sets are generated with varying levels of right censoring,
mimicking varying amounts of patient history available to providers.

Formally, our evaluation procedure is as follows. Let $N_{d}$ be the number of
patients for a particular disease $d$, and $C_{d}$ the number of symptoms
(clinical concepts) related to this disease. To evaluate growing windows with
right censoring, let $w$ be window size in days, therefore $b=\lceil
1095/w\rceil$ is the number of possible windows (buckets) of test data. For
each window $i\in\\{1,\dots,b\\}$, we increase available symptoms per patient
from day 0 (1095 days before diagnosis) to day $i*w$. Each right-censored test
data window $T_{i}$ is a $N_{d}\times C_{d}$ binary sparse matrix where
$T^{(p,s)}=1$ if patient $p$ has symptom $s$ at or before the maximum window
date $i*w$.

Iterating through each test window $T_{1},\dots,T_{b}$, we quantify fairness
gaps at each window to understand trends in bias as the diagnostic procedure
progresses. In line with established fairness literature, we measure the
equality of opportunity of the positive and negative class (recall and
specificity gap, respectively) (Zhang et al., 2020), and the precision and
accuracy gap at each test window (Table 2). For example, recall gap is the
recall of diagnosing male patients minus the recall of female patients. In
diagnosis, recall gap is important as minimizing false negatives ensures
patients with a disease are not left untreated. Our approach to analyzing
fairness gap over time is extensible to any definition of fairness relevant to
the data domain.

We also sought to quantify gender bias per model persistent across time. Chen
et al. (2018) proposes that fairness gaps (discrimination level) of a trained
model are attributable to the bias, variance, and noise of a dataset
conditioned on subgroup attributes. They experimentally suggest that
increasing training data size for a dataset can reduce fairness gaps if the
discrimination is primarily due to variance within discriminated subgroups.
Our analysis of gender bias differs by evaluating on the test data in a
longitudinal manner, decreasing the amount of right-censoring at each time
step and measuring the fairness gap over time. To measure the average fairness
gap as information accumulates over time, we leverage mean squared error (MSE)
and the overall directionality of bias over $b$ time windows, combining these
concepts into mean squared discrimination (MSD). See B for mathematical
explication.

fig:mse_recall

Figure 3: Recall MSD across phenotypes. Blue circles indicate better recall
for men; pink, for women. Circle size is proportional to the size of the
dataset (Table 1). Error bars correspond to 95% confidence intervals.
*Excludes non-melanoma skin cancer

### Results.

Analyzing fairness gaps across time steps for individual phenotypes, such as
Crohn’s disease, we find that trends in TTD disparity match observed disparity
in our previous analysis (section 3). As the amount of patient history
increases, the Crohn’s disease model’s recall for both genders increases, but
performance for men is consistently higher than women (Fig.
LABEL:fig:crohn_gender). Consistently positive recall gap (indicating better
model recall for men) contradicts our previous finding that women present
symptoms earlier than men. Fitting the recall gap using linear regression, we
observe that male-favoring recall gap decreases $-2.65\mathrm{e}^{-4}$ at each
window step (Fig. 6), demonstrating convergence in recall between genders over
time.

fig:crohn_gender

Figure 4: Gender-specific recall at varying levels of right censoring for
Crohn’s Disease.

While fine-grained per-phenotype analysis allows us to compare observed
fairness trends with hypothesized TTD disparities, we observe marked
differences between MSD for men and women across the 29 phenotypes. Using
recall MSD as a proxy for model bias we compare models by ranking phenotypes
by MSD magnitude per gender (Fig. LABEL:fig:mse_recall). The recall MSD can be
interpreted per phenotype, such as in Crohn’s, where a small male-biased MSD
contradicts our TTD finding that women present symptoms earlier. In Crohn’s,
on average women present more symptoms than men (23.86 vs 22.74 for
women/men). This might indicate that while women present the right conditions
earlier to help detect a phenotype, they also present many other conditions
that confuse the classifier, thus rendering a male-biased recall. Across all
phenotypes, recall MSD was slightly skewed towards men (17 phenotypes), and
those with less data (smaller circles) showed higher MSD magnitude (far left
or far right) (Fig. LABEL:fig:mse_recall).

##  5 Conclusion and Future Work

We propose simple metrics and experiments to explore TTD to investigate gender
disparities. We find that across 29 phenotypes, women were far more likely to
experience a longer TTD than men. Despite this fact, when training gender-
agnostic classifiers to recognize disease, we find that a majority perform
better for men than women. Furthermore, this persists across most time windows
prior to diagnosis.

Our analysis is limited to a single clinical site. We note however that we
operate in the OHDSI framework, and the methodology is extensible to other
OHDSI sites and validated phenotype definitions.

This work was supported by NLM awards R01 LM013043 and T15 LM007079.

## References

  * Chen et al. (2018) Irene Chen, Fredrik D Johansson, and David Sontag.  Why is my classifier discriminatory?  In _Advances in Neural Information Processing Systems_ , pages 3539–3550, 2018.
  * Chen et al. (2020) Irene Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi.  Ethical machine learning in health care.  _Annual Review of Biomedical Data Science_ , 2020.
  * Greenwood et al. (2018) Brad N Greenwood, Seth Carnahan, and Laura Huang.  Patient–physician gender concordance and increased mortality among female heart attack patients.  _Proceedings of the National Academy of Sciences_ , 115(34):8569–8574, 2018.
  * Launay et al. (2016) Elise Launay, Jérémie F Cohen, Patrick M Bossuyt, Pierre Buekens, Jonathan Deeks, Timothy Dye, Richard Feltbower, Andrea Ferrari, Michael Kramer, Mariska Leeflang, et al.  Reporting studies on time to diagnosis: proposal of a guideline by an international panel (rest).  _BMC medicine_ , 14(1):146, 2016.
  * Mehta et al. (2016) Laxmi S Mehta, Theresa M Beckie, Holli A DeVon, Cindy L Grines, Harlan M Krumholz, Michelle N Johnson, Kathryn J Lindley, Viola Vaccarino, Tracy Y Wang, Karol E Watson, et al.  Acute myocardial infarction in women: a scientific statement from the american heart association.  _Circulation_ , 133(9):916–947, 2016.
  * (6) OHDSI.  Observational health data sciences and informatics studies.  https://github.com/ohdsi-studies/Covid19CharacterizationCharybdis/tree/master/inst/cohort.
  * Regitz-Zagrosek (2012) Vera Regitz-Zagrosek.  Sex and gender differences in health: Science & society series on sex and science.  _EMBO reports_ , 13(7):596–603, 2012.
  * Regitz-Zagrosek and Seeland (2013) Vera Regitz-Zagrosek and Ute Seeland.  Sex and gender differences in clinical medicine.  In _Sex and Gender Differences in Pharmacology_ , pages 3–22. Springer, 2013.
  * Shaw et al. (2006) Leslee Shaw, Noel Merz, Carl Pepine, Steven Reis, Vera Bittner, Sheryl Kelsey, Marian Olson, Delia Johnson, Sunil Mankad, Barry Sharaf, et al.  Insights from the NHLBI-sponsored women’s ischemia syndrome evaluation (WISE) study: Part I: gender differences in traditional and novel risk factors, symptom evaluation, and gender-optimized diagnostic strategies.  _Journal of the American College of Cardiology_ , 47(3 Supplement):S4–S20, 2006.
  * Westergaard et al. (2019) David Westergaard, Pope Moseley, Freja Karuna Hemmingsen Sørup, Pierre Baldi, and Søren Brunak.  Population-wide analysis of differences in disease progression patterns in men and women.  _Nature communications_ , 10(1):1–14, 2019.
  * World Health Organization (2002) World Health Organization.  WHO gender policy: Integrating gender perspectives in the work of WHO.  _Geneva: World Health Organization_ , page p6, 2002.
  * Zhang et al. (2020) Haoran Zhang, Amy X Lu, Mohamed Abdalla, Matthew McDermott, and Marzyeh Ghassemi.  Hurtful words: quantifying biases in clinical contextual word embeddings.  In _Proceedings of the ACM Conference on Health, Inference, and Learning_ , pages 110–120, 2020.

##  Appendix A Disease Classification

Given the phenotype definition, a corresponding cohort is determined, i.e., a
set of patients that obtain a diagnosis for the disease of interest along with
an time of official diagnosis. The cohort is split into an 80% train and 20%
test sets, with the split maintaining the gender proportions prior to the
split. Negative examples for the classifier are drawn by finding patients from
other phenotypes that were within a 5 year age with the same gender (Table 1).

For each phenotype, we train a binary classifier using L2-penalized logistic
regression. Features consist of all extracted coded conditions up to the
diagnosis time, represented as one-hot encodings.

Fig. LABEL:fig:crohn_performance shows the performance for Crohn’s disease in
particular over the different test sets.

Phenotype | Train | Test
---|---|---
Angioedema | 458 | 114
Gastrointestinal bleeding | 4280 | 1070
Gout | 2588 | 648
Heart failure | 6388 | 1596
Hepatic failure | 1444 | 362
Prevalent Type 2 Diabetes | 28138 | 7036
Prevalent hypertension | 61258 | 15314
Prevalent chronic kidney disease | 8436 | 2110
Prevalent ESRD | 2170 | 542
Prevalent heart disease | 52938 | 13234
Prevalent malignant neoplasm* | 24638 | 6160
Prevalent HIV infection | 1502 | 376
Prevalent Dementia | 6956 | 1740
Prevalent COPD without asthma | 14130 | 3532
Prevalent Asthma without COPD | 22336 | 5584
Acute myocardial infarction events | 3856 | 964
Suicide and suicidal ideation | 4052 | 1014
Psoriasis | 2364 | 592
Psoriatic arthritis | 340 | 86
Multiple sclerosis | 408 | 102
Systemic lupus erythematosus | 432 | 108
Sjogrens syndrome | 206 | 52
Hashimoto thyroiditis | 418 | 104
Myasthenia gravis | 246 | 62
Celiac disease | 762 | 190
Scleroderma | 214 | 54
Sarcoidosis | 8352 | 2088
Ulcerative colitis | 1128 | 284
Crohn’s disease | 5074 | 1270

Table 1: Number of patients in the training and test sets for the 29
phenotypes. *Excludes non-melanoma skin cancers.

fig:crohn_performance

Figure 5: Performance of the Crohn’s diagnosis classifier when tested with
varying levels of right censoring.

##  Appendix B Fairness Gap Metrics and Results

The mean squared discrimination (MSD) can be expressed mathematically as
$\text{MSD}(M_{d})=\text{sign}(\frac{1}{b}\sum_{i=1}^{b}g_{i})*(\frac{1}{b}\sum_{i=1}^{b}g_{i}^{2})$,
where $g_{i}$ is the fairness gap (Table 2) at window $i$. The model’s MSD is
proportional to the mean squared error of recall, precision, or specificity
gap, to penalize biases towards any gender and to heavily penalize models with
larger biases; it is also sign dependent on the gap mean’s sign to signify
which gender the model exhibits better performance towards. The MSD is
constrained between [-1, 1]; since fairness gaps are between [0, 1], the MSE
will be between [0, 1], and the gender sign flips the value from -1 to 1. For
gender, positive MSD indicates model bias favoring men, while negative MSD
favors women.

Table 2 shows the definition for the different gap metrics we use. Figure 6
shows for Chron’s disease in particular, a scatter plot of the recall gaps at
varying levels of right censoring (men $-$ women).

In addition to the mean recall bias across the 29 phenotypes, Figures 8 and 7
show the mean specificity bias and the mean precision gap.

Gap | Definition
---|---
Recall Gap | $\dfrac{\text{TP}_{1}}{\text{TP}_{1}+\text{FN}_{1}}-\dfrac{\text{TP}_{2}}{\text{TP}_{2}+\text{FN}_{2}}$
Specificity Gap | $\dfrac{\text{TN}_{1}}{\text{TN}_{1}+\text{FP}_{1}}-\dfrac{\text{TN}_{2}}{\text{TN}_{2}+\text{FP}_{2}}$
Precision Gap | $\dfrac{\text{TP}_{1}}{\text{TP}_{1}+\text{FP}_{1}}-\dfrac{\text{TP}_{2}}{\text{TP}_{2}+\text{FP}_{2}}$

Table 2: Fairness gap metrics used in time-variant model fairness analysis.

fig:crohn_recall_gap

Figure 6: Gender-specific recall gaps at varying levels of right censoring for
Crohn’s Disease. Figure 7: Specificity MSD across phenotypes. Blue circles
indicate better recall for men; pink, for women. Circle size is proportional
to the size of the dataset (Table 1). Error bars correspond to 95% confidence
intervals. *Excludes non-melanoma skin cancer  Figure 8: Precision MSD across
phenotypes. Blue circles indicate better recall for men; pink, for women.
Circle size is proportional to the size of the dataset (Table 1). Error bars
correspond to 95% confidence intervals. *Excludes non-melanoma skin cancer