# DDS: Decoupled Dynamic Scene-Graph Generation Network

A S M Iftekhar*, Raphael Ruschel*, Satish Kumar, Suya You, B. S. Manjunath *
represents equal contribution.A S M Iftekhar, Raphael Ruschel, Satish Kumar,
and B.S. Manjunath are with the Department of Electrical and Computer
Engineering, University of California Santa Barbara, Santa Barbara, CA, 93106.
E-mail: {iftekhar, raphael251, satishkumar<EMAIL_ADDRESS>You is with
Army Research Laboratory, Intelligent Perception, CISD Los Angeles, CA. Email:
<EMAIL_ADDRESS>
###### Abstract

Scene-graph generation involves creating a structural representation of the
relationships between objects in a scene by predicting subject-object-relation
triplets from input data. However, existing methods show poor performance in
detecting triplets outside of a predefined set, primarily due to their
reliance on dependent feature learning. To address this issue we propose DDS–
a decoupled dynamic scene-graph generation network– that consists of two
independent branches that can disentangle extracted features. The key
innovation of the current paper is the decoupling of the features representing
the relationships from those of the objects, which enables the detection of
novel object-relationship combinations. The DDS model is evaluated on three
datasets and outperforms previous methods by a significant margin, especially
in detecting previously unseen triplets.

###### Index Terms:

Activity detection, compositional learning, zero-shot learning, scene graphs,
scene understanding.

††publicationid: pubid: This work has been submitted to the IEEE for possible
publication. Copyright may be transferred without notice, after which this
version may no longer be accessible.

##  I Introduction

Dynamic Scene-Graph (DSG) provides a graph structure presenting the
relationships among different objects in a scene. It aims to create the scene-
graph by predicting relationship triplets composed of $\langle
subject,object,relationship\rangle$ at each frame of an input video. This acts
as a foundational block for various computer vision tasks[1, 2]. Current DSG
generation systems [3, 4, 5] operate in a constrained setting where the
possible triplets are predefined for a given set of relationships and objects.
However, in a more realistic deployment scenario, it is likely that the
network will encounter triplets that it has not seen before. Therefore, a
system should be able to transfer the learned concepts of relationships and
objects to compose unseen triplets. Our analysis has shown (Table I) poor
performance in detecting these unseen triplets using the state-of-the-art
(SOTA) method. This poor performance is mainly attributed to the learning of
highly dependent feature representations of relationships and objects. The
proposed decoupled dynamic scene-graph (DDS) addresses this issue.

Fig. 1 shows the core idea of the proposed DDS network. This architecture
utilizes two different branches to learn decoupled features for relationships
and objects. As shown in the figure, DDS learns the concept of ‘ride’, ‘on’,
‘person’, ‘bicycle’, and ‘bed’ from the training examples of a ‘person riding
a bike’ and a ‘dog on the bed’ independently. The decoupled design makes DDS
look into different spatial regions for relationships and objects. These
learned concepts are transferred to successfully detect the unseen triplet
$\langle dog,bicycle,ride\rangle$, see Section V-C for more details.

Figure 1: Diagram to show the concept learning and transferring in DDS. By
focusing on different spatial regions, DDS learns the concept of relationships
(ride, on) and objects (person, bicycle, bed) independently. In the lower
section of the diagram, we show how these learned concepts are transferred and
utilized to detect the unseen triplet $\langle dog,bicycle,ride\rangle$.

Existing works [3, 4, 5] for generating DSGs follow a two-stage process.
First, an object detector localizes objects irrespective of their relationship
status. In the second stage, different attention-based [3, 4] and graph-based
[5] methods utilize features extracted from the previously localized objects
to predict relationships. In this process, the extracted features of the
objects are fed throughout the network for relationship detection. As a
result, these methods learn to associate a relationship only with a particular
combination of objects and hence perform poorly to predict unseen triplets.

In contrast, DDS ensures the learning of discriminative spatio-temporal cues
for relationships and objects. Fig. 2 shows the overview of our architecture.
It consists of two separate branches: the relation and the object branch. We
chose a Transformer based encoder-decoder [6] architecture for these branches
with two different sets of queries. Moreover, a novel temporal decoder is
added to embed temporal information into the queries. These separate sets of
queries focus on learning generalized representations for relationships and
objects from differently encoded feature maps in both temporal and spatial
domains. This is significantly better than the existing works, where the same
object features are used for both object and relationship detection. Also, DDS
does not have the dependence on the off-the-shelf object detectors like
previous works.

Our proposed model is thoroughly evaluated on the Action-Genome[7] dataset for
DSG generation, where it achieves significant performance gains compared to
the SOTA models. Additionally, we evaluate DDS on the task of static scene-
graph (SSG) generation on the HICO-DET[8] dataset and unusual SSG generation
on the UnRel[9] dataset, where DDS outperforms all the existing models in both
datasets. Finally, the proposed design choices are evaluated in an extensive
ablation study.

##  II Related Works

DDS is built on the previously developed works in SSG and DSG generation. This
section is used to review the literature in the mentioned areas along with
additional relevant publications on scene-graph generation under the
compositional setting.

###  II-A Static Scene-Graph (SSG) Generation

SSG generation is proposed by[10] for the task of image retrieval. An
entensive literature exists [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32] in this area. The initial works[21,
22, 23, 24, 25, 26] rely heavily on two-stage (object detection and then
scene-graph generation) structures. Few of these works utilize different
recurrent neural network (RNN) variants[33, 34] while other prominent
researches focus on graph structure [26, 25, 20] with attention mechanisms.
Also, many authors utilize prior knowledge[26, 29, 35] (e.g. semantic
knowledge, statistical heuristics) for SSG generation. Despite recent
improvements in SSG generation, these methods are heavily constrained by their
reliance on the object detection quality as noted in[36].

Modern works [37, 38, 39, 40, 41, 42, 43, 36, 44, 45, 46] in SSG generation
focus on utilizing a one-stage Transformer based architecture to deal with the
aforementioned issues. These approaches mainly focus on human-object
interaction (HOI) detection. HOI detection is a subtask of SSG generation
where the relationships among objects are limited to interaction verb [3],
such as hold, work, talk. These methods employ one-stage Transformer based
encoder-decoder architectures following the architecture from DETR[6]. These
architectures rely on set-based predictions to generate SSG. Among these
works, Qpic[38] uses a single encoder-decoder model while CDN[42] extends Qpic
by using sequential decoding of objects and relationships. Additionally,
MSTR[41] enables the use of multi-scale feature maps to these networks.
Another concurrent work, SSRT[36] refines the overall architecture with
spatial and semantic support features. Moreover, a recent line of research
heavily exploits the usage of very large-scale semantic knowledge engines
(e.g. CLIP[47])[46, 36, 43]. Also, few works propose to utilize different
types of post-processing steps [46] for the task. Apart from the obvious
limitation of these works being unable to utilize temporal dependencies, they
perform poorly while detecting unseen triplets. With the decoupled multi-
branch design, we significantly differ from these works by using separate sets
of queries for relationship and object detection.

###  II-B Dynamic Scene-Graph (DSG) Generation

DSG is an extension of the SSG where the scene-graph is created for videos.
This process is harder since temporal cues need to be utilized[3, 4, 5].
Current works in this area have two-stage architectures following the initial
works on SSG. Among these works, STTran[5] utilizes a temporal decoder based
on the self-attention mechanism. DSGAP[4] expands STTran with an anticipatory
pre-training paradigm. On the other hand, HORT[3] utilizes a multi-branch
design with different types of Transformers.

Both STTran and HORT use similar features for relationship and object
detection. These features come from the object bounding boxes predicted by
off-the-shelf object detectors. STTran concatenates pair-wise object features
to use as relationship features whereas HORT pools relationship features from
a joint box of object pairs. However, using similar features for relationship
and object detection forces the learning of relationships and objects to be
dependent on each other. Therefore, to ensure generalized learning, we focus
on learning the features independently.

###  II-C Compositionality in Scene-Graph Generation

Creating new compositions from base known concepts during inference is known
as compositional zero-shot learning (CZSL) under the compositional setting[48,
49, 50, 51, 52]. In this paper, we utilize this setting to evaluate our model.
Kato et al. [53] introduce CZSL in SSG generation with an embedding-based
model. Many following works[54, 55, 56] adapt different object-affordance
ideas. These works assume there exist common relationships between the
subjects and the objects. This work does not have such limited assumptions,
and as a result, is able to generate scene graphs even when the relationships
are very unusual (See Table IV).

Figure 2: Overview of DDS’s architecture. Given an input frame
$\mathbf{I}_{t}$, features are extracted by the backbone. These features are
fed to the object and the relation branch. These decoupled branches consist of
an encoder and a spatio-temporal decoder. Encoders from both branches encode
the feature maps differently and send them to the decoders. Each spatio-
temporal decoder takes a set of queries (object/relation) along with the
previous frame’s embeddings (shown by the red arrow). The output of the
spatio-temporal decoders are learned embeddings. These learned embeddings are
fed to the object and the relation heads to predict relationship triplets.
Figure 3: Design of the spatio-temporal decoders. Every spatio-temporal
decoder is composed of a temporal and a spatial decoder. Each decoder converts
a different set of queries into learned embeddings while making sure decoupled
learning in each branch.

##  III Method

This section describes the developed model for the problem of dependent
feature learning in DSG generation that makes current models perform poorly in
detecting unseen relationship triplets. This work proposes a multi-branch
network that learns distinct feature representations for relationships and
objects to improve upon this limitation. Before going into the details of the
architecture, the problem will be formulated in detail first.

###  III-A Problem Formulation

Given an input video,
$\mathbf{V}=\\{\mathbf{I}_{1},\mathbf{I}_{2},..,\mathbf{I}_{t},..,\mathbf{I}_{T}\\}$
with $T$ frames, the task in DSG generation is to predict a set of
relationship triplets, $\\{R_{1},R_{2},..,R_{t},..,R_{M}\\}$ at every frame of
the video. Every frame has $N_{M}$ number of relationship triplets. Each
relationship triplet can be presented by $\langle s,o,r_{so}\rangle$. Here,
$s$, $o$ refers to subject, object and are represented by bounding boxes and
category labels. $r_{so}$ is the relationship between $s$ and $o$. In a single
frame $\mathbf{I}_{t}$, $s$ and $o$ can have multiple relations, as shown in
the sample input-output pair in Figure 2.

In this paper, the main goal is to predict relationship triplets under the
compositional setting. In this setting the test set contains triplets that are
not present in the training set. Consider having in total $N_{o}$ number of
objects and $N_{r}$ number of relations. The training set has $N_{s}$ number
of relationship triplets composed from the mentioned $N_{o}$ objects and
$N_{r}$ relationships. On the other hand, the test set has $N_{u}$ number of
unseen triplets not present in the training set in addition to the $N_{s}$
number of seen triplets, where all the unseen triplets are composed of the
same $N_{o}$ objects and $N_{r}$ relationships.

###  III-B Technical Overview

The proposed work adopts a one-stage approach for DSG generation compared to
the current two-stage methods[3, 4, 5] as the former [37, 38, 39, 40] have
shown impressive performance in creating SSG. However, these image-based works
present poor generalization capabilities. Therefore, we propose a network that
uses a different set of queries with two branches: the relation branch and the
object branch. Each branch follows a Transformer-like encoder-decoder
architecture. Fig. 2 shows a diagram of the model, where a convolutional
neural network (CNN) extracts features from the input frame, and those are
encoded differently by the object and the relation encoders. Each spatio-
temporal decoder takes encoded features from their respective encoder in
addition to two types of inputs: queries for the current frame
($\mathbf{oQ}_{t}$, $\mathbf{rQ}_{t}$) and the embeddings
($\mathbf{oE}_{t-1}$, $\mathbf{rE}_{t-1}$) propagated from the previous frame.
As the encoded features differ for each branch, the queries learn decoupled
features for relationships and objects. The decoder outputs are the learned
object and relation spatio-temporal embeddings. These embeddings are sent to
the relation and the object heads for final predictions. Moreover, these
embeddings are propagated to the next frame of the video.

###  III-C Feature Extraction & Encoders

Consider a frame $\mathbf{I}_{t}\in\mathbb{R}^{N_{C}\times H\times W}$ at time
t of the input video $V$. Here, $N_{C},H,W$ are the number of channels,
height, and width of the frame, $\mathbf{I}_{t}$. DDS uses a CNN network as
backbone, $\mathbf{B}$ (e.g. resnet-50[57]) to extract features
$\mathbf{B}(\mathbf{I}_{t})\in\mathbb{R}^{N_{C^{\prime}}\times
H^{\prime}\times W^{\prime}}$ from the input frame. Then, $1\times 1$
convolution is used to reduce the channel dimension from $N_{C^{\prime}}$ to
$d$. After that, a flattening operation is performed and a fixed positional
embedding is added to $\mathbf{B}(\mathbf{I}_{t})$ like existing works [37,
38, 39, 40] to get the feature map
$\mathbf{F}_{t}\in\mathbb{R}^{(H^{\prime}W^{\prime})\times d^{\prime}}$. These
embeddings express each spatial position of the feature map in high
dimensions[6]. DDS uses $\mathbf{F}_{t}$ as a common feature for both the
relation and the object branches.

Both of the network’s branches have an encoder comprising of stacked multi-
head self-attention layers[58] with a feed-forward network (FFN). The output
of the encoders are two separate feature maps,

$\displaystyle\mathbf{rF}_{t}=\text{Relation Encoder}(\mathbf{F}_{t})$ (1)
$\displaystyle\mathbf{oF}_{t}=\text{Object Encoder}(\mathbf{F}_{t})$ (2)

Here, $\mathbf{rF}_{t}$, $\mathbf{oF}_{t}$ refer differently encoded versions
of the feature map $\mathbf{F}_{t}$. Our spatio-temporal decoders will utilize
these feature maps for decoupled learning.

###  III-D Spatio-Temporal Decoders

DDS’s spatio-temporal decoders convert a set of learnable queries into output
embeddings. This transformation occurs in two stages. In the first stage, the
current frame’s queries attend to the previous frame’s output embeddings to
aggregate temporal information. In the second stage, these aggregated queries
gather information from the encoded feature maps of the current frame. The
proposed multi-branch design ensures discriminative feature learning for the
queries of each branch. Each decoder consists of two small components:
temporal and spatial decoder. Please see Fig. 3 for reference.

Temporal Decoders: These decoders allow queries to leverage temporal
dependencies. Each temporal decoder takes two sets as inputs: the current
frame’s queries and the embeddings from the previous frames. For frame
$\mathbf{I}_{t}$, the current frame’s relation and object queries sets are
defined as ${\mathbf{rQ}_{t}}\in\mathbb{R}^{N_{q}\times d}$ and
$\mathbf{oQ}_{t}\in\mathbb{R}^{N_{q}\times d}$. Every query is a $d$
dimensional vector, and every branch has $N_{q}$ number of queries. Embeddings
from the previous frames for the relation and the object branches are
presented by ${\mathbf{rE}_{t-1}}\in\mathbb{R}^{N_{q}\times d}$ and
${\mathbf{oE}_{t-1}}\in\mathbb{R}^{N_{q}\times d}$, and are marked with a red
arrow in Fig.3.

Temporal decoders are made of stacked multi-head cross-attention layers[6]
with an FFN network. The cross-attention in the temporal decoders allows the
current frame’s queries to select what to learn from the previous frame’s
embeddings. The outputs of the temporal decoders are the temporally aggregated
queries $\mathbf{rA}_{t}$, $\mathbf{oA}_{t}$. They are fed to their respective
spatial decoders. In the case of the first frame of a video, the temporal
decoders directly output $\mathbf{rQ}_{t}$ and $\mathbf{oQ}_{t}$ as
$\mathbf{rA}_{t}$ and $\mathbf{oA}_{t}$ without passing them through the
cross-attention and FFN blocks as there is no previous frame in this case.

Spatial Decoders: The spatial decoders architecture is similar to the standard
Transformer decoder[6]. These decoders consist of both self-attention and
cross-attention layers along with FFN networks. Each decoder takes encoded
feature maps ($\mathbf{rF}_{t}$ or $\mathbf{oF}_{t}$) along with the
aggregated queries of the temporal decoders ($\mathbf{rA}_{t}$ or
$\mathbf{oA}_{t}$) from their respective branch as inputs. Also, these
decoders take learnable positional embeddings. These embeddings for the
relation and the object branch are defined as
$\mathbf{rPE}\in\mathbb{R}^{N_{q}\times d}$,
$\mathbf{oPE}\in\mathbb{R}^{N_{q}\times d}$.

$\displaystyle\mathbf{rE}_{t}=\text{Relation
Decoder}(\mathbf{rN}_{t},\mathbf{rPE},\mathbf{rQ}_{t})$ (3)
$\displaystyle\mathbf{oE}_{t}=\text{Object
Decoder}(\mathbf{oN}_{t},\mathbf{oPE},\mathbf{oQ}_{t})$ (4)

The outputs of the decoders are the learned spatio-temporal embeddings. They
are used in the object and the relation heads to make the final relationship
triplet predictions. Also, each spatial decoder’s output is fed to the next
frame’s temporal decoder as previous embeddings.

Both spatial and temporal decoders keep the separation in feature learning for
relationships and objects using different encoded features, set of queries,
and previous embeddings. As a result, the output embeddings from the decoders
are decoupled generalized representations.

###  III-E Object Heads

The output embeddings from the object spatio-temporal decoder,
$\mathbf{oE}_{t}$ are fed to four different FFNs. For input frame
$\mathbf{I}_{t}$, these FFNs predict subject bounding boxes,
$\mathbf{sB}_{t}\in[0,1]^{N_{q}\times 4}$, object bounding boxes,
$\mathbf{oB}_{t}\in[0,1]^{N_{q}\times 4}$, subject prediction vectors,
$\mathbf{sP}_{t}\in[0,1]^{N_{q}\times O}$, and object prediction vectors,
$\mathbf{oP}_{t}\in[0,1]^{N_{q}\times N_{o}}$. Here, $N_{q}$ is the number of
queries, and $N_{o}$ is the total number of objects.

###  III-F Relation Heads

Like the object heads, the learned output embeddings, $\mathbf{rE}_{t}$, are
fed to two FFNs that produce as output the relation prediction vectors,
$\mathbf{rP}_{t}[0,1]^{N_{q}\times N_{r}}$ and relation region bounding boxes
$\mathbf{rB}_{t}\in[0,1]^{N_{q}\times 4}$. Here, $N_{q}$ is the number of
queries, and $N_{r}$ is the total number of relationships under consideration.
Notice that the relation region bounding box is defined as the union between
the subject and object bounding boxes.

###  III-G Inference

We compose $N_{q}$ relationship pairs by one-to-one matching of
$\mathbf{sB}_{t}$ and $\mathbf{oB}_{t}$. One-to-one matching refers to
matching the $q$-th prediction from $\mathbf{sB}_{t}$ with the $q$-th
prediction from $\mathbf{oB}_{t}$. Moreover, for every prediction vector in
$\mathbf{sP}_{t}$ and $\mathbf{oP}_{t}$, the maximum confidence score is used
to create $\mathbf{sP}_{tmax}\in[0,1]^{N_{q}}$ and
$\mathbf{oP}_{tmax}\in[0,1]^{N_{q}}$ and the corresponding index is used to
determine the category label for each of the bounding boxes. For every
composed relationship pair, the final relation score prediction vectors are
calculated as:

$\mathbf{rP}_{tfinal}=\mathbf{rP}_{t}*\mathbf{sP}_{tmax}*\mathbf{oP}_{tmax}$
(5)

###  III-H Training

For training DDS, we utilize losses similar to Qpic[38]. This loss calculation
implicitly binds the two sets of queries from the relation and the object
branch. The loss calculation happens in two stages:

In the first stage, we find the bipartite matching between the predictions and
the ground truths. First, the total prediction set for the input frame
$\mathbf{I}_{t}$ as,
$\mathbf{P}_{t}=\\{\mathbf{sB}_{t},\mathbf{oB}_{t},\mathbf{sP}_{t},\mathbf{oP}_{t},\mathbf{rB}_{t},\mathbf{rP}_{t}\\}$
is generated. This yields $N_{q}$ number (equal to the number of queries in
each branch) of predictions. $N_{q}$ is chosen in such a way that it is always
greater than the number of ground truths per frame. We pad ground truths with
$\phi$ (no relationship triplet) so that it is possible to have the ground
truth set $\mathbf{G}_{t}$ with $N_{q}$ number of elements. One important
detail to note here is that there are three kinds of ground truth bounding
boxes: subject bounding boxes, object bounding boxes, and relation regions.
Ground truth relation regions refer to the union bounding boxes between the
subject and the object bounding boxes that have relations, and are only used
during the training phase. Next, each element in $\mathbf{P}_{t}$ is matched
with an element from the ground truth set, $\mathbf{G}_{t}$. The matching cost
metrics is, $\mathbf{C}\in\mathbb{R}^{[N_{q}\times N_{q}}]$. Any element
$(i,j)$ in this metrics refers to the cost to match $i^{th}$ element from
$\mathbf{P}_{t}$ with $j^{th}$ element from $\mathbf{G}_{t}$ and defined as,

${\mathbf{C}^{(i,j)}}=\eta_{b}(\mathbf{C}^{(i,j)}_{sb}+\mathbf{C}^{(i,j)}_{ob}+\mathbf{C}^{(i,j)}_{rb})+\eta_{o}\mathbf{C}^{(i,j)}_{o}+\eta_{r}\mathbf{C}^{(i,j)}_{r}$
(6)

$\mathbf{C}^{(i,j)}_{sb},\mathbf{C}^{(i,j)}_{ob},\mathbf{C}^{(i,j)}_{rb}$ are
the subject bounding box, the object bounding box and the relation region
matching costs, $\mathbf{C}^{(i,j)}_{o}$ is the object label matching cost,
and $\mathbf{C}^{(i,j)}_{r}$ is the relation label matching cost between
$i^{th}$ element from $\mathbf{P}_{t}$ and $j^{th}$ element from
$\mathbf{G}_{t}$. These costs are calculated following[38].
$\eta_{b},\eta_{o},\eta_{r}$ are fixed hyper-parameters. The Hungarian
matching algorithm[6] is used to find the optimal matching between the
predictions and the ground truths by using these cost metrics. After this
matching, every prediction is associated with a ground truth. Next, the
following loss is calculated for training the network:

$\mathcal{L}=\lambda_{g}\mathcal{L}_{GIOU}+\lambda_{l}\mathcal{L}_{L1}+\lambda_{o}\mathcal{L}_{obj}+\lambda_{r}\mathcal{L}_{rel},$
(7)

Here, $\mathcal{L}_{GIOU}$ and $\mathcal{L}_{L1}$ are the generalized
intersection over union (gIOU) and L$1$ box regression losses for the
predicted subject bounding boxes, object bounding boxes, and relation regions.
$\mathcal{L}_{obj}$ is the cross-entropy loss for subject and object label
predictions. $\mathcal{L}_{rel}$ is the binary cross-entropy loss for the
relationship label predictions. $\lambda_{o}$, $\lambda_{g}$, $\lambda_{l}$,
and $\lambda_{r}$ are the corresponding hyper-parameters.

Notice that a portion of the datasets [7, 8] fixes the subject as humans. In
this case, the subject prediction vectors and subject bounding boxes are not
used for loss calculation.

Seen Unseen R@20 R@50 R@20 R@50 STTran[5]* 33.7 36.6 0.3 4.4 DDS (Ours) 41.8
48.8 7.4 18.2

TABLE I: DDS’s performance comparison in AG test set under the compositional
setting. Both reported models are trained on the proposed small-size training
set under the compositional setting. * means the model was trained using
publicly available code. Among recent DSG generation models, only STTran’s[5]
code is publicly available. The best results are shown in bold.

Method Backbone R@20 R@50 GPNN[59] ResNet-101 33.3 42.6 PPDM[60] Hourglass-104
34.1 43.5 VRD[61] ResNet-101 22.0 32.7 IMP[33] ResNet-101 34.4 43.7 MSDN[62]
ResNet-101 34.7 43.8 Graph RCNN[63] ResNet-101 35.0 44.1 RelDN [12, 64]
ResNet-101 35.1 44.9 VCTree[12] ResNet-101 35.0 44.3 STTran[5] ResNet-101 36.2
48.8 HORT[3] ResNet-101 36.5 46.7 DSGAP[4] ResNet-101 37.9 50.1 DDS (Ours)
ResNet-50 43.3 51.5

TABLE II: DDS’s performance comparison in AG test set. Here, like other
models, DDS is trained in the full training set of AG dataset. The best
results are shown in bold. For the other models, all the reported numbers are
taken from the original publications.

Method Unseen (mAP) Seen (mAP) Full (mAP) VCL[30] 10.1 24.3 21.4 ATL[54] 9.2
24.7 21.6 FCL[56] 13.2 24.2 22.0 THID[65] 15.5 24.3 23.0 SCL[55] 19.1 30.4
28.1 DDS (Ours) 21.1 31.7 29.6

TABLE III:  DDS’s performance comparison in HICO-Det test set under RF (Rare-
First) compositional setting. The best results are shown in bold.

Method Unseen (mAP) Seen (mAP) Full (mAP) VRD[61] - - 7.2 WSL[9] - - 9.9
DUV[66] - - 13.4 DDS (Ours) 16.3 27.4 17.9

TABLE IV:  DDS’s performance comparison in UnRel test set. The best results
are shown in bold. Not reported results are marked with -.

##  IV Experiments

###  IV-A Experimental Setup

We evaluate DDS’s performance in Action Genome (AG)[7] dataset. Moreover, we
show our model’s performance in SSG generation datasets: HICO-DET[8] and
UnRel[9]. As these datasets only contain images, each sample is treated as a
single-frame video. Next, the used datasets will be presented in more detail:

Action Genome (AG)[7]: This dataset is built on top of the Charades[67]
dataset, provides frame-level annotations, and is extensively used in the
literature for DSG generation. It has 36 distinct object classes and 25
relationship classes. The object classes are common household items such as
doors, windows, and cups, and have a total (train and test set) of $476,229$
bounding boxes. The relationship classes are divided into 3 distinct sub-
types: (1) Attention relationship denotes if the subject is looking at an
object. (2) Spatial relationship denotes the spatial location of the object
with respect to the subject, for example, above, or below. (3) Contacting
relationship represents how the object is being contacted by the subject. In
total, AG provides $1,715,568$ instances of the mentioned classes contained in
$135,484$ subject-object pairs. Every subject-object pair can have multiple
relations. Also, on AG the subject class is always human.

Originally, the AG dataset provided $7,464$ videos with $166,785$ frames in
the training set and $1,737$ videos with $54,371$ frames in the test set. The
original training set contains $530$ relationship triplets. All these
relationship triplets are present in the test set. We refer to this setting as
the fully-supervised setting. As the main interest in this paper is to
evaluate DDS’s performance in the compositional setting, a new training split
of the data is created. This new proposed training set contains $6,784$ videos
with $146,517$ frames containing $421$ relationship triplets. The original
test set is not changed. It contains $499$ object-relationship, where $80$ of
them are not present in our new training set.

Type Relation Region Separate Encoders Separate Spatio-temporal Decoders
Unseen (mAP) Seen (mAP) Single branch (base network) ✗ ✗ ✗ 17.9 29.9 Multi
branch ✗ ✗ ✓ 18.7 30.5 ✗ ✓ ✓ 19.7 31.6 ✓ ✓ ✓ 21.1 31.7

TABLE V:  Impact of different components on our decoupled multi-branch design.

Unseen (mAP) Seen (mAP) DDS with o to r 19.6 30.6 DDS with r to o 19.4 28.8
DDS with separate queries 21.1 31.7

TABLE VI: Different kinds of query sharing strategy. o to r refers to object
to relation branch query sharing. r to o refers to relation to object branch
query sharing.

Relation Region Sub-Obj IoU, $\theta$ Unseen (mAP) Seen (mAP) Mixture 0 19.2
30.3 0.1 18.0 30.4 0.5 19.4 30.8 Union Box - 21.1 31.7

TABLE VII: Different kinds of relation region ground truths. Sub-Obj IoU
refers to the IoU between the subject and the object bounding boxes.

Object Spatial Decoder Relation Spatial Decoder Unseen (mAP) Seen (mAP) 3 3
20.2 31.4 3 6 20.8 31.8 6 3 21.1 31.7 6 6 18.0 31.3

TABLE VIII: Different number of layers in the object spatial decoder and the
relation spatial decoder.

HICO-Det[8]: This dataset has $80$ objects and $117$ relationship classes. The
relationships are limited to interactions such as holding, working, etc. In
the literature, this dataset is used for evaluating SSG generation performance
under compositional setting[56, 54, 55, 30]. DDS’s performance is reported
following the RF (Rare First) protocol provided by [30]. This protocol has
$37,328$ images in the training set with $480$ relationship triplets. The test
set has $9,552$ images with $480$ seen relationship triplets and $120$ unseen
relationship triplets.

UnRel[9]: This dataset provides extremely unusual SSG triplets, for example:
($\langle elephant,bike,riding\rangle$). It has $4000$ training and $1000$
test images with $100$ objects and $70$ relationships. The original train/test
split provided by the authors already provides a compositional setting where
the test set has $65$ unseen relationship triplets. The training set contains
$4000$ seen relationship triplets.

###  IV-B Evaluation Metrics

Following existing works [7, 3, 4, 5], we report our performances in AG
dataset with Recall@K metric. Here, $K=[20,50]$. We utilize the most
challenging SGDet[7] protocol to report our performances. In this protocol,
the network needs to detect relationship triplets along with subject and
object bounding boxes. There can be multiple relationship triplets between a
subject-object pair. Moreover, mAP (mean average precision) was selected to
report the performances in UnRel and HICO-Det datasets similar to current
works[9, 66, 54, 56]. Here, performances are reported in three categories:
unseen (only unseen relationship triplets), seen (only seen relationship
triplets), and full (all relationship triplets) [30]. For all datasets, a
prediction triplet from DDS is considered correct if subject and object
bounding boxes have at least $0.5$ Intersection over Union (IoU) with ground
truth bounding boxes and subject, object, and relationship labels match with
ground truth labels.

###  IV-C Implementation details

ResNet-50[57] is used as the CNN backbone. Both temporal decoders inside the
spatio-temporal decoders have a single layer. We follow Qpic’s [38] setup for
the encoders. We select $6$ layers for the object spatial decoder with $3$
layers for the relation spatial decoder. All loss coefficients in equation 6
and equation 7 are set as [38]. The number of queries in each branch is $64$.
Each query is a vector of size $256$. The model is trained with AdamW[68]
optimizer. We initialize the parameters of DDS from DETR[6] trained on
COCO[69] object detection dataset. The initial learning rate for the backbone
network is $10^{-6}$ and for the other part of the network is $10^{-5}$.

When training in the AG dataset, we drop the learning rate by $10$ times at
every $40$ epochs and utilize a batch size of $128$. DDS processes each video
frame from a single video sequentially. We utilize scale augmentation like[6].
Input frames are resized with the shortest side being at least 480 and at most
800 pixels, and longest side is at most 800.

In the other datasets[8, 9], the learning rate is dropped by $10$ times at
every $60$ epochs with a batch size of $16$. We use a scale augmentation
scheme similar to the one used for AG, except that the longest side of the
resized image is chosen as $1333$. The training schedule is selected based on
the convergence of losses. Upon acceptance, we will publicly release our
trained models and code.

##  V Results & Analysis

In this section, we first compare DDS’s performance with the SOTA models in
Section V-A. Next, a detailed study on the impact of different components of
our network is presented in Section V-B. Finally, in Section V-C qualitative
results from our model are provided.

###  V-A Comparison with the SOTA models

In the AG[7] dataset, we report DDS’s performances in Table I under the
compositional setting. In this setting, there are $\sim 12\%$ less training
data with $80$ unseen relationship triplets. We retrain the SOTA model
STTRan[5] in the mentioned setting for comparison. It is important to note
here, among the three recent DSG generation models[3, 5, 4], only STTran’s
code is publicly available, therefore limiting the capacity to evaluate other
models. DDS outperforms STTran in all recall levels in both unseen and seen
relationship triplet detection. Especially, for detecting unseen triplets DDS
achieves $4-24$ times improvement over the SOTA model. It shows the
generalization power of DDS.

Additionally, for a fair comparison with other models, we train DDS in the
full training set of AG dataset and report performance in Table II. Here, all
methods up to VCTree[12] are SSG generation methods implemented by [5]. As
expected, with similar training data like other models in Table II, DDS
achieves SOTA performance on all metrics.

In Table III and Table IV, DDS outperforms all existing methods in HICO-Det[8]
and UnRel[9] datasets. Among these methods, [65] and [55] follow a Transformer
based encoder-decoder architecture. The proposed network shows superior
performance than both of the models. In summary, DDS outperforms existing
works in both seen and unseen SSG generation (full category) in HICO-Det by
$5\%$ and in UnRel by $33\%$.

###  V-B Ablation Studies

We perform ablations for different design choices of our network in this
section. For these ablations, we report our performances in the HICO-Det[8]
dataset.

Figure 4: Qualitative results of DDS for predicting unusual relationship
triplets in UnRel[9] dataset. The subject bounding box is green and the object
bounding box is red. Our base network (single branch) fails to detect these
marked triplets. For both networks, we utilize top-$20$ predictions per
sample. Figure 5: Performance analysis of DDS over the base network. The
subject bounding box is green and the object bounding box is red. The
attention maps are visualized from the last layer of the spatio-temporal
decoder.

Multi-branch Design: We first validate our decoupled multi-branch design. The
performances are reported in Table V. The base network has one single branch
where a single set of queries is used for both relationship and object
detection. In row 1 of Table V the performance is pretty poor for the single
branch base network, especially in the unseen category. Next, a multi-branch
network is created by using two spatio-temporal decoders. Both decoders get
the same encoded features. However, the relation branch in this case doesn’t
do relation region prediction. A performance improvement is observed compared
to the single branch for this design. Similarly, with the gradual introduction
of two encoders and relation region prediction, the performance keeps
increasing. In particular, all these components yield a significant
improvement in the unseen category compared to the seen category and thus show
the impact of decoupled learning on unseen relationship triplet predictions.
We also do qualitative analysis between DDS and the base network described in
the section V-C.

Relation Region Ground Truths: As noted in Section III-F the relation branch
only produces relation region prediction during training. Ground truth
relation region is required for the training. However, ground truth relation
regions are not strictly defined by the provided annotations such as subject
and object bounding boxes. Therefore, we experiment with different settings to
generate ground truth relation regions for subject and object pairs that have
relationships:

  * •

Mixture: In this case, the intersection of the subject, and object bounding
boxes is selected as the relation region if the boxes have an IoU greater than
the $\theta$. For any other case, the relation region is defined as their
union boxes. Different values of $\theta$ are tested.

  * •

Union box: Here, the union of the subject and object bounding boxes is defined
as the relation region.

The performances of the network with different relation region ground truths
are shown in Table VII. With union boxes, DDS performs best (last row). This
may be due to the fact that the union box guarantees the inclusion of the
spatial location of the relation and therefore it can be very helpful in
detecting non-contact relationship triplets (e.g. subject looks at object
etc.).

Share Queries: DDS utilizes two different sets of queries for the relation and
the object branches. We also test DDS’s performance by sharing the queries
between the branches with two different strategies:

  * •

DDS with o to r: In this case, the output of the object spatio-temporal
decoder is fed as input relation queries to the relation spatio-temporal
decoder.

  * •

DDS with r to o: In this case, the output of the relation spatio-temporal
decoder is fed as input object queries to the object spatio-temporal decoder.

The performances are reported in Table VI. Without sharing the queries DDS
performs the best, especially in unseen categories. This matches our
hypothesis on the importance of decoupled learning by utilizing two different
sets of queries. We also notice a significant performance drop even in seen
classes when we share queries from the relation to the object branch ($2$nd
row). This shows object queries have more generalization ability than the
relation queries.

Spatial Decoders: We test with different numbers of layers for spatial
decoders. The result is shown in Table VIII. DDS performs poorly in extreme
cases (first and last row). This is expected as DDS is getting underfitted due
to the decrease in the number of layers (first row) and overfitted as the
number of layers increases (last row) in the relation branch. Also, the best
performance in the unseen category arises when the object decoder has more
layers than the relation decoder. Given that the object spatial decoder
decodes two entities (subject, object), it is reasonable to require more
layers than the relation spatial decoder.

###  V-C Qualitative Results

This section compares DDS’s performance with our base network, a single branch
network with one encoder, and one spatio-temporal decoder (details in section
V-B). This comparison is made in the UnRel[9] dataset that has very unusual
unseen relationship triplets. Fig. 4 illustrates the samples where DDS is
successful; however, our base network completely misses the relationship
triplets (predicted bounding boxes do not match with ground truth). We utilize
the most confident $20$ predictions for each sample from both networks for
this visualization.

We compare the attention maps from DDS and the base network to further analyze
our improved performances. Fig. 5 presents attention maps for the samples
where both DDS and the base network have correct predictions. The attention
maps are of the queries that predict the marked subject and object bounding
boxes from the last layer of the spatio-temporal decoders. We overlap
attention maps from both our spatio-temporal decoders to get the final
attention map. As can be seen, although both networks have correct
predictions, DDS’s attention maps cover the correct spatial region. In
contrast, the base network with only one spatio-temporal decoder has attention
on a very random portion of the object and the subject.

##  VI Conclusion

This paper proposes a multi-branch decoupled network for DSG generation. The
DDS network is comprised of two encoder-decoder based Transformer branches.
This design enables independent learning of objects and relationships, thus
enabling DDS to detect unseen relationship triplets. The effectiveness of DDS
is demonstrated through extensive experiments with DDS achieving SOTA
performance on three benchmark datasets. Moreover, the conducted ablation
studies have provided the motivation and significance for different components
of DDS. However, while successful compared to the existing works, the
quantitative results (Table II, III, IV) show room for improvement in
detecting unseen relationship triplets. Future research will focus on
improving DDS for a better generalized DSG generation.

## Acknowledgments

This research is partially supported by the following grants: US Army Research
Laboratory (ARL) under agreement number W911NF2020157; and by NSF award
SI2-SSI #1664172. The U.S. Government is authorized to reproduce and
distribute reprints for Governmental purposes notwithstanding any copyright
notation thereon. The views and conclusions contained herein are those of the
authors and should not be interpreted as necessarily representing the official
policies or endorsements, either expressed or implied, of US Army Research
Laboratory (ARL) or the U.S. Government.

## References

  * [1] O. Ulutan, S. Rallapalli, M. Srivatsa, C. Torres, and B. Manjunath, “Actor conditioned attention maps for video action detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 527–536.
  * [2] X. Liu, P. Ghosh, O. Ulutan, B. Manjunath, K. Chan, and R. Govindan, “Caesar: cross-camera complex activity recognition,” in _Proceedings of the 17th Conference on Embedded Networked Sensor Systems_ , 2019, pp. 232–244.
  * [3] J. Ji, R. Desai, and J. C. Niebles, “Detecting human-object relationships in videos,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8106–8116.
  * [4] Y. Li, X. Yang, and C. Xu, “Dynamic scene graph generation via anticipatory pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 874–13 883.
  * [5] Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Y. Yang, “Spatial-temporal transformer for dynamic scene graph generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 16 372–16 382.
  * [6] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [7] J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 236–10 247.
  * [8] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng, “Learning to detect human-object interactions,” in _2018 ieee winter conference on applications of computer vision (wacv)_. IEEE, 2018, pp. 381–389.
  * [9] J. Peyre, J. Sivic, I. Laptev, and C. Schmid, “Weakly-supervised learning of visual relations,” in _Proceedings of the ieee international conference on computer vision_ , 2017, pp. 5179–5188.
  * [10] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3668–3678.
  * [11] Y. Lu, H. Rai, J. Chang, B. Knyazev, G. Yu, S. Shekhar, G. W. Taylor, and M. Volkovs, “Context-aware scene graph generation with seq2seq transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 931–15 941.
  * [12] K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu, “Learning to compose dynamic tree structures for visual contexts,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 6619–6628.
  * [13] Y. Cong, H. Ackermann, W. Liao, M. Y. Yang, and B. Rosenhahn, “Nodis: Neural ordinary differential scene understanding,” in _European Conference on Computer Vision_. Springer, 2020, pp. 636–653.
  * [14] W. Wang, R. Wang, S. Shan, and X. Chen, “Exploring context and visual pattern of relationship for scene graph generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8188–8197.
  * [15] J. Shi, Y. Zhong, N. Xu, Y. Li, and C. Xu, “A simple baseline for weakly-supervised scene graph generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 16 393–16 402.
  * [16] W. Wang, R. Wang, and X. Chen, “Topic scene graph generation by attention distillation from caption,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15 900–15 910.
  * [17] L. Chen, H. Zhang, J. Xiao, X. He, S. Pu, and S.-F. Chang, “Counterfactual critic multi-agent training for scene graph generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4613–4623.
  * [18] M.-J. Chiou, H. Ding, H. Yan, C. Wang, R. Zimmermann, and J. Feng, “Recovering the unbiased scene graphs from the biased ones,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 1581–1590.
  * [19] H.-S. Fang, Y. Xie, D. Shao, and C. Lu, “Dirv: Dense interaction region voting for end-to-end human-object interaction detection,” in _The AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [20] C. Gao, Y. Zou, and J.-B. Huang, “ican: Instance-centric attention network for human-object interaction detection,” in _British Machine Vision Conference_ , 2018.
  * [21] Y.-L. Li, S. Zhou, X. Huang, L. Xu, Z. Ma, H.-S. Fang, Y. Wang, and C. Lu, “Transferable interactiveness knowledge for human-object interaction detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3585–3594.
  * [22] Y. Liu, Q. Chen, and A. Zisserman, “Amplifying key cues for human-object-interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 248–265.
  * [23] B. Wan, D. Zhou, Y. Liu, R. Li, and X. He, “Pose-aware multi-level feature network for human object interaction detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9469–9478.
  * [24] T. Wang, R. M. Anwer, M. H. Khan, F. S. Khan, Y. Pang, L. Shao, and J. Laaksonen, “Deep contextual attention for human-object interaction detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5694–5702.
  * [25] C. Gao, J. Xu, Y. Zou, and J.-B. Huang, “Drg: Dual relation graph for human-object interaction detection,” in _Proc. European Conference on Computer Vision (ECCV)_ , 2020.
  * [26] O. Ulutan, A. Iftekhar, and B. S. Manjunath, “Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 617–13 626.
  * [27] X. Zhong, C. Ding, X. Qu, and D. Tao, “Polysemy deciphering network for human-object interaction detection,” in _Proc. Eur. Conf. Comput. Vis_ , 2020\.
  * [28] D.-J. Kim, X. Sun, J. Choi, S. Lin, and I. S. Kweon, “Detecting human-object interactions with action co-occurrence priors,” in _European Conference on Computer Vision_. Springer, 2020, pp. 718–736.
  * [29] Y. Liu, J. Yuan, and C. W. Chen, “Consnet: Learning consistency graph for zero-shot human-object interaction detection,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 4235–4243.
  * [30] Z. Hou, X. Peng, Y. Qiao, and D. Tao, “Visual compositional learning for human-object interaction detection,” in _ECCV_ , 2020.
  * [31] H. Wang, W.-s. Zheng, and L. Yingbiao, “Contextual heterogeneous graph network for human-object interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 248–264.
  * [32] Y.-L. Li, X. Liu, X. Wu, Y. Li, and C. Lu, “Hoi analysis: Integrating and decomposing human-object interaction,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 5011–5022.
  * [33] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, “Scene graph generation by iterative message passing,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5410–5419.
  * [34] R. Zellers, M. Yatskar, S. Thomson, and Y. Choi, “Neural motifs: Scene graph parsing with global context,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5831–5840.
  * [35] A. Iftekhar, S. Kumar, R. A. McEver, S. You, and B. Manjunath, “Gtnet: Guided transformer network for detecting human-object interactions,” _arXiv preprint arXiv:2108.00596_ , 2021.
  * [36] A. Iftekhar, H. Chen, K. Kundu, X. Li, J. Tighe, and D. Modolo, “What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5353–5363.
  * [37] B. Kim, J. Lee, J. Kang, E.-S. Kim, and H. J. Kim, “Hotr: End-to-end human-object interaction detection with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 74–83.
  * [38] M. Tamura, H. Ohashi, and T. Yoshinaga, “Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 410–10 419.
  * [39] M. Chen, Y. Liao, S. Liu, Z. Chen, F. Wang, and C. Qian, “Reformulating hoi detection as adaptive set prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9004–9013.
  * [40] C. Zou, B. Wang, Y. Hu, J. Liu, Q. Wu, Y. Zhao, B. Li, C. Zhang, C. Zhang, Y. Wei _et al._ , “End-to-end human object interaction detection with hoi transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 825–11 834.
  * [41] B. Kim, J. Mun, K.-W. On, M. Shin, J. Lee, and E.-S. Kim, “Mstr: Multi-scale transformer for end-to-end human-object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 578–19 587.
  * [42] A. Zhang, Y. Liao, S. Liu, M. Lu, Y. Wang, C. Gao, and X. Li, “Mining the benefits of two-stage and one-stage hoi detection,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 209–17 220, 2021.
  * [43] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and S. Liu, “Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 123–20 132.
  * [44] J. Park, S. Lee, H. Heo, H. K. Choi, and H. J. Kim, “Consistency learning via decoding path augmentation for transformers in human object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1019–1028.
  * [45] D. Zhou, Z. Liu, J. Wang, L. Wang, T. Hu, E. Ding, and J. Wang, “Human-object interaction detection via disentangled transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 568–19 577.
  * [46] X. Qu, C. Ding, X. Li, X. Zhong, and D. Tao, “Distillation using oracle queries for transformer-based human-object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 558–19 567.
  * [47] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [48] I. Misra, A. Gupta, and M. Hebert, “From red wine to red tomato: Composition with context,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1792–1801.
  * [49] K. Wei, M. Yang, H. Wang, C. Deng, and X. Liu, “Adversarial fine-grained composition learning for unseen attribute-object recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [50] S. Purushwalkam, M. Nickel, A. Gupta, and M. Ranzato, “Task-driven modular networks for zero-shot compositional learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3593–3602.
  * [51] M. F. Naeem, Y. Xian, F. Tombari, and Z. Akata, “Learning graph embeddings for compositional zero-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 953–962.
  * [52] S. Kumar, A. Iftekhar, E. Prashnani, and B. Manjunath, “Locl: Learning object-attribute composition using localization,” _arXiv preprint arXiv:2210.03780_ , 2022.
  * [53] K. Kato, Y. Li, and A. Gupta, “Compositional learning for human object interaction,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 234–251.
  * [54] Z. Hou, B. Yu, Y. Qiao, X. Peng, and D. Tao, “Affordance transfer learning for human-object interaction detection,” in _CVPR_ , 2021.
  * [55] Z. Hou, B. Yu, and D. Tao, “Discovering human-object interaction concepts via self-compositional learning,” in _ECCV_ , 2022.
  * [56] Z. Hou, B. Yu, Y. Qiao, X. Peng, and D. Tao, “Detecting human-object interaction via fabricated compositional learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 646–14 655.
  * [57] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [58] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [59] S. Qi, W. Wang, B. Jia, J. Shen, and S.-C. Zhu, “Learning human-object interactions by graph parsing neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 401–417.
  * [60] Y. Liao, S. Liu, F. Wang, Y. Chen, C. Qian, and J. Feng, “Ppdm: Parallel point detection and matching for real-time human-object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 482–490.
  * [61] C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei, “Visual relationship detection with language priors,” in _European conference on computer vision_. Springer, 2016, pp. 852–869.
  * [62] Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang, “Scene graph generation from objects, phrases and region captions,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1261–1270.
  * [63] J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph r-cnn for scene graph generation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 670–685.
  * [64] K. Tang, Y. Niu, J. Huang, J. Shi, and H. Zhang, “Unbiased scene graph generation from biased training,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 3716–3725.
  * [65] S. Wang, Y. Duan, H. Ding, Y.-P. Tan, K.-H. Yap, and J. Yuan, “Learning transferable human-object interaction detector with natural language supervision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 939–948.
  * [66] J. Peyre, I. Laptev, C. Schmid, and J. Sivic, “Detecting unseen visual relations using analogies,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1981–1990.
  * [67] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta, “Hollywood in homes: Crowdsourcing data collection for activity understanding,” in _European Conference on Computer Vision_. Springer, 2016, pp. 510–526.
  * [68] L. Ilya, H. Frank _et al._ , “Decoupled weight decay regularization,” _Proceedings of ICLR_ , 2019.
  * [69] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.


# Burch index, summands of syzygies and linearity in resolutions

Hailong Dao  Hailong Dao
Department of Mathematics
University of Kansas
405 Snow Hall, 1460 Jayhawk Blvd.
Lawrence, KS 66045<EMAIL_ADDRESS>and  David Eisenbud  Department of
Mathematics, University of California at Berkeley and the Mathematical
Sciences Research Institute, Berkeley, CA 94720, USA<EMAIL_ADDRESS>
###### Abstract.

The Burch index is a new invariant of a local ring $R$ whose positivity
implies a kind of linearity in resolutions of $R$-modules. We show that if $R$
has depth zero and Burch index at least $2$, then any non-free 7th $R$-syzygy
contains the residue field as a direct summand. We compute the Burch index in
various cases of interest.

###### Key words and phrases:

Burch rings, Burch index, free resolutions, linearity of syzygies, summands of
syzygies

###### MSC2020 Mathematics Subject Classification:

Primary: 13D02, 13H10. Secondary: 14B99

## Introduction

Let $(R,\mathfrak{m},k)$ be a commutative, Noetherian local ring. Little is
known about the matrices that can occur in the minimal free resolutions of a
finitely generated $R$-module of infinite projective dimension. In this paper
we study conditions on $R$, expressed by an invariant that we call the _Burch
index_ , that imply that these matrices contain many elements outside the
square of the maximal ideal, or that the syzygies of $M$ contain $k$ as a
direct summand.

The second condition is generally stronger than the first: by the main theorem
of Gulliksen [8], the Koszul complex of the maximal ideal $\mathfrak{m}$ of
$R$ is a tensor factor of the minimal $R$-free resolution of $k$, so, when
$\dim_{k}(\mathfrak{m}/\mathfrak{m}^{2})\geq 2$, the entries of each matrix in
the minimal free resolution of $k$ generate $\mathfrak{m}$. It follows that
the same holds for any module with $k$ as a direct summand.

To exclude the phenomena associated with modules of finite projective
dimension, it is useful to consider the case of rings of depth 0, and the
Burch index of an arbitrary Noetherian local ring is defined in Section 1 by
reducing to this setting. Suppose that $R$ is a depth 0 local ring with
residue field $k$, and that $R=S/I$, where $(S,\mathfrak{n},k)$ is a regular
local ring. Note that $\mathfrak{n}^{2}\subset
I\mathfrak{n}:(I:\mathfrak{n})\subset\mathfrak{n}$. We define the _Burch
index_ of $R$ (relative to $S$) to be

$\operatorname{{Burch}}_{S}(R)=\dim_{k}\frac{\mathfrak{n}}{I\mathfrak{n}:(I:\mathfrak{n})}.$

Intuitively this measures the minimal generators of $I$ that are multiples of
elements in the socle of $S/I$. Our definition is inspired by ([5]) in which a
local ring $(R,\mathfrak{m},k)$ is said to be a _Burch ring_ if, in our terms,
it has Burch index $\geq 1$. It is proved there that $R$ is Burch if and only
if the second syzygy of $k$ has $k$ as a direct summand. See that paper for
background.

It turns out that if $R$ has Burch index $\geq 2$ then similar things are
true, not just for $k$, but for _every_ module of infinite projective
dimension:

######  Theorem 0.1.

Suppose that $(R,\mathfrak{m},k)$ is a local ring of depth 0 with
$\operatorname{{Burch}}(R)\geq 2$. If $M$ is an $R$-module that is not free,
then the entries of each matrix in the minimal free resolution of
$\operatorname{syz}^{R}_{5}(M)$ generate $\mathfrak{m}$. Furthermore, $k$ is a
direct summand of $\operatorname{syz}_{n}^{R}(M)$ for all $n\geq 7$.

See Theorem 3.1 for a stronger but more complicated result.

Theorem 0.1 seems to be new even in the case of the rings
$R=k[x_{1},\dots,x_{d}]/(x_{1},\dots,x_{d})^{n}$ ($n\geq 2$), which have Burch
index $d$. Theorem 0.1 implies immediately that over a ring of Burch index at
least $2$, each module has either finite projective dimension or maximal
curvature (that is, its Poincare series has the same radius of convergence as
that of the residue field). See [1] for a survey of the vast literature on
this subject. Also, any high enough (non-free) syzygy $M$ over such ring can
be used to test for finite projective dimension, in the sense that
$\operatorname{{Tor}}_{i}^{R}(M,N)=0$ for a single $i>0$ forces $N$ to have
finite projective dimension.

If $M$ is a finitely generated $R$-module, we write $J(M)$ for the ideal
generated by the entries of a minimal presentation matrix for $M$, and
$\operatorname{syz}_{n}^{R}(M)$ for the minimal $n$-th $R$-syzygy module of
$M$. Without assuming that $R$ has depth 0 (and using the definition in
Section 1 we deduce that over rings of Burch index $\geq 2$ the presentation
matrices of high syzygies contain many linear forms:

######  Corollary 0.2.

Let $(R,\mathfrak{m},k)$ be a local ring and suppose
$\operatorname{{Burch}}(R)\geq 2$. Let $M$ be any finitely generated module
over $R$ which has infinite projective dimension. For $n\geq
5+\operatorname{{depth}}R$,

$\dim_{k}\frac{\mathfrak{m}}{J(\operatorname{syz}^{R}_{n}M)+\mathfrak{m}^{2}}\leq\operatorname{{depth}}R.$

In Theorem 2.4 we draw a connection between the Burch index of an ideal $I$ in
a regular local ring $S$ and the linear entries of matrices in its $S$-free
resolution. Our results there suggest a connection between linearity of
resolution of $I$ over $S$ and the resolution of modules over $S/I$.

We conclude the paper by computing the Burch index for several special classes
of ideals: ideals in a regular local ring of dimension $2$, ideals of general
sets of points in the projective plane, ideals with almost linear resolution,
and ideals of certain fibre products. We give examples of monomial ideals to
illustrate our results.

The main results of this paper were suggested by extensive computations using
the computer algebra system Macaulay2 [10]. We wish to acknowledge the
importance of this tool, primarily produced and constantly improved by Dan
Grayson and Mike Stillman.

##  1\. Definitions and preliminary results

Throughout this paper, all rings are local and Noetherian and modules are
finitely generated. We regard positively graded algebras over a field as local
as well, with the results applying to graded modules only.

######  Definition 1.1 (_relative Burch index_).

Let $(S,\mathfrak{n},k)$ be a local ring and $I\subseteq S$ an ideal. We
define the _Burch ideal_ of an ideal $I\subset S$ to be
$\operatorname{{BI}}_{S}(I):=I\mathfrak{n}:(I:\mathfrak{n})$.

If $I=0$ or $S=S/I$ has positive depth, we set
$\operatorname{{Burch}}_{S}(I)=0.$ Otherwise
$\mathfrak{n}^{2}\subseteq\operatorname{{BI}}_{S}(I)\subseteq\mathfrak{n}$ and
we set

$\operatorname{{Burch}}_{S}(I)=\dim_{k}(\mathfrak{n}/\operatorname{{BI}}_{S}(I)).$

Our definition of the (absolute) Burch index of a local ring uses the Cohen
presentation, a residue field extension, and reduction modulo regular
sequences to reduce to the case of an ideal of codepth 0 in a regular local
ring. By a _regular presentation_ of $R$, we mean a surjective local map
$(S,\mathfrak{n},k)\to(R,\mathfrak{m},k)$ where $S$ is regular. Such a
presentation is called minimal if
$\dim_{k}\mathfrak{n}/\mathfrak{n}^{2}=\dim_{k}\mathfrak{m}/\mathfrak{m}^{2}$.

######  Definition 1.2.

Let $(R,\mathfrak{m},k)$ be a local ring and let $\hat{R}$ denote the
$\mathfrak{m}$-adic completion of $R$.

  1. (1)

If $\operatorname{{depth}}R=0$, let $\hat{R}=S/I$ be a minimal regular Cohen
presentation of $\hat{R}$. Define $\operatorname{{BI}}(R)$ to be
$\operatorname{{BI}}_{S}(I)\hat{R}\cap R$ and $\operatorname{{Burch}}(R)$ to
be $\operatorname{{Burch}}_{S}(I)$. These are well-defined by Theorem 1.3.

  2. (2)

Assume $\operatorname{{depth}}R>0$. Let
$R^{\prime}=R[z]_{(\mathfrak{m}R[z])}$. We define $\operatorname{{Burch}}(R)$
to be the maximum, over all maximal regular sequences $(\underline{x})$ in
$R^{\prime}$, of
$\\{\operatorname{{Burch}}(R^{\prime}/\underline{x}R^{\prime})\\}$

Recall that a ring $R$ with $\operatorname{{Burch}}(R)>0$ is called a Burch
ring in [5].

######  Theorem 1.3.

Let $R$ be a local ring of depth $0$ and consider a regular presentation
$R=S/I$. Then the $R$-ideal $\operatorname{{BI}}_{S}(I)R$ and the number
$\operatorname{{Burch}}_{S}(I)$ are independent of the presentation of $R$.

###### Proof.

We can assume $R$ is complete and not a field. Consider any two presentations
of $R$, $S_{1}\twoheadrightarrow R$ and $S_{2}\twoheadrightarrow R$. Then
$S=S_{1}\times_{R}S_{2}$ is also complete local, and thus by the Cohen
Structure Theorem it is a homomorphic image of another regular local ring $T$.
Hence it is enough to prove the equality of our definitions for the pair
$(T,S_{1})$ and since $T$ maps onto $S_{1}$, by induction we reduce to the
case where our presentations are coming from
$(T,\mathfrak{n}_{T})\twoheadrightarrow(S,\mathfrak{n}_{S})=T/(x)\twoheadrightarrow
R$ where $x\in\mathfrak{n}_{T}-(\mathfrak{n}_{T})^{2}$.

Suppose $R=S/I$. Let $(J,\mathfrak{m})$ be lifts of $(I,\mathfrak{n}_{S})$ to
$T$ respectively. By that we mean: the minimal generators of $J$, together
with $x$, generate the preimage of $I$ in $T$ minimally, and similarly for
$\mathfrak{m},\mathfrak{n}_{S}$. Write $\mathfrak{n}$ for $\mathfrak{n}_{T}$.

We need to show that

$J\mathfrak{m}+(x):[(J,x):\mathfrak{n}]=(J+(x))\mathfrak{n}:[(J,x):\mathfrak{n}]$

(the LHS is $\operatorname{{BI}}_{S}(I)$ and the RHS is
$\operatorname{{BI}}_{T}(I)$). Observe that
$(J+(x))\mathfrak{n}=J\mathfrak{m}+x\mathfrak{n}$. Let $A=T/J\mathfrak{m}$ and
(slightly abuse notations) $x,\mathfrak{n},H$ be the images of
$x,\mathfrak{n},(J,x):\mathfrak{n}$ in $A$. We are reduced to showing
$(x):H=x\mathfrak{n}:H$. By assumption $R$ is not a field so
$H\subset\mathfrak{n}$. Thus, if $a\in(x):H$, then
$aH\subset(x)\cap\mathfrak{n}^{2}=(x)\mathfrak{n}$, and we are done. ∎

The following result shows in particular that to compute the Burch index for
rings of positive depth one must use regular sequences consisting entirely of
elements not in the square of the maximal ideal, that is, _linear_ regular
sequences:

######  Proposition 1.4.

Let $S$ be regular and $I,J$ be ideals such that $IJ=I\cap J$ (equivalently,
$\operatorname{{Tor}}^{S}_{i>0}(S/I,S/J)=0$). If $S/I,S/J$ are both singular,
then $\operatorname{{Burch}}(S/(I+J))=0$. In particular, if $R=S/(I+f)$, where
$f$ is a nonzerodivisors in the square of the maximal ideal of $S/I$, then
$\operatorname{{Burch}}(R)=0$.

###### Proof.

Let $R=S/(I+J)$. Then $R$ is not $\operatorname{{Tor}}$-friendly, in the
language of [2]. Namely, there are $R$ modules $M,N$ of infinite projective
dimension such that $\operatorname{{Tor}}_{i>0}^{R}(M,N)=0$. On the other
hand, Burch rings are $\operatorname{{Tor}}$-friendly, by [5, Theorem 1.4]. ∎

At this point, we have enough to give basic estimates and precise values for
the Burch index of certain well known rings. See also Section 4.

######  Proposition 1.5.

Let $(R,\mathfrak{m},k)$ be a local ring.

  1. (1)

$\operatorname{{Burch}}(R)\leq\dim_{k}(\mathfrak{m}/\mathfrak{m}^{2})-\operatorname{{depth}}R$.

  2. (2)

If $R=S/J\mathfrak{n}$ where $(S,\mathfrak{n})$ is a regular local ring and
$0\neq J\subset\mathfrak{n}$, then $\operatorname{{Burch}}(R)=\dim S$.

  3. (3)

If $R$ is a Cohen-Macaulay ring of minimal multiplicity, then
$\operatorname{{Burch}}(R)=\dim_{k}(\mathfrak{m}/\mathfrak{m}^{2})-\operatorname{{depth}}R$.

  4. (4)

If $R$ is a singular hypersurface, then $\operatorname{{Burch}}(R)=1$.

###### Proof.

$(1)$ The inequality is obvious if $\operatorname{{depth}}R=0$. When
$\operatorname{{depth}}R>0$, by 1.4, to compute $\operatorname{{Burch}}(R)$,
we just need to use a regular sequence in $\mathfrak{m}-\mathfrak{m}^{2}$,
thus the statement reduces to the $\operatorname{{depth}}0$ case, whence the
assertion is obvious.

$(2)$ Clearly $\operatorname{{depth}}S/J\mathfrak{n}=0$. We have
$\operatorname{{BI}}_{S}(J\mathfrak{n})=J\mathfrak{n}^{2}:(J\mathfrak{n}:\mathfrak{n})\subseteq
J\mathfrak{n}^{2}:J\subseteq\mathfrak{n}^{2}\subseteq\operatorname{{BI}}_{S}(J\mathfrak{n})$.

$(3)$ We may assume $R$ is Artinian, and thus has the presentation
$S/\mathfrak{n}^{2}$ for some regular local ring $(S,\mathfrak{n})$, so we can
apply $(2)$ The inequality is obvious if $\operatorname{{depth}}R=0$. When
$\operatorname{{depth}}R>0$, by 1.4, to compute $\operatorname{{Burch}}(R)$,
we must use a regular sequence in $\mathfrak{m}-\mathfrak{m}^{2}$, so the
statement reduces to the $\operatorname{{depth}}0$ case.

$(4)$ By the argument above, the computation reduces to the Artinian case, so
we can assume $R=S/(t^{a})$ for some DVR $(S,(t))$ and some $a>1$, for which
the claim is clear.

∎

##  2\. Burch index and linearity

Let $(S,\mathfrak{n},k)$ be a local ring, let $I\subseteq S$ an ideal and set
$R=S/I$. Our main results link high values of Burch index to linear elements
in the $S$-free resolution of $R$ and the $R$-free resolutions of $R$-modules.

######  Definition 2.1.

Recall that we write $J(M)$ for the ideal generated by the entries of a
minimal presentation matrix for a finitely generated module $M$. We set

$\displaystyle{\rm LinSp}_{n}^{S}(M)$
$\displaystyle:=\frac{J(\operatorname{syz}^{S}_{n-1}M)+\mathfrak{n}^{2}}{\mathfrak{n}^{2}}$
$\displaystyle{\rm lin}_{n}^{S}(M)$ $\displaystyle:=\dim_{k}{\rm
LinSp}_{n}^{S}(M),$

and think of ${\rm LinSp}_{n}^{S}(M)$ as the vector space of linear forms in
the $n$-th matrix in a minimal free resolution of $M$.

In the case where the ring $S$ has depth $\geq 2$, the following result gives
another characterization of $BI_{S}(I)$.

######  Proposition 2.2.

Let $t\in\mathfrak{n}$ be a nonzerodivisor. If
$t\notin\operatorname{{BI}}_{S}(I)$ then $k$ is a direct summand of $I/tI$. If
$\operatorname{{depth}}S\geq 2$ then the converse also holds.

###### Proof.

Since $t\in\mathfrak{n}$ is regular, $tI:\mathfrak{n}\subseteq tI:(t)=I$, and
thus

$\operatorname{{{soc}}}(I/tI)=((tI:\mathfrak{n})\cap
I)/tI=(tI:\mathfrak{n})/tI.$

Thus $I/tI$ has a $k$-summand if and only if
$(tI):\mathfrak{n}\not\subseteq\mathfrak{n}I$.

If $t\notin\operatorname{{BI}}_{S}(I)=\mathfrak{n}I:(I:\mathfrak{n})$, then
$t(I:\mathfrak{n})\not\subseteq\mathfrak{n}I$, and since
$t(I:\mathfrak{n})\subseteq(tI):\mathfrak{n}$, $I/tI$ has a $k$-summand.

If $\operatorname{{depth}}S\geq 2$ then $\mathfrak{n}$ contains an element
that is a nonzerodivisor on $S/tS$, so $(tI):\mathfrak{n}=t(I:\mathfrak{n})$,
and the previous argument can be reversed. ∎

######  Remark 2.3.

Let $(S,\mathfrak{m})$ be a local ring and $M$ an $S$-module. If $t\in S$ is a
regular element on $M$ and $S$, then for each $i>0$,

$J(\operatorname{syz}_{i-1}^{S/tS}(M/tM))=\bigl{(}J(\operatorname{syz}_{i-1}^{S}(M))+tS\bigr{)}/tS.$

Thus ${\rm lin}_{i}^{S/tS}(M/tM)={\rm lin}_{i}^{S}(M)$ if the image of $t$ in
$\mathfrak{n}/\mathfrak{n}^{2}$ is outside of ${\rm LinSp}^{S}_{i}(M)$, and
otherwise ${\rm lin}_{i}^{S/tS}(M/tM)={\rm lin}_{i}^{S}(M)-1$.

The matrices in the $S$-free resolution of ideals with positive Burch have
many linear entries:

######  Theorem 2.4.

Let $(S,\mathfrak{n},k)$ be a local ring of embedding dimension $n$, and let
$I\subset\mathfrak{n}$ be an ideal. If $0<i<\operatorname{{{projdim}}}I$,
then:

  1. (1)

If $\operatorname{{Burch}}_{S}I\geq 1$ and $S$ has positive depth, then ${\rm
lin}_{i}^{S}(I)\geq n-1$.

  2. (2)

If $\operatorname{{Burch}}_{S}I\geq 2$ and
$\dim(P+\mathfrak{n}^{2})/\mathfrak{n}^{2}\leq n-2$ for each associated prime
$P$ of $S$ then ${\rm lin}_{i}^{S}(I)=n$.

###### Proof.

If $t\notin\operatorname{{BI}}_{S}(I)$ is a nonzerodivisor, then by 2.2 the
module $I/tI$ has a $k$-summand, so ${\rm
LinSp}_{i}^{S/tS}(I/tI)=\mathfrak{n}/(\mathfrak{n}^{2}+(t))$, and thus ${\rm
LinSp}_{i}^{S}(I)+\langle t\rangle=\mathfrak{n}/\mathfrak{n}^{2}$ by Remark
2.3, hence ${\rm lin}_{i}^{S}(I)\geq n-1$.

If $\operatorname{{Burch}}_{S}I\geq 1$ and $S$ has positive depth then, by
prime avoidance, we can choose such an element $t$, proving $(1)$.

If $\operatorname{{Burch}}_{S}I\geq 2$ then let the associated primes of $S$
be $P_{1},\dots,P_{a}$. Let
$V_{0}=(\operatorname{{Burch}}_{S}I)+\mathfrak{n}^{2})/\mathfrak{n}^{2}$ and
$V_{i}=(P_{i}+\mathfrak{n}^{2})/\mathfrak{n}^{2}$. For each nonzero vector
$t\in\mathfrak{n}/\mathfrak{n}^{2}$ outside of $\cup_{j=0}^{a}V_{j}$ we must
have $\langle t\rangle+{\rm LinSp}_{i}^{S}(I)=\mathfrak{n}/\mathfrak{n}^{2}$
and each $V_{j}$ has codimension at least $2$, so item (2) follows. ∎

######  Example 2.5.

We cannot deduce $\operatorname{{Burch}}(I)$ from the dimensions ${\rm
lin}_{i}^{S}(I)$. For example if
$I=(x^{3},y^{3},z^{3},x^{2}y,y^{2}z,z^{2}x)\subset S=k[[x,y,z]]$, then ${\rm
lin}_{1}(I)={\rm lin}_{2}(I)=3$, but $\operatorname{{Burch}}_{S}(I)=0$.
However, if $I$ is an ideal of projective dimension 1, then Corollary 4.1 says
that $\operatorname{{Burch}}(S/I)=\min\\{{\rm lin}_{1}^{S}(I),2\\}$.

##  3\. Burch index and $k$-direct summands

As we explained in the introduction, the condition that a module $M$ has $k$
as a direct summand is stronger than the condition that ${\rm lin}_{i}(M)$ is
the maximal ideal. Our main theorem strengthens Theorem 0.1.

######  Theorem 3.1.

Let $(R,\mathfrak{m},k)$ be a local ring of depth $0$ and embedding dimension
$\geq 2$. For every non-free $R$-module $M$:

  1. (1)

If $\operatorname{{Burch}}(R)\geq 2$ then $k$ is a direct summand of
$\operatorname{syz}^{R}_{i}(M)$ for some $i\leq 5$ and for all $i\geq 7$.

  2. (2)

If $\operatorname{{Burch}}(R)\geq 1$, and $k$ is a direct summand of
$\operatorname{syz}^{R}_{s}(\operatorname{{BI}}(R))$ for some $s\geq 1$, then
$k$ is a direct summand of $\operatorname{syz}^{R}_{i}(M)$ for some $i\leq
s+4$ and for all $i\geq s+6$.

Thus, if $R$ is Burch, then $k$ is a direct summand of all high syzygies of
non-free $R$-modules if and only if $k$ is a direct summand of some $R$-syzygy
of $\operatorname{{BI}}(R)$.

######  Remark 3.2.

The case of depth 0 and embedding dimension 1 is different, but easy to
analyze completely: in this case $\hat{R}=S/t^{n}$ for some discrete valuation
ring $(S,t)$, so $\operatorname{{BI}}_{S}(R)=t$. Any indecomposable $R$-module
has the form $M=R/t^{m}$ and the syzygies of $M$ have $k$ as direct summand in
alternate steps iff $m=1$ or $m=n-1$.

The following Propositions will be used in the proof of Theorem 3.1. Recall
the convention that the ideal of entries of a matrix with no columns is $R$.
We first show that the ideals
$J(\operatorname{syz}^{R}_{s-1}(M))+J(\operatorname{syz}_{s}^{R}(M))$, for any
$R$-module $M$, are an obstruction to socle summands:

######  Proposition 3.3.

Let $M$ be a module over the local ring $(R,\mathfrak{m})$. If

$\bigl{(}J(\operatorname{syz}^{R}_{s-1}(M))+J(\operatorname{syz}_{s}^{R}(M))\bigr{)}\subseteq
K\subsetneq\mathfrak{m}$

for some ideal $K$ of $R$ then $k$ is not a summand of
$\operatorname{syz}_{p}(K)$ for any $1\leq p\leq s-2$.

###### Proof.

Let $d_{i}$ be the $i$-th differential in a minimal free resolution of $M$.
The hypothesis is equivalent to the statement that $R/K\otimes
d_{s}=0=R/K\otimes d_{s+1}$. This implies that
$\operatorname{{Tor}}_{s}^{R}(M,R/K)$ is a nonzero free $R/K$-module, and thus
has no $k$-summand. But if $k$ were a direct summand of
$\operatorname{syz}_{p}^{R}(K)$ with $1\leq p\leq s-2$ then

$\operatorname{{Tor}}_{s}^{R}(M,R/K)=\operatorname{{Tor}}_{s-1}^{R}(M,K)=\operatorname{{Tor}}_{s-1-p}^{R}(M,\operatorname{syz}_{p}^{R}(K))$

would have a $k$-summand, a contradiction. ∎

We will use [9, Lemma 4.1], which was generalized in [5, Proposition 3.4]. For
the reader’s convenience we give the statement and proof:

######  Proposition 3.4.

Let $(S,\mathfrak{n},k)$ be a local ring, $I\subset S$ an ideal and $R=S/I$.
Let $A:F\to G$ be a map of free $S$-modules, and let $J$ be the ideal
generated by the entries of a matrix representing $A$. If
$J\subset\mathfrak{n}$ but $J\nsubseteq\operatorname{{BI}}_{S}(I)$, then $k$
is a direct summand of $\ker R\otimes A$.

###### Proof.

The hypothesis means that there is are elements $s\in I:\mathfrak{n}$ and
$t\in J$ such that $st\notin\mathfrak{n}I$. Thus for some minimal generator
$e$ of $F$, $A(se)\notin\mathfrak{n}IG$.

Write $\overline{\phantom{A}}$ for reduction mod $I$. We claim that
$\overline{se}$ is a socle element and a direct summand of $\ker\overline{A}$.
Clearly

$se\notin A^{-1}(\mathfrak{n}IG)\supset\mathfrak{n}A^{-1}(IG),$

so $\overline{se}\notin\mathfrak{n}\ker\overline{A}$. On the other hand,
$\overline{se}\in\operatorname{{{soc}}}\overline{F}$, and since
$J\subset\mathfrak{n}$ we have $\overline{se}\in\ker\overline{A}$. ∎

###### Proof of Theorem 3.1.

Let $\mathcal{L}_{2}(R)$ be the set of ideals of $R$ with
$\operatorname{{length}}(R/L)=2$.

$(1)$ Since $\operatorname{{length}}(S/\operatorname{{BI}}_{S}(I))\geq 3$, the
ideal $\operatorname{{BI}}_{S}(I)$ does not contain the preimage in $S$ of any
$L\in\mathcal{L}_{2}(R)$. By Proposition 3.4 the module $k$ is a direct
summand of $\operatorname{syz}_{2}^{R}(R/L)=\operatorname{syz}_{1}^{R}(L)$, so
Proposition 3.3 with $s=3$ gives
$J(\operatorname{syz}_{2}^{R}M)+J(\operatorname{syz}_{3}^{R}(M))=\mathfrak{m}$,
and thus not both of $J(\operatorname{syz}_{2}^{R}(M))$ and
$J(\operatorname{syz}_{3}^{R}(M))$ can be contained in
$\operatorname{{BI}}_{S}(I)$. By Proposition 3.4, $k$ is a direct summand of
$\operatorname{syz}_{4}^{R}(M)$ or $\operatorname{syz}_{5}^{R}(M)$, as
claimed.

Since $R$ has embedding dimension $\geq 2$ we have
$J(\operatorname{syz}_{i}^{R}(k))=\mathfrak{m}$ for every $i\geq 1$, and thus
$J(\operatorname{syz}_{i}^{R}(M))=\mathfrak{m}$ for every $i\geq 6$. Using
Proposition 3.4 again we see that $k$ is a summand of
$\operatorname{syz}_{i}^{R}(M)$ for each $i\geq 7$.

$(2)$ In this case $\operatorname{{length}}S/\operatorname{{BI}}_{S}(R)=2$, so
the preimage in $S$ of an ideal $L\in\mathcal{L}_{2}(R)$ contains
$\operatorname{{BI}}_{S}(R)$ if and only if it is equal to
$\operatorname{{BI}}_{S}(R)$. Thus the reasoning of part $(1)$ applies to
every $L\in\mathcal{CL}(R)$ except for $\operatorname{{BI}}(R)$ itself, to
which we can apply the hypothesis of part $(2)$. ∎

######  Example 3.5.

We can see that theorem 3.1 is sharp by taking $S=k[[a,b]],I=(a,b^{2})^{2}$.
Set $R=S/I$ and let $M=R/(a,b^{2})$. Then
$\operatorname{{BI}}_{S}(I)=(a,b^{2}),\operatorname{{Burch}}(R)=1$ and one can
see that $\operatorname{syz}_{1}^{R}(M)\cong M^{\oplus 2}$. Thus $k$ is not a
direct summand of any $R$-syzygy of $M$.

######  Example 3.6.

The bound of $5$ in Theorem 3.1 cannot be improved. Let $k=\mathbb{Z}/(101)$
and $S=k[x,y,z]$. Let $I=(x^{3},y^{3},z^{3})(x,y,z)$ and $R=S/I$. Let
$M=R/(xyz)$. Macaulay2 [10] tells us that $\operatorname{{Burch}}(R)=3$ and
$\operatorname{syz}_{5}^{R}(M)$ is the first syzygy of $M$ that has a
$k$-summand.

######  Corollary 3.7.

Let $(R,\mathfrak{m},k)$ be a local ring of embedding dimension $n$ and
$\operatorname{{depth}}R=t$. If $\operatorname{{Burch}}(R)\geq 2$, then ${\rm
lin}_{i}(M)\geq n-t$ for any module $M$ of infinite projective dimension and
each $i\geq t+5$.

###### Proof.

Suppose $\operatorname{{depth}}R=t$ and $\operatorname{{Burch}}(R)\geq 2$. If
$t=0$, we use part $(1)$ of 3.1.

The case $t>0$ follows from the definition of Burch index, Remark 2.3 and
induction. ∎

##  4\. Some computations of the Burch index

######  Theorem 4.1.

Let $(S,\mathfrak{m})$ be a regular local ring. Let $I\subset\mathfrak{m}^{2}$
be an ideal of projective dimension $1$. Then
$\operatorname{{Burch}}(S/I)=\min\\{{\rm lin}_{1}^{S}(I),2\\}$.

###### Proof.

Factoring out an appropriate regular sequence to reduce to the depth 0 case,
using 2.3 and induction, we may assume $\dim S=2$.

Let $J=J(I)$, the ideal generated by the entries of the Hilbert-Burch matrix
of $I$. Consider $t\in\mathfrak{m}-\mathfrak{m}^{2}$. By 2.2,
$t\in\operatorname{{BI}}(I)$ if and only if $k$ is not a summand of $I/tI$.
The ring $S^{\prime}:=S/tS$ is a discrete valuation ring; write $s$ for its
parameter.

The $S^{\prime}$-module $I/tI$ is a direct sum of copies of $S^{\prime}$ and
cyclic modules of the form $S^{\prime}/\mathfrak{m}^{a}S^{\prime}$ for various
$a\geq 1$. Thus, $k$ is a summand of $I/tI$ if and only if ${\rm
LinSp}_{1}^{S/tS}(I/tI)\nsubseteq\mathfrak{m}^{2}(S/tS)$ or, equivalently,
$(t)+J\nsubseteq(t)+\mathfrak{m}^{2}$. From this observation,
$\operatorname{{BI}}_{S}(I)$ is equal to: $\mathfrak{m}$ if ${\rm
lin}_{1}(I)=0$, $J$ if ${\rm lin}_{1}(I)=1$, and $\mathfrak{m}^{2}$ if ${\rm
lin}_{1}(I)=2$, and our statement follows. ∎

Here we apply the above to a geometric case:

######  Theorem 4.2.

Let $R$ be the homogenous coordinating ring of $d>1$ generic points in
$\mathbb{P}^{2}_{k}$ where $k$ is an algebraically closed field. Then
$\operatorname{{Burch}}(R)$ is

  * •

$1$ if $d=2$.

  * •

$0$ if $d=\binom{2s+1}{2}+s$ for some integer $s>0$.

  * •

$2$ in all other cases.

###### Proof.

Write $R=S/I$ where $S=k[x,y,z]$. Since $\operatorname{{{projdim}}}_{S}I=1$,
by 4.1 we just need to compute ${\rm lin}_{1}^{S}(I)$. The degrees of elements
in the Hilbert-Burch matrix $H$ of $I$ are well-known, see for instance [7,
Exercises 12, 13, pp 50]. First, write $d$ (uniquely) as
$d=\binom{s+t+1}{2}+s$, where $s,t$ are non-negative integers. The linear
forms in $H$ form a submatrix $L$ of size $(t+1)\times(t-s)$ if $t\geq s$, or
$(s-t)\times s$ if $s\geq t$. Since we are in the generic situation, the space
of linear forms has dimension $\min\\{3,n\\}$ where $n$ is the number of
entries in $L$, thus ${\rm lin}_{1}(I)=\min\\{2,n\\}$. It is easy to see that
$n=1$ if and only if $s=1,t=0$, $n=0$ if and only if $s=t$, and the assertions
are now clear. ∎

######  Proposition 4.3.

Let $(S,\mathfrak{m})$ be a polynomial ring and $I\subset\mathfrak{m}^{2}$ a
homogenous ideal of depth $1$. Suppose that the minimal generating degree of
$I:\mathfrak{m}$ is less than the minimal generating degree of $I$. Then
$\operatorname{{Burch}}(S/I)=\dim S$.

###### Proof.

If $g$ is the minimal generating degree of $I:\mathfrak{m}$, then no linear
forms can multiply a generator with degree $g$ to $I\mathfrak{m}$, which
starts in degree at least $g+2$, thus
$\operatorname{{BI}}_{S}(I)=\mathfrak{m}^{2}$, as required. ∎

If $I$ is a homogeneous ideal in a polynomial ring in $p$ variables, then we
say that $I$ the resolution of $I$ is _almost linear_ if it is linear for
$p-1$ steps where $p=\operatorname{{{projdim}}}_{S}I$.

######  Corollary 4.4.

Let $(S,\mathfrak{m})$ be a polynomial ring and $I\subset\mathfrak{m}^{2}$ a
homogeneous ideal of projective dimension $p$ ideal with almost linear
resolution. Suppose that ${\rm lin}_{p}(I)>0$. Then
$\operatorname{{Burch}}(S/I)=\dim S-\operatorname{{depth}}S/I$.

###### Proof.

We can assume $\operatorname{{depth}}S/I=0$. Let $g$ be the smallest degree of
a generator of $I$. Then the resolution assumption tells us that the socle of
$I$ has a minimal generator in degree $g-1$, and 4.3 applies.

∎

######  Example 4.5.

Corollary 4.4 cannot be improved. There are Artinian Gorenstein ideals with
almost linear resolution in each generating degree $d>1$ and number of
variables $n>2$, see for instance [6, Example 3.2]. Since the quotients are
Gorenstein and not hypersurfaces, the Burch index is $0$ (the syzygies of $k$
will never have $k$ summands, as they are indecomposable).

######  Corollary 4.6.

Let $(S,\mathfrak{m})$ be a polynomial ring and $I\subset\mathfrak{m}^{2}$ an
ideal with linear resolution. Then $\operatorname{{Burch}}(S/I)=\dim
S-\operatorname{{depth}}S/I$.

######  Example 4.7 (Fibre product).

Let $A=k[[x_{1},\dots,x_{a}]]/I$ and $B=k[[y_{1},\dots,y_{b}]]/J$ be minimal
presentations of local rings of depth $0$. Let
$S=A\times_{k}B=k[[x_{1},\dots,x_{a},y_{1},\dots,y_{b}]]/(I+J+(x_{1},\dots,x_{a})(y_{1},\dots,y_{b}))$.
Then it can be shown that
$\operatorname{{Burch}}(S)=\operatorname{{Burch}}(A)+\operatorname{{Burch}}(B)$.

## References

  * [1] L. Avramov, _Infinite resolutions_ , Six lectures on commutative algebra, 1–118, Mod. Birkhäuser Class., Birkhäuser Verlag, Basel, 2010.
  * [2] L. Avramov, S. Iyengar, S. Nasseh and S. Sather-Wagstaff, _Persistence of homology over commutative noetherian rings_ , J. Algebra, to appear.
  * [3] W. Bruns and J. Herzog, _Cohen-Macaulay Rings_ , Cambridge Studies in Advanced Mathematics, 39, Cambridge, Cambridge University Press, 1993.
  * [4] L. Burch, _On ideals of finite homological dimension in local rings_ , Proc. Cambridge Philos. Soc. 64 (1968), 941–948.
  * [5] H. Dao, T. Kobayashi and R. Takahashi, _Burch ideals and Burch rings_ , Algebra Number Theory, Algebra Number Theory 14 (2020), no. 8, 2121–2150.
  * [6] H. Dao and D. Eisenbud, _Linearity of Free Resolutions of Monomial Ideals_ , Res. Math. Sci. 9 (2022), no. 2, Paper No. 35, 15 pp.
  * [7] D. Eisenbud, _The geometry of syzygies_ , Graduate Texts in Mathematics, 229. Springer-Verlag, New York, 2005.
  * [8] Gulliksen, T., A proof of the existence of minimal $R$-algebra resolutions. Acta Math. 120 (1968) 53–58.
  * [9] A. Kustin and A. Vraciu, Totally reflexive modules that are close to Gorenstein, J. Algebra 571 (2021)190–231.
  * [10] D. R. Grayson and M. E. Stillman, Macaulay2, a software system for research in algebraic geometry, Available at https://math.uiuc.edu/Macaulay2/