# Two generalizations of Auslander–Reiten duality and applications

Arash Sadeghi  School of Mathematics, Institute for Research in Fundamental
Sciences (IPM), P.O. Box: 19395-5746, Tehran, Iran<EMAIL_ADDRESS>and  Ryo Takahashi  Graduate School of Mathematics, Nagoya University,
Furocho, Chikusaku, Nagoya, Aichi 464-8602, Japan/Department of Mathematics,
University of Kansas, Lawrence, KS 66045-7523, USA
<EMAIL_ADDRESS>https://www.math.nagoya-u.ac.jp/ takahashi/
Dedicated to Professor Mohammad T. Dibaei on the occasion of his retirement

###### Abstract.

This paper extends Auslander–Reiten duality in two directions. As an
application, we obtain various criteria for freeness of modules over local
rings in terms of vanishing of Ext modules, which recover a lot of known
results on the Auslander–Reiten conjecture.

2010 Mathematics Subject Classification. 13D07 (Primary); 13C14, 13H10
(Secondary)

Key words and phrases. Auslander–Reiten duality, $n$-torsionfree module, Ext
module, Cohen–Macaulay ring, Gorenstein ring, maximal Cohen–Macaulay module,
Auslander–Reiten conjecture, totally reflexive module, G-dimension

Sadeghi’s research was supported by a grant from IPM. Takahashi was partly
supported by JSPS Grant-in-Aid for Scientific Research 16K05098 and JSPS Fund
for the Promotion of Joint International Research 16KK0099

##  1\. Introduction

The Auslander–Reiten conjecture [6] is one of the most celebrated conjectures
in the representation theory of algebras. This long-standing conjecture is
known to hold true over several classes of algebras, including algebras of
finite representation type [6] and symmetric artin algebras with radical cube
zero [16]. This conjecture is closely related to other conjectures such as the
Nakayama conjecture [22, 6] and the Tachikawa conjecture [7, 24]. Although the
Auslander–Reiten conjecture was initially proposed over artin algebras, it
remains meaningful for arbitrary commutative noetherian rings:

######  Conjecture 1.1 (Auslander–Reiten).

Let $R$ be a commutative noetherian ring $R$ and let $M$ be a finitely
generated $R$-module. If
$\operatorname{Ext}^{i}_{R}(M,M)=0=\operatorname{Ext}^{i}_{R}(M,R)$ for all
$i\geq 1$, then $M$ is projective.

Auslander, Ding and Solberg [5] proved that Conjecture 1.1 holds for any
complete intersection local rings. Recently there has been various progress
towards Conjecture 1.1; see [2, 9, 10, 11, 12, 13, 15, 17, 18, 23] for
instance. Using Auslander–Reiten duality, Araya [1] proved that if all
Gorenstein local rings of dimension at most one satisfy the Auslander–Reiten
conjecture, then so do all Gorenstein local rings.

In this paper, we extend Auslander–Reiten duality in two directions; the
following two theorems are included in the main results of this paper.

######  Theorem 1.2.

Let $R$ be a commutative noetherian ring. Let $n\geq 1$ be an integer. Let
$M,N$ be finitely generated $R$-modules. Assume that $M$ is
$(n+1)$-torsionfree (e.g. totally reflexive) and
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{n}(R)$.

  1. (1)

There exists an exact sequence of $R$-modules

$\displaystyle 0\to$
$\displaystyle\operatorname{Ext}^{n-1}_{R}(\operatorname{Hom}_{R}(M,N),R)\to\operatorname{Ext}^{n-1}_{R}(N,M)\to\operatorname{Ext}^{n}_{R}(\operatorname{\underline{Hom}}_{R}(M,N),R)$
$\displaystyle\to$
$\displaystyle\operatorname{Ext}^{n}_{R}(\operatorname{Hom}_{R}(M,N),R).$

  2. (2)

For all integers $i\leq n-2$ one has an isomorphism of $R$-modules

$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),R)\cong\operatorname{Ext}^{i}_{R}(N,M).$

######  Theorem 1.3.

Let $(R,\mathfrak{m},k)$ be a Cohen–Macaulay local ring of dimension $d\geq 2$
with canonical module $\omega$. Let $M,N$ be maximal Cohen–Macaulay
$R$-modules such that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$.

  1. (1)

One has an isomorphism of $R$-modules

$\operatorname{\underline{Hom}}_{R}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{d-1}(M^{*},N^{\dagger}).$

  2. (2)

Suppose that $M$ is locally totally reflexive (e.g. locally free) on the
punctured spectrum of $R$. Then for all $1\leq i\leq d-2$ one has an
isomorphism of $R$-modules

$\operatorname{Ext}_{R}^{i}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{(d-1)-i}(M^{*},N^{\dagger}).$

Let us explain the notation and terminology used above. For an integer $n$, we
denote by $\operatorname{Y}^{n}(R)$ the set of prime ideals $\mathfrak{p}$
with $\operatorname{depth}R_{\mathfrak{p}}\geq n$. An $n$-torsionfree module
is a module $M$ with $\operatorname{Ext}_{R}^{i}(\operatorname{Tr}M,R)=0$ for
$1\leq i\leq n$, where $\operatorname{Tr}M$ stands for the Auslander transpose
of $M$. We denote by $\operatorname{NF}(M)$ the non-free locus of an
$R$-module $M$, that is, the set of prime ideals $\mathfrak{p}$ such that
$M_{\mathfrak{p}}$ is not $R_{\mathfrak{p}}$-free. For $R$-modules $M,N$ we
denote by $\operatorname{\underline{Hom}}_{R}(M,N)$ the stable Hom module,
namely, the quotient of $\operatorname{Hom}_{R}(M,N)$ by the homomorphisms
factoring through projective modules. Also,
$(-)^{\vee}=\operatorname{Hom}_{R}(-,\mathrm{E}_{R}(k))$,
$(-)^{*}=\operatorname{Hom}_{R}(-,R)$, and
$(-)^{\dagger}=\operatorname{Hom}_{R}(-,\omega)$ are the Matlis, algebraic and
canonical duals, respectively.

Both of the above two theorems recover the following celebrated
Auslander–Reiten duality theorem for $d\geq 2$. In fact, Theorem 1.3 is
included in a more general result (Theorem 2.8), which recovers Corollary 1.4
for all $d\geq 0$.

######  Corollary 1.4 (Auslander–Reiten duality).

Let $(R,\mathfrak{m})$ be a $d$-dimensional Gorenstein local ring. Let $M,N$
be maximal Cohen–Macaulay $R$-modules such that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$.
Then for all $i\in\mathbb{Z}$ there is an isomorphism of Tate cohomology
modules

$\mathrm{\widehat{Ext}}\mathrm{}_{R}^{i}(N,M)^{\vee}\cong\mathrm{\widehat{Ext}}\mathrm{}_{R}^{(d-1)-i}(M,N).$

There are also many other applications of the above theorems, including an
improved version of a main theorem of Celikbas and Takahashi [10, Theroem
1.1], and the following result. Recall that a commutative noetherian ring is
called generically Gorenstein if $R_{\mathfrak{p}}$ is Gorenstein for all
$\mathfrak{p}\in\operatorname{Ass}R$.

######  Corollary 1.5.

Let $R$ be a generically Gorenstein local ring of depth $t$, and let $2\leq
n\leq t$ be an integer. Let $M$ be a finitely generated $R$-module such that
$\operatorname{NF}(M)\subseteq\operatorname{Y}^{n}(R)$. Then $M$ is free if it
satisfies the following three conditions.

  1. (1)

$\operatorname{Ext}^{i}_{R}(M,M)=0$ for $n-1\leq i\leq t-1$.

  2. (2)

$\operatorname{Ext}^{i}_{R}(M,R)=0$ for $i>0$.

  3. (3)

$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,M),R)=0$ for $n\leq i\leq
t$, or $\operatorname{Hom}_{R}(M,M)$ has finite G-dimension.

It is worth mentioning that Corollary 1.5 simultaneously generalizes several
known results on the Auslander–Reiten conjecture, including the theorems of
Araya [1, Corollary 4], Ono and Yoshino [23, Theorem] and Araya, Celikbas,
Sadeghi and Takahashi [2, Corollary 1.6], and substantial parts of the
theorems of Huneke and Leuschke [17, Theorem 1.3], Goto and Takahashi [15,
Theorem 1.5(2)] and Dao, Eghbali and Lyle [13, Theorem 3.16]. Furthermore,
Corollary 1.5 yields the following result.

######  Corollary 1.6.

Let $R$ be a noetherian normal local ring of depth $t$. A finitely generated
$R$-module $M$ is free, if either of the following conditions holds.

  1. (1)

$\operatorname{Ext}^{i}_{R}(M,M)=\operatorname{Ext}^{j}_{R}(M,R)=\operatorname{Ext}^{h}_{R}(\operatorname{Hom}_{R}(M,M),R)=0$
for all $1\leq i\leq t-1$, $j\geq 1$ and $2\leq h\leq t$.

  2. (2)

$\operatorname{Ext}^{i}_{R}(M,M)=\operatorname{Ext}^{j}_{R}(M,R)=0$ for all
$1\leq i\leq t-1$ and $j\geq 1$, and that $\operatorname{Hom}_{R}(M,M)$ has
finite G-dimension.

In Section 2, we extend Auslander–Reiten duality. We prove our main theorems
in this section that yield Theorems 1.2 and 1.3 as special cases, and recover
Corollary 1.4. In Section 3, we apply our theorems to give various criteria
for freeness of modules in relation to Conjecture 1.1, including Corollaries
1.5 and 1.6. Among other things, we generalize the main theorem of [2] in this
section (Corollary 3.9).

## Convention

Throughout this paper, we assume that all rings are commutative noetherian and
all modules are finitely generated. Let $R$ be a ring. We denote by $(-)^{*}$
the $R$-dual $\operatorname{Hom}_{R}(-,R)$. If $R$ is local, then
$\mathfrak{m},k,d,t,(-)^{\vee}$ always stand for the maximal ideal of $R$, the
residue field of $R$, the (Krull) dimension of $R$, the depth of $R$, and the
Matlis dual $\operatorname{Hom}_{R}(-,\mathrm{E}_{R}(k))$, respectively.
Whenever $R$ admits a canonical module $\omega$, we denote by $(-)^{\dagger}$
the canonical dual $\operatorname{Hom}_{R}(-,\omega)$. For an $R$-module $M$,
we denote by $\mathrm{\Omega}M$ and $\operatorname{Tr}M$ the (first) syzygy
and the (Auslander) transpose of $M$. We refer the reader to [4] for details
on syzygies, transposes, $n$-torsionfree modules, totally reflexive modules
and G-dimension. For the definition and basic properties of Tate cohomology,
we refer the reader to [25, Section 7]. We say that an $R$-module $M$ is
locally free (resp. totally reflexive, of finite G-dimension) on a subset $A$
of $\operatorname{Spec}R$, provided that $M_{\mathfrak{p}}$ is free (resp.
totally reflexive, of finite G-dimension) as an $R_{\mathfrak{p}}$-module for
all $\mathfrak{p}\in A$. For an integer $n$ and an $R$-module $K$, let
$\operatorname{X}^{n}(K)$ (resp. $\operatorname{Y}^{n}(K)$) denote the set of
prime ideals $\mathfrak{p}$ of $R$ such that
$\operatorname{depth}_{R_{\mathfrak{p}}}K_{\mathfrak{p}}$ is at most (resp. at
least) $n$.

##  2\. Extending Auslander–Reiten duality to $n$-torsionfree modules

In this section, we extend Auslander–Reiten duality to $(n+1)$-torsionfree
modules for $n\geq 1$. First we present some examples of such modules.

######  Example 2.1.

Let $n\geq 1$ be an integer. An $R$-module $M$ is $(n+1)$-torsionfree in each
of the following cases.

  1. (1)

$M$ is totally reflexive.

  2. (2)

$R$ is local, $M$ is locally totally reflexive on the punctured spectrum,
$\operatorname{depth}M\geq n\geq 2$ and
$\operatorname{Ext}_{R}^{n-1}(M^{*},R)=0$.

  3. (3)

$M$ is locally of finite G-dimension on $\operatorname{X}^{n-1}(R)$ and $M$ is
a $(n+1)$st syzygy.

  4. (4)

$R$ is Cohen–Macaulay and local, $M$ is locally of finite G-dimension on the
punctured spectrum, and $M$ is a syzygy of a maximal Cohen–Macaulay module.

###### Proof.

(1) By the definition of total reflexivity
$\operatorname{Ext}^{>0}_{R}(\operatorname{Tr}M,R)=0$. Thus $M$ is
$i$-torsionfree for all $i\geq 0$.

(2) We see
$\operatorname{depth}_{R_{\mathfrak{p}}}M_{\mathfrak{p}}\geq\min\\{n,\operatorname{depth}R_{\mathfrak{p}}\\}$
for $\mathfrak{p}\in\operatorname{Spec}R$. The assumption
$\operatorname{depth}_{R}M\geq n$ especially says $n\leq d$, and
$M_{\mathfrak{p}}$ is totally reflexive for
$\mathfrak{p}\in\operatorname{X}^{n-1}(R)$. Hence $M$ is $n$-torsionfree by
[14, Proposition 2.4]. As $n\geq 2$, we have
$\operatorname{Ext}^{n+1}_{R}(\operatorname{Tr}M,R)\cong\operatorname{Ext}^{n-1}_{R}(M^{*},R)=0$.
Therefore $M$ is $(n+1)$-torsionfree.

(3) This is a consequence of [21, Theorem 43].

(4) Write $M=\mathrm{\Omega}N$ with $N$ maximal Cohen–Macaulay. Note that $N$
is also locally of finite G-dimension on the punctured spectrum. By [14,
Proposition 2.4] the module $N$ is a $d$th syzygy, and hence $M$ is a
$(d+1)$st syzygy. The assertion now follows from (3). ∎

To prove our first main result and some other results given later, we
establish a lemma.

######  Lemma 2.2.

Let $n\geq 1$ be an integer. Let $M,N,K$ be $R$-modules such that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{n}(K)$.

  1. (1)

There exists an exact sequence of $R$-modules

$0\to\operatorname{Ext}^{n-1}_{R}(\operatorname{Hom}_{R}(M,N),K)\to\operatorname{Ext}^{n-1}_{R}(M^{*}\otimes_{R}N,K)\to\operatorname{Ext}^{n}_{R}(\operatorname{\underline{Hom}}_{R}(M,N),K)\to\operatorname{Ext}^{n}_{R}(\operatorname{Hom}_{R}(M,N),K).$

  2. (2)

For all integers $i\leq n-2$ one has an isomorphism of $R$-modules
$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),K)\cong\operatorname{Ext}^{i}_{R}(M^{*}\otimes_{R}N,K)$.

###### Proof.

According to [4, Theorem (2.8)], there is an exact
sequence\diamondsuit\diamondsuit$\diamondsuit$In fact, one can add “$0\to$” at
the beginning of the exact sequence, that is,
$\operatorname{Ker}f\cong\operatorname{Tor}_{2}^{R}(\operatorname{Tr}M,N)$.

$\operatorname{Tor}_{2}^{R}(\operatorname{Tr}M,N)\to
M^{*}\otimes_{R}N\xrightarrow{f}\operatorname{Hom}_{R}(M,N)\to\operatorname{Tor}_{1}^{R}(\operatorname{Tr}M,N)\to
0.$

Set $H=\operatorname{Ker}f$,
$C=\operatorname{Tor}_{1}^{R}(\operatorname{Tr}M,N)$ and
$L=\operatorname{Im}f$. Note that the support of $H$ is contained in that of
$\operatorname{Tor}_{2}^{R}(\operatorname{Tr}M,N)$. Since
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{n}(K)$,
we see that
$\operatorname{Ext}^{i}_{R}(C,K)=0=\operatorname{Ext}^{i}_{R}(H,K)$ for all
$i<n$ by [4, Lemma (4.5)]. Dualizing the above exact sequence by $K$ yields
isomorphisms
$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),K)\cong\operatorname{Ext}^{i}_{R}(L,K)$
and
$\operatorname{Ext}_{R}^{j}(L,K)\cong\operatorname{Ext}_{R}^{j}(M^{*}\otimes_{R}N,K)$
for all $i\leq n-2$ and $j\leq n-1$, and an exact sequence

$0\to\operatorname{Ext}^{n-1}_{R}(\operatorname{Hom}_{R}(M,N),K)\to\operatorname{Ext}^{n-1}_{R}(L,K)\to\operatorname{Ext}^{n}_{R}(C,K)\to\operatorname{Ext}^{n}_{R}(\operatorname{Hom}_{R}(M,N),K).$

It remains to note from [27, Lemma (3.9)] that
$\operatorname{\underline{Hom}}_{R}(M,N)\cong\operatorname{Tor}_{1}^{R}(\operatorname{Tr}M,N)=C$.
∎

Let $K$ be an $R$-module. An $R$-module $M$ is called $n$-torsionfree with
respect to $K$ provided that
$\operatorname{Ext}^{i}_{R}(\operatorname{Tr}M,K)=0$ for all $1\leq i\leq n$.
Note that $M$ is $n$-torsionfree if and only if it is $n$-torsionfree with
respect to $R$. For $R$-modules $M,N$ we denote by $\operatorname{grade}(M,N)$
the smallest non-negative integer $n$ such that
$\operatorname{Ext}^{i}_{R}(M,N)\neq 0$. The following theorem is the first
main result of this paper, which includes Theorem 1.2 from the Introduction.

######  Theorem 2.3.

Let $n\geq 1$ be an integer. Let $M,N,K$ be $R$-modules. Assume that $M$ is
$(n+1)$-torsionfree with respect to $K$, and that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{n}(K)$.
Then the following hold.

  1. (1)

There exists an exact sequence of $R$-modules

$\displaystyle 0\to$
$\displaystyle\operatorname{Ext}^{n-1}_{R}(\operatorname{Hom}_{R}(M,N),K)\to\operatorname{Ext}^{n-1}_{R}(N,M\otimes_{R}K)\to\operatorname{Ext}^{n}_{R}(\operatorname{\underline{Hom}}_{R}(M,N),K)$
$\displaystyle\to$
$\displaystyle\operatorname{Ext}^{n}_{R}(\operatorname{Hom}_{R}(M,N),K).$

  2. (2)

For all integers $i\leq n-2$ one has an isomorphism of $R$-modules

$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),K)\cong\operatorname{Ext}^{i}_{R}(N,M\otimes_{R}K).$

###### Proof.

Thanks to Lemma 2.2, it suffices to show that
$\operatorname{Ext}^{i}_{R}(M^{*}\otimes_{R}N,K)\cong\operatorname{Ext}_{R}^{i}(N,M\otimes_{R}K)$
for all integers $i\leq n-1$. There are two spectral sequences converging to
the same point:

$E_{2}^{pq}=\operatorname{Ext}^{p}_{R}(\operatorname{Tor}_{q}^{R}(M^{*},N),K)\Longrightarrow
H^{p+q},\qquad
F_{2}^{pq}=\operatorname{Ext}^{p}_{R}(N,\operatorname{Ext}^{q}_{R}(M^{*},K))\Longrightarrow
H^{p+q}.$

As $\operatorname{grade}(\operatorname{Tor}_{i}^{R}(M^{*},N),K)\geq n$ for all
$i>0$ by [4, Lemma (4.5)], we have that $E_{2}^{pq}=0$ if $q>0$ and $p<n$, and
obtain an isomorphism $E_{2}^{i0}\cong H^{i}$ for all integers $i\leq n$.
Since $\operatorname{Ext}^{i}_{R}(\operatorname{Tr}M,K)=0$ for all $1\leq
i\leq n+1$, we have $\operatorname{Ext}^{i}_{R}(M^{*},K)=0$ for all $1\leq
i\leq n-1$. Hence $F_{2}^{pq}=0$ if $1\leq q\leq n-1$, and we get
$F_{2}^{i0}\cong H^{i}$ for all integers $i\leq n-1$. As $n+1\geq 2$, the
module $M$ is $2$-torsionfree with respect to $K$, and
$M\otimes_{R}K\cong\operatorname{Hom}_{R}(M^{*},K)$ by [4, Proposition (2.6)].
Thus there are isomorphisms

$\operatorname{Ext}^{i}_{R}(M^{*}\otimes_{R}N,K)=E_{2}^{i0}\cong H^{i}\cong
F_{2}^{i0}=\operatorname{Ext}^{i}_{R}(N,\operatorname{Hom}_{R}(M^{*},K))\cong\operatorname{Ext}^{i}_{R}(N,M\otimes_{R}K)$

for all integers $i\leq n-1$ and the proof of the theorem is completed. ∎

We obtain the following corollary as a consequence of the above theorem.

######  Corollary 2.4.

Let $R$ be a local ring, and let $M,N$ be $R$-modules. Assume that $M$ is
$(t+1)$-torsionfree and that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{t}(R)$.

  1. (1)

If
$\operatorname{Ext}_{R}^{t}(\operatorname{Hom}_{R}(M,N),R)=\operatorname{Ext}_{R}^{t-1}(N,M)=0$,
then $\operatorname{\underline{Hom}}_{R}(M,N)=0$.

  2. (2)

Suppose that $\operatorname{Hom}(M,N)$ has depth at least $2$ and finite
G-dimension. Then there exists an isomorphism

$\operatorname{Ext}_{R}^{t-1}(N,M)\cong\operatorname{Ext}_{R}^{t}(\operatorname{\underline{Hom}}_{R}(M,N),R).$

In particular, $\operatorname{Ext}_{R}^{t-1}(N,M)=0$ if and only if
$\operatorname{\underline{Hom}}_{R}(M,N)=0$.

###### Proof.

(1) If $t=0$, then $\operatorname{Hom}_{R}(\operatorname{Hom}_{R}(M,N),R)=0$,
and $\operatorname{Hom}_{R}(M,N)=0$ by [8, Exercise 1.2.27], which implies
$\operatorname{\underline{Hom}}_{R}(M,N)=0$. Let $t>0$. Theorem 2.3(1) shows
$\operatorname{Ext}^{t}_{R}(\operatorname{\underline{Hom}}_{R}(M,N),R)=0$,
while $\operatorname{grade}_{R}\operatorname{\underline{Hom}}_{R}(M,N)\geq t$
by [4, Lemma (4.5)] or [8, Proposition 1.2.10(a)]. Hence
$\operatorname{grade}_{R}\operatorname{\underline{Hom}}_{R}(M,N)>t$. Thus, if
$\operatorname{\underline{Hom}}_{R}(M,N)$ is nonzero, then its annihilator
contains an $R$-sequence of length more than $t=\operatorname{depth}R$, which
cannot occur. We must have $\operatorname{\underline{Hom}}_{R}(M,N)=0$.

(2) The last assertion is shown similarly to (1). As for the first assertion,
in view of Theorem 2.3(1), it is enough to show that
$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),R)=0$ for $i=t-1,t$.
We have
$\operatorname{Gdim}_{R}(\operatorname{Hom}_{R}(M,N))=\operatorname{depth}R-\operatorname{depth}_{R}(\operatorname{Hom}_{R}(M,N))\leq
t-2$, which implies that
$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,N),R)=0$ for all $i>t-2$.
∎

######  Remark 2.5.

In fact, one can generalize Corollary 2.4 to a $(t+1)$-torsionfree module $M$
with respect to a module $K$ satisfying
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{n}(K)$
with $n=\operatorname{depth}_{R}K$, assuming that $K$ is semidualizing for
(2). We leave it to the reader as an exercise.

Corollary 2.4 immediately recovers Corollary 1.4 in the case where $\dim R\geq
2$, as follows. Thus our Theorem 2.3 (Theorem 1.2) recovers the
Auslander–Reiten duality theorem in dimension at least two.

######  Corollary 2.6 (Auslander–Reiten duality in dimension at least two).

Let $(R,\mathfrak{m})$ be a Gorenstein local ring with $d\geq 2$. Let $M,N$ be
maximal Cohen–Macaulay $R$-modules with
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$.
Then for each $i\in\mathbb{Z}$ there is an isomorphism

$\mathrm{\widehat{Ext}}\mathrm{}_{R}^{i}(N,M)^{\vee}\cong\mathrm{\widehat{Ext}}\mathrm{}_{R}^{(d-1)-i}(M,N).$

###### Proof.

As $d\geq 2$, one can apply Corollary 2.4(2) to get
$\operatorname{Ext}^{d-1}_{R}(N,M)\cong\operatorname{Ext}^{d}_{R}(\operatorname{\underline{Hom}}_{R}(M,N),R)$,
which and [8, Corollary 3.5.9] show
$\operatorname{Ext}^{d-1}_{R}(N,M)^{\vee}\cong\operatorname{\underline{Hom}}_{R}(M,N)$.
Substituting $\mathrm{\Omega}^{i-(d-1)}N$ for $N$ (and using basic facts on
Tate cohomology [25, Section 7]), we obtain

$\mathrm{\widehat{Ext}}\mathrm{}^{i}_{R}(N,M)^{\vee}\cong\operatorname{Ext}^{d-1}_{R}(\mathrm{\Omega}^{i-(d-1)}N,M)^{\vee}\cong\operatorname{\underline{Hom}}_{R}(M,\mathrm{\Omega}^{i-(d-1)}N)\cong\mathrm{\widehat{Ext}}\mathrm{}^{(d-1)-i}_{R}(M,N),$

which give an isomorphism as in the assertion. ∎

We prepare a lemma for the proof of the second main result of this paper,
which is also be used later. The proof of the lemma is analogous to that of
[15, Lemma 2.3].

######  Lemma 2.7.

Let $R$ be a Cohen–Macaulay local ring with canonical module $\omega$. Let $M$
be an $R$-module and $N$ a maximal Cohen–Macaulay $R$-module. Let $0\leq m\leq
d$ be an integer such that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{m}(R)$.
Then
$\operatorname{Ext}^{i}_{R}(M,N^{\dagger})\cong\operatorname{Ext}^{i}_{R}(M\otimes_{R}N,\omega)$
for all integers $0\leq i\leq m$.

We are now ready to present our second main result, which includes Theorem 1.3
from the Introduction.

######  Theorem 2.8.

Let $(R,\mathfrak{m})$ be a Cohen–Macaulay local ring with canonical module
$\omega$. Let $M,N$ be $R$-modules such that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$.
Assume that $N$ is maximal Cohen–Macaulay.

  1. (1)

There is an isomorphism
$\operatorname{\underline{Hom}}_{R}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{d+1}(\operatorname{Tr}M,N^{\dagger})$.
Therefore, if $d\geq 2$, then one has

$\operatorname{\underline{Hom}}_{R}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{d-1}(M^{*},N^{\dagger}).$

  2. (2)

Suppose that $M$ is locally totally reflexive on the punctured spectrum of
$R$. Then there is an isomorphism
$\operatorname{Ext}_{R}^{i}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{d+1-i}(\operatorname{Tr}M,N^{\dagger})$
for all $1\leq i\leq d$. In particular, for all $1\leq i\leq d-2$ one has

$\operatorname{Ext}_{R}^{i}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{(d-1)-i}(M^{*},N^{\dagger}).$

###### Proof.

We claim that $E:=\operatorname{Ext}^{j}_{R}(\operatorname{Tr}M,N^{\dagger})$
has finite length for all $j\geq d$. Indeed, let $\mathfrak{p}$ be a non-
maximal prime ideal. As
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$,
either $M_{\mathfrak{p}}$ or $N_{\mathfrak{p}}$ is free. If $M_{\mathfrak{p}}$
is free, then so is $\operatorname{Tr}M_{\mathfrak{p}}$, and
$E_{\mathfrak{p}}=0$. If $N_{\mathfrak{p}}$ is free, then
$(N^{\dagger})_{\mathfrak{p}}$ is a direct sum of copies of
$\omega_{\mathfrak{p}}$, which has injective dimension at most $\dim
R_{\mathfrak{p}}<d$, and hence $E_{\mathfrak{p}}=0$.

(1) By assumption, $\operatorname{Tor}_{i}^{R}(\operatorname{Tr}M,N)$ has
finite length for all $i>0$. There are isomorphisms

$\operatorname{\underline{Hom}}_{R}(M,N)\cong\operatorname{Tor}_{1}^{R}(\operatorname{Tr}M,N)\cong\operatorname{Ext}^{d}_{R}(\operatorname{Tor}_{1}(\operatorname{Tr}M,N),\omega)^{\vee}\cong\operatorname{Ext}^{d+1}_{R}(\operatorname{Tr}M,N^{\dagger})^{\vee},$

where the first two isomorphisms are obtained by [27, Lemma (3.9)] and [8,
Corollary 3.5.9], respectively, while the last isomorphism is shown to hold by
the spectral sequence argument in [2, 2.4]. By the above claim we see that
$\operatorname{\underline{Hom}}_{R}(M,N)^{\vee}\cong\operatorname{Ext}^{d+1}_{R}(\operatorname{Tr}M,N^{\dagger})^{\vee\vee}\cong\operatorname{Ext}^{d+1}_{R}(\operatorname{Tr}M,N^{\dagger})$
by [8, Proposition 3.2.12(c)].

(2) We show the assertion by induction on $i$. There are isomorphisms

$\operatorname{Ext}_{R}^{1}(M,N)\cong\Gamma_{\mathfrak{m}}(\operatorname{Ext}_{R}^{1}(M,N))\cong\Gamma_{\mathfrak{m}}(\operatorname{Tr}M\otimes_{R}N)\cong\operatorname{Ext}_{R}^{d}(\operatorname{Tr}M\otimes_{R}N,\omega)^{\vee}\cong\operatorname{Ext}_{R}^{d}(\operatorname{Tr}M,N^{\dagger})^{\vee}.$

Let us explain each of the above isomorphisms. As
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\\{\mathfrak{m}\\}$ and
$M$ is locally totally reflexive on the punctured spectrum,
$\operatorname{Ext}^{1}_{R}(M,N)$ has finite length. Hence the first
isomorphism holds. There is an exact sequence
$0\to\operatorname{Ext}^{1}_{R}(M,N)\to\operatorname{Tr}M\otimes_{R}N\to\operatorname{Hom}_{R}((\operatorname{Tr}M)^{*},N)$
by [4, Proposition (2.6)], while
$\operatorname{Hom}_{R}((\operatorname{Tr}M)^{*},N)$ has positive depth by [8,
Exercise 1.4.19]. Thus the second isomorphism holds. The third and fourth
isomorphisms follow from [8, Corollary 3.5.9] and Lemma 2.7, respectively.
Consequently, $\operatorname{Ext}_{R}^{1}(M,N)^{\vee}$ is isomorphic to
$\operatorname{Ext}_{R}^{d}(\operatorname{Tr}M,N^{\dagger})^{\vee\vee}$, which
is isomorphic to $\operatorname{Ext}_{R}^{d}(\operatorname{Tr}M,N^{\dagger})$
by the claim given at the beginning of the proof and [8, Proposition
3.2.12(c)]. Thus the case $i=1$ is settled.

Let $i\geq 2$. Applying the induction hypothesis to $\mathrm{\Omega}M$, we
obtain isomorphisms

$\operatorname{Ext}_{R}^{i}(M,N)^{\vee}\cong\operatorname{Ext}_{R}^{i-1}(\mathrm{\Omega}M,N)^{\vee}\cong\operatorname{Ext}_{R}^{d+1-(i-1)}(\operatorname{Tr}\mathrm{\Omega}M,N^{\dagger})\cong\operatorname{Ext}_{R}^{d+1-i}(\mathrm{\Omega}\operatorname{Tr}\mathrm{\Omega}M,N^{\dagger}).$

There is an exact sequence
$0\to\operatorname{Ext}_{R}^{1}(M,R)\to\operatorname{Tr}M\to\mathrm{\Omega}\operatorname{Tr}\mathrm{\Omega}M\to
0$ by [4, Proposition (2.6)]. Since $\operatorname{Ext}_{R}^{1}(M,R)$ has
finite length and $N^{\dagger}$ is maximal Cohen–Macaulay,
$\operatorname{Ext}_{R}^{j}(\operatorname{Ext}_{R}^{1}(M,R),N^{\dagger})=0$
for all $j<d$. We obtain an isomorphism
$\operatorname{Ext}_{R}^{d+1-i}(\mathrm{\Omega}\operatorname{Tr}\mathrm{\Omega}M,N^{\dagger})\cong\operatorname{Ext}_{R}^{d+1-i}(\operatorname{Tr}M,N^{\dagger})$,
which completes the proof. ∎

Theorem 2.8 not only recovers but also refines a main result of [10] (more
precisely, [10, Theorem 1.1]), where $M$ is assumed to be locally free on the
punctured spectrum.

######  Corollary 2.9 (Celikbas–Takahashi, improved).

Let $(R,\mathfrak{m})$ be a Cohen–Macaulay local ring with a canonical module.
Let $X$ be a totally reflexive $R$-module, and let $M$ be a maximal
Cohen–Macaulay $R$-module. Assume that
$\operatorname{NF}(X)\cap\operatorname{NF}(M)\subseteq\\{\mathfrak{m}\\}$.
Then for each $i\in\mathbb{Z}$ there is an isomorphism

$\mathrm{\widehat{Ext}}\mathrm{}_{R}^{i}(X,M)^{\vee}\cong\mathrm{\widehat{Ext}}\mathrm{}_{R}^{(d-1)-i}(X^{\ast},M^{\dagger}).$

###### Proof.

We have the following isomorphisms, where the second one follows from Theorem
2.8(1) and the fact that the Ext module has finite length (use [8, Proposition
3.2.12(c)]).

$\displaystyle\mathrm{\widehat{Ext}}\mathrm{}^{i}_{R}(X,M)^{\vee}$
$\displaystyle\cong\operatorname{\underline{Hom}}_{R}(\mathrm{\Omega}^{i}X,M)^{\vee}\cong\operatorname{Ext}^{d+1}_{R}(\operatorname{Tr}\mathrm{\Omega}^{i}X,M^{\dagger})\cong\operatorname{\underline{Hom}}_{R}(\mathrm{\Omega}^{d+1}\operatorname{Tr}\mathrm{\Omega}^{i}X,M^{\dagger}),$
$\displaystyle\mathrm{\widehat{Ext}}\mathrm{}^{(d-1)-i}_{R}(X^{\ast},M^{\dagger})$
$\displaystyle\cong\operatorname{\underline{Hom}}_{R}(\mathrm{\Omega}^{(d-1)-i}\mathrm{\Omega}^{2}\operatorname{Tr}X,M^{\dagger})\cong\operatorname{\underline{Hom}}_{R}(\mathrm{\Omega}^{d+1-i}\operatorname{Tr}X,M^{\dagger}).$

It suffices to show
$\operatorname{Tr}\mathrm{\Omega}^{i}X\cong\mathrm{\Omega}^{-i}\operatorname{Tr}X$
up to free summands, which follows from [25, Proposition 7.1]. ∎

It is evident that Corollary 2.9 implies Corollary 1.4. Thus our Theorem 2.8
(Theorem 1.3) also recovers the Auslander–Reiten duality theorem (for any
dimension).

##  3\. Testing freeness by vanishing of Ext modules

In this section, we present applications of our results obtained in the
previous section to give criteria for freeness of modules over local rings and
to recover various known results about the Auslander–Reiten conjecture. To
give our first application, we establish a lemma, which will also be used
later.

######  Lemma 3.1.

Let $M$, $N$ be $R$-modules. Suppose that $M$ is reflexive (i.e.
$2$-torsionfree) and that
$\operatorname{NF}(M)\cap\operatorname{NF}(N)\subseteq\operatorname{Y}^{2}(R)$.
Then one has an isomorphism
$\operatorname{Hom}_{R}(M,N)^{*}\cong\operatorname{Hom}_{R}(N,M)$.

###### Proof.

Let $n=2$, $i=0$ and $K=R$ in Lemma 2.2(2), and then use the tensor-hom
adjunction. ∎

######  Corollary 3.2.

Let $R$ be a local ring, and let $M$ be a $(t+1)$st syzygy $R$-module. Assume
that $M$ is locally free on $\operatorname{X}^{n}(R)$ for some $0<n<t$. Then
$M$ is free, provided that one of the following three conditions is satisfied:

  1. (1)

$\operatorname{Ext}_{R}^{i}(\operatorname{Hom}_{R}(M,M),R)=\operatorname{Ext}_{R}^{j}(M,M)=0$
for all $n+1\leq i\leq t$ and $n\leq j\leq t-1$.

  2. (2)

$\operatorname{Ext}_{R}^{j}(M,M)=0$ for all $n\leq j\leq t-1$ and
$\operatorname{Gdim}_{R}\operatorname{Hom}_{R}(M,M)<\infty$.

  3. (3)

$\operatorname{Ext}_{R}^{j}(M,M)=0$ for all $n\leq j\leq t-1$ and
$\operatorname{Hom}_{R}(M,M)$ is $(t+2)$-torsionfree.

###### Proof.

First of all, note that the inequalities $0<n<t$ especially say that $t\geq
2$.

(1) We use induction on $s=t-n$. If $s=1$, then $n=t-1$. Note from Example
2.1(3) that $M$ is $(t+1)$-torsionfree. Corollary 2.4(1) implies
$\operatorname{\underline{Hom}}_{R}(M,M)=0$, which is equivalent to saying
that $M$ is free. Let $s\geq 2$.

We claim that $M$ is locally free on $\operatorname{X}^{t-1}(R)$. Indeed, pick
any prime ideal $\mathfrak{p}\in\operatorname{X}^{t-1}(R)$ and set
$t^{\prime}=\operatorname{depth}R_{\mathfrak{p}}$. If $t^{\prime}\leq n$, then
$\mathfrak{p}\in\operatorname{X}^{n}(R)$, and $M_{\mathfrak{p}}$ is
$R_{\mathfrak{p}}$-free by assumption. If $t^{\prime}>n$, then
$0<n<t^{\prime}$ and $M_{\mathfrak{p}}$ is locally free on
$\operatorname{X}^{n}(R_{\mathfrak{p}})$. As $t^{\prime}\leq t$, the module
$M_{\mathfrak{p}}$ is a $(t^{\prime}+1)$st syzygy $R_{\mathfrak{p}}$-module,
and
$\operatorname{Ext}_{R_{\mathfrak{p}}}^{i}(\operatorname{Hom}_{R_{\mathfrak{p}}}(M_{\mathfrak{p}},M_{\mathfrak{p}}),R_{\mathfrak{p}})=\operatorname{Ext}_{R_{\mathfrak{p}}}^{j}(M_{\mathfrak{p}},M_{\mathfrak{p}})=0$
for all $n+1\leq i\leq t^{\prime}$ and $n\leq j\leq t^{\prime}-1$. Since
$t^{\prime}-n<s$, we can apply the induction hypothesis to deduce that
$M_{\mathfrak{p}}$ is $R_{\mathfrak{p}}$-free. Thus the claim follows.

By Example 2.1(3) the module $M$ is $(t+1)$-torsionfree. Corollary 2.4(1)
shows $\operatorname{\underline{Hom}}_{R}(M,M)=0$, so $M$ is free.

(2) Similarly to the proof of (1), one can prove that $M$ is
$(t+1)$-torsionfree and locally free on $\operatorname{X}^{t-1}(R)$. As $M$ is
a $(t+1)$st syzygy module and $t\geq 2$, the depth lemma shows that $M$ has
depth at least two, and so does $\operatorname{Hom}_{R}(M,M)$ by [8, Exercise
1.4.19]. Now the assertion follows from Corollary 2.4(2).

(3) Again, one can show that $M$ is $(t+1)$-torsionfree. In particular, $M$ is
$2$-torsionfree. Set $H=\operatorname{Hom}_{R}(M,M)$. Lemma 3.1 implies
$H\cong H^{*}$. As $H$ is $(t+2)$-torsionfree, we have
$\operatorname{Ext}^{i}_{R}(H,R)\cong\operatorname{Ext}^{i}_{R}(H^{*},R)\cong\operatorname{Ext}^{i+2}_{R}(\operatorname{Tr}H,R)=0$
for $1\leq i\leq t$. Now the assertion follows from (1). ∎

We record various consequences of Corollary 3.2. First of all, Corollary
3.2(2) immediately recovers the theorems of Araya [1, Corollary 10], Ono and
Yoshino [23, Theorem], and the following result due to Araya, Celikbas,
Sadeghi and Takahashi [2, Corollary 1.6].

######  Corollary 3.3 (Araya–Celikbas–Sadeghi–Takahashi).

Let $R$ be a Gorenstein local ring. Let $M$ be a maximal Cohen–Macaulay
$R$-module, and $0<n<d$ an integer. Suppose that $M$ is locally free on
$\operatorname{X}^{n}(R)$ and $\operatorname{Ext}_{R}^{i}(M,M)=0$ for all
$n\leq i\leq d-1$. Then $M$ is free.

Recall that an $R$-module $M$ is called torsion-free if the natural map $M\to
M\otimes_{R}\operatorname{Q}(R)$ is injective, or in other words, each non-
zerodivisor of $R$ is also a non-zerodivisor of $M$.

######  Corollary 3.4.

Let $R$ be a Cohen–Macaulay local ring of dimension $d$ with a canonical
module $\omega$. Assume that $M$ is a $(d+1)$st syzygy $R$-module that is
locally free in codimension one. Suppose that
$\operatorname{Ext}^{i}_{R}(M,M)=0$ for $1\leq i\leq d-1$ and
$\operatorname{Hom}_{R}(M,M)\otimes_{R}\omega_{R}$ is torsion-free. Then $M$
is free.

###### Proof.

First of all, since $R$ satisfies $(\mathrm{S}_{1})$, an $R$-module is
torsion-free if and only if it satisfies $(\mathrm{S}_{1})$. Hence the
torsion-free property localizes. We argue by induction on $d$. The assertion
is evident if $d\leq 1$, so we may assume that $d>1$ and $M$ is locally free
on the punctured spectrum of $R$. Set $H=\operatorname{Hom}_{R}(M,M)$. As $H$
is locally free on the punctured spectrum, by Lemma 2.7 we have
$\operatorname{Ext}^{i}_{R}(\omega\otimes_{R}H,\omega)\cong\operatorname{Ext}^{i}_{R}(H,R)$
for all $0\leq i\leq d$. Since $\omega\otimes_{R}H$ has positive depth,
$\operatorname{Ext}_{R}^{d}(H,R)=0$ by local duality. The assertion is deduced
from Corollary 3.2(1). ∎

The following two results are also consequences of Corollary 3.2. The first
one is nothing but Corollary 1.5 from the Introduction. Both should be
compared with the theorems of Huneke and Leuschke [17, Theorem 1.3], Goto and
Takahashi [15, Theorem 1.5(2)], and Dao, Eghbali and Lyle [13, Theorem 3.16].

######  Corollary 3.5.

Let $R$ be a generically Gorenstein local ring, and let $0<n<t$ be an integer.
Let $M$ be an $R$-module which is locally free on $\operatorname{X}^{n}(R)$.
Then $M$ is free, if the following three conditions hold.

  1. (1)

$\operatorname{Ext}^{i}_{R}(M,M)=0$ for all $n\leq i\leq t-1$.

  2. (2)

$\operatorname{Ext}^{i}_{R}(M,R)=0$ for all $i>0$.

  3. (3)

$\operatorname{Ext}^{i}_{R}(\operatorname{Hom}_{R}(M,M),R)=0$ for all $n+1\leq
i\leq t$, or $\operatorname{Hom}_{R}(M,M)$ has finite G-dimension.

###### Proof.

It follows from generic Gorensteinness, (2) and [28, Corollary 1.3] that $M$
is a totally reflexive module, and a $j$th syzygy module for all $j\geq 0$.
The assertion follows from the first and second assertions of Corollary 3.2. ∎

Note that every normal ring $R$ is generically Gorenstein and every syzygy
$R$-module is locally free on $\operatorname{X}^{1}(R)$. Thus the following is
immediate from Corollary 3.5, which is nothing but Corollary 1.6 from the
Introduction.

######  Corollary 3.6.

Let $R$ be a normal local ring. An $R$-module $M$ is free if either

  1. (1)

$\operatorname{Ext}^{i}_{R}(M,M)=\operatorname{Ext}^{j}_{R}(M,R)=\operatorname{Ext}^{h}_{R}(\operatorname{Hom}_{R}(M,M),R)=0$
for all $1\leq i\leq t-1$, $j\geq 1$ and $2\leq h\leq t$, or

  2. (2)

$\operatorname{Ext}^{i}_{R}(M,M)=\operatorname{Ext}^{j}_{R}(M,R)=0$ for all
$1\leq i\leq t-1$ and $j\geq 1$, and that $\operatorname{Hom}_{R}(M,M)$ has
finite G-dimension.

Corollary 3.6 recovers the following result due to Huneke and Leuschke [17,
Theorem 0.1] and Araya [1, Corollary 4], which asserts that the
Auslander–Reiten conjecture holds for normal Gorenstein rings.

######  Corollary 3.7 (Huneke–Leuschke, Araya).

Let $R$ be a (not necessarily local) Gorenstein normal ring. Let $M$ be an
$R$-module. If
$\operatorname{Ext}_{R}^{i}(M,M)=0=\operatorname{Ext}_{R}^{i}(M,R)$ for all
$i>0$, then $M$ is projective.

Now we deal with reflexive modules $M$ such that $\operatorname{Hom}_{R}(M,M)$
is a direct sum of copies of $M$. Such modules were first studied by Auslander
[3] to give a new proof of a theorem on the purity of the branch locus for
regular local rings; see [3, Proposition 1.2 and Theorem 1.3]. The following
result especially says that the Auslander–Reiten conjecture holds true for
such modules over normal rings.

######  Corollary 3.8.

Let R be a local ring with $t\geq 2$, and let $M$ be a reflexive $R$-module
that is locally free on $\operatorname{X}^{1}(R)$. Assume that
$\operatorname{Hom}_{R}(M,M)\cong M^{\oplus n}$ for some $n>0$. If
$\operatorname{Ext}^{i}_{R}(M,R)=\operatorname{Ext}_{R}^{j}(M,M)=0$ for all
$1\leq i\leq t$ and $1\leq j\leq t-1$, then $M$ is free.

###### Proof.

We use induction on $t$. We may assume that $M$ is locally free on
$\operatorname{X}^{t-1}(R)$. By Lemma 3.1, we have $M^{\oplus
n}\cong\operatorname{Hom}_{R}(M,M)\cong\operatorname{Hom}_{R}(M,M)^{*}\cong(M^{*})^{\oplus
n}$. Since $M$ is reflexive, so is $\operatorname{Hom}_{R}(M,M)$ and

$\operatorname{Ext}_{R}^{i}(\operatorname{Tr}\operatorname{Hom}_{R}(M,M),R)\cong\operatorname{Ext}_{R}^{i-2}(\operatorname{Hom}_{R}(M,M)^{*},R)\cong\operatorname{Ext}_{R}^{i-2}(M,R)^{\oplus
n}=0$

for all $3\leq i\leq t+2$. Hence $\operatorname{Hom}_{R}(M,M)$ is
$(t+2)$-torsionfree, and so is $M$. In particular, $M$ is a $(t+2)$nd syzygy
module. Now the assertion follows from Corollary 3.2(3). ∎

Here we state an application of Theorem 2.8.

######  Corollary 3.9.

Let $R$ be a Cohen–Macaulay local ring with canonical module $\omega$. Let
$0<n<d$ be an integer, and let $M$ be an $R$-module that is locally free on
$\operatorname{X}^{n}(R)$. Then $M$ is free, if it satisfies one of the
following conditions.

  1. (1)

$M$ satisfies $(\mathrm{S}_{1})$, $M^{*}$ is maximal Cohen–Macaulay, and
$\operatorname{Ext}_{R}^{i}(M,(M^{*})^{\dagger})=0$ for all $n\leq i<d$.

  2. (2)

$\operatorname{Min}M\subseteq\operatorname{Ass}R$, $M^{*}$ is maximal
Cohen–Macaulay, and $\operatorname{Ext}_{R}^{i}(M,(M^{*})^{\dagger})=0$ for
all $n\leq i\leq d$.

  3. (3)

$M$ is a syzygy module, and
$\operatorname{Ext}_{R}^{i}(M,R)=\operatorname{Ext}_{R}^{j}(M,M\otimes_{R}\omega)=0$
for all $0<i\leq d$ and $n\leq j<d$.

###### Proof.

(1) First of all, note that the inequalities $0<n<d$ especially says $d\geq
2$.

(i) We begin with showing the assertion in the case where $M$ is locally free
on the punctured spectrum of $R$. In this case, Theorem 2.8(1) implies
$\operatorname{\underline{Hom}}_{R}(M^{*},M^{*})^{\vee}\cong\operatorname{Ext}_{R}^{d-1}(M^{**},M^{*{\dagger}})$.
Let $f:M\to M^{**}$ be the natural homomorphism. As $f$ is locally an
isomorphism on the punctured spectrum, $K=\operatorname{Ker}f$ and
$C=\operatorname{Coker}f$ have finite length. Since $M^{*{\dagger}}$ is
maximal Cohen–Macaulay, we have
$\operatorname{Ext}_{R}^{j}(K,M^{*{\dagger}})=0=\operatorname{Ext}_{R}^{j}(C,M^{*{\dagger}})=0$
for all $j<d$. It is easy to observe that the homomorphism
$\operatorname{Ext}_{R}^{d-1}(M^{**},M^{*{\dagger}})\to\operatorname{Ext}_{R}^{d-1}(M,M^{*{\dagger}})$
induced from $f$ is injective. By assumption, the latter Ext module vanishes.
Hence $\operatorname{\underline{Hom}}_{R}(M^{*},M^{*})=0$, which implies that
$M^{*}$ is free.

Consider the case $M^{*}=0$. Then $f=0$ and $M=K$. Hence $M$ has finite
length, while $M$ has positive depth as it satisfies $(\mathrm{S}_{1})$.
Therefore $M=0$. In particular $M$ is free, and we are done. Thus we may
assume $M^{*}\neq 0$. Theorem 2.8(2) implies
$\operatorname{Ext}_{R}^{d-1}(M,M^{*{\dagger}})^{\vee}\cong\operatorname{Ext}_{R}^{2}(\operatorname{Tr}M,M^{*})$,
and the former Ext module vanishes. As $M^{*}$ is a nonzero free module,
$\operatorname{Ext}_{R}^{2}(\operatorname{Tr}M,R)=0$. This means $C=0$, and we
have an exact sequence $0\to K\to M\to M^{**}\to 0$, whose last term is free.
Hence this exact sequence splits, and we get an isomorphism $M\cong K\oplus
M^{**}$. As $M$ has positive depth and $K$ has finite length, we have $K=0$.
Consequently, $M$ is free.

(ii) Now let us consider the general case. We handle this by induction on $d$.
When $d=2$, we have $n=1=d-1$, and $\operatorname{X}^{n}(R)$ coincides with
the punctured spectrum of $R$. Case (i) shows the assertion. Let $d>2$, and
pick any non-maximal prime ideal $\mathfrak{p}$. Then one can apply the
induction hypothesis to the localizations $R_{\mathfrak{p}}$ and
$M_{\mathfrak{p}}$ to deduce that $M_{\mathfrak{p}}$ is free. This means that
$M$ is locally free on the punctured spectrum, and again case (i) implies the
assertion.

(2) We use the same argument as the proof of (1). Using the assumption that
$\operatorname{Min}M\subseteq\operatorname{Ass}R$, we easily deduce that
$M^{*}_{\mathfrak{p}}\neq 0$ for all $\mathfrak{p}\in\operatorname{Supp}M$.
When $M$ is locally free on the punctured spectrum, $M^{*}$ is a free module,
and It follows from Theorem 2.8(2) that
$\operatorname{Ext}_{R}^{1}(\operatorname{Tr}M,M^{*})\cong\operatorname{Ext}_{R}^{d}(M,M^{*{\dagger}})^{\vee}=0$.
As $M^{*}\neq 0$, we have
$\operatorname{Ext}_{R}^{1}(\operatorname{Tr}M,R)=0$. This means that $M$ is a
syzygy, and in particular it satisfies $(\mathrm{S}_{1})$.

(3) Dualizing a free resolution $F$ of $M$ by $R$, we obtain an exact sequence
$0\to M^{*}\to(F_{0})^{*}\to\cdots\to(F_{d+1})^{*}$. This especially says that
$M^{*}$ is maximal Cohen–Macaulay. By [7, Corollary B4(3)] we have
$(M^{*})^{\dagger}\cong M\otimes_{R}\omega$. As $M$ is a syzygy module, it
satisfies $(\mathrm{S}_{1})$. Now the assertion follows from (1). ∎

The above result is actually a refinement of [2, Theorem 1.4 and Corollary
2.7].

######  Corollary 3.10 (Araya–Celikbas–Sadeghi–Takahashi).

Let $R,\omega,n$ be as in Corollary 3.9.

  1. (1)

An $R$-module $M$ is free if it satisfies the following conditions.

    1. (i)

$M$ is locally of finite projective dimension on $\operatorname{X}^{n}(R)$.

    2. (ii)

$M$ satisfies $(\mathrm{S}_{2})$ and $M^{\ast}$ is maximal Cohen–Macaulay.

    3. (iii)

$\operatorname{Ext}^{i}_{R}(M,(M^{\ast})^{\dagger})=0$ for all $n\leq i<d$.

  2. (2)

An $R$-module $M$ is free if it satisfies the following conditions.

    1. (i)

$M$ is locally of finite projective dimension on $\operatorname{X}^{n}(R)$.

    2. (ii)

$M$ is reflexive and $\operatorname{Ext}^{i}_{R}(M,R)=0$ for all $1\leq i\leq
d$.

    3. (iii)

$\operatorname{Ext}^{i}_{R}(M,M\otimes_{R}\omega)=0$ for all $n\leq i<d$.

###### Proof.

(1) Similarly to the beginning of the proof of [2, Theorem 1.4], the module
$M$ is locally free on $\operatorname{X}^{n}(R)$. Corollary 3.9(1) implies the
assertion.

(2) Since $M$ satisfies $(\mathrm{S}_{2})$, it is reflexive. As in the proof
of Corollary 3.9(3) the module $M^{*}$ is maximal Cohen–Macaulay. Thus the
assertion follows from (1). ∎

From here to the end of this section, we consider a rational normal surface
singularity, that is, a complete local normal domain of dimension two over an
algebraically closed field of characteristic zero which has a rational
singularity. Recall that an $R$-module $M$ is called rigid if
$\operatorname{Ext}_{R}^{1}(M,M)=0$. The following corollary is deduced from
our Theorem 2.3 and Corollary 3.2.

######  Corollary 3.11.

Let $R$ be a rational normal surface singularity. Let $M$ be a third syzygy
$R$-module. If $M$ is rigid, then $M$ is free.

###### Proof.

Set $H=\operatorname{Hom}_{R}(M,M)$. Note that $H$ is maximal Cohen–Macaulay
(see [8, Exercise 1.4.19]). The module $M$ is $3$-torsionfree by Example
2.1(3). It follows from Theorem 2.3(1) and the rigidity of $M$ that
$\operatorname{Ext}_{R}^{1}(H,R)=0$. Lemma 3.1 implies $H\cong H^{*}$. By [20,
Theorem 2.2 and Proposition 3.1(2)] the module $H$ is free. Finally, our
Corollary 3.2(1) deduces that $M$ is free. ∎

In view of Corollary 3.2(3), we are interested in when a given module is
$(d+2)$-torsionfree. Any totally reflexive module is $(d+2)$-torsionfree. If
$R$ is Gorenstein, then it is equivalent to maximal Cohen–Macaulayness. Thus
we are interested in when a non-totally reflexive module over a non-Gorenstein
ring is $(d+2)$-torsionfree. We discuss the triviality of such modules.

######  Proposition 3.12.

Let $R$ be a non-Gorenstein complete local ring with $d\geq 2$ such that $k$
is algebraically closed and has characteristic $0$. Assume that there exists
an $R$-sequence $\boldsymbol{x}=x_{1},\dots,x_{d-2}$ such that
$R/(\boldsymbol{x})$ is a rational normal surface singularity. Let $M$ be an
$R$-module such that $\operatorname{Ext}_{R}^{i}(M,R)=0$ for all $1\leq i\leq
2d$. Then $M$ is $R$-free.

###### Proof.

Set $N=\mathrm{\Omega}_{R}^{d-2}M$. It is seen from [19, Proposition 1.1.1]
that $N$ is a $(d-2)$-torsionfree module with
$\operatorname{Ext}_{R}^{i}(N,R)=0$ for $1\leq i\leq d+2$. As $N$ is a
$(d-2)$nd syzygy module, $\boldsymbol{x}$ is an $N$-sequence. Hence the Koszul
complex of $\boldsymbol{x}$ on $N$ induces an exact sequence

$0\to N\to N^{\oplus(d-2)}\to\cdots\to N^{\oplus(d-2)}\to N\to
N/\boldsymbol{x}N\to 0,$

which shows that
$\operatorname{Ext}_{R/(\boldsymbol{x})}^{i}(N/\boldsymbol{x}N,R/(\boldsymbol{x}))\cong\operatorname{Ext}_{R}^{i+d-2}(N/\boldsymbol{x}N,R)=0$
for $1\leq i\leq 4$ (see [8, Lemma 3.1.16]). It follows from [20, Proposition
3.1(1)] and [19, Proposition 1.1.1] again that
$\mathrm{\Omega}^{2}_{R/(\boldsymbol{x})}(N/\boldsymbol{x}N)$ is free over
$R/(\boldsymbol{x})$. Thus
$\operatorname{pd}_{R}N=\operatorname{pd}_{R/(\boldsymbol{x})}N/\boldsymbol{x}N$
is finite (see [8, Lemma 1.3.5]), and so is $\operatorname{pd}_{R}M$. The
vanishing assumption on Ext modules forces $M$ to be free over $R$. ∎

######  Corollary 3.13.

Let $R$ be as in Proposition 3.12. Let $M$ be a $(d+2)$-torsionfree $R$-module
such that $\operatorname{Ext}_{R}^{i}(M,R)=0$ for all $1\leq i\leq d-2$. Then
$M$ is free. In particular:

  1. (1)

When $d=2$, every $(d+2)$-torsionfree $R$-module is free.

  2. (2)

$R$ is a G-regular local ring in the sense of [26].

###### Proof.

We observe that $N:=\operatorname{Tr}\mathrm{\Omega}^{d-2}M$ satisfies
$\operatorname{Ext}_{R}^{i}(N,R)=0$ for $1\leq i\leq 2d$; see [19, Proposition
1.1.1]. Proposition 3.12 implies that $N$ is free, and hence so is
$\mathrm{\Omega}^{d-2}M$. Thus $M$ has finite projective dimension. Since $M$
is $(d+2)$-torsionfree, it is maximal Cohen–Macaulay. Therefore $M$ is free. ∎

## References

  * [1] T. Araya, The Auslander–Reiten conjecture for Gorenstein rings, Proc. Amer. Math. Soc. 137 (2009), no. 6, 1941–1944.
  * [2] T. Araya; O. Celikbas; A. Sadeghi; R. Takahashi, On the vanishing of self extensions over Cohen–Macaulay local rings, Proc. Amer. Math. Soc. (to appear), arXiv:1703.06160v1.
  * [3] M. Auslander, On the purity of the branch locus, Amer. J. Math. 84 (1962), 116–125.
  * [4] M. Auslander; M. Bridger, Stable module theory, Memoirs of the American Mathematical Society, No. 94, American Mathematical Society, Providence, R.I., 1969.
  * [5] M. Auslander; S. Ding; Ø. Solberg, Liftings and weak liftings of modules, J. Algebra, 156, (1993), 273–317.
  * [6] M. Auslander; I. Reiten, On a generalized version of the Nakayama conjecture, Proc. Amer. Math. Soc. 52 (1975), 69–74.
  * [7] L. L. Avramov; R.-O. Buchweitz; L. Şega, Extensions of a dualizing complex by its ring: commutative versions of a conjecture of Tachikawa, J. Pure Appl. Algebra 201 (2005), no. 1-3, 218–239.
  * [8] W. Bruns; J. Herzog, Cohen–Macaulay rings, revised edition, Cambridge Studies in Advanced Mathematics, 39, Cambridge University Press, Cambridge, 1998.
  * [9] O. Celikbas; K.-i. Iima; A. Sadeghi; R. Takahashi, On the ideal case of a conjecture of Auslander and Reiten, Bull. Sci. Math. 142, (2018) 94–107.
  * [10] O. Celikbas; R. Takahashi, Auslander–Reiten conjecture and Auslander–Reiten duality, J. Algebra 382 (2013), 100–114.
  * [11] L. W. Christensen; H. Holm, Algebras that satisfy Auslander’s condition on vanishing of cohomology, Math. Z. 265 (2010), no. 1, 21–40.
  * [12] L. W. Christensen; H. Holm, Vanishing of cohomology over Cohen–Macaulay rings, Manuscripta Math. 139 (2012), no. 3-4, 535–544.
  * [13] H. Dao; M. Eghbali; J. Lyle, Hom and Ext, revisited, arXiv:1710.05123v4
  * [14] M. T. Dibaei; A. Sadeghi, Linkage of modules and the Serre conditions, J. Pure Appl. Algebra 219 (2015), no. 10, 4458–4478.
  * [15] S. Goto; R. Takahashi, On the Auslander–Reiten conjecture for Cohen–Macaulay local rings, Proc. Amer. Math. Soc. 145 (2017), no. 8, 3289–3296.
  * [16] M. Hoshino, Modules without self-extensions and Nakayama’s conjecture, Arch. Math. (Basel) 43, (1984), no. 6, 493–500.
  * [17] C. Huneke; G. J. Leuschke, On a conjecture of Auslander and Reiten, J. Algebra 275 (2004), no. 2, 781–790.
  * [18] C. Huneke; L. M. Şega; N. Vraciu, Vanishing of Ext and Tor over some Cohen–Macaulay local rings, Illinois J. Math. 48 (2004), no. 1, 295–317.
  * [19] O. Iyama, Higher-dimensional Auslander–Reiten theory on maximal orthogonal subcategories, Adv. Math. 210 (2007), no. 1, 22–50.
  * [20] O. Iyama; M. Wemyss, A new triangulated category for rational surface singularities, Illinois J. Math. 55 (2011), no. 1, 325–341 (2012).
  * [21] V. Maşek, Gorenstein dimension and torsion of modules over commutative Noetherian rings, Special issue in honor of Robin Hartshorne, Comm. Algebra 28 (2000), no. 12, 5783–5811.
  * [22] T. Nakayama, On algebras with complete homology, Abh. Math. Sem. Univ. Hamburg 22 (1958), 300–307.
  * [23] M. Ono; Y. Yoshino, An Auslander–Reiten principle in derived categories, J. Pure Appl. Algebra 221 (2017), no. 6, 1268–1278.
  * [24] H. Tachikawa, Quasi-Frobenius rings and generalizations, QF-3 and QF-1 rings, Notes by Claus Michael Ringel, Lecture Notes in Mathematics, Vol. 351, Springer-Verlag, Berlin-New York, 1973.
  * [25] R. Takahashi, Remarks on modules approximated by G-projective modules, J. Algebra 301 (2006), no. 2, 748–780.
  * [26] R. Takahashi, On G-regular local rings, Comm. Algebra 36 (2008), no. 12, 4472–4491.
  * [27] Y. Yoshino, Cohen–Macaulay modules over Cohen–Macaulay rings, London Mathematical Society Lecture Note Series, 146, Cambridge University Press, Cambridge, 1990.
  * [28] Y. Yoshino, Homotopy categories of unbounded complexes of projective modules, arXiv:1805.05705v1.


# Q-DETR: An Efficient Low-Bit Quantized Detection Transformer

Sheng Xu1†, Yanjing Li1†, Mingbao Lin3, Peng Gao4, Guodong Guo5, Jinhu Lü1,2,
Baochang Zhang1,2∗
1 Beihang University 2 Zhongguancun Laboratory 3 Tencent
4 Shanghai AI Laboratory 5 UNIUBI Research, Universal Ubiquitous Co.

###### Abstract

The recent detection transformer (DETR) has advanced object detection, but its
application on resource-constrained devices requires massive computation and
memory resources. Quantization stands out as a solution by representing the
network in low-bit parameters and operations. However, there is a significant
performance drop when performing low-bit quantized DETR (Q-DETR) with existing
quantization methods. We find that the bottlenecks of Q-DETR come from the
query information distortion through our empirical analyses. This paper
addresses this problem based on a distribution rectification distillation
(DRD). We formulate our DRD as a bi-level optimization problem, which can be
derived by generalizing the information bottleneck (IB) principle to the
learning of Q-DETR. At the inner level, we conduct a distribution alignment
for the queries to maximize the self-information entropy. At the upper level,
we introduce a new foreground-aware query matching scheme to effectively
transfer the teacher information to distillation-desired features to minimize
the conditional information entropy. Extensive experimental results show that
our method performs much better than prior arts. For example, the 4-bit Q-DETR
can theoretically accelerate DETR with ResNet-50 backbone by 6.6$\times$ and
achieve 39.4% AP, with only 2.6% performance gaps than its real-valued
counterpart on the COCO dataset 1.

††footnotetext: † Equal contribution.††footnotetext: ∗ Corresponding author:
<EMAIL_ADDRESS>1 Code: https://github.com/SteveTsui/Q-DETR

##  1 Introduction

Inspired by the success of natural language processing (NLP), object detection
with transformers (DETR) has been introduced to train an end-to-end detector
via a transformer encoder-decoder [4]. Unlike early works [31, 22] that often
employ convolutional neural networks (CNNs) and require post-processing
procedures, e.g., non-maximum suppression (NMS), and hand-designed sample
selection, DETR treats object detection as a direct set prediction problem.

Despite this attractiveness, DETR usually has a tremendous number of
parameters and float-pointing operations (FLOPs). For instance, there are
39.8M parameters taking up 159MB memory usage and 86G FLOPs in the DETR model
with ResNet-50 backbone [12] (DETR-R50). This leads to an unacceptable memory
and computation consumption during inference, and challenges deployments on
devices with limited supplies of resources.

Figure 1: The histogram of query values ${\bf q}$ (blue shadow) and
corresponding PDF curves (red curve) of Gaussian distribution [17], w.r.t the
cross attention of different decoder layers in (a) real-valued DETR-R50, and
(b) 4-bit quantized DETR-R50 (baseline). Gaussian distribution is generated
from the statistical mean and variance of the query values. The query in
quantized DETR-R50 bears information distortion compared with the real-valued
one. Experiments are performed on the VOC dataset [9].

Therefore, substantial efforts on network compression have been made towards
efficient online inference [7, 44, 43, 33]. Quantization is particularly
popular for deploying on AI chips by representing a network in low-bit
formats. Yet prior post-training quantization (PTQ) for DETR [26] derives
quantized parameters from pre-trained real-valued models, which often
restricts the model performance in a sub-optimized state due to the lack of
fine-tuning on the training data. In particular, the performance drastically
drops when quantized to ultra-low bits (4-bits or less). Alternatively,
quantization-aware training (QAT) [25, 42] performs quantization and fine-
tuning on the training dataset simultaneously, leading to trivial performance
degradation even with significantly lower bits. Though QAT methods have been
proven to be very effective in compressing CNNs [27, 8] for computer vision
tasks, an exploration of low-bit DETR remains untouched.

Figure 2: Spatial attention weight maps in the last decoder of (a) real-valued
DETR-R50, and (b) 4-bit quantized DETR-R50. The green rectangle denotes the
ground-truth bounding box. Following [29], the highlighted area denotes the
large attention weights in the selected four heads in compliance with bound
prediction. Compared to its real-valued counterpart that focuses on the
ground-truth bounds, quantized DETR-R50 deviates significantly.

In this paper, we first build a low-bit DETR baseline, a straightforward
solution based on common QAT techniques [2]. Through an empirical study of
this baseline, we observe significant performance drops on the VOC [9]
dataset. For example, a 4-bit quantized DETR-R50 using LSQ [8] only achieves
76.9% AP50, leaving a 6.4% performance gaps compared with the real-valued
DETR-R50. We find that the incompatibility of existing QAT methods mainly
stems from the unique attention mechanism in DETR, where the spatial
dependencies are first constructed between the object queries and encoded
features. Then the co-attended object queries are fed into box coordinates and
class labels by a feed-forward network. A simple application of existing QAT
methods on DETR leads to query information distortion, and therefore the
performance severely degrades. Fig. 1 exhibits an example of information
distortion in query features of 4-bit DETR-R50, where we can see significant
distribution variation of the query modules in quantized DETR and real-valued
version. The query information distortion causes the inaccurate focus of
spatial attention, which can be verified by following [29] to visualize the
spatial attention weight maps in 4-bit and real-valued DETR-R50 in Fig. 2. We
can see that the quantized DETR-R50 bear’s inaccurate object localization.
Therefore, a more generic method for DETR quantization is necessary.

To tackle the issue above, we propose an efficient low-bit quantized DETR
(Q-DETR) by rectifying the query information of the quantized DETR as that of
the real-valued counterpart. Fig. 3 provides an overview of our Q-DETR, which
is mainly accomplished by a distribution rectification knowledge distillation
method (DRD). We find ineffective knowledge transferring from the real-valued
teacher to the quantized student primarily because of the information gap and
distortion. Therefore, we formulate our DRD as a bi-level optimization
framework established on the information bottleneck principle (IB). Generally,
it includes an inner-level optimization to maximize the self-information
entropy of student queries and an upper-level optimization to minimize the
conditional information entropy between student and teacher queries. At the
inner level, we conduct a distribution alignment for the query guided by its
Gaussian-alike distribution, as shown in Fig. 1, leading to an explicit state
in compliance with its maximum information entropy in the forward propagation.
At the upper level, we introduce a new foreground-aware query matching that
filters out low-qualified student queries for exact one-to-one query matching
between student and teacher, providing valuable knowledge gradients to push
minimum conditional information entropy in the backward propagation.

This paper attempts to introduce a generic method for DETR quantization. The
significant contributions in this paper are outlined as follows: (1) We
develop the first QAT quantization framework for DETR, dubbed Q-DETR. (2) We
use a bi-level optimization distillation framework, abbreviated as DRD. (3) We
observe a significant performance increase compared to existing quantized
baselines.

Figure 3: Overview of the proposed Q-DETR framework. We introduce the
distribution rectification distillation method (DRD) to refine the performance
of Q-DETR. From left to right, we respectively show the detailed decoder
architecture of Q-DETR and the learning framework of Q-DETR. The Q-Backbone,
Q-Encoder, and Q-Decoder denote quantized architectures, respectively.

##  2 Related Work

Quantization. Quantized neural networks often possess low-bit (1$\sim$4-bit)
weights and activations to accelerate the model inference and save memory. For
example, DoReFa-Net [45] exploits convolution kernels with low bit-width
parameters and gradients to accelerate training and inference. TTQ [46] uses
two real-valued scaling coefficients to quantize the weights to ternary
values. Zhuang et al. [48] present a $2\\!\sim\\!4$-bit quantization scheme
using a two-stage approach to alternately quantize the weights and
activations, providing an optimal tradeoff among memory, efficiency, and
performance. In [14], the quantization intervals are parameterized, and
optimal values are obtained by directly minimizing the task loss of the
network. ZeroQ [3] supports uniform and mixed-precision quantization by
optimizing for a distilled dataset which is engineered to match the statistics
of the batch normalization across different network layers. Xie et al. [41]
introduced transfer learning into network quantization to obtain an accurate
low-precision model by utilizing Kullback-Leibler (KL) divergence. Fang et al.
[10] enabled accurate approximation for tensor values that have bell-shaped
distributions with long tails and found the entire range by minimizing the
quantization error. Li et al. [17] proposed an information rectification
module and distribution-guided distillation to push the bit-width in a
quantized vision transformer. At the same time, we address the quantization in
DETR from the IB principle. The architectural design has also drawn increasing
attention using extra shortcut [27], and parallel parameter-free shortcuts
[25] for example.

Detection Transformer. Driven by the success of transformers [37], several
researchers have also explored transformer frameworks for vision tasks. The
first DETR [4] work introduces the Transformer structure based on the
attention mechanism for object detection. But the main drawback of DETR lies
in the highly inefficient training process. The approachh of another work
modifies the multi-head attention mechanism (MHA). Deformable-DETR [47]
constructs a sparse and point-to-point MHA mechanism using a static point-wise
query sampling method around the reference points. SMCA-DETR [11] introduces a
Gaussian-distributed spatial function before formulating a spatially modulated
co-attention. DAB-DETR [21] re-defines the query of DETR as dynamic anchor
boxes and performs soft ROI pooling layer-by-layer in a cascade manner. DN-
DETR [16] introduces query denoising into query generation, reducing the
bipartite graph matching difficulty and leading to faster convergence. Another
set of arts improves DETR methods using additional learning constraints. For
example, UP-DETR [6] proposes a novel self-supervised loss to enhance the
convergence speed and the performance of DETR.

However, prior arts mainly focus on the training efficiency of DETR, few of
which have discussed the quantization of DETR. To this end, we first build a
quantized DETR baseline and then address the query information distortion
problem based on the IB principle. Finally, a new KD method based on a
foreground-aware query matching scheme is achieved to solve Q-DETR
effectively.

##  3 The Challenge of Quantizing DETR

###  3.1 Quantized DETR baseline

We first construct a baseline to study the low-bit DETR since no relevant work
has been previously proposed. To this end, we follow LSQ+ [2] to introduce a
general framework of asymmetric activation quantization and symmetric weight
quantization:

$\displaystyle\bm{x}_{q}=$
$\displaystyle\lfloor\operatorname{clip}\\{\frac{(\bm{x}-z)}{\alpha_{x}},Q_{n}^{x},Q_{p}^{x}\\}\rceil,{\bf
w}_{q}=\lfloor\operatorname{clip}\\{\frac{{\bf w}}{\alpha_{\bf w}},Q_{n}^{\bf
w},Q_{p}^{\bf w}\\}\rceil,$ (1) $\displaystyle
Q_{a}(x)=\alpha_{x}\circ\bm{x}_{q}+z,\;\;\;\;\;\;\;\;\;\;\;\;\,Q_{w}(x)=\alpha_{\bf
w}\circ{\bf w}_{q},$

where $\operatorname{clip}\\{y,r_{1},r_{2}\\}$ clips the input $y$ with value
bounds $r_{1}$ and $r_{2}$; the $\lfloor y\rceil$ rounds $y$ to its nearest
integer; the $\circ$ denotes the channel-wise multiplication. And
$Q_{n}^{x}=-2^{a-1},Q_{p}^{x}=2^{a-1}-1$, $Q_{n}^{\bf w}=-2^{b-1},Q_{p}^{\bf
w}=2^{b-1}-1$ are the discrete bounds for $a$-bit activations and $b$-bit
weights. $x$ generally denotes the activation in this paper, including the
input feature map of convolution and fully-connected layers and input of
multi-head attention modules. Based on this, we first give the quantized
fully-connected layer as:

$\small\operatorname{Q-FC}(\bm{x})=Q_{a}({\bm{x}})\cdot Q_{w}({\bf
w})=\alpha_{x}\alpha_{\bf w}\circ(\bm{x}_{q}\odot{\bf
w}_{q}+z/\alpha_{x}\circ{\bf w}_{q}),$ (2)

where $\cdot$ denotes the matrix multiplication and $\odot$ denotes the matrix
multiplication with efficient bit-wise operations. The straight-through
estimator (STE) [1] is used to retain the derivation of the gradient in
backward propagation.

In DETR [4], the visual features generated by the backbone are augmented with
position embedding and fed into the transformer encoder. Given an encoder
output ${\bf E}$, DETR performs co-attention between object queries ${\bf O}$
and the visual features ${\bf E}$, which are formulated as:

$\displaystyle{\bf q}$ $\displaystyle=\operatorname{Q-FC}({\bf O}),\;\;{\bf
k},{\bf v}=\operatorname{Q-FC}({\bf E})$ (3) $\displaystyle{\bf A}_{i}$
$\displaystyle=\operatorname{softmax}(Q_{a}({\bf q})_{i}\cdot Q_{a}({\bf
k})_{i}^{\top}/\sqrt{d}),$ $\displaystyle{\bf D}_{i}$
$\displaystyle=Q_{a}({\bf A})_{i}\cdot Q_{a}({\bf v})_{i},$

where ${\bf D}$ is the multi-head co-attention module, i.e., the co-attended
feature for the object query. The $d$ denotes the feature dimension in each
head. More FC layers transform the decoder’s output features of each object
query for the final output. Given box and class predictions, the Hungarian
algorithm [4] is applied between predictions and ground-truth box annotations
to identify the learning targets of each object query.

Figure 4: Performance of 3/4-bit quantized DETR-R50 on VOC with different
quantized modules.

###  3.2 Challenge Analysis

Intuitively, the performance of the quantized DETR baseline largely depends on
the information representation capability mainly reflected by the information
in the multi-head attention module. Unfortunately, such information is
severely degraded by the quantized weights and inputs in the forward pass.
Also, the rounded and discrete quantization significantly affect the
optimization during backpropagation.

We conduct the quantitively ablative experiments by progressively replacing
each module of the real-valued DETR baseline with a quantized one and compare
the average precision (AP) drop on the VOC dataset [9] as shown in Fig. 4. We
find that quantizing the MHA decoder module to low bits, i.e., (1)+(2)+(3),
brings the most significant accuracy drops of accuracy among all parts of the
DETR methods, up to 2.1% in the 3-bit DETR-R50. At the same time, other parts
of DETR show comparative robustness to the quantization function.
Consequently, the critical problem of improving the quantized DETR methods is
restoring the information in MHA modules after quantization. Other qualitative
results in Fig. 1 and Fig. 2 also indicate that the degraded information
representation is the main obstacle to a better quantized DETR.

##  4 The Proposed Q-DETR

###  4.1 Information Bottleneck of Q-DETR

To address the information distortion of the quantized DETR, we aim to improve
the representation capacity of the quantized networks in a knowledge
distillation framework. Generally, we utilize a real-valued DETR as a teacher
and a quantized DETR as a student, which are distinguished with superscripts
$\mathcal{T}$ and $\mathcal{S}$, respectively.

Our Q-DETR pursues the best tradeoff between performance and compression,
which is precisely the goal of the information bottleneck (IB) method through
quantifying the mutual information that the intermediate layer contains about
the input (less is better) and the desired output (more is better) [35, 36].
In our case, the intermediate layer comes from the student, while the desired
output includes the ground-truth labels as well as the queries of the teacher
for distillation. Thus, the objective target of our Q-DETR is:

$\displaystyle\mathop{\min}_{\theta^{\mathcal{S}}}I(X;{\bf
E}^{\mathcal{S}})-\beta I({\bf E}^{\mathcal{S}},{\bf
q}^{\mathcal{S}};\bm{y}^{GT})-\gamma I({\bf q}^{\mathcal{S}};{\bf
q}^{\mathcal{T}}),$ (4)

where ${\bf q}^{\mathcal{T}}$ and ${\bf q}^{\mathcal{S}}$ represent the
queries in the teacher and student DETR methods as predefined in Eq. (3);
$\beta$ and $\gamma$ are the Lagrange multipliers [35]; $\theta^{\mathcal{S}}$
is the parameters of the student; and $I(\cdot)$ returns the mutual
information of two input variables. The first item $I(X;{\bf
E}^{\mathcal{S}})$ minimizes information between input and visual features
${\bf E}^{\mathcal{S}}$ to extract task-oriented hints [40]. The second item
$I({\bf E}^{\mathcal{S}},{\bf q}^{\mathcal{S}};\bm{y}^{GT})$ maximizes
information between extracted visual features and ground-truth labels for
better object detection. These two items can be easily accomplished by common
network training and detection loss constraints, such as proposal
classification and coordinate regression.

The core issue of this paper is to solve the third item $I({\bf
q}^{\mathcal{S}};{\bf q}^{\mathcal{T}})$, which attempts to address the
information distortion in student query via introducing teacher query as a
priori knowledge. To accomplish our goal, we first expand the third item and
reformulate it as:

$\displaystyle I({\bf q}^{\mathcal{S}};{\bf q}^{\mathcal{T}})=H({\bf
q}^{\mathcal{S}})-H({\bf q}^{\mathcal{S}}|{\bf q}^{\mathcal{T}}),$ (5)

where $H({\bf q}^{\mathcal{S}})$ returns the self information entropy expected
to be maximized while $H({\bf q}^{\mathcal{S}}|{\bf q}^{\mathcal{T}})$ is the
conditional entropy expected to be minimized. It is challenging to optimize
the above maximum & minimum items simultaneously. Instead, we make a
compromise to reformulate Eq. (5) as a bi-level issue [20, 5] that alternately
optimizes the two items, which is explicitly defined as:

$\displaystyle\mathop{\min}_{\theta}H({\bf q}^{\mathcal{S}^{*}}|{\bf
q}^{\mathcal{T}}),$ (6) $\displaystyle\operatorname{s.t.}$ $\displaystyle{\bf
q}^{\mathcal{S}^{*}}=\mathop{\arg\max}_{{\bf q}^{\mathcal{S}}}H({\bf
q}^{\mathcal{S}}).$

Such an objective involves two sub-problems, including an inner-level
optimization to derive the current optimal query ${\bf q}^{\mathcal{S}^{*}}$
and an upper-level optimization to conduct knowledge transfer from the teacher
to the student. Below, we show that the two sub-problems can be solved in the
forward & backward network propagation’s.

###  4.2 Distribution Rectification Distillation

Inner-level optimization. We first detail the maximization of self-information
entropy. According to the definition of self information entropy, $H({\bf
q}^{\mathcal{S}})$ can be implicitly expanded as:

$H({\bf q}^{\mathcal{S}})=-\int_{{\bf q}^{\mathcal{S}}_{i}\in{\bf
q}^{\mathcal{S}}}p({\bf q}^{\mathcal{S}}_{i}){\operatorname{log}p({\bf
q}^{\mathcal{S}}_{i})}.$ (7)

However, an explicit form of $H({\bf q}^{\mathcal{S}})$ can only be
parameterized with a regular distribution $p({\bf q}^{\mathcal{S}}_{i})$.
Luckily, the statistical results in Fig. 1 shows that the query distribution
tends to follow a Gaussian distribution, which is also observed in [17]. This
enables us to solve the inner-level optimization in a distribution alignment
fashion. To this end, we first calculate the mean $\mu({\bf q}^{\mathcal{S}})$
and variance $\sigma({\bf q}^{\mathcal{S}})$ of query ${\bf q}^{\mathcal{S}}$
whose distribution is then modeled as ${\bf
q}^{\mathcal{S}}\sim\mathcal{N}(\mu({\bf q}^{\mathcal{S}}),\sigma({\bf
q}^{\mathcal{S}}))$. Then, the self-information entropy of the student query
can be proceeded as:

$\displaystyle H({\bf q}^{\mathcal{S}})$
$\displaystyle=-\mathbb{E}[\operatorname{log}\mathcal{N}(\mu({\bf
q}^{\mathcal{S}}),\sigma({\bf q}^{\mathcal{S}}))]$ (8)
$\displaystyle=-\mathbb{E}[\operatorname{log}[{(2\pi{\sigma({{\bf
q}^{\mathcal{S}}})}^{2})}^{\frac{1}{2}}\operatorname{exp}(-\frac{{({\bf
q}^{\mathcal{S}}_{i}-\mu({\bf q}^{\mathcal{S}}))}^{2}}{2{\sigma({{\bf
q}^{\mathcal{S}}})^{2}}})]]$
$\displaystyle=\frac{1}{2}\operatorname{log}2\pi{\sigma({{\bf
q}^{\mathcal{S}}})}^{2}.$

The above objective reaches its maximum of $H({\bf
q}^{\mathcal{S}^{*}})=(1/2)\log 2\pi e[\sigma({{\bf
q}^{\mathcal{S}})}^{2}+\epsilon_{{\bf q}^{\mathcal{S}}}]$ when ${\bf
q}^{\mathcal{S}^{*}}=[{\bf q}^{\mathcal{S}}-\mu({\bf
q}^{\mathcal{S}})]/[{\sqrt{\sigma{({\bf q}^{\mathcal{S}})}^{2}+\epsilon_{{\bf
q}^{\mathcal{S}}}}}]$ where $\epsilon_{{\bf q}^{\mathcal{S}}}=1e^{-5}$ is a
small constant added to prevent a zero denominator. In practice, the mean and
variance might be inaccurate due to query data bias. To solve this we use the
concepts in batch normalization (BN) [34, 13] where a learnable shifting
parameter $\beta_{{{\bf q}}^{\mathcal{S}}}$ is added to move the mean value. A
learnable scaling parameter $\gamma_{{\bf q}^{\mathcal{S}}}$ is multiplied to
move the query to the adaptive position. In this situation, we rectify the
information entropy of the query in the student as follows:

$\displaystyle{\bf q}^{\mathcal{S}^{*}}$ $\displaystyle=\frac{{\bf
q}^{\mathcal{S}}-\mu({\bf q}^{\mathcal{S}})}{{\sqrt{\sigma{({\bf
q}^{\mathcal{S}})}^{2}+\epsilon_{{\bf q}^{\mathcal{S}}}}}}\gamma_{{\bf
q}^{\mathcal{S}}}+\beta_{{\bf q}^{\mathcal{S}}},$ (9)

in which case the maximum self-information entropy of student query becomes
$H({\bf q}^{\mathcal{S}^{*}})=(1/2)\log 2\pi e[(\sigma^{2}_{{\bf
q}^{\mathcal{S}}}+\epsilon_{{\bf q}^{\mathcal{S}}})/\gamma^{2}_{{\bf
q}^{\mathcal{S}}}]$. Therefore, in the forward propagation, we can obtain the
current optimal query ${\bf q}^{\mathcal{S}^{*}}$ via Eq. (9), after which,
the upper-level optimization is further executed as detailed in the following
contents.

Upper-level optimization. We continue minimizing the conditional information
entropy between the student and the teacher. Following DETR [4], we denote the
ground-truth labels by
$\bm{y}^{GT}=\\{c^{GT}_{i},b^{GT}_{i}\\}_{i=1}^{N_{gt}}$ as a set of ground-
truth objects where $N_{gt}$ is the number of foregrounds, $c_{i}^{GT}$ and
$b_{i}^{GT}$ respectively represent the class and coordinate (bounding box)
for the $i$-th object. In DETR, each query is associated with an object.
Therefore, we can obtain $N$ objects for teacher and student as well, denoted
as
$\bm{y}^{\mathcal{S}}=\\{c^{\mathcal{S}}_{j},b^{\mathcal{S}}_{j}\\}_{j=1}^{N}$
and
$\bm{y}^{\mathcal{T}}=\\{c^{\mathcal{T}}_{j},b^{\mathcal{T}}_{j}\\}_{j=1}^{N}$.

The minimization of the conditional information entropy requires the student
and teacher objects to be in a one-to-one matching. However, it is problematic
for DETR due primarily to the sparsity of prediction results and the
instability of the query’s predictions [16]. We propose a foreground-aware
query matching to rectify “well-matched” queries to solve this. Concretely, we
match the ground-truth bounding boxes with this student to find the maximum
coincidence as:

$\displaystyle G_{i}=\mathop{\max}_{1\leq j\leq
N}\operatorname{GIoU}(b^{GT}_{i},b^{\mathcal{S}}_{j}),$ (10)

where $\operatorname{GIoU}(\cdot)$ is the generalized intersection over union
function [32]. Each $G_{i}$ reflects the “closeness” of student proposals to
the $i$-th ground-truth object. Then, we retain highly qualified student
proposals around at least one ground truth to benefit object recognition [38]
as:

$\displaystyle\begin{split}\small
b_{j}^{\mathcal{S}}=\left\\{\begin{array}[]{ll}{b}_{j}^{\mathcal{S}},\\!\\!\\!\\!&{\operatorname{GIoU}}(b^{GT}_{i},b^{\mathcal{S}}_{j})>\tau
G_{i},\,\,\forall\;i\\\
\varnothing,&\text{otherwise},\end{array}\right.\end{split}$ (11)

where $\tau$ is a threshold controlling the proportion of distilled queries.
After removing object-empty ($\varnothing$) queries in
$\tilde{\bm{q}}^{\mathcal{S}}$, we form a distillation-desired query set of
students denoted as $\tilde{{\bm{q}}}^{\mathcal{S}}$ associated with its
object set
$\tilde{{\bm{y}}}^{\mathcal{S}}=\\{\tilde{c}^{\mathcal{S}}_{j},\tilde{b}^{\mathcal{S}}_{j}\\}_{j=1}^{\tilde{N}}$.
Correspondingly, we can obtain a teacher query set
$\tilde{{\bm{y}}}^{\mathcal{T}}=\\{\tilde{c}^{\mathcal{T}}_{j},\tilde{b}^{\mathcal{T}}_{j}\\}_{j=1}^{\tilde{N}}$.
For the $j$-th student query, its corresponding teacher query is matched as:

$\tilde{c}^{\mathcal{T}}_{j},\tilde{b}^{\mathcal{T}}_{j}=\mathop{\arg\max}_{\tilde{c}^{\mathcal{T}}_{k},\tilde{b}^{\mathcal{T}}_{k}}\sum^{N}_{k=1}\mu_{1}\operatorname{GIoU}(\tilde{b}^{\mathcal{S}}_{j},b^{\mathcal{T}}_{k})-\mu_{2}\|\tilde{b}^{\mathcal{S}}_{j}-b^{\mathcal{T}}_{k}\|_{1},$
(12)

where $\mu_{1}=2$ and $\mu_{2}=5$ control the matching function, values of
which is to follow [4].

Finally, the upper-level optimization after rectification in Eq. (6) becomes:

$\displaystyle\mathop{\min}_{\theta}H(\tilde{{\bf
q}}^{\mathcal{S}^{*}}|\tilde{{\bf q}}^{\mathcal{T}}).$ (13)

Optimizing Eq. (13) is challenging. Alternatively, we minimize the norm
distance between $\tilde{\bf q}^{\mathcal{S}^{*}}$ and $\tilde{{\bf
q}}^{\mathcal{T}}$, optima of which, _i.e._ , $\tilde{\bf
q}^{\mathcal{S}^{*}}=\tilde{\bf q}^{\mathcal{T}}$, is exactly the same with
that in Eq. (13). Thus, the final loss for our distribution rectification
distillation loss becomes:

$\mathcal{L}_{DRD}(\tilde{{\bf q}}^{\mathcal{S}^{*}},\tilde{{\bf
q}}^{\mathcal{T}})=\mathbb{E}[\|\tilde{\bf D}^{\mathcal{S}^{*}}-\tilde{\bf
D}^{\mathcal{T}}\|_{2}],$ (14)

where we use the Euclidean distance of co-attented feature $\tilde{\bf D}$
(see Eq. 3) containing the information query $\tilde{\bf q}$ for optimization.

In backward propagation, the gradient updating drives the student queries
toward their teacher hints. Therefore we accomplish our distillation. The
overall training losses for our Q-DETR model are:

$\small\mathcal{L}=\mathcal{L}_{GT}(\bm{y}^{GT},\bm{y}^{\mathcal{S}})+\lambda\mathcal{L}_{DRD}(\tilde{{\bf
q}}^{\mathcal{S}^{*}},\tilde{{\bf q}}^{\mathcal{T}}),$ (15)

where $L_{GT}$ is the common detection loss for missions such as proposal
classification and coordinate regression [4], and $\lambda$ is a tradeoff
hyper-parameter.

##  5 Experiments

In this section, we evaluate the performance of the proposed Q-DETR mode using
popular DETR [4] and SMCA-DETR [11] models. To the best of our knowledge,
there is no publicly available source code on quantization-aware training of
DETR methods at this point, so we implement the baseline and LSQ [8] methods
ourselves.

###  5.1 Datasets and Implementation Details

Datasets. We first conduct the ablative study and hyper-parameter selection on
the PASCAL VOC dataset [9], which contains natural images from 20 different
classes. We use the VOC trainval2012, and VOC trainval2007 sets to train our
model, which contains approximately 16k images, and the VOC test2007 set to
evaluate our Q-DETR, which contains 4952 images. We report COCO-style metrics
for the VOC dataset: AP, AP50 (default VOC metric), and AP75. We further
conduct the experiments on the COCO 2017 [18] object detection tracking.
Specifically, we train the models on COCO train2017 and evaluate the models on
COCO val2017. We list the average precision (AP) for IoUs$\in[0.5:0.05:0.95]$,
designated as AP, using COCO’s standard evaluation metric. For further
analyzing our method, we also list AP50, AP75, APs, APm, and APl.

(a) Effect of $\tau$ and $\lambda$.

(b) Mutual information curves.

Figure 5: (a) We select $\tau$ and $\lambda$ using 4-bit Q-DETR-R50 on VOC.
(b) The mutual information curves of $I(X;{\bf E})$ and $I(\bm{y}^{GT};{\bf
E},{\bf q})$ (Eq. 4) on the information plane. The red curves represent the
teacher model (DETR-R101). The orange, green, red, and purple lines represent
the 4-bit baseline, 4-bit baseline + DA, 4-bit baseline + FQM, and 4-bit
baseline + DA + FQM (4-bit Q-DETR).

Implementation Details. Our Q-DETR is trained with the DETR [4] and SMCA-DETR
[11] framework. We select the ResNet-50 [12] and modify it with Pre-Activation
structures and RPReLU [25] function following [23]. PyTorch [30] is used for
implementing Q-DETR. We run the experiments on 8 NVIDIA Tesla A100 GPUs with
$80$ GB memory. We use ImageNet ILSVRC12 [15] to pre-train the backbone of a
quantized student. The training protocol is the same as the employed
frameworks [4, 11]. Specifically, we use a batch size of 16. AdamW [28] is
used to optimize the Q-DETR, with the initial learning rate of $1e^{-4}$. We
train for 300/500 epochs for the Q-DETR on VOC/COCO dataset, and the learning
rate is multiplied by 0.1 at the 200/400-th epoch, respectively. Following the
SMCA-DETR, we train the Q-SMCA-DETR for 50 epochs, and the learning rate is
multiplied by 0.1 at the 40-th epoch on both the VOC and COCO datasets. We
utilize a multi-distillation strategy, where we save the encoder and decoder
network as real-valued at the first stage. Then we train the fully quantized
DETR at the second stage, where we load the weight from the checkpoint of
first stage. We select real-valued DETR-R101 (84.5% AP50 on VOC and 43.5% AP
on COCO) and SMCA-DETR-R101 (85.3% AP50 on VOC and 44.4% AP on COCO) as
teacher network.

###  5.2 Ablation Study

Hyper-parameter selection. As mentioned above, we select hyper-parameters
$\tau$ and $\lambda$ in this part using the 4-bit Q-DETR model. We show the
model performance (AP50) with different setups of hyper-parameters
$\\{\tau,\lambda\\}$ in Fig. 5 (a), where we conduct ablative experiments on
the baseline + DA (AP50=78.8%). As can be seen, the performances increase
first and then decrease with the increase of $\tau$ from left to right. Since
$\tau$ controls the proportion of selected distillation-desired queries, we
show that the full-imitation ($\tau=0$) performs worse than the vanilla
baseline with no distillation ($\tau=1$), showing query selection is
necessary. The figure also shows that the performances increase first and then
decrease with the increase of $\tau$ from left to right. The Q-DETR obtains
better performances with $\tau$ set as 0.5 and 0.6. With the varying value of
$\lambda$, we find $\\{\lambda,\tau\\}$ = {2.5, 0.6} boost the performance of
Q-DETR most, achieving 82.7% AP on VOC test2007. Based on the ablative study
above, we set hyper-parameters $\tau$ and $\lambda$ as 0.6 and 2.5 for the
experiments in this paper.

Effectiveness of components. We show quantitative improvements of components
in Q-DETR in Tab. 1. As shown in Tab. 1, the quantized DETR baseline suffers a
severe performance drop on AP50 (13.6%, 6.5%, and 5.3% with 2/3/4-bit,
respectively). DA and FQM improve the performance when used alone, and the two
techniques further boost the performance considerably when combined. For
example, the DA improves the 2-bit baseline by 1.9%, and the FQM achieves a
5.2% performance improvement. While combining the DA and FQM, the performance
improvement achieves 6.7%.

Information analysis. We further show the information plane following [39] in
Fig. 5(b). We adopt the test AP50 to quantify $I(\bm{y}^{GT};{\bf E},{\bf
q})$. We employ a reconstruction decoder to decode the encoded feature ${\bf
E}$ to reconstruct the input and quantify $I(X;{\bf E})$ using the $\ell_{1}$
loss. As shown in Fig. 5(b), the curve of the larger teacher DETR-R101 is
usually on the right of the curve of small student models, which indicates a
greater ability of information representation. Likewise, the purple line
(Q-DETR-R50) is usually on the right of the three left curves, showing the
information representation improvements with the proposed methods.

Table 1: Evaluating the components of Q-DETR-R50 on the VOC dataset. #Bits (W-A-Attention) denotes the bit-width of weights, activations, and attention activations. DA denotes the distribution alignment module. FQM denotes foreground-aware query matching. Method | #Bits | AP50 | #Bits | AP50 | #Bits | AP50
---|---|---|---|---|---|---
Real-valued | 32-32-32 | 83.3 | - | - | - | -
Baseline | 4-4-8 | 78.0 | 3-3-8 | 76.8 | 2-2-8 | 69.7
+DA | 4-4-8 | 78.8 | 3-3-8 | 78.0 | 2-2-8 | 71.6
+FQM | 4-4-8 | 81.5 | 3-3-8 | 80.9 | 2-2-8 | 74.9
| +DA+FQM
---
(Q-DETR)
4-4-8 | 82.7 | 3-3-8 | 82.1 | 2-2-8 | 76.4
Table 2: We report AP, AP50, and AP75 ($\%$) with state-of-the-art quantization methods on DETR and SMCA-DETR using VOC test2007. #Bits (W-A-Attention) denotes the bit-width of weights, activations, and attention activations. Model | Method | #Bits | AP | AP50 | AP75
---|---|---|---|---|---
DETR-R50 | Real-valued | 32-32-32 | 59.5 | 83.3 | 64.7
Percentile | 8-8-8 | 54.7 | 79.2 | 60.1
VT-PTQ | 57.6 | 82.3 | 63.1
LSQ | 4-4-8 | 49.7 | 76.9 | 53.0
Baseline | 51.3 | 78.0 | 54.1
Q-DETR | 57.1 | 82.7 | 61.5
LSQ | 3-3-8 | 47.0 | 75.3 | 49.1
Baseline | 49.2 | 76.8 | 51.8
Q-DETR | 56.8 | 82.1 | 61.2
LSQ | 2-2-8 | 42.6 | 68.2 | 44.8
Baseline | 44.0 | 69.7 | 45.8
Q-DETR | 50.7 | 76.4 | 54.1
SMCA-DETR -R50 | Real-valued | 32-32-32 | 56.7 | 83.7 | 62.0
Percentile | 8-8-8 | 54.7 | 79.2 | 60.1
VT-PTQ | 55.9 | 83.0 | 61.3
LSQ | 4-4-8 | 49.6 | 78.6 | 53.4
Baseline | 50.7 | 79.5 | 55.4
Q-DETR | 56.2 | 83.3 | 61.6
LSQ | 3-3-8 | 47.7 | 76.5 | 51.7
Baseline | 49.9 | 77.5 | 53.6
Q-DETR | 54.3 | 82.6 | 59.5
LSQ | 2-2-8 | 42.3 | 69.7 | 44.8
Baseline | 43.9 | 70.4 | 46.1
Q-DETR | 50.2 | 76.7 | 52.6

###  5.3 Results on PASCAL VOC

We first compare our method with the 2/3/4-bit baseline and LSQ [8] based on
the same frameworks for object detection task with the VOC dataset. We also
report the detection performance of the 8-bit post-training quantization
networks, such as percentile [19], VT-PTQ [26]. We use the input resolution
following [4], i.e. 1333$\times$800\. We mainly discuss the AP50 (default VOC
metric) in the following.

We evaluate the proposed Q-DETR on DETR-R50 models in Tab. 2. For the DETR-R50
model, compared with the 8-bit PTQ method, our 4-bit Q-DETR achieves a much
larger compression ratio than 8-bit VT-PTQ, but with a bit of performance
improvement (82.7% vs. 82.3%). Also, the proposed method boosts the
performance of 2/3/4-bit baseline by 6.7%, 5.3%, and 4.7% with the same
architecture and bit-width, which significantly validates the effectiveness of
our method.

Table 3: Comparison with state-of-the-art quantization methods using DETR and SMCA-DETR on COCO val2017. #Bits (W-A-Attention) denotes bit-width of weights, activations, and attention activations. Model | Method | #Bits | Size(MB) | OPs(G) | AP | AP50 | AP75 | APs | APm | APl
---|---|---|---|---|---|---|---|---|---|---
DETR-R50 | Real-valued | 32-32-32 | 159.32 | 85.51 | 42.0 | 62.4 | 44.2 | 20.5 | 45.8 | 61.1
Percentile | 8-8-8 | 39.83 | 23.01 | 38.6 | - | - | - | - | -
VT-PTQ | 41.2 | - | - | - | - | -
LSQ | 4-4-8 | 19.92 | 13.02 | 33.3 | 53.7 | 33.9 | 12.8 | 37.0 | 51.6
Baseline | 34.1 | 55.3 | 35.4 | 14.3 | 38.0 | 53.8
Q-DETR | 39.4 | 60.2 | 41.4 | 17.7 | 43.4 | 59.9
LSQ | 3-3-8 | 15.03 | 7.61 | 31.0 | 52.3 | 32.1 | 11.3 | 33.9 | 48.5
Baseline | 32.3 | 52.2 | 32.9 | 12.3 | 35.4 | 50.3
Q-DETR | 36.1 | 55.9 | 37.5 | 14.6 | 39.4 | 55.2
LSQ | 2-2-8 | 10.03 | 5.32 | 24.7 | 44.6 | 26.5 | 6.3 | 25.3 | 42.7
Baseline | 26.6 | 46.6 | 26.5 | 8.4 | 28.2 | 44.4
Q-DETR | 31.4 | 51.3 | 31.6 | 11.6 | 34.3 | 49.6
SMCA-DETR-R50 | Real-valued | 32-32-32 | 164.75 | 86.65 | 41.0 | 62.2 | 43.6 | 21.9 | 44.3 | 59.1
Percentile | 8-8-8 | 41.19 | 23.66 | 37.5 | 58.5 | 40.1 | 17.6 | 39.1 | 55.9
VT-PTQ | 40.2 | 61.0 | 42.6 | 20.3 | 42.9 | 57.7
LSQ | 4-4-8 | 20.59 | 13.48 | 33.9 | 55.0 | 35.0 | 13.2 | 37.2 | 51.4
Baseline | 35.0 | 56.4 | 36.4 | 15.6 | 38.3 | 52.5
Q-DETR | 38.3 | 59.7 | 39.8 | 17.7 | 41.7 | 56.8
LSQ | 3-3-8 | 15.68 | 8.05 | 30.1 | 52.6 | 31.4 | 11.9 | 33.4 | 46.6
Baseline | 31.8 | 53.7 | 32.6 | 12.6 | 35.2 | 49.8
Q-DETR | 35.0 | 56.3 | 36.9 | 15.0 | 39.0 | 53.1
LSQ | 2-2-8 | 10.84 | 4.54 | 23.9 | 42.2 | 24.2 | 9.4 | 26.2 | 37.5
Baseline | 25.4 | 44.3 | 25.2 | 8.4 | 27.2 | 40.3
Q-DETR | 30.5 | 51.8 | 31.8 | 12.0 | 33.2 | 48.0

Besides, our method generates convincing results on SMCA-DETR. As shown in
Tab. 2, the performance of the proposed Q-DETR with SMCA-DETR-R50 outperforms
the 2/3/4-bit Baseline method by 6.3% , 5.1% and 3.8% on AP50, a large margin.
Compared with 8-bit post-training quantization methods, our method achieves a
significantly higher compression rate and comparable performance.

###  5.4 Results on COCO

We further show comparison on the large-scale COCO [18] dataset. We compare
our method with the 2/3/4-bit baseline and LSQ [8] based on the same
frameworks. We also report the detection performance of the 8-bit post-
training quantization networks, such as percentile [19] , VT-PTQ [26]. The AP
with different IoU thresholds, and AP of objects with varying scales are all
reported in Tab. 3.

Tab. 3 lists the comparison of several quantization approaches and detection
frameworks in computing complexity, storage cost. Our Q-DETR significantly
accelerates computation and reduces storage requirements for various
detectors. We follow [40] to calculate memory usage, by adding 32$\times$ the
number of real-valued weights and $a\times$ the number of quantized weights in
the $a$-bit networks. The number of operations (OPs) is calculated in the same
way as [40]. Current CPUs can handle both bit-wise XNOR and bit-count
operations in parallel. The respective number of FLOPs adds
$\\{\frac{1}{32},\frac{1}{16},\frac{1}{8}\\}$ of the number of
$\\{$2,3,4$\\}$-bit multiplications equals the OPs following [24].

We summarize the experimental results on COCO val2017 of Q-DETR-R50 from lines
2 to 17 in Tab. 3. For the DETR-R50 model, compared with the 8-bit PTQ method,
our 4-bit Q-DETR achieves a much larger acceleration than the 8-bit VT-PTQ but
with an acceptable performance gap. Also, the proposed method boosts the
performance of 2/3/4-bit baseline by 4.8%, 3.8% and 5.1% AP with the same
architecture and bit-width, which is significant on the large-scale COCO
dataset. Compared with the real-valued counterparts, the proposed 2/3/4-bit
Q-DETR achieves computation acceleration and storage savings by
16.07$\times$/11.23$\times$/6.57$\times$ and
15.88$\times$/10.60$\times$/7.99$\times$. The above results are of great
significance in the real-time inference of object detection. All of the
improvements have impacts on object detection.

For the SMCA-DETR-R50 model, we observe similar performance improvements and
compression ratios. For example, the 4-bit Q-SMCA-DETR-R50 theoretically
accelerates 6.42$\times$ with only a 2.7% performance gap compared with the
real-valued counterpart, which is significant for real-time DETR methods.

##  6 Conclusion

This paper introduces a novel method for training quantized DETR (Q-DETR) with
knowledge distillation to rectify the query distribution. Q-DETR generalizes
the information bottleneck (IB) principle and leads a bi-level distribution
rectification distillation. We effectively employ a distribution alignment
module to solve inner-level and a foreground-aware query matching scheme to
solve upper level. As a result, Q-DETR significantly boosts performance of
low-bit DETR. Extensive experiments show that Q-DETR surpasses state-of-the-
arts in DETR quantization.

##  7 Acknowledgements

This work was supported by National Natural Science Foundation of China under
Grant 62141604, 62076016, Beijing Natural Science Foundation L223024.

## References

  * [1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [2] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In Proc. of CVPR Workshops, pages 696–697, 2020.
  * [3] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Zeroq: A novel zero shot quantization framework.  In Proc. of CVPR, pages 13169–13178, 2020.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proc. of ECCV, pages 213–229, 2020.
  * [5] Benoît Colson, Patrice Marcotte, and Gilles Savard.  An overview of bilevel optimization.  Annals of operations research, 153(1):235–256, 2007.
  * [6] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  Up-detr: Unsupervised pre-training for object detection with transformers.  In Proc. of CVPR, pages 1601–1610, 2021.
  * [7] Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas.  Predicting parameters in deep learning.  In Proc. of NeurIPS, pages 2148–2156, 2013.
  * [8] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha.  Learned step size quantization.  arXiv preprint arXiv:1902.08153, 2019.
  * [9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338, 2010.
  * [10] Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun.  Post-training piecewise linear quantization for deep neural networks.  In Proc. of ECCV, pages 69–86, 2020.
  * [11] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of detr with spatially modulated co-attention.  In Proc. of ICCV, pages 3621–3630, 2021.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. of CVPR, pages 770–778, 2016.
  * [13] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proc. of ICML, pages 448–456, 2015.
  * [14] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In Proc. of CVPR, pages 4350–4359, 2019.
  * [15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Proc. of NeurIPS, pages 1097–1105, 2012.
  * [16] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang.  Dn-detr: Accelerate detr training by introducing query denoising.  In Proc. of CVPR, pages 13619–13627, 2022.
  * [17] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo.  Q-vit: Accurate and fully quantized low-bit vision transformer.  In Proc. of NeurIPS, pages 1–12, 2022.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proc. of ECCV, pages 740–755, 2014.
  * [19] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou.  Fq-vit: Post-training quantization for fully quantized vision transformer.  In Proc. of IJCAI, pages 1173–1179, 2021.
  * [20] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin.  Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [21] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  pages 1–19, 2022.
  * [22] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In Proc. of ECCV, pages 21–37, 2016.
  * [23] Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen.  Nonuniform-to-uniform quantization: Towards accurate quantization via generalized straight-through estimation.  In Proc. of CVPR, pages 4942–4952, 2022.
  * [24] Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang-Ting Cheng.  Bi-real net: Binarizing deep network towards real-network performance.  International Journal of Computer Vision, 128(1):202–219, 2020\.
  * [25] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng.  Reactnet: Towards precise binary neural network with generalized activation functions.  In Proc. of ECCV, pages 143–159, 2020.
  * [26] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  Proc. of NeurIPS, pages 1–12, 2021.
  * [27] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng.  Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm.  In Proc. of ECCV, pages 722–737, 2018.
  * [28] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In Proc. of ICLR, pages 1–18, 2017.
  * [29] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In Proc. of ICCV, pages 3651–3660, 2021.
  * [30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In Proc. of NeurIPS Workshops, pages 1–4, 2017.
  * [31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2016.
  * [32] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proc. of CVPR, pages 658–666, 2019.
  * [33] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In Proc. of ICLR, pages 1–13, 2015.
  * [34] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In Proc. of NeurIPS, pages 1–11, 2018.
  * [35] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  arXiv:1703.00810, 2017.
  * [36] Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  arXiv preprint physics/0004057, 2000.
  * [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. of NeurIPS, pages 1–11, 2017.
  * [38] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng.  Distilling object detectors with fine-grained feature imitation.  In Proc. of CVPR, pages 4933–4942, 2019.
  * [39] Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang.  Revisiting locally supervised learning: an alternative to end-to-end training.  In Proc. of ICLR, pages 1–21, 2021.
  * [40] Ziwei Wang, Ziyi Wu, Jiwen Lu, and Jie Zhou.  Bidet: An efficient binarized object detector.  In Proc. of CVPR, pages 2049–2058, 2020.
  * [41] Zheng Xie, Zhiquan Wen, Jing Liu, Zhiqiang Liu, Xixian Wu, and Mingkui Tan.  Deep transferring quantization.  In Proc. of ECCV, pages 625–642, 2020.
  * [42] Sheng Xu, Yanjing Li, Tiancheng Wang, Teli Ma, Baochang Zhang, Peng Gao, Yu Qiao, Jinhu Lü, and Guodong Guo.  Recurrent bilinear optimization for binary neural networks.  In Proc. of ECCV, pages 19–35, 2022.
  * [43] Sheng Xu, Yanjing Li, Bohan Zeng, Teli Ma, Baochang Zhang, Xianbin Cao, Peng Gao, and Jinhu Lü.  Ida-det: An information discrepancy-aware distillation for 1-bit detectors.  In Proc. of ECCV, pages 346–361, 2022.
  * [44] Sheng Xu, Junhe Zhao, Jinhu Lu, Baochang Zhang, Shumin Han, and David Doermann.  Layer-wise searching for 1-bit detectors.  In Proc. of CVPR, pages 5682–5691, 2021.
  * [45] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  arXiv preprint arXiv:1606.06160, 2016.
  * [46] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally.  Trained ternary quantization.  In Proc. of ICLR, pages 1–10, 2017.
  * [47] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In Proc. of ICLR, pages 1–16, 2020.
  * [48] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid.  Towards effective low-bitwidth convolutional neural networks.  In Proc. of CVPR, pages 7920–7928, 2018.