v_1 \mapsto v_2 \hspace{.5cm} v_2 \mapsto (v_1 \wedge v_2) \cap (v_3 \wedge u^{-1}v_1) \hspace{.35cm} v_3 \mapsto v_3
\end{equation}

while preserving the local system and the affine flag $F$.

We have similarly a birational automorphism $\sigma \in {\rm Bir}(\mathcal{A}_{{\rm SL}_4,D_{2,1}})$ defined via

$$v_1 \mapsto v_2 \hspace{.5cm} v_2 \mapsto (v_1 \wedge v_2) \cap (u^{-1}v_1 \wedge u^{-1}v_2 \wedge u^{-2}v_1) . $$

The transformation $\sigma$ is a quasi cluster automorphism of the corresponding cluster algebra. It acts on the cluster algebra while preserving the sets of diagram invariants, web invariants, tree invariants, and forest invariants.

One checks that $\sigma$ is a quasi cluster automorphism by providing an explicit sequence permutation-mutation sequence which matches Definition <ref>.
The required sequences are

\begin{align}
&(B\,  \rho^2(A) \, \rho( A) \, \rho( B))\circ  \mu_{B}\mu_{\rho A} \text{ for the once-punctured triangle, $k=3$} \\
&(A\,  \rho(A) \, \rho( B) )\circ (C\,  \rho(C) )\circ \mu_{C}\mu_{\rho A}\mu_{\rho B} \mu_C  \text{ for the once-punctured bigon, $k=4$}.
\end{align}

In both cases, we wrote the required permutation of vertices in cycle notation.

If $T$ is a tensor diagram, then one can compute $\sigma_*([T])$ by “plugging in” the boundary edges of $T$ to a tensor-diagrammatical gadget which computes the formulas (<ref>). This plugging in process does not create any interior cycles on boundary vertices, so $\sigma$ preserves the sets of forest and tree invariants.  The crossings which are created by plugging in can be planarized using the crossing removal relation (only one of the terms in the crossing removal relation survives), so $\sigma$ also preserves the set of web invariants.

Now we sketch the proof of Proposition <ref> with further details filled in on a case by case basis below.

In all three cases, there are only finitely many $P$-clusters and the Conjecture <ref> can be verified by hand and by inspection. We have listed these $P$-clusters up to $W$-action in (<ref>), (<ref>), and (<ref>).

In the finite type example, we compute below all cluster variables and clusters explicitly as tagged tensor diagram invariants. One can see the validity of our two cluster combinatorics conjectures by inspection.

In the two elliptic $D_4$ examples, consider the group generated by the quasi cluster automorphism $\sigma$ identified above, together with the cyclic shift map $\rho$, the duality map $\ast$, and the Weyl group action at the puncture. This group acts on the cluster algebra by (quasi) cluster automorphisms and preserves the sets of tagged web invariants and tagged forest invariants.

We argue in both cases elliptic $D_4$ examples that there are only finitely many clusters modulo the action of this group. When $k=3$ example, we identify explicitly these finitely many clusters and argue the following extra steps. First, every cluster monomial in this finite list of clusters is a tagged web invariant and a tagged forest invariant. Second, if $x,y$ are cluster variables in this finite list, and if the weights of $x$ and $y$ at the puncture are not $W$-sortable, then $x$ and $y$ are in fact root-conjugate taggings of the same underlying tensor diagram. Third, if $x$ appears in our finite list, then $x$ is cluster compatible with any of its root-conjugate taggings. (It is not important that a cluster witnessing this fifth assertion appears in our finite list.) The first of these statements
proves Conjecture  <ref> while the second and third statements proves Conjecture <ref>.

It would be possible to carry out the same three steps alluded to in the last paragraph of the above proof in the case of $\mathscr{A}'({\rm SL}_4,D_{2,1})$, thereby verifying our cluster combinatorics conjectures for this cluster algebra. The number of clusters which needed to be explicitly computed was fairly large so we did not carry this out.

§.§ The case of $\mathscr{A}'({\rm SL}_3,D_{2,1})$

This is a type $D_4$ cluster algebra with two frozen variables, $16$ cluster variables, and $50$ clusters. We depict the underlying tensor diagrams (up to dihedral and $W$- action) in Figure <ref>.
The diagrams $B$ and $A$ both can be tagged in three ways and have two dihedral images. The diagram $x$ has weight zero at the puncture and has four dihedral images. The two dihedral images of the last diagram $f$ are the frozen variables. We have drawn $x$ in its tree form but we can obtain its planar form by applying the crossing removal relation and noting that one of the two terms vanishes because it has a boundary 2-cycle.

Each of the four variables $\{A,B,\rho(A),\rho(B)\}$ is compatible with exactly two out of the four cluster variables $\{x,\rho(x),\tau(x),\tau\rho(x)\}$ in both senses of compatibility (namely, cluster compatibility and also the compatibility notion that two web invariants are compatible when their product is again a web invariant). Specifically $B$ is compatible with $\rho(x)$ and $\rho(\tau(x))$, $\rho(B)$ is compatible with $x$ and $\tau(x)$,
$A$ is compatible with $x$ and $\rho(\tau(x))$, and $\rho(A)$ is compatible with $\rho(x)$ and $\tau(x)$.

Figure <ref> shows all 50 clusters in this cluster algebra, grouped in concentric circle “levels” with 6, 12, 24, 8 clusters respectively. Equation (<ref>) lists the $P$-clusters.
Clusters in the outermost level have $P$-cluster in the first row of (<ref>) up to $W$-action, those in the next level have $P$-cluster in rows two or three of (<ref>) up to $W$-action, those in the next level have $P$-cluster in rows four or five, and those in the innermost have $P$-cluster in the last two rows.

Mutation from the first to second level is an instance of the flattening relation (<ref>) with $\ell(S) = 1$, e.g exchanging $\rho(B)$ for $s_2(B)$. Mutating from the second to the third level is an instance of the skein relations, e.g. exchanging $\rho(A)$ for $\rho \tau (x)$. There are also mutations within the third level which implement the dosp mutation $12|3 \leftrightarrow 12|3$. Mutation from the third to fourth level is an instance of $\ell(S) =2$ flattening relation followed by additional skein relation, e.g. exchanging $A$ for $s_2s_1(B)$. Specifically, the flattening relation yields a linear combination $[T]-2[T']+[T'']$ after applying the binomial theorem to $(u^{-1}-1)^2$. One of the invariants $[T'']$ is equal to 0. The tensor diagram $[T]$ has a self-crossing and can be expressed as $[T] = [T''']+3[T']$ using the crossing removal relation and the fact that ${\rm tr}(u) =3$. The right hand side of the flattening relation then becomes a sum of two terms $[T''']+[T']$
and these two terms match the two terms appearing the exchange relation. Finally, there are mutations within the fourth level exchanging two dihedral images of $x$ (these are consequences of the skein relations).

[gray, out = 160, in = -90] (0,0) to (-.7,1);
[gray, out = 90, in = 200] (-.7,1) to (0,2);
[gray, out = 20, in = -90] (0,0) to (.7,1);
[gray, out = 90, in = -20] (.7,1) to (0,2);
at (0,2) $\bullet$;
at (0,0) $\bullet$;
at (0,1.06) $\circ$;
at (0,.94) $\bullet$;
at (0,-.5) $B$;

[gray, out = 160, in = -90] (3,0) to (2.3,1);
[gray, out = 90, in = 200] (2.3,1) to (3,2);
[gray, out = 20, in = -90] (3,0) to (3.7,1);
[gray, out = 90, in = -20] (3.7,1) to (3,2);
[out = -30, in = 90] (3,2) to (3.5,1.1);
[out = 30, in = -90] (3,0) to (3.5,.95);
at (3,2) $\bullet$;
at (3,0) $\bullet$;
at (3,1.06) $\circ$;
at (3,.94) $\bullet$;
at (3,-.5) $A$;
at (3.5,1.0) $\circ$;

[gray, out = 160, in = -90] (6,0) to (5.3,1);
[gray, out = 90, in = 200] (5.3,1) to (6,2);
[gray, out = 20, in = -90] (6,0) to (6.7,1);
[gray, out = 90, in = -20] (6.7,1) to (6,2);
[out = -30, in = 90] (6,2) to (6.5,1.1);
[out = 30, in = -90] (6,0) to (6.5,.95);
[out = -100, in = -90] (6.45,1.0) to (5.65,.95);
[out = 90, in = 180] (5.65,.95) to (6.0,1.45);
[out = 0, in = 70] (6.0,1.45) to (6,0);
at (6,2) $\bullet$;
at (6,0) $\bullet$;
at (6,-.5) $x$;
at (6,1.06) $\circ$;
at (6,.94) $\bullet$;
at (6.5,1.06) $\circ$;
[gray, out = 160, in = -90] (9,0) to (8.3,1);
[gray, out = 90, in = 200] (8.3,1) to (9,2);
[gray, out = 20, in = -90] (9,0) to (9.7,1);
[gray, out = 90, in = -20] (9.7,1) to (9,2);
[out = -30, in = 90] (9,2) to (9.5,1.1);
[out = 30, in = -90] (9,0) to (9.5,.95);
[out = -100, in = -90] (9.45,1.0) to (8.55,.85);
[out = 90, in = 205] (8.55,.85) to (9.0,2);
at (9.5,1.06) $\circ$;
at (9,2) $\bullet$;
at (9,0) $\bullet$;
at (9,-.5) $f$;
at (9,1.06) $\circ$;
at (9,.94) $\bullet$;

The tensor diagrams underlying mutable and frozen variables in $\mathscr{A}'({\rm SL}_3,D_{2,1})$, which has finite cluster type $D_4$.

§.§ The case of $\mathscr{A}'({\rm SL}_3,D_{3,1})$

This cluster algebra has the same cluster type as the cluster algebra associated to a four-punctured sphere $S_{0,4}$. The underlying quiver mutation class consists of four quivers. There are six combinatorial types of tagged triangulations cf. [1]. We use the nomenclature for these six combinatorial types given in loc. cit. and note that
types $I$ and $VI$ have the same underlying quiver, as do types II and V. We distinguish between
unlabeled and labeled triangulations – in the former we consider the arcs in a triangulation as a set while in the latter we think of these arcs as an ordered tuple. Any two tagged unlabeled triangulations of the same combinatorial type are related by an element of the tagged mapping class group ${\rm MCG}^\bowtie(S_{0,4})$.

We denote by $T_0$ be the following labeled triangulation of $S_{0,4}$, where we use $p_1,\dots,p_4$ to denote the four punctures and think of the sphere as $\mathbb{R}^2 \cup \{\infty\}$.

\begin{equation}
\begin{tikzpicture}
\node at (-2,0) {$T_0 = $};
\node at (0,0) {\small $p_4$};
\node at (90:1.25cm) {\small $p_1$};
\node at (-30:1.25cm) {\small $p_2$};
\node at (210:1.25cm) {\small $p_3$};
\draw (-30:.2cm)--(-30:1.05cm);
\draw (210:.2cm)--(210:1.05cm);
\draw (90:.2cm)--(90:1.05cm);
\draw (105:1.15cm)--(195:1.15cm);
\draw (225:1.15cm)--(-45:1.15cm);
\draw (75:1.15cm)--(-15:1.15cm);
\node at (35:1.05cm) {\tiny $6$};
\node at (-90:1.05cm) {\tiny $2$};
\node at (145:1.05cm) {\tiny $4$};
\node at (80:.75cm) {\tiny $3$};
\node at (-45:.75cm) {\tiny $5$};
\node at (195:.75cm) {\tiny $1$};
\node at (2.5,.8) {$1: \, A$};
\node at (2.63,.3) {$3: \, \rho A$};
\node at (2.7,-.2) {$5: \, \rho^2 A$};
\node at (4.2,.8) {$2: \, B$};
\node at (4.33,.3) {$4: \, \rho B$};
\node at (4.4,-.2) {$6: \, \rho^2 B$};
\end{tikzpicture}
\end{equation}

To the right, we have identified the 6 initial cluster arcs for $S_{0,4}$ with the 6 initial cluster variables for $\mathscr{A}'({\rm SL}_3,D_{3,1})$  in such a way that the mutable parts of the quivers are identified. We denote by $T_1$ the result of notching all arcs in $T_0$ at the puncture $p_1$.

Fixing the above matching of initial seeds, we obtain a bijection between cluster variables (resp. clusters) for $\mathscr{A}'({\rm SL}_3,\mathbb{D}_{3,1})$ and tagged arcs (resp. tagged triangulations) in $S_{0,4}$.  If $\Sigma$ is a seed for $\mathscr{A}'({\rm SL}_3,D_{3,1})$ we denote by $T(\Sigma)$ its corresponding tagged triangulation. In particular, every cluster for $\mathscr{A}'({\rm SL}_3,D_{3,1})$ has one of the six aforementioned combinatorial types.

Figures <ref> and <ref> together illustrate this bijection between cluster variables in specific instances. The bijection is equivariant with respect to the group of dihedral symmetries of $D_{3,1}$, where in the case of $S_{0,4}$ we view the punctures  $p_1,p_2,p_3$ as the vertices of a triangle whose puncture is $p_4$. We have broken the two figures up so that the ${\rm SL}_3$ tensor diagrams appearing in Figure <ref> have nonzero weight at the puncture (hence, $W$ acts on them), while those in Figure <ref> have zero weight. We computed the tensor diagrams in this table by performing mutations using skein relations and the flattening relation. We omit the details.

[green] (-30:1.25cm)–(90:1.25cm);
[rotate = 35, green] at (80:1.0cm) $\bowtie$;
[rotate = 35, green] at (-20:1.0cm) $\bowtie$;
[rotate = -60, red] at (-.202,.02) $\bowtie$;
[rotate = 0]at (-.35,-.4) $A$;
[rotate = 30, red] at (-.7,.1) $s_1(A)$;
[rotate = -60, green] at (.85,.35) $s_1s_2(A)$;
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
[xshift = 3.7cm]
[rotate = 0, green] at (.1,.2) $\bowtie$;
[rotate = 0,red] at (-.1,1.1) $\bowtie$;
[rotate = 0,green] at (.1,1.1) $\bowtie$;
[rotate = 0]at (0,-.8) $B$;
[rotate = 0, red] at (-.6,.4) $s_2(B)$;
[rotate = 0, green] at (.6,.4) $s_2s_1(B)$;
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

[xshift = 7.4cm]
[out = 135, in = 90] (90:1.25) to (-1.5,-.5);
[out = -90, in = -90] (-1.5,-.5) to (0,0);
[red, out = 115, in = 90] (90:1.25) to (-1.7,-.5);
[red, out = -90, in = -90] (-1.7,-.5) to (0,-.3);
[red](0,-.3) to (0,0);
[green, out = 90, in = 180] (210:1.25) to (0,.2);
[green, out = 0, in = 90] (0,.2) to (-30:1.25);
[rotate = 0, green] at (-1.05,-.4) $\bowtie$;
[rotate = 0, green] at (1.05,-.4) $\bowtie$;
[rotate = 0, red] at (0,-.35) $\bowtie$;
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
at (-.75,.85) $A'$;
[rotate = 45, red] at (-1.20,1.25) $s_1(A')$;
[green] at (.8,.35) $s_1s_2(A')$;

[xshift = 11.1cm]
[out = 210, in = 90] (0,1.25) to (-.34,-.14);
[out = -90, in = 210] (-.34,-.14) to (-30:1.25);
[red,out = 15, in = 0] (0,0) to (.2,1.65);
[red,out = 180, in = 60] (.2,1.65) to (210:1.25);
[green,out = 5, in = 0] (.15,0) to (.3,1.85);
[green,out = 180, in = 80] (.3,1.85) to (210:1.25);
[rotate = -45, green] at (.3,.05) $\bowtie$;
[rotate = -5, green] at (-1.05,-.2) $\bowtie$;
[rotate = -10, red] at (-.95,-.45) $\bowtie$;
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
at (-.5,-.65) $B'$;
[red] at (1.0,.15) $s_2(B')$;
[rotate = 0,green] at (1.6,.5) $s_2s_1(B')$;

[yshift = 3.1cm]
[out = 90, in = 0] (.5,.59) to (0,1.25);
[out = 100, in = -10] (-30:1.25) to (.55,.5);
at (.75,.75) $A$;
at (0,.08) $\bullet$;
at (0,-.08) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
at (.5,.5) $\circ$;

[xshift = 3.7cm, yshift = 3.1cm]
at (.2,.75) $B$;
at (0,-.08) $\bullet$;
at (0,.08) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

[xshift = 7.4cm, yshift = 3.1cm]
at (0,.5) $\circ$;
[out = 90, in = 180] (210:1.25) to (-.1,.5);
[out = 90, in = 0] (-30:1.25) to (.1,.5);
at (0,.08) $\bullet$;
at (0,-.08) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
at (.5,.7) $A'$;

[xshift = 11.1cm, yshift = 3.1cm]
at (-.5,.3) $\circ$;
at (.6,-.35) $\circ$;
at (.5,.3) $\bullet$;
[out = 200, in = 145] (0,1.25) to (-.55,.35);
[out = 110, in = 180] (210:1.25) to (-.58,.28);
[out = 80, in = 90] (-.45,.3) to (.5,.3);
[out = 20, in = -100] (210:1.25) to (.6,-.4);
[out = 80, in = -10] (.62,-.3) to (.55,.25);

at (0,-.08) $\bullet$;
at (0,.08) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;
at (.65,.75) $B'$;

The top row shows plain ${\rm SL}_3$ tensor diagrams $A,B,A',B'$ in the once-punctured triangle, each of which has three images under ${\rm SL}_3$-Weyl group action at the puncture. In the bottom row, we depict the three tagged arcs in $S_{0,4}$ which correspond to the each of these $W$-orbits.

[xshift = 7.4cm]
[rotate = 90] at (-.8,-.63) $\bowtie$;
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

[xshift = 11.1cm]
[rotate = -30] at (-.3,1.1) $\bowtie$;
[out = 195, in = 0] (-30:1.25) to (.2,-1);
[out = 180, in = 190] (.2,-1) to (90:1.25);
at (0,0) $\bullet$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

[yshift = 0cm]
at (-.3,.5) $\circ$;
[out = 210, in = 90] (-.35,.5) to (210:1.25);
[out = 90, in = 225] (-.3,.57) to (90:1.25);
[out = 00, in = 90] (-.25,.5) to (.5,0);
[out = -90, in = 10] (.5,0) to (210:1.25);
at (0,.08) $\bullet$;
at (0,-.08) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

[xshift = 3.7cm, yshift = 0cm]
[out = 210, in = 90] (-.35,.5) to (210:1.25);
[out = 90, in = 225] (-.3,.57) to (90:1.25);
[out = 00, in = 90] (-.25,.5) to (.5,0);
[out = -90, in = -70] (.5,0) to (-.5,-.2);
[out = 110, in = 200] (-.5,-.2) to (90:1.25);
at (0,-.08) $\bullet$;
at (-.3,.5) $\circ$;
at (90:1.25cm) $\bullet$;
at (-30:1.25cm) $\bullet$;
at (210:1.25cm) $\bullet$;

The first and second ${\rm SL}_3$ tensor diagrams in the once-punctured triangle correspond respectively to the third and fourth tagged arcs in $S_{0,4}$. Each of these has six dihedral images.

Altogether, there are six plain ${\rm SL}_3$ tensor diagrams appearing in these figures versus 4 plain ${\rm SL}_2$ arcs. Each plain arc has three dihedral images and can be tagged in four ways, for a total of 48 tagged arcs appearing in the two figures. The ${\rm SL}_3$ diagrams in the first figure have 3 dihedral images and can be tagged in 3 ways, while those in the second figure have six dihedral images and cannot be tagged, again for a total of 48 diagrams.

Each quasi cluster automorphism of $\mathscr{A}'({\rm SL}_3,D_{3,1})$ induces an element of the cluster modular group for $S_{0,4}$. We list some instances of this correspondence in the table below:

Aut($\mathcal{A}'$)   ${\rm MCG}^\bowtie(S_{0,4})$   Sequence
$\rho$    $120^\circ$ rotation about $p_4$     (135)(246)

$\sigma$    half-twist $\substack{p_2 \\ \leftrightarrows \\ p_1}$      $(2534)\mu_{23}$

$s_1$    tag-change at $p_4$     $(35)\mu_{1351}$

$s_2$    N/A     $(46)\mu_{2462}$

$w_0 \ast \rho$    tag-change at all punctures

The third column shows the permutation-mutation sequence out of the initial seed which induces the corresponding automorphism. The notation $\mu_{23}$ means to mutate at vertex 2 and then at vertex 3. The quasi automorphism in the last row is the Donaldson-Thomas transformation, whose mutation sequence we have omitted due to its length. The quasi automorphism in the fourth row does not correspond to a tagged mapping class, but rather an “exotic” element of the cluster modular group for $S_{0,4}$ matching up a type I triangulation with a type VI triangulation.

We now carry out the three additional steps from Proposition <ref>, thereby verifying all of our conjectures for $\mathscr{A}'({\rm SL}_3,D_{3,1})$.

We begin by explaining that there are only finitely many clusters modulo $H'$ action, where $H'$ is the group defined in the proof of the above proposition. Then we carry out the three extra steps from the proof of the proposition. There is a group homomorphism $B_3 \to {\rm MCG}(S_{0,4})$ from the braid group on three strands to the mapping class group of a four-punctured sphere. The image of this homomorphism is the subgroup of mapping classes which fix the fourth puncture, see <cit.>. This subgroup has
index four inside ${\rm MCG}(S_{0,4})$ with cosets distinguished by which puncture is brought to $p_4$. Looking at the above table, the elements $\sigma$ and $\rho \sigma \rho^{-1}$ are the images of the Artin generators under this homomorphism, thus the group $H:= \langle\sigma, \rho \rangle$ has index four in ${\rm MCG}(S_{0,4})$. There are clearly only finitely many clusters modulo the action of this group. There will be even fewer clusters modulo the action of the larger group $H'$.

To carry out the three extra steps from the proof of the proposition, we explicitly identify a finite list of clusters which exhaust all clusters modulo $H'$-action.

One can check that the ${\rm MCG}(S_{0,4})$-orbit of the unlabeled triangulation $T_0$ coincides with its $H$-orbit. (The mutations needed to bring a given $p_i$, $i=1,2,3$ to the North pole are the same mutations needed to perform a half-twist. The two mapping classes differ only by a permutation of variables.)

One can also check that there are six ways to mutate from a type I triangulation to a type II triangulation, four ways to mutate from a type II to a type IV, one way to mutate from a type IV to a type V, two ways to mutate from a type V to a type VI, and one way to mutate from a type V to a type III. All other mutations move weakly “backwards” to an earlier combinatorial type.  Altogether, given a particular type I triangulation $T$, we get $6 \cdot 4 \cdot 1 \cdot 2+6\cdot 4 \cdot 1 = 72$ mutation sequences which efficiently pass from a given type I triangulation to a triangulation of any other type. Note that some steps in these mutation sequences commute so these 72 mutations sequences do not yield 72 distinct clusters. We refer to the clusters reachable from $T$ by one of these 72 mutations sequences as the club of $T$.

We claim that every tagged triangulation of $S_{0,4}$ is in the club of either $T_0$ or $T_1$ modulo $H'$-action. Indeed, suppose that $T$ has type $I$ and let $T'$ be it underlying plain triangulation. By two paragraphs previous, there exists an element $h_1 \in H$ such that $h_1(T') $ is a relabeling of $T_0$. We can choose $h_2 \in \langle {\rm DT},s_1\rangle $ (i.e., in the group generated by tag-changing at $p_4$ or at all the $p_i$'s) such that $h_2h_1(T) \in \{T_0,T_1\}$ as unlabeled tagged triangulations. Thus, if $T$ has type I, then we can move it using $H'$ to either $T_0$ or $T_1$. If we consider a tagged triangulation $T'$ of some other type, then $T'$ is in the club of some type I triangulation $T$. Choosing $h_1$ and $h_2$ as in the previous sentences so that $h_2h_1(T) \in \{T_0,T_1\}$, then $h_2h_1(T')$ is in the club of either $T_0$ or $T_1$ as claimed.

The clubs of $T_0$ and $T_1$ serves as our list of finitely many clusters modulo $H'$-action.

Now we carry out the first extra step, i.e. we check that every cluster monomial drawn from the clubs of $T_0$ and $T_1$ is a tagged web invariant and a tagged forest invariant. This is a straightforward but lengthy check using Figures <ref> and <ref>. Every tagged triangulation in these two clubs is built out of the 48 tagged arcs dihedrally related to these two figures. One must verify that when a collection of such tagged arcs are pairwise compatible then so are their corresponding tensor diagrams, where the notion of compatibility for ${\rm SL}_3$ tensor diagrams is that their product is a tagged web invariant.
The proof of the second extra step is illustrated in Figure <ref>: in the clubs of $T_0$ and $T_1$, the cluster variables which have nonzero weight are listed in the top of the figure, and each of these is compatible with its three taggings (because the three tagged arcs below each diagram are compatible tagged arcs).

The proof of the third extra step is another straightforward check using Figure <ref>. For example, the tensor diagram $A$ has weight $\omega_1$ at the puncture, so it is not $W$-sortable with tensor diagrams of weight $e_2$, $e_3$, or $e_2+e_3$. Looking right in the figure, the diagrams that have this weight and which are not root-conjugate to $A$ are $s_1s_2(A')$, $s_1(A')$, $s_2s_1(B')$, and $s_2s_1(B)$. We can see that the tagged arc corresponding to $A$ is not compatible with any of the four tagged arcs which correspond to these cluster variables.

The $P$-clusters in $\mathscr{A}'({\rm SL}_3,D_{3,1})$ are listed below up to $W$-action:

\begin{equation}\label{eq:Pclusters2}
\begin{tikzpicture}
\node at (0,0) {$
\begin{tabular}{l|c}
$P$-cluster & dosp \\ \hline
$e_1 \times 3, e_1+e_2 \times 3$ & $1|2|3$ \\ \hline
$e_1 \times 2, e_1+e_2 \times 4$ &  \\ \hline
$e_1 \times 4, e_1+e_2 \times 2$ &  \\ \hline
$e_1 \times 2, e_1+e_2 \times 3, 0$ &   \\ \hline
$e_1 \times 3, e_1+e_2 \times 2,0$ &  \\ \hline
$e_1 \times 2, e_1+e_2 \times 2,0\times 2$ &  \\ \hline
$e_1,e_2, e_1+e_2 \times 4$ & $12|3$ \\ \hline
$e_1,e_2, e_1+e_2 \times 3,0$ &   \\ \hline
$e_1,e_2, e_1+e_2 \times 2,0 \times 2$ &  \\ \hline
$e_1,e_2, e_1+e_2 ,1\times 3$ &  \\
\end{tabular}$};
\node at (7,1)
$P$-cluster & dosp \\ \hline
$e_1\times 4,e_1+e_2, e_1+e_3 $ & $1|23$ \\ \hline
$e_1\times 3,e_1+e_2, e_1+e_3,1$ &   \\ \hline
$e_1\times 2,e_1+e_2, e_1+e_3,0\times 2$ &  \\ \hline
$e_1\times 1,e_1+e_2, e_1+e_3,0\times 3$ &  \\ \hline
$e_1,e_2,e_3,0\times 3$ & $123^+$ \\ \hline
$e_1+e_2,e_1+e_3,e_2+e_3,0\times 3$ & $123^-$ \\ \hline
\end{tabular}$};
\end{tikzpicture}
\end{equation}

Using the above argument, it is not hard to see that every cluster variable in $\mathscr{A}'({\rm SL}_3,D_{3,1})$ is a tagged tree invariant, and that every cluster monomial factors as a product of these using arborization moves and the four extra moves from Figure <ref>, as predicted in Conjecture <ref>. We do not know how to prove the converse directions of these statements, e.g. that a web diagram which arborizes to a tree diagram indeed determines a cluster variable.

§.§ The case of $\mathscr{A}'({\rm SL}_4,D_{2,1})$

This cluster algebra also has the same cluster type as a four-punctured sphere. We follow the style of argument from the previous section making the needed modifications. We take our initial triangulation $T_0$ to be the following labeled tagged triangulation of $S_{0,4}$, which is a type V triangulation:

\begin{equation}
\begin{tikzpicture}
\node at (-2.7,.9) {$T_0 = $};
\node at (0,2) {$p_1$};
\node at (0,0) {$p_2$};
\node at (1,1) {$p_3$};
\node at (-1,1) {$p_4$};
\draw (-.2,1.8)--(-.8,1.2);
\draw (-.2,1.9)--(-.8,1.3);
\draw (.2,1.8)--(.8,1.2);
\draw (.2,1.9)--(.8,1.3);
\draw (0,.2)--(0,1.8);
\draw [out = 170, in = -90] (-.2,0) to (-1.5,1);
\draw [out = 90, in = 190] (-1.5,1) to (-.2,2);
\node at (.2,1) {\tiny $2$};
\node at (-1.7,1) {\tiny $5$};
\node at (.55,1.85) {\tiny $4$};
\node at (.3,1.5) {\tiny $1$};
\node at (-.65,1.75) {\tiny $3$};
\node at (-.3,1.5) {\tiny $6$};

\node [rotate = -45] at (-.7,1.4) {$\bowtie$};
\node [rotate = 45] at (.7,1.4) {$\bowtie$};
\node at (2.63,1.5) {$1: \, A$};
\node at (2.63,1.0) {$3: \, C$};
\node at (2.73,.5) {$5: \, \rho B$};
\node at (4.3,1.5) {$2: \, B$};
\node at (4.4,1.0) {$4: \, \rho C$};
\node at (4.4,.5) {$6: \, \rho A$};
\end{tikzpicture}
\end{equation}

We identify the initial tagged arcs with initial cluster variables for $\mathcal{A}'_{{\rm SL}_4,D_{2,1}}$ as indicated to the right. This induces the following correspondence between quasi cluster automorphisms of $\mathcal{A}'_{{\rm SL}_4,D_{2,1}}$ and elements of the cluster modular group of $\mathscr{A}(S_{0,4})$:

Aut($\mathcal{A}'$)   ${\rm MCG}^\bowtie(S_{0,4})$   Sequence
$\rho$    lift of $(34)$     (16)(34)(25)

$\sigma \circ \rho$     Dehn twist about $p_1,p_4$      $(152)\mu_{4214}$

$\rho \circ \sigma$     Dehn twist about $p_1,p_3$     $(625)\mu_{3563}$

$s_1 \circ s_3$     lift of $(12)(34)$

$s_2$   tag change at $p_2$      $(25)\mu_{25}$

$\ast$    tag change at both $p_3,p_4$     (14)(36)

tag-change at all punctures

There is a group homomorphism from ${\rm  MCG}(S_{0,4})$ to the symmetric group on four symbols keeping track of how the punctures are permuted. In the middle column above, “lift of $(34)$” indicates a mapping class whose image under this homomorphism is the transposition $(34)$, etc. The two Dehn twists in this table are pure mapping classes, i.e. their image is the identity permutation.

We now explain that there are only finitely many clusters modulo the action of the group defined in the proof of Proposition <ref>.

The two compositions $\rho \circ \sigma$ and $\sigma \circ \rho$ correspond to Dehn twists about simple closed curves with geometric intersection number two. Any two such mapping classes generate the pure mapping class group of $S_{0,4}$ , see  <cit.>. There are clearly finitely many tagged triangulations  of $S_{0,4}$ modulo the pure mapping class group.

We note that we can reduce the size of the finite check needed to verify our conjectures by precomposing with $\rho$,$s_1 \circ s_3$,$s_2$, and $\ast$, as these quasi cluster transformations correspond to certain non-pure mapping classes and tag-changing transformations.

The $P$-clusters for $\mathscr{A}'({\rm SL}_4,D_{2,1})$ are listed below up to $W$ action, this time in multiplicative notation for space-saving purposes:

\begin{equation}\label{eq:Pclusters3}
\begin{tikzpicture}
\node at (0,0) {
\begin{tabular}{l|l}
$P$-cluster & dosp\\\hline \rule{0pt}{0.85\normalbaselineskip}
$a \times 2,ab \times 2,abc \times 2$ & $1|2|3|4$ \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a,b,ab\times 2, abc\times 2 $ & $12|3|4$\\
$a,b,ab,abc\times 3$ &  \\
$a,b,ab,abc\times 2,1$ &  \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a \times 2,ab\times 2 ,abc,abd$ & $1|2|34$  \\
$a\times 3 ,ab,abc,abd$ &  \\
$a\times 2,ab,abc,abd,1$ &  \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a\times 2,ab,ac,abc \times 2$ & $1|23|4$  \\
$a,ab,ac,abc\times 3$ &   \\
$a\times 3,ab,ac,abc$ &   \\
$a,ab,ac,abc\times 2,1$ & \\
$a\times 2,ab,ac,abc,1$ & \\
\hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a,b,ab,ab,abc,abd$ & $12|34$ \\
$a,b,ab,abc,abd,1$ &   \\
$a,b,abc,abd,1\times 2$ & \\  \end{tabular}};
\node at (7.7,.45) {\begin{tabular}{l|l}
$P$-cluster & dosp\\\hline \rule{0pt}{0.85\normalbaselineskip}
\rule{0pt}{0.85\normalbaselineskip} \hspace{-.425cm} $a,b,c,abc,abc,1$ &$123^+|4$ \\
$a,b,c,abc,1\times 2$ &  \\
$a,b,c,abc\times 3$ &  \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $ab,ac,bc,abc \times 3$ & $123^-|4$ \\
$ab,ac,bc,abc \times 2, 1$ &  \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a \times 3, ab,ac,ad$ & $1|234^+$  \\
$a\times 2,ab,ac,ad,1$ &  \\ \hline \rule{0pt}{0.85\normalbaselineskip}
\hspace{-.25cm} $a\times 2,abc,abd,acd,1$ & $1|234^-$  \\
$a,abc,abd,acd,1\times 2$ &  \\
$a\times 3,abc,abd,acd$ &   \\
\hline \rule{0pt}{0.85\normalbaselineskip} \hspace{-.35cm}
$abc,abd,acd,bcd,1\times 2$ & $1234^-$\\ \hline
\rule{0pt}{0.85\normalbaselineskip}\hspace{-.15cm}
$a,b,c,d,1\times 2$ & $1234^+$ \end{tabular}};
\end{tikzpicture}
%\caption{The 27 combinatorial types of $\omega$-clusters. }
\end{equation}

[1]
E. Barnard, E. Meehan, N. Reading, S. Viel,
Universal geometric coefficients for the four-punctured sphere.
Ann. Comb. 22 (2018), no. 1, 1–44.

[2]
S. C. Billey, W. Jockusch, R. P. Stanley,
Some combinatorial properties of Schubert polynomials,
|sl J. Algebraic Combin.  2 (1993), no. 4, 345–374.

[3]
S. Cautis, J. Kamnitzer, S. Morrison,
Webs and Quantum Skew Howe Duality,
Math. Ann. 360 (2014), no. 1–2, 351–390.

[4]
D. Douglas, Z. Sun,
Tropical Fock-Goncharov coordinates for SL$_3$-webs on surfaces I: construction,
arXiv:2011.01768, (2021).

[5]
B. Farb, D. Margalit,
A Primer on Mapping Class Groups,
Princeton Mathematical Series, 49. Princeton University Press, Princeton, NJ,  (2012).

[6]
V. V. Fock and A. Goncharov,
Moduli spaces of local systems and higher Teichmüller Theory,
Publ. Math. Inst. Hautes. Etudes. Sci.  no. 103 (2006) 1–211.

[7]
S. Fomin, P. Pylyavskyy,
Tensor diagrams and cluster algebras,
Adv. Math. (2016) 717–787.

[8]
——, Webs on surfaces, rings of invariants, and cluster algebras,
Proc. Natl. Acad. Sci. USA   no. 111 (2014), no. 27, 9680–9687.

[9]
S. Fomin, M. Shapiro, D. Thurston,
Cluster algebras and triangulated surfaces. Part I: Cluster complexes,
Acta Math. 201 (2008), no. 1, 83–146.

[10]
S. Fomin, D. Thurston,
Cluster algebras and triangulated surfaces. Part II: Lambda lengths,
Mem. Amer. Math. Soc. 255 (2018), no. 1223, v+97 pp.

[11]
S. Fomin, L. Williams, A. Zelevinsky,
Cluster Algebras, in preparation, arXiv:1707.07190, (2020).

[12]
C. Fraser,
Braid group symmetries of Grassmannian cluster algebras,
Selecta Math.  (N.S.) 26 (2020), no. 2, Paper No. 17, 51 pp.

[13]
Quasi-homomorphisms of cluster algebras,
Adv. in Appl. Math 81, (2016) 40–77.

[14]
C. Frohman, A. Sikora,
SU(3)-skein algebras and webs on surfaces,
Math. Z., to appear, arXiv:2002.08151, (2021).

[15]
A. Goncharov, L. Shen,
Donaldson-Thomas transformations for moduli spaces of G-local systems,
Adv. Math. 327 (2018), 225–348.

[16]
——, Geometry of canonical bases and mirror symmetry,
Invent. Math.  202 (2015), no. 2, 487–633.

[17]
——, Quantum geometry of moduli spaces of local systems and representation theory
arXiv:1904.10491 (2020).

[18]
J. E. Grabowski,
Graded cluster algebras,
J. Algebraic Combin. 42 (2015), no. 4, 1111–1134.

[19]
H. K. Kim,
SL$_3$-laminations as bases for PGL$_3$ cluster varieties for surfaces,
arXiv:2011.14765 (2021).

[20]
R. Inoue, T. Ishibashi, H. Oya,
Cluster realizations of Weyl groups and higher Teichmüller theory,
arXiv:1902.02716 (2020), to appear in
Selecta Math.

[21]
R. Inoue, T. Lam, P. Pylyavskyy,
On the cluster nature and quantization of geometric R-matrices,
Publ. Res. Inst. Math. Sci.  55 (2019), no. 1, 25–78.

[22]
T. Ishibashi, W.Yuasa,
Skein and cluster algebras of marked surfaces without punctures for $\mathfrak{sl}_3$,
arXiv:2101.00643 (2021).

[23]
R. Kedem,
$Q$-systems as cluster algebras,
J. Phys. A 41 (2008), no. 19.

[24]
G. Kuperberg,
Spiders for rank 2 Lie algebras, Comm. Math. Phys. 180, no. 1 (1996), 109–151.

[25]
I. Le,
Cluster structures on higher Teichmüller spaces for classical groups,
Forum Math. Sigma 7 (2019), Paper No. e13.

[26]
I. Le, C. Fraser,
Tropicalization of positive Grassmannians,
Selecta Math. (N.S.) 25 (2019), no. 5, Paper No. 75, 55 pp.

[27]
T. Magee,
Littlewood-Richardson coefficients via mirror symmetry for cluster varieties,
Proc. Lond. Math. Soc. (3) 121 (2020), no. 3, 463–512.

[28]
M. R. Mills,
On the relationship between green-to-red sequences, local-acyclicity, and upper cluster algebras,
arXiv:1804.00479, (2018).

[29]
G. Muller,
Skein and cluster algebras of marked surfaces,
Quantum Topol.  7 (2016), no. 3, 435–503.

[30]
G. Musiker, R. Schiffler, L. Williams,
Bases for cluster algebras from surfaces,
Compos. Math.  149 (2013), no. 2, 217–263.

[31]
P. Pylyavskyy,
Zamolodchikov integrability via rings of invariants,
Journal of Integrable Systems Volume 1, Issue 1, January 2016, xyw010.

[32]
G. Schrader, A. Shapiro,
A cluster realization of Uq(sln) from quantum character varieties,
Invent. Math. 216 (2019), no. 3, 799–846.

[33]
L. Shen,
Duals of semisimple Poisson-Lie groups and cluster theory of moduli spaces of $G$-local systems,
arXiv.2003.07901, (2021), to appear in IMRN.

[34]
B. Sturmfels,
Algorithms in Invariant Theory,
Springer-Verlag, 1993.


# Cal-DETR: Calibrated Detection Transformer

Muhammad Akhtar Munir1,2 , Salman Khan1,3, Muhammad Haris Khan1,
Mohsen Ali2, Fahad Shahbaz Khan1,4
1Mohamed bin Zayed University of AI, 2Information Technology University,
3Australian National University, 4Linköping University
<EMAIL_ADDRESS>
###### Abstract

Albeit revealing impressive predictive performance for several computer vision
tasks, deep neural networks (DNNs) are prone to making overconfident
predictions. This limits the adoption and wider utilization of DNNs in many
safety-critical applications. There have been recent efforts toward
calibrating DNNs, however, almost all of them focus on the classification
task. Surprisingly, very little attention has been devoted to calibrating
modern DNN-based object detectors, especially detection transformers, which
have recently demonstrated promising detection performance and are influential
in many decision-making systems. In this work, we address the problem by
proposing a mechanism for calibrated detection transformers (Cal-DETR),
particularly for Deformable-DETR, UP-DETR and DINO. We pursue the train-time
calibration route and make the following contributions. _First_ , we propose a
simple yet effective approach for quantifying uncertainty in transformer-based
object detectors. _Second_ , we develop an uncertainty-guided logit modulation
mechanism that leverages the uncertainty to modulate the class logits. _Third_
, we develop a logit mixing approach that acts as a regularizer with
detection-specific losses and is also complementary to the uncertainty-guided
logit modulation technique to further improve the calibration performance.
Lastly, we conduct extensive experiments across three in-domain and four out-
domain scenarios. Results corroborate the effectiveness of Cal-DETR against
the competing train-time methods in calibrating both in-domain and out-domain
detections while maintaining or even improving the detection performance. Our
codebase and pre-trained models can be accessed at
https://github.com/akhtarvision/cal-detr.

##  1 Introduction

Deep neural networks (DNNs) have enabled notable advancements in computer
vision, encompassing tasks such as classification [29, 6, 44, 11], object
detection [2, 53, 5, 38, 43, 51], and semantic segmentation [47, 42, 3]. Due
to their high predictive performance, DNNs are increasingly adopted in many
real-world applications. However, it has been shown that DNNs often make
highly confident predictions even when they are incorrect [10]. This
misalignment between the predictive confidence and the actual likelihood of
correctness undermines trust in DNNs and can lead to irreversible harm in
safety-critical applications, such as medical diagnosis [7, 41], autonomous
driving [9], and legal decision-making [49].

Among others, an important factor behind the overconfident predictions of DNNs
is the training with zero-entropy target vectors, which do not encompass
uncertainty information [12]. An early class of methods aimed at reducing the
model’s miscalibration proposes various post-hoc techniques [10, 15, 16, 37,
21]. Typically, a single temperature scaling parameter is used to re-scale the
logits of a trained model which is learned using a held-out validation set.
Despite being effective and simple, they are architecture-dependent and
require a held-out set which is unavailable in many real-world scenarios [28].
Recently, we have seen an emergence of train-time calibration methods. They
are typically based on an auxiliary loss that acts as a regularizer for
predictions during the training process [12, 28, 25, 19, 33]. For instance,
[28] imposes a margin constraint on the logit distances in the label smoothing
technique, and [12] develops an auxiliary loss that calibrates the confidence
of both the maximum class and the non-maximum classes. These train-time
approaches reveal improved calibration performance than various existing post-
hoc methods.

We note that majority of the DNN calibration methods target the classification
task [10, 15, 16, 37, 21, 12, 28, 25, 19]. Strikingly, much little attention
has been directed towards studying the calibration of object detectors [33].
Object detection is a fundamental task in computer vision whose application to
safety-critical tasks is growing rapidly. Furthermore, most of the existing
methods focus on calibrating in-domain predictions. However, a deployed model
could be exposed to constantly changing distributions which could be radically
different from its training distribution. Therefore, in this paper, we aim to
investigate the calibration of object detection methods both for _in-domain_
and _out-domain_ predictions. Particularly, we consider the recent
transformer-based object detection methods [5, 53, 51] as they are the current
state-of-the-art with a simpler detection pipeline. Moreover, they are gaining
increasing attention from both the scientific community and industry
practitioners.

In this paper, inspired by the train-time calibration paradigm, we propose an
uncertainty-guided logit modulation and a logit mixing approach to improve the
calibration of detection transformers (dubbed as Cal-DETR). In order to
modulate the logits, we propose a new and simple approach to quantify
uncertainty in transformer-based object detectors. Unlike existing approaches
[8, 23], it does not require any architectural modifications and imposes
minimal computational overhead. The estimated uncertainty is then leveraged to
modulate the class logits. Next, we develop a mixing approach in the logit
space which acts as a regularizer with detection-specific losses to further
improve the calibration performance. Both uncertainty-guided logit modulation
and logit mixing as a regularizer are complementary to each other. We perform
extensive experiments with three recent transformer-based detectors
(Deformable-DETR (D-DETR) [53], UP-DETR [5] and DINO [51]) on three in-domains
and four out-domain scenarios, including the large-scale object detection MS-
COCO dataset [27]. Our results show the efficacy of Cal-DETR against the
established train-time calibration methods in improving both in-domain and
out-domain calibration while maintaining or even improving detection
performance.

##  2 Related Work

Vision transformer-based object detection: Vision transformers are enabling
new state-of-the-art performances for image classification [6, 46], semantic
segmentation [47, 42], and object detection [2, 53, 5, 51]. The work of DETR
[2] introduced the first-ever transformer-based object detection pipeline to
the vision community by eliminating post-processing steps like non-maximum
suppression with object matching to particular ground truth using bipartite
matching. However, DETR is very slow in convergence and performs poorly on
small objects, and to address these issues, Deformable-DETR [53] was proposed
that is based on a deformable attention module as opposed to vanilla self-
attention in DETR [2]. The work of [5] further enhanced the performance of
this object detection pipeline by pretraining the encoder-decoder parameters.
Recently, DINO [51] improved detection performance with contrastive denoising
training, look forward twice approach and integrated effective initialization
of anchors.

Model calibration: Model calibration requires the perfect alignment of model’s
predictive confidence with the actual likelihood of its correctness. One of
the earliest techniques for model calibration is known as post-hoc temperature
scaling (TS) [10], which is a variant of Platt scaling [37]. The idea is to
re-scale the logits with a single temperature parameter (T) which is learned
using a held-out validation set. However, a limitation of TS is that the
selection of temperature parameter T value is reliant on the network
architecture and data [28]. And, TS has been shown to be effective for
calibrating in-domain predictions. To extend TS to domain-drift scenarios,
[45] proposed to perturb the held-out validation set before the post-hoc
processing step. Dirichlet calibration (DC) [20] was extended from beta
calibration [21] to a multi-class setting. It uses an extra neural network
layer that log-transforms the miscalibrated probabilities, and the parameters
for this layer are learned using the validation set. Despite being effective
and computationally inexpensive, post-hoc calibration methods demand the
availability of a held-out validation set, which is a difficult requirement to
be met for many real-world use cases.

Train-time calibration methods tend to engage model parameters during training
by proposing an auxiliary loss function that is optimized along with the task-
specific loss function [12, 28]. Some methods aim at maximizing the entropy of
the predictive distribution either explicitly or implicitly. For instance,
[35] formulated an auxiliary term, that is based on negative of entropy, which
is minimized during training to alleviate the impact of overconfident
predictions. Along similar lines, [32] investigated that the label smoothing
(LS) technique is also capable of reducing miscalibration, and [31]
demonstrated that the focal loss (FL) can improve the model calibration. These
methods are effective, however, their logit-distance constraints can often
lead to non-informative solutions. To circumvent this, [28] proposed a margin-
based generalization of logit-distance constraints to achieve a better balance
between discriminative and calibration performance. [12] proposed to calibrate
the confidence of both predicted and non-predicted classes by formulating an
auxiliary loss term known as the multi-class difference of confidence and
accuracy. All aforementioned calibration techniques target classification
task, recently [33] developed a train-time calibration loss for object
detection which imposes constraints on multi-class predictions and the
difference between the predicted confidence and the IoU score.

Uncertainty estimation and model calibration: Owing to the computational
intractability of Bayesian inference, different approximate inference methods
have been proposed, such as variational inference [1, 30], and stochastic
expectation propagation [14]. The work of [34] evaluated predictive
uncertainty of models under distributional shift and investigates its impact
on the accuracy-uncertainty trade-off. [19] leveraged the predictive
uncertainty of model and proposed to align this with accuracy, and [18]
induced soft bins and makes calibration error estimates differentiable. It is
also possible to quantify uncertainty by ensemble learning that takes into
account empirical variance of model predictions. It allows better
discriminative performance and improved predictive uncertainty estimates
leading to well-calibrated models. Ensembles can emerge with difference in
weights initializations and random shuffling of training examples [23] and
Monte Carlo (MC) dropout [8]. However, ensemble-based uncertainty estimation
techniques could be computationally expensive, especially with complex models
and large-scale datasets. In this work, we propose a simple and
computationally efficient technique to quantify uncertainty in a transformer-
based object detection pipeline.

##  3 Method

This section begins by discussing the preliminaries, including the problem
settings and the background knowledge relevant to calibrating object
detectors. We then detail our proposed approach, Cal-DETR, encompassing
uncertainty-guided logit modulation and logit mixing for calibrating the
recent vision transformer-based (ViT) object detection methods (Fig. 1).

Figure 1: _Main architecture:_ An image is passed through a feature extractor
CNN and unmodified transformer’s encoder structure, whereas the decoder is
modified to get the model well calibrated. Logits are modulated based on
uncertainty quantification that exploits the inbuilt design of the decoder
without extra computational cost (Sec. 3.3.1). This module is followed by a
logit mixing mechanism which is comprised of a regularizer ($\mathcal{L}_{R}$)
to obtain calibrated predictions (Sec. 3.3.2). <category:confidence> are
depicted on the right side of figure. The baseline detector (D-DETR [53])
outputs wrong dog prediction with a relatively high confidence, however, our
Cal-DETR provides accurate cat prediction with a high confidence. Green boxes
are accurate predictions whereas Red (dashed) box is inaccurate prediction.
Blue bounding boxes represent ground truths for corresponding detections.
$\mathcal{L}_{cls}$ and $\mathcal{L}_{loc}$ are classification and
localization losses respectively [5, 53, 51].

###  3.1 Preliminaries

Assume a joint distribution $\mathcal{S}(\mathcal{X},\mathcal{Y})$, where
$\mathcal{X}$ is an input space and $\mathcal{Y}$ is the corresponding label
space. Let $\mathcal{D}=\\{(\mathbf{x}_{i},\hat{\mathbf{y}}_{i})_{i=1}^{N}\\}$
be the dataset sampled from this joint distribution consisting of pairs of
input image $\mathbf{x}_{i}\in\mathcal{X}$ and corresponding label
$\hat{\mathbf{y}_{i}}\in\mathcal{Y}$. The input image belongs to
$\mathbf{x}_{i}\in\mathbb{R}^{H\times W\times Z}$, where $H$ is the height,
$W$ is the width and $Z$ is the number of channels. For each input image
$\mathbf{x}_{i}$, we have the corresponding ground truth label,
$\hat{\mathbf{y}}_{i}\in\mathbb{R}^{5}$, and it consists of class label and
the bounding box coordinates $(\hat{c},\hat{x},\hat{y},\hat{w},\hat{h})$. Here
$\hat{c}$ is the class label $\in\\{1,2,...,C\\}$ and
$\hat{x},\hat{y},\hat{w},\hat{h}$ are the bounding box
$\hat{\mathbf{b}}\in\mathbb{R}^{4}$ coordinates. Next, we define the
calibration of a model in classification and object detection tasks.

A model is said to be calibrated when its accuracy perfectly aligns with the
predictive confidence. An uncalibrated model is prone to making over-confident
and under-confident predictions. To mitigate these problems, for an image, a
model should be able to predict the class confidence that matches with the
actual likelihood of its correctness. Let $\mathcal{H}_{cls}$ be a
classification model which predicts a class label $\bar{c}$ with the
confidence score $\bar{z}$. Now, the calibration for the classification task
can be defined as: $\mathbb{P}(\bar{c}=\hat{c}|\bar{z}=z)=z,~{}\text{s.t.,
}z\in[0,1]$ [22]. This expression shows that, to achieve the perfect
calibration in classification, the predicted confidence and accuracy must
match.

Let $\mathcal{H}_{det}$ be the detection model, which predicts a bounding box
$\bar{\mathbf{b}}$ and the corresponding class label $\bar{c}$ with the
confidence score $\bar{z}$. The calibration for a detection task is defined as
$\mathbb{P}(F=1|\bar{z}=z)=z,~{}\forall z\in[0,1]$ [22]111This calibration
definition of object detectors can be extended after including box
information.. Where $F=1$ denotes the true prediction of the object detection
model which occurs when the predicted and the ground truth classes are the
same, and the Intersection over Union (IoU) of both the predicted and ground
truth boxes are above a pre-defined threshold $k$ i.e.
$\mathbbm{1}[IoU(\bar{\mathbf{b}},\hat{\mathbf{b}})\geq
k]\mathbbm{1}[\bar{c}=\hat{c}]$. $F=0$ denotes the false prediction and
together with $F=1$, the precision is determined. In object detection, to
achieve perfect calibration, the precision of a model must match the predicted
confidence. In the next subsection, we revisit the formulae for measuring
calibration.

###  3.2 Measuring Calibration

Classification: For classification problems, miscalibration is measured by
computing Expected Calibration Error (ECE) [10]. It computes the expected
divergence of accuracy from all possible confidence levels as:
$\mathbb{E}_{\bar{z}}[|\mathbb{P}(\bar{c}=\hat{c}|\bar{z}=z)-z|]$. In
practice, the ECE metric is approximated by (1) first dividing the space of
confidence scores into B equal-sized bins, (2) then computing the (absolute)
difference between the average accuracy and the average confidence of samples
falling into a bin, and finally (3) summing all the differences after
weighting each difference by the relative frequency of samples in a bin.

$\displaystyle\mathrm{ECE}=\sum_{b=1}^{B}\frac{|S(b)|}{|D|}\left|\mathrm{accuracy}(b)-\mathrm{confidence}(b)\right|,$
(1)

where, $\mathrm{accuracy}(b)$ and $\mathrm{confidence}(b)$ represent the
average accuracy and the average confidence of samples in the $b^{th}$ bin,
respectively. $S(b)$ is a set of samples in the $b^{th}$ bin and $|D|$ is the
total number of samples.

Object Detection: Similar to ECE, the Detection Expected Calibration Error
(D-ECE) is measured as the expected divergence of precision from confidence,
at all confidence levels [22]. It can be expressed as
$\mathbb{E}_{\bar{s}}[|\mathbb{P}(F=1|\bar{z}=z)-z|]$. As in Eq.(1),
confidence is a continuous variable, therefore we can approximate D-ECE:

$\displaystyle\mathrm{D\text{-}ECE}=\sum_{b=1}^{B}\frac{|S(b)|}{|D|}\left|\mathrm{precision}(b)-\mathrm{confidence}(b)\right|,$
(2)

where $S(b)$ contains a set of object instances in $b^{th}$ bin. Note that,
the D-ECE considers only the precision vs. confidence, and not the mean
Average Precision (mAP). The mAP requires the computation of recall for which
detections are not present, and so there are no associated confidences.

###  3.3 Cal-DETR: Uncertainty-Guided Logit Modulation & Mixing

Modern object detection methods occupy a central place in many safety-critical
applications, so it is of paramount importance to study their calibration.
Recently, vision transformer-based (ViT) object detectors (e.g., Deformable-
DETR (D-DETR) [53], UP-DETR [5] and DINO [51]) are gaining importance for both
industry practitioners and researchers owing to their state-of-the-art
performance and relatively simple detection pipeline. Therefore, we aim to
study the calibration of ViT-based object detectors. Towards improving
calibration in transformer-based object detectors, we first describe our
uncertainty-guided logit modulation approach (Sec. 3.3.1) and then propose a
new logit mixing regularization scheme (Sec.3.3.2).

####  3.3.1 Uncertainty-Guided Logit Modulation

We first briefly review the DETR pipeline [2], which is a popular ViT-based
object detection pipeline. Recent transformer-based detectors [5, 53, 51]
largely borrow the architecture of DETR pipeline [2]. Then, we describe our
new uncertainty quantification approach and how it is leveraged to modulate
the logits from the decoder to eventually calibrate confidences.

Revisiting DETR pipeline: DETR [2] was proposed as the first ViT-based object
detection pipeline with the motivation of eliminating the need for hand-
designed components in existing object detection pipelines. It consists of a
transformer encoder-decoder architecture with a set-based Hungarian loss which
ensures unique predictions against each ground truth bounding box. Assuming
the existence of feature maps $f\in\mathbb{R}^{H\times W\times C}$ which are
generated from ResNet [11] backbone, DETR relies on the standard transformer
encoder-decoder architecture to project these feature maps as the features of
a set of object queries. A feed-forward neural network (FFN) in a detection
head regresses the bounding box coordinates and the classification confidence
scores.

Quantifying uncertainty in DETR pipeline: Some recent studies have shown that
the overall (output) predictive uncertainty of the model is related to its
calibration [23, 19]. So estimating a model’s uncertainty could be helpful in
aiding the confidence predictions to align with the precision of a detection
model. To this end, we propose a new uncertainty quantification approach for
DETR pipeline after leveraging the innate core component of transformer
architecture i.e. the decoder. It is a simple technique that requires no
changes to the existing architecture and yields minimal computational cost,
unlike Monte Carlo dropout [8] and Deep Ensembles [23]. Specifically, let
$\mathrm{O}_{\mathrm{D}}\in\mathbb{R}^{L\times M\times Q\times C}$ and
$\mathrm{O}_{\mathrm{D}}^{L}\in\mathbb{R}^{M\times Q\times C}$ be the outputs
of all decoder layers and final decoder layer of the detection transformer,
respectively. Where $L$ is the number of decoder layers, $M$ is the mini-
batch, $Q$ is the number of queries, and $C$ represents the number of classes.
To quantify the uncertainty for every class logit $u_{c}$, we compute the
variance along the $L$ dimension which consists of multiple outputs of decoder
layers. In other words, the variation in the output of decoder layers reflects
the uncertainty for a class logit $u_{c}$. Overall, after computing variance
along the $L^{th}$ dimension in $\mathrm{O}_{\mathrm{D}}$, we obtain a tensor
$\mathbf{u}\in\mathbb{R}^{M\times Q\times C}$, which reflects the uncertainty
for every class logit across query and minibatch dimensions.

Logit modulation: We convert the uncertainty $\mathbf{u}$ into certainty
using: $1-\tanh(\mathbf{u})$, where $\tanh$ is used to scale the uncertainty
values to $[0,1]$ from $[0,\inf]$, and use this certainty to modulate the
logits output from the final layer as:

$\displaystyle\mathrm{\tilde{O}}_{\mathrm{D}}^{L}=\mathrm{O}_{\mathrm{D}}^{L}\otimes(1-\tanh(\mathbf{u})),$
(3)

where $\mathrm{\tilde{O}}_{\mathrm{D}}^{L}$ is the uncertainty-modulated
decoder output and then this becomes the input to the loss function.
$1-\tanh(\mathbf{u})$ indicates that if an output in logit space shows higher
uncertainty, it will be down-scaled accordingly. In other words, a higher
uncertainty implicitly implies a likely poor calibration of the output logit
value (i.e. the pre-confidence score).

####  3.3.2 Logit Mixing for Confidence Calibration

In addition to uncertainty-guided logit modulation, we develop a new logit
mixing approach which acts as a regularizer with the detection-specific losses
for further improving the confidence calibration. We propose logit mixing as
the mixing in the logit space of an object detector which is inspired by the
input space mixup for classification [50]. In a DETR pipeline, this logit
space is of dimensions $Q\times C$, where $Q$ is the number of object queries
and $C$ is the number of classes. We only consider positive queries for
performing logit mixing as they are responsible for the detection of objects.
In object detection, the instance-level object information is important as we
consider the confidence scores associated with the positive class assigned to
the bounding boxes.

In our logit mixing scheme, we first build a prototypical representation by
computing the mean across all positive queries, which is then used to achieve
the mixing for any given positive query. For example, given three positive
queries, let $Q_{p}$ = $\\{Q_{1},Q_{2},Q_{3}\\}\subset Q$ (total number of
queries) belonging to the object classes and let $\tilde{Q}$ be a prototypical
representation of all the positive queries, we formulate the expression as:

$\displaystyle\acute{Q}_{i}=\alpha
Q_{i}+(1-\alpha)\tilde{Q}.~{}~{}~{}~{}~{}~{}~{}\forall~{}Q_{i}\in Q_{p}$ (4)

Now, $\\{\acute{Q}_{1},\acute{Q}_{2},\acute{Q}_{3}\\}$ are the mixed versions
of $\\{Q_{1},Q_{2},Q_{3}\\}$ in the logit space. $\alpha$ is the mixing weight
for logits. For the selection of $\alpha$, more details are in Sec. 4.2.

We employ a regularization loss by leveraging the proposed logit mixing
output, denoted as $\acute{\psi}(\mathbf{x}_{i})$, which is expressed as:
$\mathcal{L}_{R}(p(\bar{c}|\acute{\psi}(x_{i}),\acute{c}_{i}$), where
$\acute{c}_{i}$ contains the labels after smoothing with mixing values.
Instead of one-hot labels, $\alpha$ determines the weightage of classes for
object instances. Specifically $\alpha$ value for a given positive query
contributes to smoothing the label while remaining positive queries that
contribute to forming a prototypical representation share 1-$\alpha$ value to
smooth corresponding labels uniformly. The loss $\mathcal{L}_{R}$ is used
together with the task-specific classification loss
$\mathcal{L}_{cls}(p(\bar{c}|\psi(x_{i})),\hat{c}_{i}$) to obtain a joint loss
formulation as: $\mathcal{L}_{cls}+\lambda\mathcal{L}_{R}$, where
${\psi}(\mathbf{x}_{i})$ denotes the non-mixed logits and $\lambda$ controls
the contribution of regularization loss. The proposed regularizer loss tends
to capture the information from the vicinity of object instances in the logit
space by mixing logits and constructing a non-zero entropy target
distribution. This penalizes the model for producing overconfident predictions
by possibly reducing overfitting [52, 36]. Our empirical results (Sec. 4.1)
show that this not only improves model calibration but also improves the
overall detection performance in most scenarios and it is also complementary
to our uncertainty-guided logit modulation technique.

##  4 Experiments & Results

Datasets: To perform experiments, we utilize various in-domain and out-domain
benchmark datasets. Details are as follows: MS-COCO [27] is a large-scale
object detection dataset for real images containing 80 classes. It splits into
118K train images, 41K test images, and 5K validation images. The train set
(train2017) is used for training while the validation set (val2017) is
utilized for evaluation. CorCOCO contains a similar set of images as present
in val2017 of MS-COCO but with the corrupted version [13]. Random corruptions
with random severity levels are introduced for evaluation in an out-domain
scenario. Cityscapes [4] is an urban driving scene dataset contains 8 classes
person, rider, car, truck, bus, train, motorbike, and bicycle. The training
set consists of 2975 images while the 500 validation images are used as the
evaluation set. Foggy Cityscapes [39] is a foggy version of Cityscapes, 500
validation images simulated with severe fog are used for evaluation in this
out-domain scenario. BDD100k [48] contains 100k images from which 70K are
training images, 20K are for the test set and 10K for the validation set.
Categories are similar to Cityscapes and for evaluation, we consider the
validation set and make a subset of daylight images as an out-domain scenario,
reducing the set to 5.2K images. Sim10k [17] contains 10k synthetic images
with car category from which the training split is 8K images and for the
evaluation set, the split is 1K images.

Datasets (validation post-hoc): For the given scenarios, we opt separate
validation datasets for the post-hoc calibration method (temperature scaling).
In the MS-COCO scenario, we utilize the Object365 [40] validation dataset
reflecting similar categories. For the Cityscapes scenario, a subset of the
BDD100k train set is used as a validation set. And for the Sim10k scenario, we
use the remaining split of images.

Detection Calibration Metric: For the calibration of object detectors, we
report our results with the evaluation metric, which is the detection expected
calibration error (D-ECE) [22]. It is computed for both in-domain and out-
domain scenarios.

Implementation Details: We use transformer-based state-of-art-detectors
Deformable-DETR (D-DETR), UP-DETR, and DINO as baselines respectively, which
are uncalibrated. With the respective in-domain training datasets, we train
D-DETR with our proposed method and compare it with baseline, post-hoc
approach [10], and train time calibration losses [28, 12, 33] integrated with
D-DETR. We find empirically over the validation set and use the logit mixing
parameters $\alpha_{1}=0.9$, $\alpha_{2}=0.1$ and $\lambda=0.5$ (see Sec.
4.2). For the classification loss, the focal loss is incorporated, and for
localization, losses are generalized IoU and L1 as described in [26, 2]. For
more implementation details, we refer readers to D-DETR [53], UP-DETR [5] and
DINO [51].

Table 1: Deformable-DETR (D-DETR) Calibration: MS-COCO & CorCOCO. Calibration results along with the detection performance. Cal-DETR improves calibration as compared to baseline, other train-time losses, and TS methods.  Methods Datasets | In-Domain (COCO) | Out-Domain (CorCOCO)
---|---|---
| D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ D-DETR}}$ [53] | 12.8 | 44.0 | 62.9 | 10.8 | 23.9 | 35.8
Temp. Scaling [10] | 14.2 | 44.0 | 62.9 | 12.3 | 23.9 | 35.8
MDCA [12] | 12.2 | 44.0 | 62.9 | 11.1 | 23.5 | 35.3
MbLS [28] | 15.7 | 44.4 | 63.4 | 12.4 | 23.5 | 35.3
TCD [33] | 11.8 | 44.1 | 62.9 | 10.4 | 23.8 | 35.6
Cal-DETR (Ours) | 8.4 | 44.4 | 64.2 | 8.9 | 24.0 | 36.3
Table 2: UP-DETR Calibration: MS-COCO & CorCOCO. Results are reported on Epoch150 setting and Cal-DETR shows improvement in calibration as compared to other methods including baseline.  Methods Datasets | In-Domain (COCO) | Out-Domain (CorCOCO)
---|---|---
| D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ UP-DETR}}$ [5] | 25.5 | 37.1 | 57.3 | 27.5 | 19.9 | 32.5
Temp. Scaling [10] | 26.4 | 37.1 | 57.3 | 28.2 | 19.9 | 32.5
MDCA [12] | 26.5 | 36.9 | 57.5 | 28.6 | 19.4 | 31.8
MbLS [28] | 25.4 | 36.3 | 57.0 | 26.5 | 20.1 | 32.9
TCD [33] | 25.2 | 36.4 | 57.2 | 26.8 | 19.1 | 31.5
Cal-DETR (Ours) | 23.9 | 37.3 | 57.8 | 25.5 | 20.0 | 32.6
Figure 2: D-DETR Calibration: MS-COCO & Cityscapes as in-domains and respective Cor-COCO & Foggy Cityscapes as out-domains. Our proposed method (Cal-DETR) effectively decreases D-ECE compared to methods: baseline, post-hoc, and train-time losses.  Table 3: DINO Calibration: MS-COCO & CorCOCO. Results are reported on the Epoch12 (4-scale) setting and Cal-DETR shows improvement in calibration as compared to other methods including baseline.  Methods Datasets | In-Domain (COCO) | Out-Domain (CorCOCO)
---|---|---
| D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ DINO}}$ [51] | 15.5 | 49.0 | 66.6 | 13.2 | 27.3 | 39.2
Temp. Scaling [10] | 15.1 | 49.0 | 66.6 | 14.3 | 27.3 | 39.2
MDCA [12] | 16.3 | 48.6 | 66.1 | 14.0 | 26.7 | 38.4
MbLS [28] | 19.1 | 48.6 | 66.0 | 15.9 | 26.9 | 38.4
TCD [33] | 15.6 | 48.5 | 66.1 | 12.9 | 26.8 | 38.5
Cal-DETR (Ours) | 11.7 | 49.0 | 66.5 | 10.6 | 27.5 | 39.3
Table 4: D-DETR Calibration: Results are shown in comparison with baseline, post-hoc, and train-time loss methods. Cal-DETR shows improvement in detection calibration on evaluation sets of the in-domain dataset (Cityscapes) and out-domain datasets (Foggy Cityscapes & BDD100k). AP and AP0.5 are also reported. Methods Datasets | In-Domain (Cityscapes) | Out-Domain (Foggy Cityscapes) | Out-Domain (BDD100k)
---|---|---|---
| D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ D-DETR}}$ [53] | 13.8 | 26.8 | 49.5 | 19.5 | 17.3 | 29.3 | 11.7 | 10.2 | 21.9
Temp. Scaling [10] | 12.6 | 26.8 | 49.5 | 14.6 | 17.3 | 29.3 | 24.5 | 10.2 | 21.9
MDCA [12] | 13.4 | 27.5 | 49.5 | 17.1 | 17.7 | 30.3 | 14.2 | 10.7 | 22.7
MbLS [28] | 12.1 | 27.3 | 49.7 | 20.0 | 17.1 | 29.1 | 11.6 | 10.5 | 22.7
TCD [33] | 11.9 | 28.3 | 50.8 | 14.3 | 17.6 | 30.3 | 12.8 | 10.5 | 22.2
Cal-DETR (Ours) | 8.4 | 28.4 | 51.4 | 11.9 | 17.6 | 29.8 | 11.4 | 11.1 | 23.9
Table 5: D-DETR Calibration: Cal-DETR results in improved calibration performance over both in-domain and out-domain datasets, Sim10K and BDD100K respectively. The evaluation set contains car category. AP and AP0.5 are also reported.  Methods Datasets | In-Domain (Sim10k) | Out-Domain (BDD100k)
---|---|---
|  D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ |  D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ D-DETR}}$ [53] | 10.3 | 65.9 | 90.7 | 7.3 | 23.5 | 46.6
Temp. Scaling [10] | 15.7 | 65.9 | 90.7 | 10.5 | 23.5 | 46.6
MDCA [12] | 10.0 | 64.8 | 90.3 | 8.8 | 22.7 | 45.7
MbLS [28] | 22.5 | 63.8 | 90.5 | 16.8 | 23.4 | 47.4
TCD [33] | 8.6 | 66.4 | 90.4 | 6.6 | 23.1 | 46.0
Cal-DETR (Ours) | 6.2 | 65.9 | 90.8 | 6.3 | 23.8 | 46.5
Figure 3: Reliability Diagrams: Baseline [53] vs. Cal-DETR (Ours): Reliability
plots on MS-COCO dataset.

###  4.1 Main Results

Real and Corrupted: We evaluate the calibration performance on large-scale
datasets (MS-COCO as in-domain and its image corrupted version as out-domain,
CorCOCO). The CorCOCO evaluation set is constructed by incorporating random
severity and random corruption levels from [13]. Our method not only provides
lower calibration errors but also shows improved detection performance. For
D-DETR, in comparison to the baseline uncalibrated model, our method improves
calibration of 4.4%$\downarrow$ (in-domain) and 1.9%$\downarrow$ (out-domain).
In this dataset, we observe that classification-based train time calibration
losses (MbLS [28] & MDCA [12]) are sub-optimal for the detection problems and
worse than the baseline in some scenarios. We show that our method outperforms
train time losses and achieves better calibration, e.g. in comparison with
detection-based calibration loss (TCD [33]) improvements are 3.4%$\downarrow$
(in-domain) and 1.5%$\downarrow$ (out-domain). For UP-DETR, our method
improves calibration from baseline by 1.6%$\downarrow$ (in-domain) and
2.0%$\downarrow$ (out-domain). For more results, we refer to Fig. 2, Tab. 4 &
Tab. 4. We also provide the results on COCO (in-domain) and CorCOCO (out-
domain) with the DINO model, as shown in Tab. 4. Our Cal-DETR improves the
calibration performance of this strong baseline for both in-domain and out-
domain settings.

Weather: Our method improves calibration over the baseline, post-hoc, and
train-time losses with significant margins. Also, with our method, detection
performance improves or is competitive with better calibration scores. Over
baseline, we achieve an improvement of 5.4%$\downarrow$ (in-domain) and
7.6%$\downarrow$ (out-domain). Tab. 4 and Fig. 2 show further results.

Urban Scene: We show that our method improves calibration and/or competitive,
especially in the out-domain. For example, with MDCA [12], we improve
calibration by 5.0%$\downarrow$ (in-domain) and 2.8%$\downarrow$ (out-domain).
Detection performance with our method is also improved. With comparison to
[33], our method improves calibration by 1.4%$\downarrow$. For more results,
see Tab. 4.

Synthetic and Real: This scenario evaluates Sim10k as in-domain dataset and
BDD100k subset reflecting same category with Sim10k as out-domain dataset.
Over baseline, improvement of calibration for in-domain is 4.1%$\downarrow$
and for out-domain 1.0%$\downarrow$. Compared to MbLS [28], our method shows
better calibration with the improvement of 16.3%$\downarrow$ (in-domain) and
10.5%$\downarrow$ (out-domain). Tab. 5 shows further results.

Reliability diagrams: Fig. 3 plots reliability diagrams on selected MS-COCO
classes. It visualizes the D-ECE, and a perfect detection calibration means
that precision and confidence are aligned.

Table 6: Impact of Components: Calibration results on MS-COCO & CorCOCO along with the detection performance are shown component-wise. Logit Modulation (L${}_{\text{mod}}$) and Logit Mixing (L${}_{\text{mix}}$).  Methods | In-Domain (COCO) | Out-Domain (CorCOCO)
---|---|---
L${}_{\text{mod}}$ | L${}_{\text{mix}}$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
|  | 12.8 | 44.0 | 62.9 | 10.8 | 23.9 | 35.8
✓ |  | 11.0 | 44.1 | 62.8 | 9.9 | 23.6 | 35.5
| ✓ | 9.5 | 43.5 | 63.3 | 9.2 | 23.3 | 35.3
✓ | ✓ | 8.4 | 44.4 | 64.2 | 8.9 | 24.0 | 36.3
Table 7: Impact of logit mixing and loss weightage with Cityscapes split
(train/validation).

Cal-DETR | In-Domain
---|---
|  D-ECE $\downarrow$ | AP | AP0.5
$\alpha_{1}=0.9,~{}\alpha_{2}=0.1$ | 8.1 | 25.0 | 46.8
$\alpha_{1}=0.8,~{}\alpha_{2}=0.2$ | 8.0 | 24.6 | 46.5
$\alpha_{1}=0.7,~{}\alpha_{2}=0.3$ | 10.1 | 25.0 | 46.7
$\alpha_{1}=0.6,~{}\alpha_{2}=0.4$ | 10.3 | 24.3 | 46.0
(a) Impact of logit mixing $\alpha_{1}$ and $\alpha_{2}=1-\alpha_{1}$ values
in Cal-DETR. We observe improvement in detection performance by raising
$\alpha$ values. We see that calibration performance is competitive with high
$\alpha$ values.

Cal-DETR | In-Domain
---|---
|  D-ECE $\downarrow$ | AP | AP0.5
$\lambda=0.25$ | 10.4 | 24.6 | 46.1
$\lambda=0.325$ | 10.4 | 24.7 | 46.8
$\lambda=0.5$ | 8.1 | 25.0 | 46.8
$\lambda=0.75$ | 8.9 | 24.6 | 47.0
$\lambda=1.0$ | 8.7 | 24.4 | 46.4
(b) Loss weightage $\lambda$ values impact in Cal-DETR. We observe little
improvement in detection performance (AP) on $\lambda=0.5$ value. With the
same value, we see that calibration performance is competitive.

###  4.2 Ablations & Analysis

We study ablation and sensitivity experiments with our proposed method. This
section presents the impact of the components involved in our proposed method.
For the choice of hyperparameters and sensitivity experiments, we split the
training set of Cityscapes into train/validation sets (80%/20%).

Impact of Components: We separately investigate the function of logit
modulation (L${}_{\text{mod}}$) and logit mixing (L${}_{\text{mix}}$) to see
the impact on calibration performance. In Tab. 4.1, the calibration
performance is improved over the baseline while incorporating individual
components in our Cal-DETR. When we combine both, it further improves
calibration performance along with the detection performance. We believe that
this improvement could be due to greater diversity in logit space, since
beyond mixed logits, we have access to non-mixed logits as well.

Logit Mixing weightage: We study the logit mixing weightage used in our
proposed method on a separate validation set. Keeping the range of
$\alpha_{1}\in[0.6,0.9]$, we observe in the experiments that the best trade-
off of detection performance and model calibration is obtained at
$\alpha_{1}=0.9$. We notice that calibration performance is competitive for
higher $\alpha_{1}$ values but an overall improvement is observed for
$\alpha_{1}=0.9$. Tab. 7(a) shows the behavior of $\alpha$ in Cal-DETR.

Choice of single $\alpha$: Several mixup strategies operate to mix an input
image using another randomly selected image, so it usually involves two
samples in the process. In our approach, we perform a query instance/object
level mixup in the logit space. We first build a prototypical representation
using all positive queries which is then used to achieve a mixup for a given
positive query. Owing to this difference from conventional mixup strategies
operating, our experiment with conventional mixup leads to suboptimal results.
The subpar performance is potentially because the logit mixing suppresses the
object representation with lower arbitrary $\alpha$ simultaneously having
dominant prototypical representation from multiple positive queries. It is to
be noted that the sampling strategy (Cal-DETRS) still works better when
compared to other calibration methods. We report the results in the Tab. 8. We
empirically find $\beta$ hyperparameter on beta distribution using the split
of Cityscapes dataset. For our reported result, we choose $\beta$=0.3 using
this procedure.

Table 8: Choice of singe $\alpha$: Cal-DETR outperforms its variant with sampling strategy (Cal-DETRS) but it still works better when compared to other calibration methods. Methods Datasets | In-Domain (Cityscapes) | Out-Domain (Foggy Cityscapes)
---|---|---
| D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$ | D-ECE $\downarrow$ | AP $\uparrow$ | AP0.5 $\uparrow$
Baseline${}_{\textit{ D-DETR}}$ [53] | 13.8 | 26.8 | 49.5 | 19.5 | 17.3 | 29.3
TCD [33] | 11.9 | 28.3 | 50.8 | 14.3 | 17.6 | 30.3
Cal-DETRS (Ours) | 11.7 | 28.2 | 52.3 | 13.2 | 17.3 | 29.6
Cal-DETR (Ours) | 8.4 | 28.4 | 51.4 | 11.9 | 17.6 | 29.8

Regularizer Loss weightage: The impact of loss weightage is studied over the
validation set, which shows that $\lambda=0.5$ is a better choice to use with
the regularizer function. Detection performance is increased with this setting
along with the calibration performance. The trend shows that for a higher
value of $\lambda$, calibration is more competitive (see Tab. 7(b)).

Figure 4: D-UCE for MS-COCO evaluation set. Uncertainty calibration error is
reduced as compared to the baseline.

D-UCE complements D-ECE:  To investigate the role of uncertainty calibration
which complements confidence calibration, we analyze the detection expected
uncertainty calibration error (D-UCE) [24] in Fig. 4. The formulation is
described in the supplementary material.

Limitation:  Our approach could struggle with the model calibration for long-
tailed object detection and is a potential future direction.

##  5 Conclusion

In this paper, we tackled the problem of calibrating recent transformer-based
object detectors by introducing Cal-DETR. We proposed a new technique for
estimating the uncertainty in transformer-based detector, which is then
utilized to develop an uncertainty-guided logit modulation technique. To
further enhance calibration performance, we developed a new logit mixing
strategy which acts a regularizer on detections with task-specific loss.
Comprehensive experiments on several in-domain and out-domain settings
validate the effectiveness of our method against the existing train-time and
post-hoc calibration methods. We hope that our work will encourage new
research directions for calibrating object detectors.

## Acknowledgement

The computational resources were provided by the National Academic
Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the
Swedish Research Council through grant agreement no. 2022-06725, and by the
Berzelius resource, provided by the Knut and Alice Wallenberg Foundation at
the National Supercomputer Center.

## References

  * Blundell et al. [2015] Blundell, C., J. Cornebise, K. Kavukcuoglu, and D. Wierstra (2015).  Weight uncertainty in neural network.  In International Conference on Machine Learning, pp. 1613–1622.
  * Carion et al. [2020] Carion, N., F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020).  End-to-end object detection with transformers.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp. 213–229. Springer.
  * Chen et al. [2018] Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille (2018).  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40(4), 834–848.
  * Cordts et al. [2016] Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele (2016).  The cityscapes dataset for semantic urban scene understanding.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Dai et al. [2021] Dai, Z., B. Cai, Y. Lin, and J. Chen (2021).  Up-detr: Unsupervised pre-training for object detection with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1601–1610.
  * Dosovitskiy et al. [2021] Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021).  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations.
  * Dusenberry et al. [2020] Dusenberry, M. W., D. Tran, E. Choi, J. Kemp, J. Nixon, G. Jerfel, K. Heller, and A. M. Dai (2020).  Analyzing the role of model uncertainty for electronic health records.  In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204–213.
  * Gal and Ghahramani [2016] Gal, Y. and Z. Ghahramani (2016).  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pp. 1050–1059. PMLR.
  * Grigorescu et al. [2020] Grigorescu, S., B. Trasnea, T. Cocias, and G. Macesanu (2020).  A survey of deep learning techniques for autonomous driving.  Journal of Field Robotics 37(3), 362–386.
  * Guo et al. [2017] Guo, C., G. Pleiss, Y. Sun, and K. Q. Weinberger (2017).  On calibration of modern neural networks.  In International Conference on Machine Learning, pp. 1321–1330. PMLR.
  * He et al. [2016] He, K., X. Zhang, S. Ren, and J. Sun (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hebbalaguppe et al. [2022] Hebbalaguppe, R., J. Prakash, N. Madan, and C. Arora (2022, June).  A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16081–16090.
  * Hendrycks and Dietterich [2019] Hendrycks, D. and T. Dietterich (2019).  Benchmarking neural network robustness to common corruptions and perturbations.  International Conference on Learning Representations (ICLR).
  * Hernández-Lobato and Adams [2015] Hernández-Lobato, J. M. and R. Adams (2015).  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In International Conference on Machine Learning, pp. 1861–1869.
  * Islam et al. [2021] Islam, M., L. Seenivasan, H. Ren, and B. Glocker (2021).  Class-distribution-aware calibration for long-tailed visual recognition.  arXiv preprint arXiv:2109.05263.
  * Ji et al. [2019] Ji, B., H. Jung, J. Yoon, K. Kim, and Y. Shin (2019).  Bin-wise temperature scaling (bts): Improvement in confidence calibration performance through simple scaling techniques.  2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 4190–4196.
  * Johnson-Roberson et al. [2017] Johnson-Roberson, M., C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan (2017).  Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 746–753. IEEE.
  * Karandikar et al. [2021] Karandikar, A., N. Cain, D. Tran, B. Lakshminarayanan, J. Shlens, M. C. Mozer, and B. Roelofs (2021).  Soft calibration objectives for neural networks.  Advances in Neural Information Processing Systems 34, 29768–29779.
  * Krishnan and Tickoo [2020] Krishnan, R. and O. Tickoo (2020).  Improving model calibration with accuracy versus uncertainty optimization.  Advances in Neural Information Processing Systems.
  * Kull et al. [2019] Kull, M., M. Perello Nieto, M. Kängsepp, T. Silva Filho, H. Song, and P. Flach (2019).  Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.  Advances in neural information processing systems 32.
  * Kull et al. [2017] Kull, M., T. Silva Filho, and P. Flach (2017).  Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers.  In Artificial Intelligence and Statistics, pp. 623–631. PMLR.
  * Kuppers et al. [2020] Kuppers, F., J. Kronenberger, A. Shantia, and A. Haselhoff (2020).  Multivariate confidence calibration for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 326–327.
  * Lakshminarayanan et al. [2016] Lakshminarayanan, B., A. Pritzel, and C. Blundell (2016).  Simple and scalable predictive uncertainty estimation using deep ensembles.  arXiv preprint arXiv:1612.01474.
  * Laves et al. [2019] Laves, M.-H., S. Ihler, K.-P. Kortmann, and T. Ortmaier (2019).  Well-calibrated model uncertainty with temperature scaling for dropout variational inference.  arXiv preprint arXiv:1909.13550.
  * Liang et al. [2020] Liang, G., Y. Zhang, X. Wang, and N. Jacobs (2020).  Improved trainable calibration method for neural networks on medical imaging classification.  In British Machine Vision Conference (BMVC).
  * Lin et al. [2017] Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár (2017).  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pp. 2980–2988.
  * Lin et al. [2014] Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014).  Microsoft coco: Common objects in context.  In European conference on computer vision, pp. 740–755. Springer.
  * Liu et al. [2022] Liu, B., I. Ben Ayed, A. Galdran, and J. Dolz (2022, June).  The devil is in the margin: Margin-based label smoothing for network calibration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 80–88.
  * Liu et al. [2021] Liu, Z., Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021, October).  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022.
  * Louizos and Welling [2016] Louizos, C. and M. Welling (2016).  Structured and efficient variational deep learning with matrix gaussian posteriors.  In International Conference on Machine Learning, pp. 1708–1716.
  * Mukhoti et al. [2020] Mukhoti, J., V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania (2020).  Calibrating deep neural networks using focal loss.  Advances in Neural Information Processing Systems 33, 15288–15299.
  * Müller et al. [2019] Müller, R., S. Kornblith, and G. E. Hinton (2019).  When does label smoothing help?  Advances in neural information processing systems 32.
  * Munir et al. [2022] Munir, M. A., M. H. Khan, M. S. Sarfraz, and M. Ali (2022).  Towards improving calibration in object detection under domain shift.  In Advances in Neural Information Processing Systems.
  * Ovadia et al. [2019] Ovadia, Y., E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek (2019).  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems 32.
  * Pereyra et al. [2017] Pereyra, G., G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton (2017).  Regularizing neural networks by penalizing confident output distributions.  arXiv preprint arXiv:1701.06548.
  * Pinto et al. [2022] Pinto, F., H. Yang, S.-N. Lim, P. Torr, and P. K. Dokania (2022).  Using mixup as a regularizer can surprisingly improve accuracy & out-of-distribution robustness.  In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), Advances in Neural Information Processing Systems.
  * Platt et al. [1999] Platt, J. et al. (1999).  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  Advances in large margin classifiers 10(3), 61–74.
  * Ren et al. [2015] Ren, S., K. He, R. Girshick, and J. Sun (2015).  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pp. 91–99.
  * Sakaridis et al. [2018] Sakaridis, C., D. Dai, and L. Van Gool (2018).  Semantic foggy scene understanding with synthetic data.  International Journal of Computer Vision 126(9), 973–992.
  * Shao et al. [2019] Shao, S., Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun (2019).  Objects365: A large-scale, high-quality dataset for object detection.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8429–8438.
  * Sharma et al. [2017] Sharma, M., O. Saha, A. Sriraman, R. Hebbalaguppe, L. Vig, and S. Karande (2017).  Crowdsourcing for chromosome segmentation and deep classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 34–41.
  * Strudel et al. [2021] Strudel, R., R. Garcia, I. Laptev, and C. Schmid (2021).  Segmenter: Transformer for semantic segmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7262–7272.
  * Tian et al. [2019] Tian, Z., C. Shen, H. Chen, and T. He (2019).  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE international conference on computer vision, pp. 9627–9636.
  * Tolstikhin et al. [2021] Tolstikhin, I. O., N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy (2021).  Mlp-mixer: An all-mlp architecture for vision.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, Volume 34, pp. 24261–24272.
  * Tomani et al. [2021] Tomani, C., S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner (2021).  Post-hoc uncertainty calibration for domain drift scenarios.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124–10132.
  * Touvron et al. [2021] Touvron, H., M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021).  Training data-efficient image transformers & distillation through attention.  In International conference on machine learning, pp. 10347–10357. PMLR.
  * Xie et al. [2021] Xie, E., W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo (2021).  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in Neural Information Processing Systems 34, 12077–12090.
  * Yu et al. [2020] Yu, F., H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell (2020).  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636–2645.
  * Yu and Alì [2019] Yu, R. and G. S. Alì (2019).  What’s inside the black box? ai challenges for lawyers and researchers.  Legal Information Management 19(1), 2–13.
  * Zhang et al. [2018] Zhang, H., M. Cisse, Y. N. Dauphin, and D. Lopez-Paz (2018).  mixup: Beyond empirical risk minimization.  In International Conference on Learning Representations.
  * Zhang et al. [2023] Zhang, H., F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum (2023).  DINO: DETR with improved denoising anchor boxes for end-to-end object detection.  In The Eleventh International Conference on Learning Representations.
  * Zhang et al. [2021] Zhang, L., Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou (2021).  How does mixup help with robustness and generalization?  In International Conference on Learning Representations.
  * Zhu et al. [2021] Zhu, X., W. Su, L. Lu, B. Li, X. Wang, and J. Dai (2021).  Deformable {detr}: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations.