###  8.2.  Multilinear maximal singular integral operators

In this section, we will consider the weighted estimates of the commutators of
multilinear maximal operator $T^{*}$, which is defined as

$T^{*}(\vec{f})(x):=\sup_{\delta>0}\left|\int_{\sum_{i=1}^{m}\left|y_{i}-x\right|^{2}>\delta^{2}}K\left(x,y_{1},\ldots,y_{m}\right)f_{1}\left(y_{1}\right)\cdots
f_{m}\left(y_{m}\right)d\vec{y}\right|,$

for $x\notin\bigcap_{j=1}^{m}\operatorname{supp}f_{j}$ and each $f_{j}\in
L_{c}^{\infty}(j=1,\ldots,m)$, where $d\vec{y}=dy_{1}\cdots dy_{m}$ and
$K(x,\vec{y})$ is an $m$-linear Calderón-Zygmund kernel of type $\omega,$
which is defined in Subsection 8.1. In this subsection, we consider the
following commutators of multilinear maximal singular integral operators:

$T_{\vec{b}}^{*}(\vec{f})(x):=\sum_{j=1}^{m}\sup_{\delta>0}\left|\int_{\sum_{i=1}^{m}\left|y_{i}-x\right|^{2}>\delta^{2}}\left(b_{j}(x)-b_{j}\left(y_{j}\right)\right)K\left(x,\vec{y}\right)f_{1}\left(y_{1}\right)\cdots
f_{m}\left(y_{m}\right)d\vec{y}\right|.$

Now, we use [65, Theorems 1.8], Theorem 1.8-1.10 and Theorem 1.18 to conclude
the following results.

######  Theorem 8.4.

Let $T$ be an $m$-linear $\omega$-Calderón-Zygmund operator with $\omega\in$
$\log$-Dini$(1,0).$ If $\vec{b}\in\mathrm{BMO}^{m},$ then we have:

  1. (a).

Let $\vec{w}=\left(w_{1},\ldots,w_{m}\right)$ and
$u=\prod_{i=1}^{m}w_{i}^{1/m}$. If $\vec{w}\in A_{\vec{1}}$ and $v\in
A_{\infty},$ then there exists $t>1$ depending only on $v$, such that

$\left\|\frac{T^{*}_{\vec{b}}(\vec{f})}{v}\right\|_{L^{\frac{1}{m},\infty}(uv^{\frac{1}{m}})}\lesssim
K_{0}^{2+6m}[v^{\frac{1}{m}}]_{A_{t}}^{2+4m}\|\vec{b}\|_{\mathrm{BMO}}\left\|\frac{\mathcal{M}_{L(\log
L)}(\vec{f})}{v}\right\|_{L^{\frac{1}{m},\infty}(uv^{\frac{1}{m}})},$

where
$K_{0}=4C_{n}p_{0}p_{0}^{\prime}([u]_{A_{1}}+2^{p_{0}-1}C_{n}^{t}[v^{\frac{1}{m}}]_{A_{t}}^{2}[u]_{A_{1}}^{p_{0}-1})+1$
with $p_{0}=2^{n+3}(t-1)[u]_{A_{1}}+1$ and
$\|\vec{b}\|_{\mathrm{BMO}}:=\sup_{1\leq j\leq
m}\left\|b_{j}\right\|_{\mathrm{BMO}}.$

  2. (b).

For any $0<p<\infty,w\in A_{\infty},$

$\int_{\mathbb{R}^{n}}\left|T^{*}_{\vec{b}}(\vec{f})(x)\right|^{p}w(x)dx\lesssim\|\vec{b}\|_{\mathrm{BMO}}[w]_{A_{\infty}}^{p}[w]_{A_{\infty}}^{\max\\{2,p\\}}\int_{\mathbb{R}^{n}}\left(\mathcal{M}_{L(\log
L)}(\vec{f})(x)\right)^{p}w(x)dx,$

where $\|\vec{b}\|_{\mathrm{BMO}}:=\sup_{1\leq j\leq
m}\left\|b_{j}\right\|_{\mathrm{BMO}}.$

  3. (c).

Let $\phi$ be a $N$-function with sub-multiplicative property. For any
$1<r<\infty,$

    1. (1)

if $r<i_{\phi}<\infty,$ then there exists constant $\alpha$ such that for
every $1<q<\frac{i_{\phi}}{r}$ and $w\in A_{q},$

$\displaystyle\int_{\mathbb{R}^{n}}\phi\left(T^{*}_{\vec{b}}(\vec{f})(x)\right)w(x)dx\lesssim$
$\displaystyle[w]_{A_{\infty}}^{2\alpha
C_{1}+1}\|\vec{b}\|_{\mathrm{BMO}}^{1+\alpha C_{1}}$
$\displaystyle\times\left(\prod_{i=1}^{m}\int_{\mathbb{R}^{n}}\phi^{m}\left([w]_{A_{q}}^{\frac{1}{qr}}\left|f_{i}(x)\right|\right)w(x)dx\right)^{\frac{1}{m}};$

    2. (2)

if $1<i_{\phi}\leq r,$ then there exists constant $\alpha$ such that for every
$1<q<i_{\phi}$ and $w\in A_{q},$

$\displaystyle\int_{\mathbb{R}^{n}}\phi\left(T^{*}_{\vec{b}}(\vec{f})(x)\right)w(x)dx\lesssim$
$\displaystyle[w]_{A_{\infty}}^{2\alpha
C_{1}+mC_{1}+2}\|\vec{b}\|_{\mathrm{BMO}}^{1+\alpha C_{1}}$
$\displaystyle\times\left(\prod_{i=1}^{m}\int_{\mathbb{R}^{n}}\phi^{m}\left([w]_{A_{q}}^{\frac{2}{q}}\left|f_{i}(x)\right|\right)w(x)dx\right)^{\frac{1}{m}}.$

###  8.3.  Multilinear pseudo-differential operators

Given a function $\sigma$ on $\mathbb{R}^{n}\times\mathbb{R}^{nm}$, the
$m$-linear pseudo-differential operator $T_{\sigma}$ is defined by

$T_{\sigma}(\vec{f})(x):=\int_{(\mathbb{R}^{n})^{m}}\sigma(x,\vec{\xi})e^{2\pi
ix\cdot\left(\xi_{1}+\cdots+\xi_{m}\right)}\widehat{f_{1}}\left(\xi_{1}\right)\cdots\widehat{f_{m}}\left(\xi_{m}\right)d\vec{\xi}$

for all $f_{i}\in\mathscr{S}\left(\mathbb{R}^{n}\right),i=1,\ldots,m$, where
$d\vec{\xi}=d\xi_{1}\cdots d\xi_{m},$ and $\widehat{f}$ is the Fourier
transform of the function $f$ defined by

$\widehat{f}(\xi)=\int_{\mathbb{R}^{n}}f(x)e^{-2\pi ix\cdot\xi}dx.$

Given $m\in\mathbb{N},$ $r\in\mathbb{R}$ and $0\leq\rho,\delta\leq 1.$ We say
a smooth function $\sigma$ belongs to the Hörmander class
$S_{\rho,\delta}^{r}(n,m)$ if for each triple of multi-indices
$\alpha:=\left(\alpha_{1},\ldots,\alpha_{n}\right)$ and
$\beta_{1},\ldots,\beta_{m}$, there exists a constant $C_{\alpha,\beta}$ such
that

$\left|\partial_{x}^{\alpha}\partial_{\xi_{1}}^{\beta_{1}}\cdots\partial_{\xi_{m}}^{\beta_{m}}\sigma(x,\vec{\xi})\right|\leq
C_{\alpha,\beta}\left(1+\left|\xi_{1}\right|+\cdots+\left|\xi_{m}\right|\right)^{r-\rho\sum_{j=1}^{m}\left|\beta_{j}\right|+\delta|\alpha|}.$

It was shown in [63] that if $\sigma\in S_{\rho,\delta}^{r}(n,m)$, then the
iterated commutator of the pseudo-differential operator $T_{\sigma}$ is
weighted bounded from $L^{p_{1}}\times\cdots\times L^{p_{m}}$ to $L^{p}$ with
$1/p=1/p_{1}+\cdots+1/p_{m}$. Using Theorem 1.4, 1.8 and Theorem 1.18,
combined with Theorem 1.1 in [63], we know that both Hypothesis 1 and
Hypothesis 2 hold. Therefore, we obtain

######  Theorem 8.5.

Let $T_{\sigma}$ be an m-linear pseudo-differential operator, $\sigma\in
S_{\rho,\delta}^{r}(n$, $m)$ with $0\leq\rho,\delta\leq 1$ and $r<2n(\rho-1)$.
Let $I=\\{i_{1},\ldots,i_{l}\\}=\\{1,\ldots,l\\}\subseteq\\{1,\ldots,m\\}.$ If
$\vec{b}\in\mathrm{BMO}^{l},$ then the following hold:

  1. (a).

Let $Q_{0}$ be a cube and $f_{s}\in L_{c}^{\infty}\left(\mathbb{R}^{n}\right)$
such that $\operatorname{supp}\left(f_{s}\right)\subset Q_{0}$ for $1\leq
s\leq m.$ If $w\in A_{\infty},$ then there are constants $\alpha,c>0$
independent of $w$ such that

$\displaystyle w\left(\left\\{x\in
Q_{0}:\left|T_{\sigma,\vec{b}}(\vec{f})(x)\right|>t\mathcal{M}_{L(\log
L)}(\vec{f})(x)\right\\}\right)$ $\displaystyle\hskip 142.26378pt\qquad\leq
ce^{-{\frac{\alpha}{[w]_{A_{\infty}}}}\left({\frac{t}{\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}}}\right)^{\frac{1}{l+1}}}w(Q_{0}),\quad
t>0.$

  2. (b).

For any $0<p<\infty,w\in A_{\infty},$

$\int_{\mathbb{R}^{n}}\left|T_{\sigma,\vec{b}}(\vec{f})(x)\right|^{p}w(x)dx\lesssim\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}[w]_{A_{\infty}}^{pl}[w]_{A_{\infty}}^{\max\\{2,p\\}}\int_{\mathbb{R}^{n}}\left(\mathcal{M}_{L(\log
L)}(\vec{f})(x)\right)^{p}w(x)dx.$

  3. (c).

Let $\phi$ be a $N$-function with sub-multiplicative property. For any
$1<r<\infty,$

    1. (1)

if $r<i_{\phi}<\infty,$ then there exists constant $\alpha$ such that for
every $1<q<\frac{i_{\phi}}{r}$ and $w\in A_{q},$

$\displaystyle\int_{\mathbb{R}^{n}}\phi\left(T_{\sigma,\vec{b}}(\vec{f})(x)\right)w(x)dx\lesssim$
$\displaystyle[w]_{A_{\infty}}^{(l+1)(\alpha
C_{1}+1)}\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}^{1+\alpha C_{1}}$
$\displaystyle\times\left(\prod_{i=1}^{m}\int_{\mathbb{R}^{n}}\phi^{m}\left([w]_{A_{q}}^{\frac{1}{qr}}\left|f_{i}(x)\right|\right)w(x)dx\right)^{\frac{1}{m}};$

    2. (2)

if $1<i_{\phi}\leq r,$ then there exists constant $\alpha$ such that for every
$1<q<i_{\phi}$ and $w\in A_{q},$

$\displaystyle\int_{\mathbb{R}^{n}}\phi\left(T_{\sigma,\vec{b}}(\vec{f})(x)\right)w(x)dx\lesssim$
$\displaystyle[w]_{A_{\infty}}^{(l+1)(\alpha
C_{1}+1)+1+mC_{1}}\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}^{1+\alpha C_{1}}$
$\displaystyle\times\left(\prod_{i=1}^{m}\int_{\mathbb{R}^{n}}\phi^{m}\left([w]_{A_{q}}^{\frac{2}{q}}\left|f_{i}(x)\right|\right)w(x)dx\right)^{\frac{1}{m}}.$

###  8.4.  Higher order Calderón commutators

In this subsection, we apply our results to derive the quantitative weak
$A_{\infty}$ decay estimates, mixed weak type estimates and Fefferman-Stein
inequalities with arbitrary weights of higher order Calderón commutators on
$\mathbb{R}$. These operators and its higher-order counterpart first appeared
in the investigation of Cauchy integrals along Lipschitz curves, in the proof
of the $L^{2}$ boundedness of the latter.

For our purpose, we first present some definitions. Given functions
$A_{1},\ldots,A_{m}$ defined on $\mathbb{R},$ let
$a_{j}=\frac{dA_{j}}{dt},j=1,\ldots,m$. The higher order Calderón commutators
are defined by

$\mathcal{C}_{m+1}\left(a_{1},\ldots,a_{m},f\right)(x):=\text{ p.v.
}\int_{\mathbb{R}}\frac{\prod_{j=1}^{m}\left(A_{j}(x)-A_{j}(y)\right)}{(x-y)^{m}}f(y)dy.$

Using the method in [22, p. 2106], we can rewrite $\mathcal{C}_{m+1}$ in the
form of the multilinear singular integral as follows.

$\mathcal{C}_{m+1}\left(a_{1},\ldots,a_{m},f\right)(x)=\int_{\mathbb{R}^{m+1}}K(x,y_{1},\ldots,y_{m+1})f\left(y_{m+1}\right)\prod_{j=1}^{m}a_{j}\left(y_{j}\right)dy_{1}\cdots
dy_{m+1},$

where the kernel

(8.1)
$K(x,y_{1},\ldots,y_{m+1})=\frac{(-1)^{me\left(y_{m+1}-x\right)}}{\left(x-y_{m+1}\right)^{m+1}}\prod_{j=1}^{m}\chi_{\left(x\wedge
y_{m+1},x\vee y_{m+1}\right)}\left(y_{j}\right)$

with $x\wedge y=\min\\{x,y\\}$, $x\vee y=\max\\{x,y\\},$ and

$e(x)=\begin{cases}1,&x>0,\\\ 0,&x<0.\end{cases}$

Whenever $\left|x-x^{\prime}\right|\leq\frac{1}{8}\min_{1\leq j\leq
m+1}\left|x-y_{j}\right|$, it was shown in [29] that

$|K(x,y_{1},\ldots,y_{m+1})|\lesssim\frac{1}{\left(\sum_{j=1}^{m+1}\left|x-y_{j}\right|\right)^{m+1}}$

and

$\left|K(x,y_{1},\ldots,y_{m+1})-K(x^{\prime},y_{1},\ldots,y_{m+1})\right|\lesssim\frac{\left|x-x^{\prime}\right|}{\left(\sum_{j=1}^{m+1}\left|x-y_{j}\right|\right)^{m+2}}.$

It is natural to generalize $\mathcal{C}_{m+1}$ to multilinear version as
following.

(8.2)
$\mathscr{C}(\vec{f})(x):=\int_{\mathbb{R}^{m+1}}K(x,y_{1},\ldots,y_{m+1})\prod_{j=1}^{m+1}f_{j}\left(y_{j}\right)dy_{1}\cdots
dy_{m+1},$

where the kernel $K$ is given in (8.1).

With [8, Theorem 2.24 ] in hand, both Hypothesis 1 and Hypothesis 2 hold. One
can obtain the following results by Theorems 1.4, 1.10 and 1.14.

######  Theorem 8.6.

Let $\mathscr{C}$ be the operator in (8.2) with the kernel $K$ given by (8.1).
If $I=\\{i_{1},\ldots,i_{l}\\}=\\{1,\ldots,l\\}\subseteq\\{1,\ldots,m+1\\},$
$\vec{b}\in\mathrm{BMO}^{l},$ then the following statement are true

  1. (a).

Let $w\in A_{\infty}^{\text{weak}},$ $Q_{0}$ be an interval and $f_{s}\in
L_{c}^{\infty}\left(\mathbb{R}\right)$ such that
$\operatorname{supp}\left(f_{s}\right)\subset Q_{0}$ for $1\leq s\leq m+1.$
Then there are constants $\alpha,c>0$ independent of $w$ such that

$\displaystyle w\left(\left\\{x\in
Q_{0}:\left|\mathscr{C}_{\vec{b}}(\vec{f})(x)\right|>t\mathcal{M}_{L(\log
L)}(\vec{f})(x)\right\\}\right)$ $\displaystyle\hskip 142.26378pt\qquad\leq
ce^{-{\frac{\alpha}{[w]_{A_{\infty}}^{weak}+1}}\left({\frac{t}{\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}}}\right)^{\frac{1}{l+1}}}w(2Q_{0}),\quad
t>0.$

  2. (b).

Let $\vec{w}=\left(w_{1},\ldots,w_{m+1}\right)$ and
$u=\prod_{i=1}^{m+1}w_{i}^{1/(m+1)}$. If $\vec{w}\in A_{\vec{1}}$ and $v\in
A_{\infty},$ then there exists $t>1$ depending only on $v$, such that

$\left\|\frac{\mathscr{C}_{\vec{b}}(\vec{f})}{v}\right\|_{L^{\frac{1}{m+1},\infty}(uv^{\frac{1}{m+1}})}\lesssim\mathcal{K}\prod_{s=1}^{l}\|b_{s}\|_{\mathrm{BMO}}\left\|\frac{\mathcal{M}_{L(\log
L)}(\vec{f})}{v}\right\|_{L^{\frac{1}{m+1},\infty}(uv^{\frac{1}{m+1}})},$

where $\mathcal{K}=K_{0}^{2l+6(m+1)}[v^{\frac{1}{m+1}}]_{A_{t}}^{2l+4(m+1)}$
with

$K_{0}=Cp_{0}p_{0}^{\prime}([u]_{A_{1}}+2^{p_{0}-1}C^{t}[v^{\frac{1}{m+1}}]_{A_{t}}^{2}[u]_{A_{1}}^{p_{0}-1})+1,p_{0}=16(t-1)[u]_{A_{1}}+1.$

  3. (c).

Let $1<p_{1},\ldots,p_{m+1}<\infty$ and
$\frac{1}{p}=\frac{1}{p_{1}}+\cdots+\frac{1}{p_{m+1}}.$ Assume that $m\geq 1$
and for all weights
$\vec{w}=\left(w_{1},\ldots,w_{m+1}\right),\nu_{\vec{w}}=\prod_{s=1}^{m+1}w_{s}^{p/p_{s}},$
$b_{s}\in\mathrm{BMO}_{p_{s}}(w_{s})\cap\mathrm{BMO}$ with $1\leq s\leq l.$ If
$0<p\leq 1,$ then

$\left\|\mathscr{C}_{\vec{b}}(\vec{f})\right\|_{L^{p}\left(\nu_{\vec{w}}\right)}\leq
C\|\vec{b}\|_{\mathrm{BMO}}^{*}\prod_{s=1}^{m+1}\left\|f_{s}\right\|_{L^{p_{s}}(Mw_{s})},$

where $C$ is independent of $\vec{w}$ and $\vec{b},$ and

$\|\vec{b}\|_{\mathrm{BMO}}^{*}=\max_{\vec{\gamma}\in\\{1,2\\}^{l}}\\{\prod_{s:\gamma_{s}=1}\|b_{s}\|_{\mathrm{BMO}_{p_{s}}(w_{s})}\prod_{s:\gamma_{s}=2}\|b_{s}\|_{\mathrm{BMO}}\\}.$

###  8.5.  Stein’s square functions

The Stein’s square function $G_{\alpha}$ is defined by

$G_{\alpha}f(x)=\left(\int_{0}^{\infty}\left|\frac{\partial}{\partial
t}B_{\alpha}^{t}f(x)\right|^{2}tdt\right)^{1/2},\qquad\hbox{for }\alpha>0,$

where $B_{\alpha}^{t}$ is the Bochner-Riesz multiplier
$\widehat{B_{\alpha}^{t}f}(\xi)=\left(1-\frac{|\xi|^{2}}{t^{2}}\right)_{+}^{\alpha}\widehat{f}(\xi).$
A simple calculation gives that

$\frac{\partial}{\partial
t}B_{\alpha}^{t}f(x)=\frac{2\alpha}{t}\int_{\mathbb{R}^{n}}\frac{|\xi|^{2}}{t^{2}}\left(1-\frac{|\xi|^{2}}{t^{2}}\right)_{+}^{\alpha-1}\widehat{f}(\xi)e^{2\pi
ix\xi}d\xi.$

Let
$\widehat{K_{t}^{\alpha}}(\xi)=\frac{|\xi|^{2}}{t^{2}}\left(1-\frac{|\xi|^{2}}{t^{2}}\right)_{+}^{\alpha-1}$.
Then, $G_{\alpha}$ can be rewritten as

$G_{\alpha}f(x)=\left(\int_{0}^{\infty}\left|K_{t}^{\alpha}*f(x)\right|^{2}\frac{dt}{t}\right)^{1/2}.$

The function $G_{\alpha}$ was first introduced by Stein [60] to study $L^{2}$
properties of the maximal Bochner-Riesz operator and deduce almost everywhere
convergence for Bochner-Riesz means of Fourier series. Invoking [10, Theorem
1.1] and Theorems 1.10 and 1.18, we know Hypothesis 2 hold and thus we may
obtain the following results.

######  Theorem 8.7.

Let $\alpha>\frac{n+1}{2},$ then the following hold:

  1. (a).

If $w\in A_{1}$ and $v\in A_{\infty}$, then

$\left\|\frac{G_{\alpha}f}{v}\right\|_{L^{1,\infty}\left(wv\right)}\lesssim\left\|f\right\|_{L^{1}\left(w\right)}.$

  2. (b).

Let $\phi$ be an $N$-function belonging to $\Delta_{2}$ and $w\in
A_{i_{\phi}}$. If $i_{\phi}>1$, then there exists $C_{0}>0$ such that

$\int_{\mathbb{R}^{n}}\phi\left(\left|G_{\alpha}f(x)\right|\right)w(x)dx\lesssim
C(\phi,w)\int_{\mathbb{R}^{n}}\phi(|f(x)|)w(x)dx,$

where

$C(\phi,w)=\begin{cases}[w]_{A_{\infty}}^{1+\alpha
C_{1}},&C_{0}[w]_{A_{i_{\phi}}}^{1/i_{\phi}}<2,\\\ [w]_{A_{\infty}}^{1+\alpha
C_{1}}\left([w]_{A_{i_{\phi}}}^{1/i_{\phi}}\right)^{C_{1}},&C_{0}[w]_{A_{i_{\phi}}}^{1/i_{\phi}}\geq
2.\end{cases}$

## References

  * [1] T. C. Anderson and B. Hu, A unified method for maximal truncated Calderón-Zygmund operators in general function spaces by sparse domination, Proc. Edinb. Math. Soc. 63(2) (2020) 229-247.
  * [2] T. C. Anderson, T. Hytönen and O. Tapiola, Weak $A_{\infty}$ weights and weak reverse Hölder property in a space of homogeneous type, Journal of Geometric Analysis 27 (2017), 95-119.
  * [3] C. Benea, F. Bernicot and T. Luque, Sparse bilinear forms for Bochner Riesz multipliers and applications, Trans. Lond. Math. Soc. 4 (2017), 110–128.
  * [4] Á. Bényi, J. M. Martell, K. Moen, E. Stachura and R. H. Torres, Boundedness results for commutators with BMO functions via weighted estimates: a comprehensive approach, Math. Ann. 376 (2020), 61-102.
  * [5] F. Bernicot, D. Frey and S. Petermichl, Sharp weighted norm estimates beyond Calderón-Zygmund theory, Anal. PDE 9 (2016), 1079-1113.
  * [6] B. Bojarski and T. Iwaniec, $p$-harmonic equation and quasiregular mappings, in Partial Differential Equations (Warsaw, 1984), Banach Center Publications, Vol. 19, PWN, Warsaw, 1987, pp. 25-38.
  * [7] S. M. Buckley, Estimates for operator norms on weighted spaces and reverse Jensen inequalities, Trans. Am.Math.Soc.340(1) (1993), 253–272.
  * [8] M. Cao, G. Ibanez-Firnkorn, I. Rivera-Rios, Q. Xue and K. Yabuta, A class of multilinear bounded oscillation operators on measure spaces and applications, Math. Ann. to appear.
  * [9] M. Cao, Q. Xue and K. Yabuta, Weak and strong type estimates for the multilinear pseudo-differential operators, J. Funct. Anal. 278 (2020), no. 10, 108454, 46 pp.
  * [10] M. J. Carro, C. Domingo-Salazar, Stein’s square function $G_{\alpha}$ and sparse operators, J. Geom. Anal. 27(2) (2017), 1624-1635.
  * [11] M. J. Carro, C. Pérez, F. Soria, J. Soria, Maximal functions and the control of weighted inequalities for the fractional integral operator, Indiana Univ. Math. J. 54 (2005), no. 3, 627–644.
  * [12] S. Chanillo and R. L. Wheeden, Some weighted norm inequalities for the area integral, Indiana Univ. Math. J. 36 (1987), no. 2, 277–294.
  * [13] S. Choudhary, K. Jotsaroop, S. Shrivastava and K. Shuin, Bilinear Bochner–Riesz Square Function and Applications, J Fourier Anal Appl 29, 63 (2023), https://doi.org/10.1007/s00041-023-10049-9.
  * [14] R. R. Coifman, C. Fefferman, Weighted norm inequalities for maximal functions and singular integrals, Studia Math. 51 (1974), 241-250.
  * [15] J. M. Conde-Alonso, A. Culiuc, F. Di Plinio and Y. Ou, A sparse domination principle for rough singular integrals, Anal. PDE. 10(5) (2017) 1255-1284.
  * [16] J. M. Conde-Alonso and G. Rey, A pointwise estimate for positive dyadic shifts and some applications, Math. Ann. 365(3–4) (2016), 1111-1135.
  * [17] D. Cruz-Uribe, J. M. Martell, C. Pérez, Weighted weak-type inequalities and a conjecture of Sawyer, Int. Math. Res. Not. 30 (2005) 1849-1871.
  * [18] D. Cruz-Uribe, J. M. Martell, C. Pérez, Sharp weighted estimates for classical operators, Adv. Math. 229(1) (2012) 408–441.
  * [19] G. Curbera, J.Cuerva, J. Martell and C. Pérez, Extrapolation with weights, rearrangement invariant function spaces, modular inequalities and applications to singular integrals, Adv. Math. 203 (2006) 256-318.
  * [20] W. Damián, A.K. Lerner, C. Pérez, Sharp weighted bounds for multilinear maximal functions and Calderón-Zygmund operators, J. Fourier Anal. Appl. 21 (2015), no.1, 161-181.
  * [21] Y. Ding, X. Lai, Weak type $(1,1)$ bound criterion for singular integrals with rough kernel and its applications, Trans. Amer. Math. Soc. 371 (2019), no. 3, 1649-1675.
  * [22] X. Duong, L. Grafakos and L. Yan, Multilinear operators with non-smooth kernels and commutators of singular integrals, Trans. Amer. Math. Soc. 362 (2010), 2089-2113.
  * [23] C. Fefferman and E. M. Stein, Some maximal inequalities, Amer. J. Math. 93 (1971), 107-115.
  * [24] N. Fujii, Weighted bounded mean oscillation and singular integrals, Math. Jpn. 22 (1977), no. 5, 529–534.
  * [25] L. Grafakos, Classical Fourier Analysis, GTM249, 3rd Edition, Springer, New York, 2014.
  * [26] L. Grafakos, Modern Fourier Analysis, GTM250, 3rd Edition, Springer, New York, 2014.
  * [27] L. Grafakos and R. H. Torres, Multilinear Calderón–Zygmund theory, Adv. Math. 165 (2002), 124–164.
  * [28] C. Hoang and K. Moen, Muckenhoupt-Wheeden conjectures for sparse operators, Arch. Math. (Basel) 109 (2017), no. 1, 49-58.
  * [29] G. Hu and Y. Zhu, Weighted norm inequalities with general weights for the commutator of Calderón, Acta Math. Sin. Engl. Ser. 29 (2013), 505-514.
  * [30] T. Hytönen, The sharp weighted bound for general Calderón-Zygmund operators, Ann. Math. 175 (2012), 1473-1506.
  * [31] G. H. Ibáñez-Firnkorn and I. P. Rivera-Ríos, Sparse and weighted estimates for generalized Hörmander operators and commutators, Monatsh. Math. 191(1) (2020) 125-173.
  * [32] G. H. Ibáñez-Firnkorn and I. P. Rivera-Ríos, Mixed weak type inequalities in Euclidean spaces and in spaces of homogeneous type, Math. Nachr. (2023), 1-37.
  * [33] G. A. Karagulyan, Exponential estimates for the Calderón-Zygmund operator and related problems of Fourier series, Mat. Zametki 71(3) (2002), 398–411.
  * [34] V. Kokilashvili, M. Krbec, Weighted Inequalities in Lorentz and Orlicz Spaces, World Scientific, Singapore, 1991.
  * [35] A. K. Lerner, A simple proof of the $A_{2}$ conjecture, Int. Math. Res. Not. 24(14) (2013) 3159-3170.
  * [36] A. K. Lerner, On an estimate of Calderón-Zygmund operators by dyadic positive operators, J. Anal. Math. 121 (2013) 141-161.
  * [37] A. K. Lerner, F. Nazarov, Intuitive dyadic calculus: the basics, Expo. Math. 37(3), (2019) 225-265.
  * [38] A. K. Lerner, S. Ombrosi, C. Pérez, R. H. Torres and R. Trujillo-González, New maximal functions and multiple weights for the multilinear Calderón-Zygmund theory, Adv. Math. 220 (2009), 1222-1264.
  * [39] A. K. Lerner, S. Ombrosi, I. P. Rivera-Ríos, On pointwise and weighted estimates for commutators of Calderón-Zygmund operators, Adv. Math. 319 (2017), 153-181.
  * [40] K. Li, Sparse domination theorem for multilinear singular integral operators with $L^{r}$-Hörmander condition, Michigan Math. J. 67(2) (2018), 253–265.
  * [41] K. Li, S. Ombrosi, B. Picardi, Weighted mixed weak-type inequalities for multilinear operators, Stud. Math. 244 (2019) 203-215.
  * [42] G. Lu, P. Zhang, Multilinear Calderón-Zygmund operators with kernels of Dini’s type and applications, Nonlinear Analysis, 107 (2014), 92-117.
  * [43] B. Muckenhoupt, R. L. Wheeden, Some weighted weak-type inequalities for the Hardy-Littlewood maximal function and the Hilbert transform, Indiana Univ. Math. J. 26(5) (1977), 801–816.
  * [44] S. Ombrosi, C. Pérez, E. Rela, I. P. Rivera-Ríos, A note on generalized Fujii-Wilson conditions and BMO spaces, Israel J. Math. 238(2) (2020), 571–591.
  * [45] C. Ortiz-Caraballo, C. Pérez, E. Rela, Exponential decay estimates for singular integral operators, Math. Ann. 357 (2013), 1217-1243.
  * [46] C. Pérez, Endpoint estimates for commutators of singular integral operators, J. Funct. Anal. 128 (1995) 163-185.
  * [47] C. Pérez, Sharp estimates for commutators of singular integrals via iterations of the Hardy-Littlewood maximal function, J. Fourier Anal. Appl. 3 (1997) 743-756.
  * [48] C. Pérez, Weighted norm inequalities for singular integral operators, J. London Math. Soc. (2) 49 (1994), no. 2, 296-308.
  * [49] C. Pérez, A Course on Singular Integrals and Weights, Advanced Courses in Mathematics CRM Barcelona. Birkhäuser, Basel (2013).
  * [50] C. Pérez, G. Pradolini, R. H. Torres and R. Trujillo-González, End-point estimates for iterated commutators of multilinears ingular integrals, Bull. Lond. Math. Soc., 46 (2014), 26-42.
  * [51] C. Pérez, I. P. Rivera-Ríos, Borderline weighted estimates for commutators of singular integrals, Isr. J. Math. 217 (2017) 435-475.
  * [52] C. Pérez, I. P. Rivera-Ríos, Three observations on commutators of singular integral operators with BMO functions, Harmonic analysis, partial differential equations, Banach spaces, and operator theory. Vol. 2, 287-304, Assoc. Women Math. Ser., 5, Springer, Cham (2017).
  * [53] C. Pérez, E. Roure-Perdices, Sawyer-type inequalities for Lorentz spaces, Math. Ann. 383 (2022), 493-528.
  * [54] C. Pérez, R. H. Torres, Sharp maximal function estimates for multilinear singular integrals, Contemp. Math. 320 (2003), 323-331.
  * [55] R. Rahm, Borderline weak-type estimates for sparse bilinear forms involving $A_{\infty}$ maximal functions, J. Math. Anal. Appl. 504 (2021), no. 1, Paper No. 125372, 10 pp.
  * [56] M. M. Rao, Z. Ren, Theory of Orlicz Spaces, volume 146 of Monographs and Textbooks in Pure and Applied Mathematics. Marcel Dekker, Inc., New York, 1991
  * [57] M. C. Reguera, C. Thiele, The Hilbert transform does not map $L^{1}(Mw)$ to $L^{1,\infty}(w)$, Math. Res. Lett. 19 (1) (2012), 1-7.
  * [58] E. Sawyer, A weighted weak type inequality for the maximal function, Proc. Am. Math. Soc. 93(4) (1985), 610-614.
  * [59] E. T. Sawyer, Two weight norm inequalities for certain maximal and integral operators, in Harmonic Analysis (Minneapolis, MN, 1981), Lecture Notes in Mathematics, Vol. 908, Springer, Berlin–New York, 1982, pp. 102-127.
  * [60] E. M. Stein, Localization and summability of multiple Fourier series, Acta Math. 100 (1958), 92–147.
  * [61] J. Tan, Q. Xue, Weighted variation inequalities for singular integrals and commutators in rearrangement invariant Banach and quasi-Banach spaces, Acta Math. Sin. (Engl. Ser.) 39 (2023), no. 7, 1389-1413.
  * [62] J. Tan, Q. Xue, Quantitative weighted estimates for the multilinear pseudo-differential operators in function spaces, submitted, https://doi.org/10.48550/arXiv.2312.08938.
  * [63] Y. Wen, H. Wu and Q. Xue, A note on multilinear pseudo-differential operators and iterated commutators, Bull. Korean Math. Soc. 57(4) (2020), 851–864.
  * [64] Y. Wen, H. Wu and Q. Xue, Sparse domination and weighted inequalities for the $\rho$-variation of singular integrals and commutators, J. Geom. Anal. 32 (2022), no. 12, 30 pp.
  * [65] C. Zhang, Two weighted estimates for the commutators of multilinear maximal singular integral operators with Dini-type kernels, Acta Math. Sin. (Engl. Ser.) 39 (2023), no. 3, 459-480.


# Segment Anything Model (SAM) Enhances Pseudo- Labels for Weakly Supervised
Semantic Segmentation

Tianle Chen 1 Zheda Mai 111footnotemark: 1 Ruiwen Li2 Wei-lun Chao 1
1The Ohio State University
2EAIGLE Inc
(mai.145<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contributions.

###### Abstract

Weakly supervised semantic segmentation (WSSS) aims to bypass the need for
laborious pixel-level annotation by using only image-level annotation. Most
existing methods rely on Class Activation Maps (CAM) to derive pixel-level
pseudo-labels and use them to train a fully supervised semantic segmentation
model. Although these pseudo-labels are class-aware, indicating the coarse
regions for particular classes, they are not object-aware and fail to
delineate accurate object boundaries. To address this, we introduce a simple
yet effective method harnessing the Segment Anything Model (SAM), a class-
agnostic foundation model capable of producing fine-grained instance masks of
objects, parts, and subparts. We use CAM pseudo-labels as cues to select and
combine SAM masks, resulting in high-quality pseudo-labels that are both
class-aware and object-aware. Our approach is highly versatile and can be
easily integrated into existing WSSS methods without any modification. Despite
its simplicity, our approach shows consistent gain over the state-of-the-art
WSSS methods on both PASCAL VOC and MS-COCO datasets.

##  1 Introduction

Semantic segmentation, a task aiming to assign a semantic label to each image
pixel [38], has found wide applications in various fields, such as medical
imaging [5], remote sensing [53] and autonomous driving [17]. The success of
deep learning techniques and the availability of large-scale pixel-level
annotations have greatly boosted the performance of semantic segmentation in
recent years [27]. However, acquiring pixel-level annotations is daunting due
to its laborious and costly nature. As an alternative, weakly supervised
semantic segmentation (WSSS) seeks to train a segmentation model with cheaper
yet weaker annotations such as bounding boxes [39, 29, 44, 45], scribbles
[34], points [24, 6], and image-level class labels [28, 55, 19, 48]. Among
existing approaches, image-level WSSS has gained widespread popularity due to
the abundance of image-level annotations online or in various vision datasets
[41, 36] and the availability of strong pre-trained classifiers [18, 42].

As image-level labels do not provide location information for each object
class, most of the existing WSSS methods leverage Class Activation Maps (CAM)
[56] to derive location cues. These approaches typically follow a four-stage
learning process. First, they train a classification model with image-level
labels. Then, based on the intermediate feature maps and their weights to a
class, CAMs are generated as the coarse estimate of the class location.
Subsequently, the initial CAMs are refined with post-processing techniques,
such as pixel affinity-based methods [2, 33] or saliency guidance [30, 7, 52],
to create pixel-level pseudo-labels. Finally, a semantic segmentation model
[10, 8] is trained using the pseudo-labels as pixel-level supervision. The
efficacy of WSSS greatly relies on the accuracy of pseudo-labels. However, it
is widely recognized that the CAM-derived pseudo-labels often suffer from
_partial activation_ [3, 47], activating the most discriminative region
instead of the entire object area, and _false activation_ [50, 21], wrongly
activating the background around the object. (Figure 1 shows partial and false
activation examples.) _In other words, CAM-derived pseudo-labels lack the
awareness of objects, resulting in poor contours that drastically deviate from
object boundaries._

Figure 1: Illustration of how SAM addresses partial and false activation on
PASCAL VOC 2012 train set: (A) original images; (B) pseudo-labels generated by
a SOTA image-level WSSS method, CLIMS [50]; (C) masks from SAM; (D) SAM
enhanced pseudo-labels; (E) ground-truth labels.

In this paper, we investigate a novel approach to addressing this issue,
directly incorporating object boundary information into pseudo-label
generation. Concretely, we leverage the advent of segmentation foundation
models [46, 25, 59], in particular, the Segment Anything Model (SAM) [25],
which is capable of producing fine-grained, class-agnostic masks of objects,
parts, or subparts. We hypothesize that the quality of the resulting pseudo-
labels can be greatly enhanced by appropriately integrating the coarse class
location from CAM and the object boundary information from SAM.

To this end, we propose SAM Enhanced pseudo-labels (SEPL). SEPL uses CAM-
derived pseudo-labels for a particular class as the seed signals to select the
most relevant masks from SAM. The union of these masks, which encompass both
class and object information, is then treated as the enhanced pseudo-labels
for training semantic segmentation models. More specifically, SEPL consists of
two stages: mask assignment and mask selection (see Figure 2). During mask
assignment, each SAM mask is assigned to the class (of the image-level
annotation) whose CAM-derived pseudo-labels have the largest intersection with
the mask. During mask selection, SAM masks with substantial overlap with the
CAM-derived pseudo-labels are chosen to address _false activation_ , given
that background masks typically manifest minimal overlap. Meanwhile, we also
select SAM masks that substantially encompass the CAM-derived pseudo-labels,
targeting the challenge of _partial activation_. Given the precise alignment
of SAM masks to object boundaries, we find substantial enhancements in
mitigating partial and false activations in the existing pseudo-labels, as
depicted in Figure 1.

SEPL is remarkably versatile as it can be seamlessly integrated into existing
WSSS methods without modifying the original methods. Despite its simplicity,
SEPL achieves a notable improvement in the mean Intersection over Union (mIoU)
of pseudo-labels and ground-truth labels compared to eleven state-of-the-art
WSSS methods, with an average gain of 5.33% and 3.12% on the train set of
PASCAL VOC 2012 dataset [15] and MS COCO 2014 [36], respectively. As far as we
know, this is the first study to investigate the potential of SAM in the
context of WSSS. We hope this work will pave the way for applying segmentation
foundation models in diverse computer vision applications.

Related work. Due to the page limit, we leave it in Appendix A.

##  2 SAM Enhanced Pseudo Labels

###  2.1 Preliminary

Following the standard setup, each training image $\bm{X}\in{H\times W\times
C}$ is associated with only an image-level label vector
$\bm{y}=\left[y_{1},y_{2},\ldots,y_{K}\right]^{\top}\in\\{0,1\\}^{K}$ for $K$
classes, where $y_{k}=1$ indicates the presence of class $k$ in $\bm{X}$ and 0
otherwise. Upon training a classifier $f$ with this dataset, WSSS methods feed
an image to $f$ and obtain the Class Activation Maps (CAM)
$\mathcal{M}=\left[\bm{M}_{1},\cdots\bm{M}_{K}\right]$, where $\bm{M}_{k}\in
R^{H\times W}$ highlights the discriminative image regions utilized by $f$ to
identify class $k$. Post-processing techniques, such as AffinityNet [3] and
IRNet [1], further refine $\bm{M}_{k}$ to produce pseudo-labels
$\bm{P}\in\\{0,1,\cdots,K\\}^{H\times W}$, where pixel is mapped to either a
class label in $\\{1,\cdots,K\\}$ or 0 for background regions.
$\bm{P}_{k}\in\\{0,k\\}^{H\times W}$ represents the pseudo-labels for class
$k$. Subsequently, a fully supervised semantic segmentation network (e.g., [9,
8]) is trained with $\bm{P}$. SAM takes an image $\bm{X}\in{H\times W\times
C}$ as input and returns a list of masks capturing either a subpart, a part or
an entire object:
$\mathcal{S}=\left[\bm{S}_{0},\bm{S}_{1},\cdots,\bm{S}_{L}\right]$ where
$\bm{S}_{l}\in\\{0,1\\}^{H\times W}$ and $L$ is the number of masks. It is
noteworthy that different images may receive various mask quantities from SAM.

Figure 2: Illustration of the SEPL pipeline. SEPL comprises two stages, mask
assignment and mask selection. Based on the intersection between each SAM mask
and pseudo-labels, a mask is assigned to the class with the largest
intersection. For each mask, two metrics are computed: $o_{s}$, the fraction
of the mask overlapped by pseudo-labels, and $o_{p}$, the fraction of pseudo-
labels overlapped by the mask. A mask is retained as an enhanced pseudo-label
if either metric surpasses the designated threshold.

###  2.2 Approach

While CAM-derived pseudo-labels are class-aware and identify discriminative
regions for individual classes, they often fail to delineate accurate object
boundaries. In contrast, SAM is able to precisely segment most parts or
objects in a class-agnostic manner. Bridging their capabilities, we propose
SAM Enhanced Pseudo Labels (SEPL) to harness the potential of SAM for pseudo-
label enhancement.

Our approach takes pseudo-labels $[\bm{P}_{1},\cdots,\bm{P}_{K}]$ and SAM
masks $[\bm{S}_{0},\cdots,\bm{S}_{L}]$ as input and returns a list of enhanced
pseudo-labels $[\hat{\bm{P}_{1}},\cdots,\hat{\bm{P}_{K}}]$. Specifically, it
consists of two stages, mask assignment and mask selection as elucidated in
Figure 2. During the mask assignment phase, we compute the intersection
between each SAM mask $\bm{S}_{l}$ and pseudo-labels $\bm{P}_{k}$ for every
class $k\in\\{1,\cdots,K\\}$. Each $\bm{S}_{l}$ is assigned to the class with
the largest intersection area and masks without any overlap with existing
pseudo-labels are disregarded. After processing all SAM masks, we obtain a
mask assignment list $A=[A_{1},\cdots,A_{K}]$ where $A_{k}$ contains the masks
assigned to class $k$.

It is widely known that the CAM-derived pseudo-labels often suffer from false
activation [50, 21] and partial activation [3, 47]. Our mask selection
strategy aims to address them by selecting the most relevant masks based on
the overlaps between SAM masks and pseudo-labels. False activation arises when
pseudo-labels encapsulate the target object along with a marginal section of
the surrounding background. Thus, the masks for the entire target object (or
its parts) should predominantly align with the pseudo-labels, whereas masks
for the background should exhibit minimal overlap with the pseudo-labels. To
mitigate false activation, we select masks demonstrating extensive coverage by
the pseudo-labels. Conversely, partial activation arises when pseudo-labels
only cover the most discriminative part instead of the entire object.
Therefore, if a mask covers the majority of the pseudo-labels, it is
indicative that this mask likely represents the complete object and should be
retained for enhanced pseudo-labels to address partial activation.

Based on the intuitions mentioned above, we iterate through every mask
assigned to class $k$. For each mask $\bm{S}$, we compute $o_{s}$, the
fraction of mask $\bm{S}$ overlapped by pseudo-labels $\bm{P}_{k}$, and
$o_{p}$, the fraction of pseudo-labels $\bm{P}_{k}$ overlapped by mask
$\bm{S}$. A mask $\bm{S}$ is preserved as an enhanced pseudo-label if:

  1. 1.

$o_{s}>t_{1}$ where $t_{1}=0.5$ indicates at least 50% of the mask is covered
by the pseudo-labels

  2. 2.

$o_{p}>t_{2}$ where $t_{2}=0.85$ indicates at least 85% of the pseudo-labels
is covered by the mask

If the initial pseudo-labels are not covered by any SAM masks, we will keep
them unchanged in the enhanced pseudo-labels. The overall algorithm is
summarized in Algorithm 1.

Algorithm 1 SAM Enhanced Pseudo-Labels (SEPL) for One Image

Input: Pseudo labels $[\bm{P}_{1},\cdots,\bm{P}_{K}]$, Masks
$[\bm{S}_{0},\cdots,\bm{S}_{L}]$,
threshold $t_{1}=0.5$, threshold $t_{2}=0.85$
Output Enhanced pseudo-labels $[\hat{\bm{P}_{1}},\cdots,\hat{\bm{P}_{K}}]$

procedure SEPL($[\bm{P}_{1},\cdots,\bm{P}_{K}]$,
$[\bm{S}_{0},\cdots,\bm{S}_{L}]$)

$A=[A_{1},\cdots,A_{K}]$ where $A_{k}=\\{\\}$ $\triangleright$ $A_{k}$ stores
the masks assigned to class $k$

for $l$ from $0$ to $L$ do $\triangleright$ Mask assignment

$k^{\star}=\underset{k}{\arg\max}\;\text{Intersect}(\bm{S}_{l},\bm{P}_{k})$

$A_{k^{\star}}\leftarrow A_{k^{\star}}\cup\\{\bm{S}_{l}\\}$

for $k$ from $1$ to $K$ do $\triangleright$ Mask selection

if $\bm{P}_{k}==\\{0\\}^{H\times W}$ then

continue

$tmp=\\{\\}$ $\triangleright$ $tmp$ stores enhanced pseudo-labels for class
$k$

for each mask $S$ in $A_{k}$ do

$o_{s}=\frac{\text{Intersect}(S,\bm{P}_{k})}{\text{nonzero\\_area}(S)}$
$\triangleright$ fraction of mask $S$ covered by pseudo-labels $\bm{P}_{k}$

$o_{p}=\frac{\text{Intersect}(S,\bm{P}_{k})}{\text{nonzero\\_area}(\bm{P}_{k})}$
$\triangleright$ fraction of pseudo-labels $\bm{P}_{k}$ covered by mask $S$

if $o_{s}>t_{1}$ or $o_{p}>t_{2}$ then

$tmp\leftarrow tmp\cup\\{S\\}$

if $tmp==\\{\\}$ then

$tmp\leftarrow tmp\cup\\{\bm{P}_{k}\\}$

$\hat{\bm{P}_{k}}\leftarrow$ Merge masks in $tmp$ with element-wise OR and
assign $k$ to nonzero elements

##  3 Experiment

###  3.1 Experimental setup

#### Datasets and Evaluation Metric

We evaluate our proposed framework on the PASCAL VOC 2012 [15] and MS COCO
2014 dataset [35]. The dataset details can be found in Appendix subsection
B.1. We only used image-level ground-truth labels during pseudo-labels
generation. The mean Intersection over Union (mIoU) is adopted as the
evaluation metric for all experiments. To demonstrate the quality of the
pseudo-labels, we evaluate them on the VOC and COCO training set.

#### Pseudo Labels

In our experiments, we generate pseudo-labels using several SOTA WSSS methods,
including: Recurseed [21], L2G [20], CLIPES [37], RCA [58], EPS [31], CLIMS
[50], TransCAM [33], PPC [14], SIPE [11], and PuzzleCAM [22]. The detailed
introductions of them can be found in Appendix subsection B.2.

#### Implementation Details

SAM masks are generated with the official code [25]. The hyperparameters used
for inference can be found in Appendix subsection B.3. $t_{1}$ and $t_{2}$ by
default are set to 0.5 and 0.85 respectively. We use Deeplab V2 (ResNet-101)
[9] as the fully supervised semantic segmentation model.

###  3.2 Quantitative Evaluation and Comparison

Figures Figure 3 and Figure 4 illustrate the enhanced pseudo-label quality
achieved on the PASCAL VOC and MS COCO with our SEPL algorithm. SEPL
consistently and significantly elevates the quality of pseudo-labels across
various original WSSS methods. Moreover, the impact of this enhancement
extends beyond the pseudo-label quality. Utilizing the enhanced pseudo-labels
for training supervised semantic segmentation models(DeepLab V2) yields
notable improvements in performance. The models, when trained on the enhanced
pseudo-labels, consistently outperform those trained on original pseudo-
labels. Table 1 and Table 2 provide a detailed quantitative comparison of
these performances across both PASCAL VOC and MS COCO datasets. More detailed
results can be found in Appendix Appendix C.

Figure 3: Pseudo labels quality on PASCAL VOC 2012. Original: pseudo-labels
from original SOTA WSSS methods. SEPL: pseudo-labels enhanced by SEPL. The
improvement after enhancement is indicated in green.

Method |  Recurseed [21] |  L2G [20] |  CLIPES [37] |  RCA [58] |  EPS [31] |  CLIMS [50] |  TransCAM [33] |  PPC+EPS [14] |  PPC+SEAM [14] |  SIPE [11] |  PuzzleCAM [22] |  FS [8]
---|---|---|---|---|---|---|---|---|---|---|---|---
Origin | 71.38 | 69.38 | 70.48 | 69.48 | 68.16 | 69.33 | 68.10 | 70.30 | 65.01 | 67.14 | 65.84 | 76.48
SEPL | 72.89 | 72.41 | 73.06 | 69.70 | 72.13 | 71.11 | 69.93 | 71.93 | 68.25 | 69.67 | 68.89

Table 1: Performance of Deeplab V2 (ResNet-101) trained on pseudo-labels
without post-processing: Original method vs. SAM-enhanced SEPL. Evaluated on
the VOC $val$ set. FS: full supervision

Figure 4: Pseudo-label quality on COCO $train$ set: Original from WSSS methods
vs. SEPL-enhanced. Green indicates enhancements.

Method |  L2G [20] |  SIPE [11] |  EPS [31] |  CLIPES [37] |  FS [8]
---|---|---|---|---|---
Origin | 43.06 | 41.53 | 39.06 | 46.29 | 55.04
SEPL | 46.39 | 45.19 | 41.55 | 47.90

Table 2: Reult of Deeplab V2 (ResNet101) without CRF on COCO $val$ set:
Original vs. SEPL-enhanced pseudo-labels. FSS: full supervision

##  4 When does SAM not help?

Upon analyzing instances where SEPL was ineffective, we attributed the
shortcomings primarily to three sources: the initial pseudo-labels, the SAM
masks, and our enhancement algorithm.

#### Initial pseudo-labels

Since we leverage the initial pseudo-labels as the anchors to find relevant
masks, if they activate on incorrect objects or fail to activate on the target
objects, the SAM masks won’t offer any enhancement. In fact, they may
detrimentally affect the quality of the pseudo-labels, as shown in Figure 6.
To address this, we turn to better WSSS methods for more precise pseudo-
labels.

#### SAM masks

While SAM effectively processes most images in VOC and COCO, it occasionally
falters. As depicted in Figure 6, SAM might sometimes overlook certain parts
of images or erroneously group several objects as a single segment. Fine-
tuning SAM’s inference hyperparameters could be a potential remedy to enhance
segmentation outcomes.

#### Enhancement Algorithm

While our current algorithm is proficient in many scenarios, it falters in
specific situations. As depicted in Figure 7, certain SAM masks within an
image overshadow and envelop smaller masks. An example is column (g) entirely
covers the masks in columns (e) and (f). Since pseudo-labels often suffer from
partial activation, our existing algorithm inclines towards selecting larger
masks to produce a more complete pseudo-label. Yet, this approach poses a
dilemma: the absence of clarity on whether these masks encapsulate multiple
objects or just a singular entity. As illustrated in the first row of Figure
7, the mask in (g) spans both the boat and the sea. Blindly opting for the
larger mask risks deteriorating the quality of pseudo-labels. A promising
resolution could lie in treating the SAM masks as nodes in a tree, leveraging
their inherent hierarchical structure. This tree-based approach might
facilitate a more discerning selection of the appropriate masks.

Figure 5: Examples of pseudo-labels activate on incorrect objects or fail to
activate on the target objects. Adding SAM mask may detrimentally affect the
quality of the pseudo-labels

Figure 6: Example of SAM’s failures. The first row shows that SAM overlooks
certain parts in an image and the second row shows SAM erroneously groups
several objects as a single mask

Figure 7: Certain SAM masks within an image envelop smaller masks. The SAM
mask in column (g) entirely covers the masks in columns (e) and (f), which
poses a challenge as we cannot ascertain if a mask represents multiple objects
or just one.

##  5 Conclusion

This paper presents a pioneer investigation into the application of SAM as a
foundation model in WSSS. By leveraging SAM’s class-agnostic capability of
producing fine-grained instance masks, we use CAM pseudo-labels as cues to
select and combine SAM masks, resulting in high-quality pseudo-labels that are
both class-aware and object-aware. Our approach is highly versatile and can be
easily integrated into existing WSSS methods without any modification. Despite
its simplicity, our approach shows consistent improvement over the SOTA WSSS
methods on both PASCAL VOC and MS-COCO datasets. We anticipate that this study
will catalyze the adoption of segmentation foundational models across a broad
spectrum of computer vision tasks.

## Acknowledgments

This research is supported in part by NSF (IIS-2107077, OAC2118240, and
OAC-2112606) and Cisco Research. We are thankful for the computational
resources of the Ohio Supercomputer Center

## References

  * [1] Jiwoon Ahn, Sunghyun Cho and Suha Kwak “Weakly supervised learning of instance segmentation with inter-pixel relations” In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 2209–2218
  * [2] Jiwoon Ahn and Suha Kwak “Learning Pixel-Level Semantic Affinity With Image-Level Supervision for Weakly Supervised Semantic Segmentation” In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018
  * [3] Jiwoon Ahn and Suha Kwak “Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4981–4990
  * [4] Nikita Araslanov and Stefan Roth “Single-stage semantic segmentation from image labels” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4253–4262
  * [5] Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad and Ghassan Hamarneh “Deep semantic segmentation of natural and medical images: a review” In _Artificial Intelligence Review_ 54 Springer, 2021, pp. 137–178
  * [6] Amy Bearman, Olga Russakovsky, Vittorio Ferrari and Li Fei-Fei “What’s the point: Semantic segmentation with point supervision”, 2016, pp. 549–565
  * [7] Arslan Chaudhry, Puneet K. Dokania and Philip H.. Torr “Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation”, 2017 arXiv:1707.05821 [cs.CV]
  * [8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy and Alan L Yuille “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs” In _TPAMI_ , 2017
  * [9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy and Alan L Yuille “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs” In _IEEE transactions on pattern analysis and machine intelligence_ 40.4 IEEE, 2017, pp. 834–848
  * [10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy and Alan L. Yuille “Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs” In _ICLR_ , 2015
  * [11] Qi Chen, Lingxiao Yang, Jian-Huang Lai and Xiaohua Xie “Self-Supervised Image-Specific Prototype Exploration for Weakly Supervised Semantic Segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 4288–4298
  * [12] Junsuk Choe, Seungho Lee and Hyunjung Shim “Attention-based dropout layer for weakly supervised single object localization and semantic segmentation” In _IEEE transactions on pattern analysis and machine intelligence_ 43.12 IEEE, 2020, pp. 4256–4271
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold and Sylvain Gelly “An image is worth 16x16 words: Transformers for image recognition at scale” In _arXiv preprint arXiv:2010.11929_ , 2020
  * [14] Ye Du, Zehua Fu, Qingjie Liu and Yunhong Wang “Weakly supervised semantic segmentation by pixel-to-prototype contrast” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4320–4329
  * [15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn and Andrew Zisserman “The pascal visual object classes (voc) challenge” Springer, 2010, pp. 303–338
  * [16] Junsong Fan, Zhaoxiang Zhang, Tieniu Tan, Chunfeng Song and Jun Xiao “Cian: Cross-image affinity net for weakly supervised semantic segmentation” In _Proceedings of the AAAI Conference on Artificial Intelligence_ 34.07, 2020, pp. 10762–10769
  * [17] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck and Klaus Dietmayer “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges” In _IEEE Transactions on Intelligent Transportation Systems_ 22.3 IEEE, 2020, pp. 1341–1360
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep Residual Learning for Image Recognition”, 2015 arXiv:1512.03385 [cs.CV]
  * [19] Peng-Tao Jiang, Qibin Hou, Yang Cao, Ming-Ming Cheng, Yunchao Wei and Hong-Kai Xiong “Integral object mining via online attention accumulation”, 2019, pp. 2070–2079
  * [20] Peng-Tao Jiang, Yuqi Yang, Qibin Hou and Yunchao Wei “L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16886–16896
  * [21] Sang Hyun Jo, In Jae Yu and Kyung-Su Kim “RecurSeed and CertainMix for weakly supervised semantic segmentation” In _arXiv preprint arXiv:2204.06754_ , 2022
  * [22] Sanghyun Jo and In-Jae Yu “Puzzle-cam: Improved localization via matching partial and full features” In _2021 IEEE International Conference on Image Processing (ICIP)_ , 2021, pp. 639–643 IEEE
  * [23] Tsung-Wei Ke, Jyh-Jing Hwang and Stella X Yu “Universal weakly supervised segmentation by pixel-to-segment contrastive learning” In _arXiv preprint arXiv:2105.00957_ , 2021
  * [24] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein and Bernt Schiele “Simple does it: Weakly supervised instance and semantic segmentation” Computer Vision Foundation / IEEE, 2017, pp. 876–885
  * [25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg and Wan-Yen Lo “Segment anything” In _arXiv preprint arXiv:2304.02643_ , 2023
  * [26] Alexander Kolesnikov and Christoph H Lampert “Seed, expand and constrain: Three principles for weakly-supervised image segmentation” In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , 2016, pp. 695–711 Springer
  * [27] Fahad Lateef and Yassine Ruichek “Survey on semantic segmentation using deep learning techniques” In _Neurocomputing_ 338 Elsevier, 2019, pp. 321–348
  * [28] Jungbeom Lee, Eunji Kim and Sungroh Yoon “Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation” Computer Vision Foundation / IEEE, 2021, pp. 4071–4080
  * [29] Jungbeom Lee, Jihun Yi, Chaehun Shin and Sungroh Yoon “BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation”, 2021 arXiv:2103.08907 [cs.CV]
  * [30] Seungho Lee, Minhyun Lee, Jongwuk Lee and Hyunjung Shim “Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation” In _CVPR_ , 2021
  * [31] Seungho Lee, Minhyun Lee, Jongwuk Lee and Hyunjung Shim “Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 5495–5505
  * [32] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst and Yun Fu “Tell me where to look: Guided attention inference network” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9215–9223
  * [33] Ruiwen Li, Zheda Mai, Zhibo Zhang, Jongseong Jang and Scott Sanner “Transcam: Transformer attention-based cam refinement for weakly supervised semantic segmentation” In _Journal of Visual Communication and Image Representation_ 92 Elsevier, 2023, pp. 103800
  * [34] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He and Jian Sun “Scribblesup: Scribble-supervised convolutional networks for semantic segmentation” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3159–3167
  * [35] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C Lawrence Zitnick “Microsoft coco: Common objects in context” In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 2014, pp. 740–755 Springer
  * [36] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár and C. Zitnick “Microsoft Coco: Common Objects in Context” In _Computer Vision – ECCV 2014_ , 2014, pp. 740–755 DOI: 10.1007/978-3-319-10602-1_48
  * [37] Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu and Xiaofei He “CLIP Is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023, pp. 15305–15314
  * [38] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz and Demetri Terzopoulos “Image Segmentation Using Deep Learning: A Survey” In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44.7, 2022, pp. 3523–3542 DOI: 10.1109/TPAMI.2021.3059968
  * [39] Youngmin Oh, Beomjun Kim and Bumsub Ham “Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation” In _CVPR_ , 2021
  * [40] Lixiang Ru, Yibing Zhan, Baosheng Yu and Bo Du “Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16846–16855
  * [41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei “ImageNet Large Scale Visual Recognition Challenge”, 2015 arXiv:1409.0575 [cs.CV]
  * [42] Karen Simonyan and Andrew Zisserman “Very Deep Convolutional Networks for Large-Scale Image Recognition”, 2015 arXiv:1409.1556 [cs.CV]
  * [43] Krishna Kumar Singh and Yong Jae Lee “Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization” In _2017 IEEE international conference on computer vision (ICCV)_ , 2017, pp. 3544–3553 IEEE
  * [44] Chunfeng Song, Yan Huang, Wanli Ouyang and Liang Wang “Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation” Computer Vision Foundation / IEEE, 2019, pp. 3136–3145
  * [45] Weixuan Sun, Jing Zhang and Nick Barnes “3d guided weakly supervised semantic segmentation” In _Proceedings of the Asian Conference on Computer Vision_ , 2020
  * [46] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen and Tiejun Huang “Seggpt: Segmenting everything in context” In _arXiv preprint arXiv:2304.03284_ , 2023
  * [47] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan and Xilin Chen “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12275–12284
  * [48] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao and Shuicheng Yan “Object region mining with adversarial erasing: A simple classification to semantic segmentation approach” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1568–1576
  * [49] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao and Shuicheng Yan “Object region mining with adversarial erasing: A simple classification to semantic segmentation approach” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1568–1576
  * [50] Jinheng Xie, Xianxu Hou, Kai Ye and Linlin Shen “CLIMS: cross language image matching for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4483–4492
  * [51] Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid and Dan Xu “Multi-class token transformer for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4310–4319
  * [52] Qi Yao and Xiaojin Gong “Saliency guided self-attention network for weakly and semi-supervised semantic segmentation” In _IEEE Access_ , 2020
  * [53] Xiaohui Yuan, Jianfang Shi and Lichuan Gu “A review of deep learning methods for semantic segmentation of remote sensing imagery” In _Expert Systems with Applications_ 169 Elsevier, 2021, pp. 114417
  * [54] Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun and Kaizhu Huang “Reliability does matter: An end-to-end weakly supervised semantic segmentation approach” In _Proceedings of the AAAI Conference on Artificial Intelligence_ 34.07, 2020, pp. 12765–12772
  * [55] Tianyi Zhang, Guosheng Lin, Weide Liu, Jianfei Cai and Alex Kot “Splitting vs. merging: Mining object regions with discrepancy and intersection loss for weakly supervised semantic segmentation”, 2020
  * [56] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva and Antonio Torralba “Learning Deep Features for Discriminative Localization” In _Computer Vision and Pattern Recognition_ , 2016
  * [57] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva and Antonio Torralba “Learning deep features for discriminative localization” In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929
  * [58] Tianfei Zhou, Meijie Zhang, Fang Zhao and Jianwu Li “Regional semantic contrast and aggregation for weakly supervised semantic segmentation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4299–4309
  * [59] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao and Yong Jae Lee “Segment everything everywhere all at once” In _arXiv preprint arXiv:2304.06718_ , 2023

## Appendix

##  Appendix A Related Work

###  A.1 Weakly Supervised Semantic Segmentation (WSSS)

Recent approaches in weakly supervised semantic segmentation (WSSS) often rely
on Class Activation Maps (CAM) [57] to generate pixel-level pseudo-labels.
These pseudo-labels are then used to train the segmentation model in a fully
supervised manner. However, CAM often exhibits a bias towards the most
discriminative regions of the target object which limits the quality of the
pseudo-labels. To overcome this challenge, recent works mainly focus on
generating high-quality CAMs with integral activation on the entire object
regions.

Early-stage works [43, 49, 32] encourage the network to discover less
activated object parts via adversarial erasing. In addition to the
classification loss typically used in the WSSS framework, specific loss
functions such as SEC loss [26], equivariance regularization [47], and
contrastive loss [23, 14] have been exploited in narrowing the gap between the
pixel-level and image-level supervisions. Some works also introduce network
modules to address the partial activation problem of CAM: SEAM [47] leverages
pixel-level semantic affinities with a pixel correlation module; CIAN [16]
exploits the additional information from related images with a cross-image
affinity module. Recent methods based on Vision Transformer [13] including
[33, 51] aim to uncover more comprehensive object regions by exploring the
global information from the attention of the transformer network. Most of
these works follow the multi-stage framework, where a post-processing step is
necessary for refining and improving the initial pseudo-labels generated from
CAM.

###  A.2 Post-Processing in WSSS

Although there are end-to-end WSSS solutions [40, 4, 54] available, most of
the recent works still rely on some post-processing techniques to enhance the
initial pseudo-labels to achieve superior performance. Among these techniques,
two widely utilized methods for refining pseudo-labels are AffinityNet [3] and
IRNet [1]. AffinityNet trains a network from CAM to predict the semantic
affinities and uses it for propagating local activations, whereas IRNet [1]
learns and predicts semantic affinities more effectively by leveraging class
boundary maps. Despite the substantial improvement in the pseudo-labels, these
methods require training a separate network. The computational cost involved
can be a significant barrier, especially when working with large-scale
datasets. Additionally, the careful tuning of hyperparameters to obtain
accurate foreground and background pixels for training these networks can slow
down the entire training pipeline. This requirement for meticulous parameter
tuning not only adds complexity to the process but also limits the
applicability and scalability of these post-processing techniques.

##  Appendix B Experiment Details

###  B.1 Dataset Details

Experiments are conducted on two publicly available datasets, PASCAL VOC 2012
[15]and MS COCO 2014 [36]. The PASCAL VOC 2012 dataset contains 20 semantic
categories and the background. It is split into three sets, the training,
validation, and test sets, each containing 1464, 1449, and 1456 images,
respectively. Following the standard setting, we also use the augmented
training set, yielding a total of 10582 training images. The MS COCO 2014
dataset has 80 semantic categories. Following [12], the images without target
categories are excluded from the dataset, remaining 82081 training images and
40137 validation images. We report the mean Intersection-over-Union (mIoU),
precision, and recall for evaluation. To demonstrate the quality116 of the
pseudo-labels, we evaluate them on the VOC and COCO training set.

###  B.2 Baselines

  * •

CLIMS: [50] A Cross-Language Image Matching framework leveraging natural
language supervision to activate complete object regions and suppress related
open background regions for improved CAM quality in WSSS.

  * •

SIPE: [11] Self-supervised Image-specific Prototqualitativeype Exploration,
which tailors prototypes for each image to capture complete regions,
optimizing feature representation and enabling self-correction for improved
WSSS performance.

  * •

PPC: [14] A weakly-supervised pixel-to-prototype contrast method providing
pixel-level supervisory signals, executed across and within different views of
an image to enhance the quality of pseudo masks for WSSS.

  * •

TransCAM: [33] A Conformer-based solution that refines CAM by leveraging
attention weights from the transformer branch of the Conformer, capturing both
local features and global representations for WSSS.

  * •

RecurSeed: [21] An approach that alternately reduces non-detections and false-
detections through recursive iterations, implicitly finding an optimal
junction and leveraging a novel data augmentation method, EdgePredictMix, for
improved WSSS performance.

  * •

L2G [20] A simple online local-to-global knowledge transfer framework for high
quality object attention mining. It first leverages a local classification
network to extract attentions from multiple local patches randomly cropped
from the input image. Then, it utilizes a global network to learn
complementary attention knowledge across multiple local attention maps online.

  * •

CLIPES [37] An approach that leverages CLIP to improve pseudo-label
generation, refinement and final segmentation model training.

  * •

RCA [58] RCA is equipped with a regional memory bank to store massive, diverse
object patterns appearing in training data, which acts as strong support for
exploration of dataset-level semantic structure.

  * •

EPS [31] EPS learns from pixel-level feedback by combining two weak
supervisions; the image-level label provides the object identity via the
localization map and the saliency map from the off-the-shelf saliency
detection model offers rich boundaries

  * •

PuzzleCAM [22] PuzzleCAM minimizes differences between the features from
separate patches and the whole image. It consists of a puzzle module and two
regularization terms to discover the most integrated region in an object.

###  B.3 Implementation Details

#### SAM Inference Hyperparameters

For our experiments, we adopted the standard settings of the SAM model as
provided in their official repository. However, we made modifications to two
specific hyperparameters to tailor the model’s behavior to our needs:

  * •

–pred-iou-thresh was set from None to 0.86.

  * •

–stability-score-thresh was set from None to 0.92.

By adjusting these thresholds, our objective was to enable SAM to produce a
wider and more diverse range of masks for selection via our algorithm.
Importantly, these modifications did not have a detrimental effect on the
inference speed, ensuring efficiency was maintained throughout the process.
For inference, we employed the default pretrained ViT-H SAM model.

##  Appendix C Extra Results

Figure 8 illustrates the qualitative improvements of the pseudo-labels
enhanced by SEPL in recall, precision, and mIoU, respectively, which are
calculated based on the average of all samples. Table 3 shows the quantitative
improvement of the pseudo-labels enhanced by SEPL for recall and precision.

Method |  Recurseed |  L2G |  CLIPES |  RCA |  EPS |  CLIMS |  TransCAM |  PPC+EPS |  PPC+SEAM |  SIPE |  PuzzleCAM
---|---|---|---|---|---|---|---|---|---|---|---
Pseudo Label Precision | 85.10 | 78.86 | 83.81 | 81.53 | 78.10 | 80.97 | 80.46 | 82.73 | 78.93 | 76.75 | 83.29
SEPL Precision | 85.73 | 84.57 | 85.96 | 82.13 | 84.85 | 83.29 | 81.89 | 84.32 | 81.76 | 79.86 | 83.93
Precision Delta | 0.63 | 5.71 | 2.15 | 0.60 | 6.75 | 2.32 | 1.43 | 1.59 | 2.83 | 3.11 | 6.39
Pseudo Label Recall | 85.85 | 87.35 | 85.36 | 71.04 | 84.93 | 83.19 | 84.83 | 86.91 | 83.45 | 87.68 | 76.77
SEPL Recall | 91.75 | 92.05 | 91.58 | 79.37 | 90.94 | 89.12 | 90.07 | 92.02 | 90.36 | 91.12 | 86.22
Recall Delta | 5.90 | 4.70 | 6.22 | 8.33 | 6.01 | 5.93 | 5.24 | 5.11 | 6.91 | 3.44 | 9.45

Table 3: Pseudo label precision and recall improvements by incorporating SEPL

(a) Recall

(b) Precision

(c) mIoU

Figure 8: Improvements for pseudo-labels: (a) Recall, (b) Precision, (c) mIoU.

###  C.1 Apply SAM on CAM without post-processing

As mentioned in Section subsection A.2, most recent works still rely on some
post-processing techniques to enhance the initial CAM, aiming to procure more
precise pseudo-labels. However, these enhancement procedures often demand
substantial computational overhead and extended training durations. Such
constraints can potentially hinder the broad-scale deployment of WSSS on
extensive datasets. Since the initial CAM also provides an estimation of
object localization, our proposed SEPL can be directly applied to the initial
CAM. This approach circumvents the need for additional post-processing steps,
leading to appreciable reductions in both training duration and computational
demands in the WSSS pipeline.

As shown in Figure 9, the initial CAM consistently benefits from SAM masks.
More interestingly, CAM+SAM can reach and even surpass the quality of pseudo-
labels obtained after conventional post-processing. This finding suggests that
SAM has the potential to replace time-consuming post-processing steps,
offering a more efficient solution to WSSS tasks. Figure 10 illustrates the
qualitative improvements of the initial CAM by SEPL in recall, precision, and
mIoU, respectively.

Figure 9:  By directly utilizing iinitial CAM with SAM, we achieved comparable
performance to that of post-processed pseudo-labels enhanced by SAM. This
finding suggests that SAM can be used as a substitute for post-processing
modules, resulting in a marked acceleration of the entire WSSS training
pipeline

(a) Recall

(b) Precision

(c) mIoU

Figure 10: Improvements for CAM without post-processing: (a) Recall, (b)
Precision, (c) mIoU.


# The Life and Death of Unwanted Bits: Towards Proactive Waste Data Management
in Digital Ecosystems

Ragib Hasan, Randal Burns
Department of Computer Science, Johns Hopkins University
3400 N. Charles Street, Baltimore, MD 21218, United States
<EMAIL_ADDRESS>

###### Abstract

Our everyday data processing activities create massive amounts of data. Like
physical waste and trash, unwanted and unused data also pollutes the digital
environment by degrading the performance and capacity of storage systems and
requiring costly disposal. In this paper, we propose using the lessons from
real life waste management in handling waste data. We show the impact of waste
data on the performance and operational costs of our computing systems. To
allow better waste data management, we define a waste hierarchy for digital
objects and provide insights into how to identify and categorize waste data.
Finally, we introduce novel ways of reusing, reducing, and recycling data and
software to minimize the impact of data wastage.

##  1 Introduction

In real life, all human activities produce unwanted, unusable, or useless by-
products. Such worthless objects are considered to be waste or trash. Waste
products impact the environment and ecosystem by using up or polluting
resources, degrading performance of physical processes, and requiring
expensive cleanup. To deal with waste in the ecosystem, various waste
management techniques have been developed over the years [10, 11]. These
techniques aim at reducing the production of waste, repurposing the waste or
its components, and efficient disposal of waste.

In today’s world, we are increasingly living virtual lives – creating,
processing and consuming data in the form of digital objects. A computing
system is similar to a real life ecosystem. In a digital ecosystem, data and
applications that consume and produce data interact and use the physical
hardware components and resources. Like real life ecosystems, a digital
ecosystem has finite resources such as storage, compute cycles, and network
bandwidth. Also, consumer applications share and compete for these resources.
No matter how much “illusion” of infinite resources various abstractions
provide, in reality resources in a digital ecosystem are not infinite.
Storing, transferring, and disposing of data consume these resources. However,
all data in a given system are not equally important or useful, and often a
significant amount of data in a system can be unwanted or unused content. Such
waste data consumes resources but provide no value to the corresponding
digital ecosystem. Unless we introduce responsible waste data management
practices, such waste data will misuse resources and make a significant impact
on the digital ecosystem.

What happens to these unwanted bits? Typical approaches to managing waste data
include compression and/or deletion of such unwanted data. However, these
processes come at a price – disposal of waste data consumes resources in the
form of energy used to delete data, tying up compute cycles, blocking I/O,
etc. Disposal via deletion also causes degradation of performance and reduces
the lifetime of storage components (such as Flash storage). We need better
waste management techniques to handle unwanted data. In this paper, we argue
about the need to examine waste data in a systematic manner. We posit that
successful real life waste management techniques can be effectively adapted to
handle waste data. The contributions of this paper are as follows:

We present a definition of waste data in digital ecosystems.

We show the impact of waste data on reduction and degradation of system
capacity.

We introduce a hierarchy for waste data management techniques.

We advocate the need for an integrated approach for managing waste data and
discuss how well known real life waste management principles can be adapted
for this. The rest of the paper is organized as follows: Section 2 presents a
definition of waste data. We discuss the impact of waste data in Section 3. We
introduce a hierarchy for waste data management and explore the use of
different real life waste management principles for managing waste data in
Section 4. Finally, we discuss related work in Section 5 and conclude in
Section 6.

##  2 Defining Waste Data

In real life, the definition of waste is somewhat subjective, as what is waste
to one system can be considered valuable resources by another system. Various
authorities and agencies have defined waste in different ways. For example,
the Basel convention and the European Union define waste as something that is
or will be discarded by the holder [1, 2]. The Organization for Economic
Cooperation and Development (OECD) defines waste as materials that are by-
products of regular processing, which have no use to the creator and which are
disposed of [9]. Pongrácz et al. [10, 11] provided a definition of waste based
on their classification of objects – an object is considered to be waste if it
is unintentionally created, or the user has used up the object, or the
object’s quality has degraded, or the object is unwanted by the user.

Similarly, providing the definition of waste data in a computing system is
difficult. Informally, a data object or software can be considered to be waste
data by a user if it has no utility for the user in the given context. To
provide a formal definition of waste data, we leverage the definition of
physical waste given by Pongrácz et al. [10, 11]. In particular, we use
Pongrácz et al.’s classification scheme to define waste data as data belonging
to any one or more of the following categories:

Unintentional data.  Data unintentionally created, as a side effect or by-
product of a process, with no purpose.

Used data.  Good data that has served its purpose and is no longer useful to
the user.

Degraded data.  Data that has degraded in quality such that it is no longer
useful to the user.

Unwanted data.  Data that was never useful to the user. Next, we discuss each
of these waste data categories with examples.

Unintentional data.  Almost all data processing applications generate
unintentional by-products. We define a data object to be a by-product if it is
not included in the final set of data objects produced by the application. For
example, the goal of LaTeX compilation is to generate output .pdf or .ps files
from source text and images. However, when LaTeX is executed, it automatically
creates a number of temporary data objects and files that assist in
compilation. In the context of LaTeX compilation, these files (such as .aux,
.bbl, .log) can be considered to be unintentional by-products that assist in
the production of the final data product (e.g., .ps or .pdf).

Used data.  In most cases, input data is no longer useful to the user once
computations have been performed over it. For example, an aggregation
operation can use data from many sensors. The sensor readings may be useful to
the user performing the aggregation operation only until the computation is
over. After that, the input data may become useless, and hence considered to
be waste data.

Degraded data.  When data gets corrupted, it can become unusable, and
therefore be considered as waste data by a user. Also, when other changes make
data or software obsolete, it can be considered to have degraded and therefore
marked as waste data by the user. For example, newer software releases can
make old versions of the software and the related files obsolete.

Unwanted data.  This class of waste data includes data that may or may not be
of high quality, but is not relevant to the user at all to begin with. For
example, software documentation in an unknown language can be of good quality
but still be irrelevant to a non-speaker.

##  3 Impact of Waste Data

In the physical reality, waste has adverse impact on the environment of the
ecosystem. The presence of waste pollutes the ecosystem, causing economic,
social, and operational impacts. We argue that in the same manner, waste data
affects a computing system by consuming resources without providing value, and
by degrading system performance and components. Storage, processing, and
transfer of data require the use of system resources such as disk space, cpu
cycles, and network or I/O bandwidth. Disposing of the waste data by deletion
also impacts the life of storage devices and incurs energy and time overheads.

Storage Consumption.  Waste data consumes a lot of storage space. For example,
creating and editing a small text file in vi causes a temporary swap file to
be created. To illustrate the amount of temporary waste data created by source
code compilation, we compiled Openssl 1.0.0a on a Linux workstation.
Compilation of Openssl produces about 13.6 MB of target binary code. However,
it also produces about 44.5 MB of temporary object code that is not part of
the installation. From the viewpoint of the user, these temporary object files
produced during compilation are waste data. Such unwanted data consumes a lot
of space and needs to be deleted.

The overall amount of unused and dead data in a given system is not small. We
wrote a Perl script to determine the percentage of files that have never been
accessed since last modification. We ran the script on three different
platforms – an Apple MacBook used as a personal laptop, a Ubuntu Linux
desktop, and a student lab server running Fedora Linux. The results are show
on Table 1.

Platform | MacBook | Desktop | Server
---|---|---|---
% of files | 20.6 | 47.4 | 57.1
% of used space | 98.5 | 38.1 | 99.5
Table 1: Analysis of files in a MacBook, a desktop workstation, and a student
lab server. In all cases, a large number (20.6%–57.1%) of files have never
been accessed since last modification.

In all three cases, a large fraction of files in the system have never been
accessed since last modification, reflecting the results from previous work in
the area [4, 13]. In terms of space usage, these files amounted from 38% to as
high as 99% of the total used space on the machines. This shows that the
amount of waste data in a system is quite significant.

Reducing Device Lifetime.  Disposal of waste data via deletion can impact the
lifetime of storage devices. For example, Flash based storage devices
typically have a maximum number of write cycles. Multi-Level Cell (MLC) flash
devices support a maximum of 1,000–10,000 write/erase cycles per cell while
Single-Level Cell (SLC) flash devices support up to 100,000 write-erase cycles
per cell [16]. Waste or by-product data brings no value, but uses up flash
storage write cycles, reducing the lifetime of such storage devices. As flash-
based solid state storage becomes popular, especially in mobile devices, we
need to ensure that waste data write / erase cycles do not impact the lifetime
of such storage.

Performance Degradation.  Presence of unwanted waste data can degrade system
performance. For example, in a file system, the extra storage space consumed
by waste data may cause unnecessary fragmentation and use up available inodes.
If waste data can be identified and not stored by the system, we can reduce
the load and fragmentation greatly. For example, Table 1 shows that 98.5% of
the total space used by files in the laptop was actually consumed by files
that were never accessed since last modification. By storing these files
separate from the frequently accessed files, we can vastly improve system
performance. Deletion also takes up CPU cycles and consumes energy – a fact
which is significant in low-powered mobile devices.

##  4 Managing Waste Data

How do we deal with waste data? Storage and deletion of waste data is costly
in terms of energy and space consumed. Therefore, we need effective strategies
to handle waste data. To provide a guideline for waste data management, we
turned to the techniques used in real life waste management. We argue in this
paper that these lessons from real life waste management are equally effective
in managing waste data in digital environments. We start our discussion by
presenting a hierarchy of waste data management methods. Then we discuss some
specific approaches application designers and system architects can adopt to
minimize the impact of waste data.

Figure 1: The Waste Data Management Hierarchy. Processes at the top are more
preferable.

###  4.1 Waste data hierarchy

In dealing with waste in the natural environment, a waste hierarchy is widely
used to classify and organize waste management schemes according to their
usefulness and impact [17]. We propose adapting the waste hierarchy from real
life waste management to develop guidelines for choosing waste data management
schemes. Besides the “three R’s” (reduce, reuse, recycle), we use the
additional steps of recovery and disposal in our scheme, leveraging the five-
step waste management hierarchy described in [1]. We show the waste data
hierarchy in Figure 1, and describe the steps below:

Reduce.  At the top of the waste data hierarchy is reduction of waste, which
refers to reducing the amount of waste data generated in the system. We opine
that this is the most favorable option, since less waste will cause the least
overhead on the system. Applications should be designed with waste-reduction
in mind and only store the desired output data in the disk. In-memory caching
of temporary values and content-based addressing can help reduce the amount of
waste data produced by applications. Operating systems and file systems can
provide incentives to applications that produce less waste data and
punishments to those that produce a lot of waste data (we discuss this later
in this section).

Reuse.  In the next layer, we have reuse of waste, which refers to reusing the
waste data for other purposes. Schemes that can be classified as data reuse
include data deduplication [19], where the content of waste data objects can
be used by the deduplication scheme to achieve better compression ratios.
Another example is the reuse of translation memories in machine translation,
where the information from one translation session can be used to enrich
global translation capabilities. Google’s Translation Toolkit already allows
this type of data reuse, and it has been used successfully for translation of
English Wikipedia articles into African and South Asian languages [6]. For
degraded data, regeneration [18] or restoration [14] can be used to recover
data quality.

Recycle.  Slightly less preferable than reuse is to recycle waste data, where
data objects can be broken up and used for different objects. While it is
difficult to fathom what it means to recycle application specific data for
other purposes, we can definitely recycle waste containing application code.
When an obsolete software package is going to be removed, we can extract the
usable components from it and use them for other applications.

Recover.  Sometimes, the waste data cannot be recycled or reused. A
possibility of still gaining some utility from such data is to recover
information. For example, used log files can be anonymized and shared or
analyzed for getting high-level views. Obsolete data can also be mined to
gather patterns about historical trends.

Dispose.  At the bottom of the hierarchy sits schemes for Disposal of data,
through deletion. This is costly in terms of time and energy spent deleting
data objects. So, we opine that deletion should be the absolute last recourse
in managing waste data.

Above the hierarchy is the ideal state of zero waste, where careful system
design results in production of no waste data. Below this hierarchy of schemes
for waste data management lies another approach not shown in the hierarchy –
physical elimination. Sometimes, the five schemes may not be enough or
feasible for managing waste data. For example, the data may be stored in
physically immutable media, and hence not subject to any of the above schemes.
Also, security issues and regulations may require physical elimination of the
storage media. This can be achieved by incinerating, degaussing, or destroying
the storage media. However, this has the worst impact on the natural
environment as any such disposal would impact the physical ecosystem.

Next, we discuss some specific strategies and best practices for waste data
management.

###  4.2 Some schemes for managing waste data

Taking a cue from successful real life waste management strategies, we propose
several schemes for managing waste data in digital ecosystems. For this, we
leverage the concepts of waste hierarchy and extended producer responsibility
[7].

Digital landfills.  A digital landfill is the equivalent of real-life
landfills, where unwanted data can be disposed of without additional cost
associated with deletion. For this, we propose using a semi-volatile storage
device. Such a storage device would store data, but gradually unwanted data
objects will fade automatically and the storage space can be reclaimed. This
type of device can be implemented on a volatile storage medium using a least-
recently-used (LRU) scheme, where data which has not been used recently is
allowed to fade, while more frequently used data is refreshed.

Waste penalties for applications.  A waste penalty can be imposed on
applications that create large amounts of waste data. For example, the
operating system can penalize an application that creates a lot of temporary
files by reducing its I/O bandwidth or schedule it to receive fewer CPU
cycles. This gives applications incentives to act responsibly in creating
waste data. This concept is equivalent to the Pay-as-You-Throw scheme and the
polluter-pays principle used in real life waste management [5].

Extensive system-wide Deduplication and Micro-modular software.  A big problem
with recycling old or unwanted software is that software libraries are not
usually written to allow extraction of small amounts of code. Shared dynamic
link libraries do allow code sharing among multiple applications [8]. However,
they do not allow removal of unused routines to retain only the routines that
are used. When recycling a library, it is therefore not possible to extract
usable routines from it. To allow recycling old code, we propose breaking up
software code libraries into micro-modules in the level of individual routines
or algorithms, which can be extracted from the library when recycling old
code.

##  5 Related Work

While researchers have explored different storage management issues, the
systematic management of waste data has received little interest. Information
lifecycle management (ILM) has been used by the storage industry to determine
optimal management of data objects throughout their lifecycle [12]. A major
challenge in ILM is to design valuation schemes to determine the importance of
information. Chen presented such a scheme based on file access patterns [3].
We can use such schemes to identify waste data. Zadok et al. [18] advocated
for the need to reduce storage consumption through application of regeneration
and smart space reclamation policies, in order to increase device lifetimes
and available storage. Researchers have also analyzed existing systems to
identify typical usage patterns. An early work by Satyanarayanan [13]
introduced the notion of functional lifetime (f-lifetime) for files, defined
as the difference between a file’s age and the time since its last access.
Files with lower f-lifetimes are less useful, since the gap between their
creation/last modification times and last access times are short. In a study
of file systems, Douceur et al. [4] showed that 44% of the files in the
studied systems had an f-lifetime of zero (the percentage was higher, at 67%,
for technical support systems), indicating that these files have not been
accessed at all since last modification. This agrees with our findings
presented in Section 3. Vogels found that file lifetimes are often quite short
– almost 80% files are actually deleted within 4 seconds [15]. The very short
lifetime indicates that the usefulness of these files ends quickly. Deleting
these files is costly, and application designers should rethink their I/O to
prevent the creation of such waste data. Finally, researchers have developed
techniques such as software refactoring and reuse [8], and data deduplication
[19], which can be applied in different stages of the waste data hierarchy to
reduce the impact of waste data.

##  6 Conclusion

For many years, the abundance of storage space and decreasing storage costs
have allowed us to ignore the adverse impact of waste data on our digital
ecosystems. But as we start dealing with massive quantities of data, we need
to manage waste data in order to reduce overheads and energy costs, and
improve efficiency. In this paper, we defined the waste data problem and
proposed using techniques from real life waste management to minimize the
impact of waste data on our computing environment. Our waste data management
hierarchy can be used to determine the preferable option in dealing with waste
data in different applications. We also advocated the adoption of responsible
application behavior and best practices in reducing the impact of waste data.
We posit that software engineering techniques as well as hardware
architectures will need to be adapted with waste data minimization,
management, and recycling in mind, in order to build a efficient and
sustainable digital ecosystem.

## Acknowledgements

This work was supported by the National Science Foundation under Grant
#0937060 to the Computing Research Association for the CIFellows Project.

## References

  * [1] The Waste Framework Directive.  European Economic Community Directive (75/442/EEC), 1975.
  * [2] E. Baker, E. Bournay, A. Harayama, and P. Rekacewicz.  Vital waste graphics.  United Nations Environment Program, 2004.
  * [3] Y. Chen.  Information valuation for information lifecycle management.  In Proc. of the 2nd International Conf. on Autonomic Computing, pages 135–146, 2005.
  * [4] J. R. Douceur and W. J. Bolosky.  A large-scale study of file-system contents.  SIGMETRICS Perform. Eval. Rev., 27(1):59–70, 1999.
  * [5] EPA.  Pay-as-you-throw.  Online at http://epa.gov/osw/conserve/tools/payt, 2009.
  * [6] M. Galvez.  Translating wikipedia.  Google Translate Blog, http://googletranslate.blogspot.com/2010/07/translating-wi%kipedia.html, 2010.
  * [7] R. Lifset.  Take it back: extended producer responsibility as a form of incentive-based environmental policy.  J. Resour. Manage. Technol., 21(4):163–175, 1993.
  * [8] H. Mili, F. Mili, and A. Mili.  Reusing software: Issues and research directions.  IEEE Trans. on Software Eng, 21(6):528–562, 2002.
  * [9] OECD.  Glossary of statistical terms.  Online at http://stats.oecd.org/glossary/detail.asp?ID=2896.
  * [10] E. Pongrácz, P. Phillips, and R. Keiski.  Evolving the theory of waste management-implications to waste minimization.  In Proc. of the Waste minimization and Resources Use Optimization Conference, pages 61–67, 2004.
  * [11] E. Pongrácz and V. Pohjola.  Re-defining waste, the concept of ownership and the role of waste management.  Resources, Conservation and Recycling, 40(2):141–153, 2004.
  * [12] D. Reiner, G. Press, M. Lenaghan, D. Barta, and R. Urmston.  Information lifecycle management: the EMC perspective.  In Proc. of ICDE, pages 804–807, 2004.
  * [13] M. Satyanarayanan.  A study of file sizes and functional lifetimes.  ACM SIGOPS Operating Systems Review, 15(5):96–108, 1981.
  * [14] B. Schroeder, S. Damouras, and P. Gill.  Understanding latent sector errors and how to protect against them.  In Proc. of the 8th USENIX FAST, pages 1–14, 2010.
  * [15] W. Vogels.  File system usage in windows nt 4.0.  In Proc. of ACM SOSP, pages 93–109, 1999.
  * [16] Y. Wang, D. Liu, M. Wang, Z. Qin, Z. Shao, and Y. Guan.  RNFTL: a reuse-aware NAND flash translation layer for flash memory.  SIGPLAN Not., 45(4):163–172, 2010.
  * [17] D. Wilson.  Stick or carrot?: The use of policy measures to move waste management up the hierarchy.  Waste Management & Research, 14(4):385, 1996.
  * [18] E. Zadok, J. Osborn, A. Shater, C. Wright, K. Muniswamy-Reddy, and J. Nieh.  Reducing storage management costs via informed user-based policies.  In Proc. of IEEE MSST, pages 193–197, 2004.
  * [19] B. Zhu, K. Li, and H. Patterson.  Avoiding the disk bottleneck in the data domain deduplication file system.  In Proc. of the 6th USENIX FAST, pages 1–14, 2008.