# Supplementary Material for
Non-monotonic temperature dependence and first-order phase transition of
relaxation times in molecular spin

Le Tuan Anh Ho<EMAIL_ADDRESS>Department of Chemistry, National
University of Singapore, 3 Science Drive 3 Singapore 117543  Liviu Ungur
Department of Chemistry, National University of Singapore, 3 Science Drive 3
Singapore 117543  Liviu F. Chibotaru<EMAIL_ADDRESS>Theory of
Nanomaterials Group, Katholieke Universiteit Leuven, Celestijnenlaan 200F,
B-3001 Leuven, Belgium

###### Contents

  1. I Density matrix elements of the excited doublets/singlets in the stationary limit
    1. I.1 Diagonal terms
    2. I.2 Off-diagonal terms
    3. I.3 Summary
  2. II Redfield master equation for the ground doublet
    1. II.1 Diagonal terms
    2. II.2 Off-diagonal terms
    3. II.3 Summary
  3. III Magnetization relaxation of a molecular spin at resonance
    1. III.1 Relaxation at transition point $\gamma=\Delta_{1}$
    2. III.2 Relaxation in the positive proximity of the transition point $\gamma=\Delta_{1}+\delta\gamma$
  4. IV Magnetization relaxation of a molecular spin in general case
  5. V Magnetization relaxation fitting: single-, bi-, or tri-exponential form?

##  I Density matrix elements of the excited doublets/singlets in the
stationary limit

In the stationary limit of excited doublets/singlets $m^{\mathrm{th}}$, we
have:

$\frac{d\rho_{mm}}{dt}=\frac{d\rho_{m^{\prime}m^{\prime}}}{dt}=0,\,\frac{d\rho_{mm^{\prime}}}{dt}=\frac{d\rho_{m^{\prime}m}}{dt}=0.$
(1)

If we consider the density matrix elements corresponding to the ground doublet
as parameters, ones corresponding to excited levels can be found by solving
this system of equations. Here all equations are formulated in localized basis
[Ho2022a].

###  I.1 Diagonal terms

The symmetry between $\ket{m}$ and $\ket{m^{\prime}}$ as “up” and “down”
states corresponding to $m^{\mathrm{th}}$ doublet allows us to approximate the
form of the diagonal density matrix elements for excited doublets/singlets of
the investigated spin system as:

$\displaystyle\rho_{mm}$
$\displaystyle=A_{mm}^{11}\rho_{11}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{11^{\prime}}\rho_{11^{\prime}}+A_{mm}^{1^{\prime}1}\rho_{1^{\prime}1},$
(2) $\displaystyle\rho_{m^{\prime}m^{\prime}}$
$\displaystyle=A_{mm}^{11}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{11}+A_{mm}^{11^{\prime}}\rho_{1^{\prime}1}+A_{mm}^{1^{\prime}1}\rho_{11^{\prime}},$
(3)

where $A_{mn}^{ij}$ are parameters. These parameters can be found by
numerically solving the system of equations (1). However, here we focus more
on their properties instead of their value since these are not necessary for
our work hereinafter.

In the stationary limit, we have
$d\rho_{mm}/dt=d\rho_{m^{\prime}m^{\prime}}/dt=0$ for excited
doublets/singlets. This also implies
$d\left(\rho_{mm}+\rho_{m^{\prime}m^{\prime}}\right)/dt=0$ and
$d\left(\rho_{11}+\rho_{1^{\prime}1^{\prime}}\right)/dt=0$, accordingly,

$\displaystyle\left(A_{mm}^{11^{\prime}}+A_{mm}^{1^{\prime}1}\right)\frac{d\rho_{11^{\prime}r}}{dt}=0,$
(4)

at any moment $t$. This is satisfied if and only if
$A_{mm}^{11^{\prime}}+A_{mm}^{1^{\prime}1}=0$. Since
$\rho_{mm^{\prime}}=\rho_{m^{\prime}m}^{*}$ and $\rho_{mm}$,
$\rho_{m^{\prime}m^{\prime}}$ are real, it is easy to see that
$A_{mm}^{11^{\prime}}=\left(A_{mm}^{1^{\prime}1}\right)^{*}$. These two
equations results in:

$\mathrm{Re}\left(A_{mm}^{11^{\prime}r}\right)=0,$ (5)

thus we can rewrite Eq. (2) and (3) as:

$\displaystyle\rho_{mm}=A_{mm}^{11}\rho_{11}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{11^{\prime}i}\rho_{11^{\prime}i},$
(6)
$\displaystyle\rho_{m^{\prime}m^{\prime}}=A_{mm}^{11}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{11}-A_{mm}^{11^{\prime}i}\rho_{11^{\prime}i},$
(7)

where we have defined
$A_{mm}^{11^{\prime}i}\equiv-2\,\mathrm{Im}\left(A_{mm}^{11^{\prime}}\right)$
and $\rho_{11^{\prime}r}$, $\rho_{11^{\prime}i}$ are respective real and
imaginary part of $\rho_{11^{\prime}}$.

###  I.2 Off-diagonal terms

Similarly, off-diagonal density matrix elements for excited doublets/singlets
can take the form:

$\displaystyle\rho_{mm^{\prime}}=A_{mm^{\prime}}^{11}\rho_{11}+A_{mm^{\prime}}^{1^{\prime}1^{\prime}}\rho_{1^{\prime}1^{\prime}}+A_{mm^{\prime}}^{11^{\prime}}\rho_{11^{\prime}}+A_{mm^{\prime}}^{1^{\prime}1}\rho_{1^{\prime}1},$
(8)
$\displaystyle\rho_{m^{\prime}m}=A_{mm^{\prime}}^{11}\rho_{1^{\prime}1^{\prime}}+A_{mm^{\prime}}^{1^{\prime}1^{\prime}}\rho_{11}+A_{mm^{\prime}}^{11^{\prime}}\rho_{1^{\prime}1}+A_{mm^{\prime}}^{1^{\prime}1}\rho_{11^{\prime}}.$
(9)

Since $\rho_{mm^{\prime}}=\rho_{m^{\prime}m}^{*}$, we have:

$A_{mm^{\prime}}^{11}=\left(A_{mm^{\prime}}^{1^{\prime}1^{\prime}}\right)^{*},\,A_{mm^{\prime}}^{1^{\prime}1^{\prime}}=\left(A_{mm^{\prime}}^{11}\right)^{*},\,A_{mm^{\prime}}^{11^{\prime}}=\left(A_{mm^{\prime}}^{11^{\prime}}\right)^{*},\,A_{mm^{\prime}}^{1^{\prime}1}=\left(A_{mm^{\prime}}^{1^{\prime}1}\right)^{*}.$
(10)

In other words, $A_{mm^{\prime}}^{11^{\prime}}$ and
$A_{mm^{\prime}}^{1^{\prime}1}$ are real numbers while $A_{mm^{\prime}}^{11}$
and $A_{mm^{\prime}}^{1^{\prime}1^{\prime}}$ are complex conjugate.

We thus can easily write the real part $\rho_{mm^{\prime}r}$ of
$\rho_{mm^{\prime}}$:

$\rho_{mm^{\prime}r}=\frac{1}{2}\left[\left(A_{mm^{\prime}}^{11}+A_{mm^{\prime}}^{1^{\prime}1^{\prime}}\right)\left(\rho_{11}+\rho_{1^{\prime}1^{\prime}}\right)+\left(A_{mm^{\prime}}^{11^{\prime}}+A_{mm^{\prime}}^{1^{\prime}1}\right)\left(\rho_{11^{\prime}}+\rho_{1^{\prime}1}\right)\right].$
(11)

Since at equilibrium,
$\rho_{mm^{\prime}r}^{\left(eq\right)}=\rho_{11^{\prime}}^{\left(eq\right)}=\rho_{1^{\prime}1}^{\left(eq\right)}\approx
0$ whereas
$\rho_{11}^{\left(eq\right)}\approx\rho_{1^{\prime}1^{\prime}}^{\left(eq\right)}\neq
0$, we have
$\left(A_{mm^{\prime}}^{11}+A_{mm^{\prime}}^{1^{\prime}1^{\prime}}\right)\approx
0$. Thus,

$\rho_{mm^{\prime}r}=A_{mm^{\prime}}^{11^{\prime}r}\rho_{11^{\prime}r},$ (12)

where we have defined $A_{mm^{\prime}}^{11^{\prime}r}\equiv
A_{mm^{\prime}}^{11^{\prime}}+A_{mm^{\prime}}^{1^{\prime}1}$.

Similarly for $\rho_{mm^{\prime}i}$, we have

$\displaystyle\rho_{mm^{\prime}i}$
$\displaystyle=A_{mm^{\prime}}^{11i}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)+A_{mm^{\prime}}^{11^{\prime}i}\rho_{11^{\prime}i},$
(13)

where
$A_{mm^{\prime}}^{11i}\equiv\mathrm{Im}\left(A_{mm^{\prime}}^{11}\right)$ and
$A_{mm}^{11^{\prime}i}\equiv
A_{mm^{\prime}}^{11^{\prime}}-A_{mm^{\prime}}^{1^{\prime}1}$.

###  I.3 Summary

In summary we have the following approximations for density matrix elements of
the excited doublets/singlets in stationary limit:

$\displaystyle\rho_{mm}=A_{mm}^{11}\rho_{11}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{11^{\prime}i}\rho_{11^{\prime}i},$
(14)
$\displaystyle\rho_{m^{\prime}m^{\prime}}=A_{mm}^{11}\rho_{1^{\prime}1^{\prime}}+A_{mm}^{1^{\prime}1^{\prime}}\rho_{11}-A_{mm}^{11^{\prime}i}\rho_{11^{\prime}i},$
(15)
$\displaystyle\rho_{mm^{\prime}r}=A_{mm^{\prime}}^{11^{\prime}r}\rho_{11^{\prime}r},$
(16)
$\displaystyle\rho_{mm^{\prime}i}=A_{mm^{\prime}}^{11i}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)+A_{mm^{\prime}}^{11^{\prime}i}\rho_{11^{\prime}i},$
(17)

where all above $A_{mn}^{ij}$ are real parameters.

##  II Redfield master equation for the ground doublet

From the semi-secular approximated Redfield equation for the general
$d\rho_{mm}/dt$ and $d\rho_{mm^{\prime}}/dt$ in the main text, the equation
for density matrix elements of the ground doublet can be straightforwardly
derived:

$\displaystyle\frac{d\rho_{11}}{dt}$
$\displaystyle=\left(\Delta_{1i}\rho_{11^{\prime}r}-\Delta_{1r}\rho_{11^{\prime}i}\right)+\sum_{n^{\mathrm{th}}}\left(R_{11,nn}\rho_{nn}+R_{11,n^{\prime}n^{\prime}}\rho_{n^{\prime}n^{\prime}}\right)+2\sum_{n^{\mathrm{th}}}\left(R_{11,nn^{\prime}}^{r}\rho_{nn^{\prime}r}-R_{11,nn^{\prime}}^{i}\rho_{nn^{\prime}i}\right),$
(18) $\displaystyle\frac{d\rho_{1^{\prime}1^{\prime}}}{dt}$
$\displaystyle=-\left(\Delta_{1i}\rho_{11^{\prime}r}-\Delta_{1r}\rho_{11^{\prime}i}\right)+\sum_{n^{\mathrm{th}}}\left(R_{1^{\prime}1^{\prime},nn}\rho_{nn}+R_{1^{\prime}1^{\prime},n^{\prime}n^{\prime}}\rho_{n^{\prime}n^{\prime}}\right)+2\sum_{n^{\mathrm{th}}}\left(R_{1^{\prime}1^{\prime},nn^{\prime}}^{r}\rho_{nn^{\prime}r}-R_{1^{\prime}1^{\prime},nn^{\prime}}^{i}\rho_{nn^{\prime}i}\right),$
(19) $\displaystyle\frac{d\rho_{11^{\prime}r}}{dt}$
$\displaystyle=-\frac{\Delta_{1i}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)+W_{1}\rho_{11^{\prime}i}+\sum_{n^{\mathrm{th}}}\left(R_{11^{\prime},nn}^{r}\rho_{nn}+R_{11^{\prime},n^{\prime}n^{\prime}}^{r}\rho_{n^{\prime}n^{\prime}}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\sum_{n^{\mathrm{th}}}\left[\left(R_{11^{\prime},nn^{\prime}}^{r}+R_{1^{\prime}1,nn^{\prime}}^{r}\right)\rho_{nn^{\prime}r}-\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)\rho_{nn^{\prime}i}\right],$
(20) $\displaystyle\frac{d\rho_{11^{\prime}i}}{dt}$
$\displaystyle=\frac{\Delta_{1r}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-W_{1}\rho_{11^{\prime}r}+\sum_{n^{\mathrm{th}}}\left(R_{11^{\prime},nn}^{i}\rho_{nn}+R_{11^{\prime},n^{\prime}n^{\prime}}^{i}\rho_{n^{\prime}n^{\prime}}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\sum_{n^{\mathrm{th}}}\left[\left(R_{11^{\prime},nn^{\prime}}^{i}-R_{1^{\prime}1,nn^{\prime}}^{i}\right)\rho_{nn^{\prime}r}+\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)\rho_{nn^{\prime}i}\right].$
(21)

where $R_{ij,mn}$ are matrix elements of the Redfield super-operator
[Garanin2011, Ho2017].

###  II.1 Diagonal terms

Substituting $\rho_{mm}$, $\rho_{m^{\prime}m^{\prime}}$,
$\rho_{mm^{\prime}r}$, and $\rho_{mm^{\prime}i}$ from the above section, Eqs.
(6, 7, 12, 13) into the semi-secular approximated Redfield equation of
$d\rho_{11}/dt$, Eq. (18) then simplifying gives:

$\displaystyle\frac{d\rho_{11}}{dt}$
$\displaystyle=\left(\Delta_{1i}\rho_{11^{\prime}r}-\Delta_{1r}\rho_{11^{\prime}i}\right)+\sum_{n^{\mathrm{th}}}\left(R_{11,nn}\rho_{nn}+R_{11,n^{\prime}n^{\prime}}\rho_{n^{\prime}n^{\prime}}\right)+2\sum_{n^{\mathrm{th}}}R_{11,nn^{\prime}}\rho_{nn^{\prime}r}$
$\displaystyle=\left(\Delta_{1i}+2R_{11,11^{\prime}}^{r}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{r}A_{nn^{\prime}}^{11^{\prime}r}\right)\rho_{11^{\prime}r}$
$\displaystyle\qquad+\left[-\Delta_{1r}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,nn}-R_{11,n^{\prime}n^{\prime}}\right)A_{nn}^{11^{\prime}i}-2R_{11,11^{\prime}}^{i}-2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11^{\prime}i}\right]\rho_{11^{\prime}i}$
$\displaystyle\qquad+\left[R_{11,11}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,nn}A_{nn}^{11}+R_{11,n^{\prime}n^{\prime}}A_{nn}^{1^{\prime}1^{\prime}}\right)-2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right]\rho_{11}$
$\displaystyle\qquad+\left[R_{11,1^{\prime}1^{\prime}}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,nn}A_{nn}^{1^{\prime}1^{\prime}}+R_{11,n^{\prime}n^{\prime}}A_{nn}^{11}\right)+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right]\rho_{1^{\prime}1^{\prime}}$
$\displaystyle\equiv\Delta_{1i}^{\prime}\rho_{11^{\prime}r}-\Delta_{1r}^{\prime}\rho_{11^{\prime}i}+\frac{1}{2}\left(\Gamma_{e}\rho_{1^{\prime}1^{\prime}}-\Gamma_{e}^{\prime}\rho_{11}\right).$
(22)

Similarly we have for $d\rho_{1^{\prime}1^{\prime}}/dt$:

$\displaystyle\frac{d\rho_{1^{\prime}1^{\prime}}}{dt}$
$\displaystyle=\left(-\Delta_{1i}+2R_{1^{\prime}1^{\prime},11^{\prime}}^{r}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{1^{\prime}1^{\prime},nn^{\prime}}^{r}A_{nn^{\prime}}^{11^{\prime}r}\right)\rho_{11^{\prime}r}$
$\displaystyle\qquad+\left[\Delta_{1r}-2R_{1^{\prime}1^{\prime},11^{\prime}}^{i}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{1^{\prime}1^{\prime},nn}-R_{11,nn}\right)A_{nn}^{11^{\prime}i}-2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{1^{\prime}1^{\prime},nn^{\prime}}^{i}A_{nn^{\prime}}^{11^{\prime}i}\right]\rho_{11^{\prime}i}$
$\displaystyle\qquad+\left[R_{1^{\prime}1^{\prime},11}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{1^{\prime}1^{\prime},nn}A_{nn}^{11}+R_{1^{\prime}1^{\prime},n^{\prime}n^{\prime}}A_{nn}^{1^{\prime}1^{\prime}}\right)-2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{1^{\prime}1^{\prime},nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right]\rho_{11}$
$\displaystyle\qquad+\left[R_{1^{\prime}1^{\prime},1^{\prime}1^{\prime}}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{1^{\prime}1^{\prime},nn}A_{nn}^{1^{\prime}1^{\prime}}+R_{1^{\prime}1^{\prime},n^{\prime}n^{\prime}}A_{nn}^{11}\right)+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{1^{\prime}1^{\prime},nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right]\rho_{1^{\prime}1^{\prime}}$
$\displaystyle\equiv-\Delta_{1i}^{\prime\prime}\rho_{11^{\prime}r}+\Delta_{1r}^{\prime\prime}\rho_{11^{\prime}i}+\frac{1}{2}\left(\Gamma_{e}^{\prime\prime}\rho_{11}-\Gamma_{e}^{\prime\prime\prime}\rho_{1^{\prime}1^{\prime}}\right).$
(23)

At equilibrium we have $d\rho_{11}/dt=d\rho_{1^{\prime}1^{\prime}}/dt=0$.
Supposing that the energy bias of the ground doublet is much smaller than the
temperature, i.e. $\rho_{11}^{\left(eq\right)}=\rho_{11}^{\left(eq\right)}$
and
$\rho_{11^{\prime}r}^{\left(eq\right)}=\rho_{11^{\prime}i}^{\left(eq\right)}=0$,
we further obtain:

$\displaystyle\Gamma_{e}=\Gamma_{e}^{\prime},$ (24)
$\displaystyle\text{$\Gamma_{e}^{\prime\prime}$=$\Gamma_{e}^{\prime\prime\prime}$},$
(25)

In addition, within the stationary limit we have
$d\left(\rho_{11}+\rho_{1^{\prime}1^{\prime}}\right)/dt=0$ at any time $t$, it
is straightforward from Eqs. (22-23) that

$\displaystyle\Delta_{1r}^{\prime\prime}=\Delta_{1r}^{\prime},$ (26)
$\displaystyle\Delta_{1i}^{\prime\prime}=\Delta_{1i}^{\prime},$ (27)
$\displaystyle\Gamma_{e}=\Gamma_{e}^{\prime}=\Gamma_{e}^{\prime\prime}=\Gamma_{e}^{\prime\prime\prime}.$
(28)

Accordingly, we can reduce Eq. (22) and (23) further to obtain:

$\displaystyle\frac{d\rho_{11}}{dt}$
$\displaystyle=\Delta_{1i}^{\prime}\rho_{11^{\prime}r}-\Delta_{1r}^{\prime}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{1^{\prime}1^{\prime}}-\rho_{11}\right),$
(29) $\displaystyle\frac{d\rho_{1^{\prime}1^{\prime}}}{dt}$
$\displaystyle=-\Delta_{1i}^{\prime}\rho_{11^{\prime}r}+\Delta_{1r}^{\prime}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right).$
(30)

where:

$\displaystyle\Delta_{1r}^{\prime}$
$\displaystyle\equiv\Delta_{1r}-\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,nn}-R_{11,n^{\prime}n^{\prime}}\right)A_{nn}^{11^{\prime}i}+2R_{11,11^{\prime}}^{i}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11^{\prime}i}\approx\Delta_{1r},$
(31) $\displaystyle\Delta_{1i}^{\prime}$
$\displaystyle\equiv\Delta_{1i}+2R_{11,11^{\prime}}^{r}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{r}A_{nn^{\prime}}^{11^{\prime}r}\approx\Delta_{1i},$
(32) $\displaystyle\Gamma_{e}$
$\displaystyle=2\left[R_{11,1^{\prime}1^{\prime}}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,n^{\prime}n^{\prime}}A_{nn}^{11}+R_{11,nn}A_{nn}^{1^{\prime}1^{\prime}}\right)+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right].$ (33)

and the approximated version:

$\displaystyle\frac{d\rho_{11}}{dt}$
$\displaystyle=\Delta_{1i}\rho_{11^{\prime}r}-\Delta_{1r}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{1^{\prime}1^{\prime}}-\rho_{11}\right),$
(34) $\displaystyle\frac{d\rho_{1^{\prime}1^{\prime}}}{dt}$
$\displaystyle=-\Delta_{1i}\rho_{11^{\prime}r}+\Delta_{1r}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right).$
(35)

###  II.2 Off-diagonal terms

Similar to $d\rho_{11}/dt$ and $d\rho_{1^{\prime}1^{\prime}}/dt$, by
substituting $\rho_{mm}$, $\rho_{m^{\prime}m^{\prime}}$,
$\rho_{mm^{\prime}r}$, and $\rho_{mm^{\prime}i}$ from Eqs. (6, 7, 12, 13) into
the semi-secular approximated Redfield equation of $d\rho_{11^{\prime}r}/dt$,
we obtain the simplified equation for $d\rho_{11^{\prime}r}/dt$:

$\displaystyle\frac{d\rho_{11^{\prime}r}}{dt}$
$\displaystyle=\left[-\frac{\Delta_{1i}}{2}+R_{11^{\prime},11}^{r}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{r}A_{nn}^{11}+R_{11^{\prime},n^{\prime}n^{\prime}}^{r}A_{nn}^{1^{\prime}1^{\prime}}\right)-\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11i}\right]\rho_{11}$
$\displaystyle\qquad+\left[\frac{\Delta_{1i}}{2}+R_{11^{\prime},1^{\prime}1^{\prime}}^{r}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{r}A_{nn}^{1^{\prime}1^{\prime}}+R_{11^{\prime},n^{\prime}n^{\prime}}^{r}A_{nn}^{11}\right)+\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11i}\right]\rho_{1^{\prime}1^{\prime}}$
$\displaystyle\qquad+\left[R_{11^{\prime},11^{\prime}}^{r}+R_{1^{\prime}1,11^{\prime}}^{r}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn^{\prime}}^{r}+R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11^{\prime}r}\right]\rho_{11^{\prime}r}$
$\displaystyle\qquad+\left[W_{1}-\left(R_{11^{\prime},11^{\prime}}^{i}+R_{1^{\prime}1,11^{\prime}}^{i}\right)+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{r}A_{nn}^{11^{\prime}i}-R_{11^{\prime},n^{\prime}n^{\prime}}^{r}A_{nn}^{11^{\prime}i}\right)-\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11^{\prime}i}\right]\rho_{11^{\prime}i}$
$\displaystyle\equiv$
$\displaystyle-\frac{\Delta_{1i}^{\prime\prime\prime}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}^{\prime}\rho_{11^{\prime}r}+W_{1}^{\prime}\rho_{11^{\prime}i}.$
(36)

where

$\displaystyle\Delta_{1i}^{\prime\prime\prime}$
$\displaystyle\equiv\Delta_{1i}-2R_{11^{\prime},11}^{r}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left[\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11i}-\left(R_{11^{\prime},nn}^{r}A_{nn}^{11}+R_{11^{\prime},n^{\prime}n^{\prime}}^{r}A_{nn}^{1^{\prime}1^{\prime}}\right)\right]\approx\Delta_{1i}$
(37) $\displaystyle\gamma_{11^{\prime}}^{\prime}$
$\displaystyle\equiv-\left[R_{11^{\prime},11^{\prime}}^{r}+R_{1^{\prime}1,11^{\prime}}^{r}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn^{\prime}}^{r}+R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11^{\prime}r}\right]\approx-
R_{11^{\prime},11^{\prime}}^{r}\equiv\gamma_{11^{\prime}}$ (38) $\displaystyle
W_{1}^{\prime}$ $\displaystyle\equiv
W_{1}-\left(R_{11^{\prime},11^{\prime}}^{i}+R_{1^{\prime}1,11^{\prime}}^{i}\right)+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{r}A_{nn}^{11^{\prime}i}-R_{11^{\prime},n^{\prime}n^{\prime}}^{r}A_{nn}^{11^{\prime}i}\right)-\left(R_{11^{\prime},nn^{\prime}}^{i}+R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11^{\prime}i}\approx
W_{1}$ (39)

Meanwhile, the equation for $d\rho_{11^{\prime}i}/dt$ is:

$\displaystyle\frac{d\rho_{11^{\prime}i}}{dt}$
$\displaystyle=\left[\frac{\Delta_{1r}}{2}+R_{11^{\prime},11}^{i}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{i}A_{nn}^{11}+R_{11^{\prime},n^{\prime}n^{\prime}}^{i}A_{nn}^{1^{\prime}1^{\prime}}\right)+\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11i}\right]\rho_{11}$
$\displaystyle+\left[-\frac{\Delta_{1r}}{2}+R_{11^{\prime},1^{\prime}1^{\prime}}^{i}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{i}A_{nn}^{1^{\prime}1^{\prime}}+R_{11^{\prime},n^{\prime}n^{\prime}}^{i}A_{nn}^{11}\right)-\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11i}\right]\rho_{1^{\prime}1^{\prime}}$
$\displaystyle+\left[-W_{1}+\left(R_{11^{\prime},11^{\prime}}^{i}-R_{1^{\prime}1,11^{\prime}}^{i}\right)+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn^{\prime}}^{i}-R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11^{\prime}r}\right]\rho_{11^{\prime}r}$
$\displaystyle+\left[\left(R_{11^{\prime},11^{\prime}}^{r}-R_{1^{\prime}1,11^{\prime}}^{r}\right)+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{i}A_{nn}^{11^{\prime}i}-R_{11^{\prime},n^{\prime}n^{\prime}}^{i}A_{nn}^{11^{\prime}i}\right)+\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11^{\prime}i}\right]\rho_{11^{\prime}i}$
$\displaystyle\equiv\frac{\Delta_{1r}^{\prime\prime\prime}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}^{\prime\prime}\rho_{11^{\prime}i}-W_{1}^{\prime\prime}\rho_{11^{\prime}r}.$
(40)

where

$\displaystyle\Delta_{1r}^{\prime\prime\prime}$
$\displaystyle\equiv\Delta_{1r}+2R_{11^{\prime},11}^{i}+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left[\left(R_{11^{\prime},nn}^{i}A_{nn}^{11}+R_{11^{\prime},n^{\prime}n^{\prime}}^{i}A_{nn}^{1^{\prime}1^{\prime}}\right)+\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11i}\right]\approx\Delta_{1r}$
(41) $\displaystyle\gamma_{11^{\prime}}^{\prime\prime}$
$\displaystyle\equiv-\left[\left(R_{11^{\prime},11^{\prime}}^{r}-R_{1^{\prime}1,11^{\prime}}^{r}\right)+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn}^{i}A_{nn}^{11^{\prime}i}-R_{11^{\prime},n^{\prime}n^{\prime}}^{i}A_{nn}^{11^{\prime}i}\right)+\left(R_{11^{\prime},nn^{\prime}}^{r}-R_{1^{\prime}1,nn^{\prime}}^{r}\right)A_{nn^{\prime}}^{11^{\prime}i}\right]\approx-
R_{11^{\prime},11^{\prime}}^{r}\equiv\gamma_{11^{\prime}}$ (42) $\displaystyle
W_{1}^{\prime\prime}$ $\displaystyle\equiv
W_{1}-\left(R_{11^{\prime},11^{\prime}}^{i}-R_{1^{\prime}1,11^{\prime}}^{i}\right)-\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11^{\prime},nn^{\prime}}^{i}-R_{1^{\prime}1,nn^{\prime}}^{i}\right)A_{nn^{\prime}}^{11^{\prime}r}\approx
W_{1}$ (43)

The approximated equations of $d\rho_{11^{\prime}r}/dt$ and
$d\rho_{11^{\prime}i}/dt$ then are:

$\displaystyle\frac{d\rho_{11^{\prime}r}}{dt}$
$\displaystyle=-\frac{\Delta_{1i}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}\rho_{11^{\prime}r}+W_{1}\rho_{11^{\prime}i},$
(44) $\displaystyle\frac{d\rho_{11^{\prime}i}}{dt}$
$\displaystyle=\frac{\Delta_{1r}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}\rho_{11^{\prime}i}-W_{1}\rho_{11^{\prime}r}.$
(45)

###  II.3 Summary

Within the stationary limit for excited doublets/singlets, the approximated
Redfield master equation for the ground doublet, and relaxation of the spin
system, can be summarized as follows:

$\displaystyle\frac{d\rho_{11}}{dt}$
$\displaystyle=\Delta_{1i}\rho_{11^{\prime}r}-\Delta_{1r}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{1^{\prime}1^{\prime}}-\rho_{11}\right),$
(46) $\displaystyle\frac{d\rho_{1^{\prime}1^{\prime}}}{dt}$
$\displaystyle=-\Delta_{1i}\rho_{11^{\prime}r}+\Delta_{1r}\rho_{11^{\prime}i}+\frac{1}{2}\Gamma_{e}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right),$
(47) $\displaystyle\frac{d\rho_{11^{\prime}r}}{dt}$
$\displaystyle=-\frac{\Delta_{1i}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}\rho_{11^{\prime}r}+W_{1}\rho_{11^{\prime}i},$
(48) $\displaystyle\frac{d\rho_{11^{\prime}i}}{dt}$
$\displaystyle=\frac{\Delta_{1r}}{2}\left(\rho_{11}-\rho_{1^{\prime}1^{\prime}}\right)-\gamma_{11^{\prime}}\rho_{11^{\prime}i}-W_{1}\rho_{11^{\prime}r}.$
(49)

where

$\displaystyle\Gamma_{e}$
$\displaystyle=2\left[R_{11,1^{\prime}1^{\prime}}+\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}\left(R_{11,n^{\prime}n^{\prime}}A_{nn}^{11}+R_{11,nn}A_{nn}^{1^{\prime}1^{\prime}}\right)+2\sum_{n^{\mathrm{th}}\neq
1^{\mathrm{st}}}R_{11,nn^{\prime}}^{i}A_{nn^{\prime}}^{11i}\right].$

As the states $\ket{1}$ and $\ket{1^{\prime}}$ have opposite magnetic moments,
in order to find the magnetization relaxation rate of the spin system, we can
change the variables $\rho_{11}$ and $\rho_{1^{\prime}1^{\prime}}$ to
$X_{1}\equiv\rho_{11}-\rho_{1^{\prime}1^{\prime}}$. The above differential
equations can thus be rewritten as:

$\displaystyle\frac{dX_{1}}{dt}=-\Gamma_{e}X_{1}-2\left(\Delta_{1r}\rho_{11^{\prime}i}-\Delta_{1i}\rho_{11^{\prime}r}\right),$
(50)
$\displaystyle\frac{d\rho_{11^{\prime}r}}{dt}=-\gamma_{11^{\prime}}\rho_{11^{\prime}r}+W_{1}\rho_{11^{\prime}i}-\frac{\Delta_{1i}}{2}X_{1},$
(51)
$\displaystyle\frac{d\rho_{11^{\prime}i}}{dt}=-W_{1}\rho_{11^{\prime}r}-\gamma_{11^{\prime}}\rho_{11^{\prime}i}+\frac{\Delta_{1r}}{2}X_{1},$
(52)

or in the matrix form:

$\frac{d\overrightarrow{\rho}}{dt}=\Phi\overrightarrow{\rho},$ (53)

where the transition rate matrix $\Phi$ and $\overrightarrow{\rho}$,
respectively, are:

$\Phi=\left(\begin{array}[]{ccc}-\Gamma_{e}&2\Delta_{1i}&-2\Delta_{1r}\\\
-\Delta_{1i}/2&-\gamma_{11^{\prime}}&W_{1}\\\
\Delta_{1r}/2&-W_{1}&-\gamma_{11^{\prime}}\end{array}\right),\,\overrightarrow{\rho}=\left(\begin{array}[]{c}X_{1}\\\
\rho_{11^{\prime}r}\\\ \rho_{11^{\prime}i}\end{array}\right).$ (54)

Three eigenvalues of the matrix $\Phi$ are corresponding rates of three
relaxation modes.

We also have the corresponding characteristic polynomial:

$\lambda^{3}-\lambda^{2}(2\gamma_{11^{\prime}}+\Gamma_{e})+\lambda\left(\gamma_{11^{\prime}}^{2}+2\gamma_{11^{\prime}}\Gamma_{e}+\Omega_{1}^{2}\right)-\left(\gamma_{11^{\prime}}^{2}\Gamma_{e}+\gamma_{11^{\prime}}\Delta_{1}^{2}+\Gamma_{e}W_{1}^{2}\right)=0,$
(55)

where

$\displaystyle\Omega_{1}$ $\displaystyle=\sqrt{\Delta_{1}^{2}+W_{1}^{2}},$
(56) $\displaystyle\Delta_{1}^{2}$
$\displaystyle=\Delta_{1r}^{2}+\Delta_{1i}^{2}.$ (57)

##  III Magnetization relaxation of a molecular spin at resonance

Relaxation of the magnetization at resonance $W_{1}=0$ can be obtained by
diagonalizing the transition rate matrix Eq. (54). Defining
$\gamma\equiv\left(\gamma_{11^{\prime}}^{\prime}-\Gamma_{e}\right)/2$, the
corresponding relaxation rates and eigenvectors then are:

$\displaystyle\lambda_{1}$
$\displaystyle=\Gamma_{e}+\gamma-\sqrt{\gamma^{2}-\Delta_{1}^{2}},\qquad$
$\displaystyle\qquad\overrightarrow{\rho}_{1}$
$\displaystyle=\left(2\frac{\gamma+\sqrt{\gamma^{2}-\Delta_{1}^{2}}}{\Delta_{1r}},-\frac{\Delta_{1i}}{\Delta_{1r}},1\right),$
(58) $\displaystyle\lambda_{2}$
$\displaystyle=\Gamma_{e}+\gamma+\sqrt{\gamma^{2}-\Delta_{1}^{2}},\qquad\qquad$
$\displaystyle\overrightarrow{\rho}_{2}$
$\displaystyle=\left(2\frac{\gamma-\sqrt{\gamma^{2}-\Delta_{1}^{2}}}{\Delta_{1r}},-\frac{\Delta_{1i}}{\Delta_{1r}},1\right),$
(59) $\displaystyle\lambda_{3}$
$\displaystyle=\gamma_{11^{\prime}}^{\prime},\qquad\qquad$
$\displaystyle\overrightarrow{\rho}_{3}$
$\displaystyle=\left(0,\frac{\Delta_{1r}}{\Delta_{1i}},1\right).$ (60)

If at $t=0$, the system is prepared so that $X_{1}=1$ and
$\rho_{11^{\prime}r}=\rho_{11^{\prime}i}=0$, we have

$\displaystyle\left(\begin{array}[]{c}X_{1}\\\ \rho_{11^{\prime}r}\\\
\rho_{11^{\prime}i}\end{array}\right)=\frac{\Delta_{1r}}{4\sqrt{\gamma^{2}-\Delta_{1}^{2}}}\left(\begin{array}[]{c}2\frac{\gamma+\sqrt{\gamma^{2}-\Delta_{1}^{2}}}{\Delta_{1r}}e^{-\lambda_{1}t}-2\frac{\gamma-\sqrt{\gamma^{2}-\Delta_{1}^{2}}}{\Delta_{1r}}e^{-\lambda_{2}t}\\\
\frac{\Delta_{1i}}{\Delta_{1r}}\left(e^{-\lambda_{2}t}-e^{-\lambda_{1}t}\right)\\\
e^{-\lambda_{1}t}-e^{-\lambda_{2}t}\end{array}\right).$ (67)

Accordingly, the magnetization as function of time is:

$\displaystyle M\left(t\right)$
$\displaystyle\sim\mathrm{Tr}\left(S_{z}\rho\right)=\sum_{m^{\mathrm{th}}}s_{m}\left(\rho_{mm}-\rho_{m^{\prime}m^{\prime}}\right)$
$\displaystyle=\left[s_{1}+\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}\left(A_{mm}^{11}-A_{mm}^{1^{\prime}1^{\prime}}\right)\right]X_{1}+\left[2\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}A_{mm}^{11^{\prime}i}\right]\rho_{11^{\prime}i}$
$\displaystyle=\frac{1}{2\sqrt{\gamma^{2}-\Delta_{1}^{2}}}\left\\{\left(\gamma+\sqrt{\gamma^{2}-\Delta_{1}^{2}}\right)\left[s_{1}+\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}\left(A_{mm}^{11}-A_{mm}^{1^{\prime}1^{\prime}}\right)\right]+\Delta_{1r}\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}A_{mm}^{11^{\prime}i}\right\\}e^{-\lambda_{1}t}$
$\displaystyle-\frac{1}{2\sqrt{\gamma^{2}-\Delta_{1}^{2}}}\left\\{\left(\gamma-\sqrt{\gamma^{2}-\Delta_{1}^{2}}\right)\left[s_{1}+\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}\left(A_{mm}^{11}-A_{mm}^{1^{\prime}1^{\prime}}\right)\right]+\Delta_{1r}\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}A_{mm}^{11^{\prime}i}\right\\}e^{-\lambda_{2}t}$ (68)

Defining:

$\displaystyle r$ $\displaystyle\equiv\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}\frac{s_{m}A_{mm}^{11^{\prime}i}}{s_{1}+\sum_{m^{\mathrm{th}}\neq
1^{\mathrm{st}}}s_{m}\left(A_{mm}^{11}-A_{mm}^{1^{\prime}1^{\prime}}\right)},$
(69) $\displaystyle\Gamma_{1,2}^{\mathrm{tn}}$
$\displaystyle\equiv\gamma\mp\sqrt{\gamma^{2}-\Delta_{1}^{2}},$ (70)

we obtain:

$\displaystyle M\left(t\right)$
$\displaystyle=\frac{M_{0}e^{-\Gamma_{e}t}}{2\sqrt{\gamma^{2}-\Delta_{1}^{2}}}\left[\left(\Gamma_{2}^{\mathrm{tn}}+\Delta_{1r}r\right)e^{-\Gamma_{1}^{\mathrm{tn}}t}-\left(\Gamma_{1}^{\mathrm{tn}}+\Delta_{1r}r\right)e^{-\Gamma_{2}^{\mathrm{tn}}t}\right],$
(71)

where $M_{0}\equiv M\left(t=0\right)$. Considering that the correlation
between the diagonal matrix elements of the excited states and the off-
diagonal density matrix elements of the ground doublet $A_{mm}^{11^{\prime}i}$
is typically weak, the above can be further approximated as:

$\displaystyle M\left(t\right)$
$\displaystyle=\frac{M_{0}e^{-\Gamma_{e}t}}{2\sqrt{\gamma^{2}-\Delta_{1}^{2}}}\left[\Gamma_{2}^{\mathrm{tn}}e^{-\Gamma_{1}^{\mathrm{tn}}t}-\Gamma_{1}^{\mathrm{tn}}e^{-\Gamma_{2}^{\mathrm{tn}}t}\right].$
(72)

###  III.1 Relaxation at transition point $\gamma=\Delta_{1}$

At transition point $\gamma=\Delta_{1}$ (at some $T_{0}$), with the initial
condition $X_{1}=1$ and $\rho_{11^{\prime}r}=\rho_{11^{\prime}i}=0$, we have

$\displaystyle
X_{1}\left(t\right)=(1+\Delta_{1}t)e^{-\left(\Delta_{1}+\Gamma_{e}\right)t},$
(73)
$\displaystyle\rho_{11^{\prime}r}=-\frac{1}{2}\Delta_{1i}te^{-\left(\Delta_{1}+\Gamma_{e}\right)t},$
(74)
$\displaystyle\rho_{11^{\prime}i}=\frac{1}{2}\Delta_{1r}te^{-\left(\Delta_{1}+\Gamma_{e}\right)t},$
(75) $\displaystyle
M\left(t,T_{0}\right)=M_{0}e^{-\left(\Gamma_{e,0}+\Delta_{1}\right)t}\left[1+\Delta_{1}t\right].$
(76)

###  III.2 Relaxation in the positive proximity of the transition point
$\gamma=\Delta_{1}+\delta\gamma$

Supposing that $\gamma_{11^{\prime}}\left(T_{0}+\delta
T\right)=\gamma_{11^{\prime},0}+\chi\delta T$, $\Gamma_{e}\left(T_{0}+\delta
T\right)=\Gamma_{e,0}+\xi\delta T$, accordingly
$\gamma=\frac{1}{2}\left[\left(\gamma_{11^{\prime},0}-\Gamma_{e,0}\right)+\left(\chi-\xi\right)\delta
T\right]$=$\Delta_{1}+\zeta\delta T$. From Eq. (72)), we have

$\displaystyle M\left(t,T_{0}+\delta T\right)$
$\displaystyle=M\left(t,T_{0}\right)+\delta M\left(t,T_{0},\delta T\right)$
(77)

where $M\left(t,T_{0}\right)$ from Eq. (76) and

$\delta M\left(t,T_{0},\delta T\right)\equiv
M_{0}e^{-\left(\Gamma_{e,0}+\Delta_{1}\right)t}\delta
Tt\left[\frac{1}{6}\left(\chi-\xi\right)\Delta_{1}^{2}t^{2}-\xi\Delta_{1}t-\xi\right],$
(78)

which allow us to find the time $\tau_{M}$ from which $\delta
M\left(t\right)>0$:

$\displaystyle\tau_{M}=\frac{\sqrt{3}\xi}{\Delta_{1}\left(\chi-\xi\right)}\left(\sqrt{3}+\sqrt{1+\frac{2\chi}{\xi}}\right),$
(79)

The condition for $\tau_{M}$ in the observable time $\Delta_{1}^{-1}$ can be
easily found

$\frac{\sqrt{3}\xi}{\left(\chi-\xi\right)}\left(\sqrt{3}+\sqrt{1+\frac{2\chi}{\xi}}\right)\leq
1\Longleftrightarrow\frac{\xi}{\chi}\leq\frac{1}{13},$ (80)

or,

$\zeta/\xi\geq 6.$ (81)

##  IV Magnetization relaxation of a molecular spin in general case

The quantity $X_{1}\left(t\right)$ in Eqs. (50-52) can be found in the form
$X_{1}\left(t\right)=e^{-\Gamma_{e}t}\sum_{i=1}^{3}c_{i}e^{-\Gamma_{i}t}$
where $c_{i}$ depends on the initial conditions and $\Gamma_{i}$ are as
follows [Ho2022c]:

$\displaystyle\Gamma_{1}$
$\displaystyle=\frac{4\gamma}{3}-\frac{3\Omega_{1}^{2}-4\gamma^{2}}{3S}+\frac{1}{3}S,$
(82) $\displaystyle\Gamma_{2}$
$\displaystyle=\frac{4\gamma}{3}+\frac{1+i\sqrt{3}}{6}\frac{3\Omega_{1}^{2}-4\gamma^{2}}{S}-\frac{1-i\sqrt{3}}{6}S,$
(83) $\displaystyle\Gamma_{3}$
$\displaystyle=\frac{4\gamma}{3}+\frac{1-i\sqrt{3}}{6}\frac{3\Omega_{1}^{2}-4\gamma_{1}^{2}}{S}-\frac{1+i\sqrt{3}}{6}S,$
(84)

where

$\displaystyle S\equiv$
$\displaystyle\sqrt[3]{9\gamma\left(\Delta_{1}^{2}-2W_{1}^{2}\right)-8\gamma^{3}+3\sqrt{3}\sqrt{D}},$
(85) $\displaystyle D\equiv$ $\displaystyle
16\gamma^{4}W_{1}^{2}+\gamma^{2}\left(8W_{1}^{4}-20W_{1}^{2}\Delta_{1}^{2}-\Delta_{1}^{4}\right)+\Omega_{1}^{6}$
(86)

Using the initial condition
$\left(X_{1},\rho_{11^{\prime}r},\rho_{11^{\prime}i}\right)|_{t=0}=\left(1,0,0\right)$
we have the following equations for the constants $c_{i}$:

$\displaystyle\sum_{i=1}^{3}c_{i}=1,\quad\sum_{i=1}^{3}\Gamma_{i}c_{i}=0,\quad\sum\Gamma_{i}^{2}c_{i}=-\Delta_{1}^{2},$
(87)

which leads to [Ho2022c]:

$\displaystyle c_{1}$
$\displaystyle=\frac{\Gamma_{2}\Gamma_{3}-\Delta_{1}^{2}}{\left(\Gamma_{1}-\Gamma_{2}\right)\left(\Gamma_{1}-\Gamma_{3}\right)},$
(88) $\displaystyle c_{2}$
$\displaystyle=\frac{\Gamma_{1}\Gamma_{3}-\Delta_{1}^{2}}{\left(\Gamma_{2}-\Gamma_{1}\right)\left(\Gamma_{2}-\Gamma_{3}\right)},$
(89) $\displaystyle c_{3}$
$\displaystyle=\frac{\Gamma_{1}\Gamma_{2}-\Delta_{1}^{2}}{\left(\Gamma_{3}-\Gamma_{1}\right)\left(\Gamma_{3}-\Gamma_{2}\right)},$
(90)

where $\Gamma_{i}$, $i=1,2,3$, are given above. Time-dependent magnetization
$M\left(t\right)$ of a molecular spin is approximately proportional to
population difference in the ground doublet $X_{1}\left(t\right)$:

$\displaystyle M\left(t\right)$ $\displaystyle\approx
M_{0}e^{-\Gamma_{e}t}\left[\frac{\Gamma_{2}\Gamma_{3}-\Delta_{1}^{2}}{\left(\Gamma_{1}-\Gamma_{2}\right)\left(\Gamma_{1}-\Gamma_{3}\right)}e^{-\Gamma_{1}t}+\frac{\Gamma_{1}\Gamma_{3}-\Delta_{1}^{2}}{\left(\Gamma_{2}-\Gamma_{1}\right)\left(\Gamma_{2}-\Gamma_{3}\right)}e^{-\Gamma_{2}t}+\frac{\Gamma_{1}\Gamma_{2}-\Delta_{1}^{2}}{\left(\Gamma_{3}-\Gamma_{1}\right)\left(\Gamma_{3}-\Gamma_{2}\right)}e^{-\Gamma_{3}t}\right],$
(91)

where $\Gamma_{e}$ is given in Eq. (33) and $\Gamma_{i}$, $i=1,2,3,$ are given
in Eqs. (82-84). Here, the first part $M_{0}e^{-\Gamma_{e}t}$ is magnetization
relaxation without the tunneling effect in the ground doublet while the latter
embraces this effect.

##  V Magnetization relaxation fitting: single-, bi-, or tri-exponential form?

In general, magnetization relaxation in intermediate and low temperature
regimes are a combination of three relaxation modes whose corresponding rates
are a simple addition of “classical” relaxation processes (Orbach, Raman, and
direct process) $\Gamma_{e}$ and the ones resulting from the quantum tunneling
in the ground (quasi) doublet. However, depending on the temperature and the
energy bias in the ground doublet, one relaxation mode may still relax much
slower than the remaining. Consequently, using one relaxation rate
characterizing for the whole relaxation of the molecular spin may still be a
good approximation for these cases. In contrast, from the resonance case
$W_{1}=0$, it is clearly that in the proximity of the transition decoherence
rate where $\gamma=\Delta_{1}$, relaxation of the magnetization must be
described using a bi-exponential formula. Hence, the question is in which
conditions the single-, bi-, or tri-exponential formula should be used.

From our companion work [Ho2022c], we know that at the transition point
$\gamma_{0}$ at least two lowest quantum tunneling rates share the same value
of the real part. Just for a summary, here is dependence of the transition
decoherence rate $\gamma_{0}$ on $\Delta_{1}$ and $W_{1}$:

$\displaystyle\gamma_{0}$ $\displaystyle=$
$\displaystyle\begin{cases}\Delta_{1}&\text{for\,\,\,}W_{1}=0,\\\
\frac{\left(\sqrt{\Delta_{1}^{2}-8W_{1}^{2}}+3\Delta_{1}\right)\sqrt{4W_{1}^{2}-\Delta_{1}\sqrt{\Delta_{1}^{2}-8W_{1}^{2}}+\Delta_{1}^{2}}}{8\sqrt{2}W_{1}}&\text{for\,\,\,}0\leq
W_{1}<\frac{\Delta_{1}}{2\sqrt{2}},\\\
\frac{3}{2\sqrt{2}}\sqrt{\Delta_{1}^{2}-2W_{1}^{2}}&\text{for\,\,\,}\frac{\Delta_{1}}{2\sqrt{2}}\leq
W_{1}<\frac{\Delta_{1}}{\sqrt{2}},\\\
0&\text{for\,\,\,}W_{1}\geq\frac{\Delta_{1}}{\sqrt{2}}.\end{cases}$ (92)

As can be seen, for $W_{1}\leq 1/\sqrt{2}$, we can easily say that in the
proximity of this special point $\gamma_{0}$, the magnetization relaxation
should be fit using bi- or tri-exponential formula. However, remember that as
long as (the real part of) relaxation rates are at the same order of magnitude
then tri-exponential formula of the magnetization relaxation still needs to be
used, we define $\gamma_{n}$ at which
$\mathrm{Re}\left(\Gamma_{2,3}\right)=n\mathrm{Re}\left(\Gamma_{1}\right)\sim\mathcal{O}\left(\Gamma_{1}\right)$
where $1<n\leq 10$. From the governing equation of the magnetization
relaxation in spin system above and/or its characteristic polynomial Eq. (55),
we can obtain the value for $\gamma_{n}$:

$\gamma_{n}=\frac{1}{2\sqrt{2}}\frac{2n+1}{2n-1}\mathrm{Re}\left[\sqrt{\left(2n-1\right)\Delta_{1}^{2}-2W_{1}^{2}}\right],\text{
for\,\,\,}W_{1}\geq\frac{\Delta_{1}}{2\sqrt{2}},$ (93)

from this value of $\gamma_{n}$, which corresponds to a specific temperature
$T_{n}$, lowering temperature requires a tri-exponential formula for the
description of the magnetization relaxation of a molecular spin.
Unfortunately, for $W_{1}\leq\Delta_{1}/2\sqrt{2}$, we do not have a simple
formula for $\gamma_{n}$. We thus resort to a numerical visualization of
$\gamma_{n}$. Fig. 1 hence presents the dependence of $\gamma_{n}$ on $W_{1}$
in $\Delta_{1}$ unit for different value of $n$.

Figure 1: Transition decoherence rate $\gamma_{0}$ ($n=1$) and $\gamma_{n}$,
$n=3,5,10$, as a function of the energy bias $W_{1}$ in the ground doublet in
$\Delta_{1}=1$ unit. The dashed lines are $\gamma_{n}$ from Eq. (93) applied
for any $W_{1}$.

As can be seen, the numerical calculation shows that Eq. (93) (dashed lines)
still gives a very good approximation of $\gamma_{n}$ at low value of $W_{1}$.
Hence, we can take:

$\displaystyle\gamma_{n}$
$\displaystyle\approx\frac{1}{2\sqrt{2}}\frac{2n+1}{2n-1}\mathrm{Re}\left[\sqrt{\left(2n-1\right)\Delta_{1}^{2}-2W_{1}^{2}}\right]\,\,\,\forall
W_{1}.$ (94)

In summary, for the purpose of magnetization relaxation fitting:

  * •

For $W_{1}=0$, the bi-exponential formula should be use in the proximity of
the transition decoherence rate $\gamma_{0}=\Delta_{1}$ and for all
$\gamma\leq\gamma_{0}$.

  * •

For $0<W_{1}\leq\Delta_{1}/\sqrt{2}$, the tri-exponential formula for the
magnetization relaxation, Eq. (91), needs to be used in the positive proximity
of the transition decoherence rate $\gamma_{0}$ (from $\gamma_{0}$ to
$\gamma_{n}$) and for all $\gamma\leq\gamma_{0}$.

  * •

For $W_{1}>\Delta_{1}/\sqrt{2}$, the tri-exponential formula for the
magnetization relaxation needs to be used for $\gamma<\gamma_{n}$ where
$\gamma_{n}$ is given by Eq. (93).

  * •

For all other cases, single-exponential formula can be used.

It should be noted that when $W_{1}>\Delta_{1}/2\sqrt{2}$, the governing
equation for the quantum tunneling process gives two complex conjugates and
one real root, experimental data can be fitted either by using Eq. (91) or
another equivalent one, but easier for numerical fitting, whose all parameters
are real numbers:

$M\left(t\right)=M_{0}e^{-\Gamma_{e}t}\left[\frac{\Gamma_{2r}^{2}+\Gamma_{2i}^{2}-\Delta_{1}^{2}}{\left(\Gamma_{1}-\Gamma_{2r}\right)^{2}+\Gamma_{2i}^{2}}e^{-\Gamma_{1}t}+\left(1-\frac{\left(\Gamma_{2i}^{2}+\Gamma_{2r}^{2}\right)-\Delta_{1}^{2}}{\left(\Gamma_{1}-\Gamma_{2r}\right)^{2}+\Gamma_{2i}^{2}}\right)e^{-\Gamma_{2r}t}\cos\Gamma_{2i}t\right.\\\
\left.+\frac{\Gamma_{1}\left(\Gamma_{1}\Gamma_{2r}-\Gamma_{2r}^{2}+\Gamma_{2i}^{2}\right)-\left(\Gamma_{1}-\Gamma_{2r}\right)\Delta_{1}^{2}}{\Gamma_{2i}\left[\left(\Gamma_{1}-\Gamma_{2r}\right)^{2}+\Gamma_{2i}^{2}\right]}e^{-\Gamma_{2r}t}\sin\Gamma_{2i}t\right]\\\
\equiv
M_{0}\left[(1-a)e^{-\left(\Gamma_{1}+\Gamma_{e}\right)t}+e^{-\left(\Gamma_{2r}+\Gamma_{e}\right)t}\left(a\cos\Gamma_{2i}t+b\sin\Gamma_{2i}t\right)\right].$
(95)

In principle, from the fitted parameter, the tunneling splitting of the ground
doublet can be found.


11institutetext:  Wuhan University, Wuhan Hubei 430072, China 22institutetext:
Princeton University, Princeton NJ 08544, USA

# D3 Scaling Up Deepfake Detection by Learning from Discrepancy

Yongqi Yang Equal contribution.11 Zhihao Qian⋆ 11 Ye Zhu 22 Yu Wu 11

###### Abstract

The boom of Generative AI brings opportunities entangled with risks and
concerns. In this work, we seek a step toward a universal deepfake detection
system with better generalization and robustness, to accommodate the
responsible deployment of diverse image generative models. We do so by first
scaling up the existing detection task setup from the one-generator to
multiple-generators in training, during which we disclose two challenges
presented in prior methodological designs. Specifically, we reveal that the
current methods tailored for training on one specific generator either
struggle to learn comprehensive artifacts from multiple generators or tend to
sacrifice their ability to identify fake images from seen generators (i.e.,
_In-Domain_ performance) to exchange the generalization for unseen generators
(i.e., _Out-Of-Domain_ performance). To tackle the above challenges, we
propose our Discrepancy Deepfake Detector (D${}^{\textbf{3}}$) framework,
whose core idea is to learn the universal artifacts from multiple generators
by introducing a parallel network branch that takes a distorted image as extra
discrepancy signal to supplement its original counterpart. Extensive scaled-up
experiments on the merged UFD and GenImage datasets with six detection models
demonstrate the effectiveness of our framework, achieving 5.3% accuracy
improvement in the OOD testing compared to the current SOTA methods while
maintaining the ID performance.

##  1 Introduction

Recent advances in generative modeling have marked a new era of Generative AI,
which is transforming society as it progresses from our research community
into the real world. The widespread appeal of generative AI technologies among
the general populace can be attributed to the significant achievements in
scaling generative models—such as BigGAN [4], DALL$\cdot$E [35], and
StableDiffusion [36] — to unprecedented model sizes and leveraging expansive,
open-world datasets. However, with these opportunities come inherent risks;
these advancements also raise concerns about diverse aspects including data
privacy, AI ethics, social fairness, and regulatory compliance [15]. Among
these challenges, the risk of malicious use of synthetic data stands out as
one of the particularly pressing issues that demand focused research
attention.

Deepfake detection, which aims at discerning between real and generated fake
images, acts as a crucial defense in responsibly deploying large generative
models in the real world. The effectiveness of these detection systems largely
depends on their generalization capability; ideally, they should reliably
identify any synthesized image as fake, regardless of the generating source
(i.e., the deep generative models from which they are generated). Accordingly,
current deepfake detection methods [32; 45; 29; 5; 46; 10] usually follow a
standard task setup for evaluating the performance, which involves training on
images from one specific generator and testing on images from various unseen
generators. As a natural step towards a universal deepfake detection
foundation model, in this work, we propose to scale up this current setup from
_“train-on-one and test-on-many”_ to _“train-on-many and test-on-many”_ , to
accommodate the rapid development pace from the generative modeling side and
better align with the real-world scenario.

However, in the process of scaling up existing deepfake detection models to
multiple generators, we encounter two challenges that have not yet been
adequately addressed in prior literature, limiting the generalization
performance and robustness of an expanded detection system. _First_ , we
reveal that the existing methodology designs tailored for the _“train-on-one
and test-on-many”_ setup struggle to learn the comprehensive and universal
artifacts when the fake images in training present different fingerprint
patterns from more than one generator. For instance, one of the current SOTA
detection models, DIRE from ICCV’23 [46], tends to learn the most obvious
generator-specific artifacts (e.g., fingerprints of diffusion models [21; 42])
but overlooks the subtle but invariant common artifacts shared by other
generative AI models (e.g., fingerprints from GAN-based generators [17]),
leading to inferior testing performance on the images produced by unseen
generators. As detailed in Sec. 4, our experimental results show that, when
scaling up to 8 training generators and 12 unseen generators, DIRE [46]
achieves 97.6% on in-domain (ID) testing, but performs significantly worse on
the critical out-of-domain (OOD) testing with only 68.4% accuracy. _Second_ ,
our empirical analysis further suggests that prior methods tend to comprise
the trade-off between training and testing performance, by “underfitting” the
ID training to barter for better OOD generalization ability in testing. As a
concrete example, another SOTA detection method, UFD from CVPR’23 [32],
achieves better generalization on OOD testing with an accuracy of 81.4%
compared to DIRE [46]. However, the better OOD performance of UFD comes with
the expense of the lower ID testing performance, resulting in only 86.6% on
the ID accuracy.

To tackle the above challenges on the way of scaling up the deepfake detection
methods to a universal system, we introduce our D${}^{\textbf{3}}$ framework,
terminology from Discrepancy Deepfake Detector, for an extended setup under
the _“train-on-many and test-on-many”_ scenario. The secret recipe of
D${}^{\textbf{3}}$ lies within the core idea of exploiting and learning the
universal artifacts among various deep generators, which intuitively
facilitates the learning and improves the testing robustness. Specifically,
unlike the existing methods [45; 5; 32] that feed the detection model only
with either real or generated images, we adopt a two-branch design to provide
D${}^{\textbf{3}}$ with an extra corrupted image corresponding to its original
counterpart, as illustrated in Fig. 1 and detailed in Sec. 3. The distorted
image, which is obtained via operations such as patch shuffling, flipping, and
rotation, deconstructs the unique fingerprint from a specific generator and
serves as a discrepancy signal to supplement the original image. This unique
methodological design encourages the proposed D${}^{\textbf{3}}$ framework to
effectively capture the invariant artifacts from the image pair. We further
improve the robustness and generalization ability of our D${}^{\textbf{3}}$ by
extracting the image features from the CLIP encoder [34], and enhance their
interactions via self-attention layers before making the final prediction.

We perform extensive experiments by gradually scaling up six deepfake
detection methods (including [32; 5; 29; 45; 46] and our proposed
D${}^{\textbf{3}}$) from conventional setup to 8 training generators and 12
testing unseen generators, on the UniversalFakeDetect (UFD) [32] and the
GenImage [52] datasets. Each detection model is trained till convergence to
ensure a fair comparison. Our comprehensive experiments and analysis not only
explicitly disclose the two previously mentioned issues, but also demonstrate
the effectiveness of our proposed D${}^{\textbf{3}}$ by outperforming SOTA
methods with 5.3% on the OOD testing mean accuracy.

In summary, our work has the following contributions:

  * •

We propose to scale up the current deepfake detection setup from _one-
generator_ to _multiple-generators_ in training to accommodate the fast
development pace from the generative modeling side and disclose two factors in
the existing methods that limit the generalization ability and robustness in
this extended detection scenario.

  * •

We introduce our Discrepancy Deepfake Detector (D${}^{\textbf{3}}$) framework
to address the above challenges, whose core idea is to learn the general
artifacts among multiple generators with an extra distorted image as the
discrepancy signal via a parallel network branch.

  * •

We perform extensive experiments, by gradually scaling up the current setup to
8 generators, and achieve 5.3% improvement in the OOD testing performance
compared to SOTA methods, while maintaining the ID performance.

##  2 Related Work

With the swift advancement of generative models [18; 27; 21; 43; 36; 13; 39]
and customization methods [38; 16; 47; 53; 26; 49], synthesized images are
becoming increasingly realistic, making it challenging for the human eye to
differentiate between real and fake images. To counter potential malicious
usage, a considerable amount of research has been devoted to detecting
generated images, yielding impressive identification accuracy.

###  2.1 Artifact-based Detection Methods

These methods explicitly extract artifacts in generated images in a
preprocessing way. They usually directly estimate the artifacts [50; 22],
compute the residual noise between the original image and its reconstructed
one [30; 29; 46], or calculate gradients [44]. [30] was the first to reveal
that GANs leave specific fingerprints on the generated images. This was
achieved by extracting the noise residual through an appropriate denoising
filter, demonstrating its potential for forensic analysis. [50] proposed a GAN
simulator to emulate the artifacts produced by the common pipeline shared by
several popular GAN models. [22] proposed a fingerprint generator to cover the
fingerprints of various GAN models to achieve generalized detection ability
and avoid data dependency. LNP [29] pretrain a reconstruction network with
only real images to extract the Learned Noise Patterns (LNP), which can
explicitly embody artifacts. DIRE [46] used the prior from a pre-trained
diffusion model to determine whether an image is within the distribution of
fake images. This was achieved by computing the Diffusion Reconstruction Error
(DIRE). [44] employed a pre-trained CNN model to extract the gradients of
target images as the representation of artifacts.

However, their artifact extraction and simulation capabilities are limited by
the prior knowledge derived solely from particular models. Therefore, they can
not deal with samples generated by unfamiliar generators, making it difficult
to be universally applied to real-world applications.

###  2.2 Learning-based Detection Methods

This type of method considers directly learning the artifacts from raw images.
They aim to design a better network to extract more discriminative features.
CNNDet [45] directly trained ResNet50 as the classifier on their proposed
semantic-aligned dataset, containing fake images generated by ProGAN. Their
methods exhibited strong generalization capabilities to other CNN-based
generative models. Patchfor [5] used a fully convolutional patch-based
classifier, emphasizing local patches over global structure. [19] modified the
ResNet50 network with two fewer down-sampling layers to improve the detection
performance. However, the learning-based methods directly train the entire
network to learn the presence or absence of actual artifact patterns. They
often overlook features inherent to real images, which can lead to overfitting
and poor generalization.

###  2.3 CLIP-based Detection Methods

To mitigate the shortcomings of the learning-based methods, this type of
method leverages the frozen feature space of CLIP to extract more
generalizable features. UFD [32] first highlighted that a feature space not
explicitly learned for generated image detection is more effective due to its
unbiased decision boundaries. Consequently, they performed detection in the
pre-trained CLIP feature space using linear probing or nearest neighbor
methods, demonstrating impressive generalization performance on unseen
generated images. Taking a further step in this direction, [10] trained SVMs
with the features from the penultimate layer and achieved commendable
performance despite a limited amount of well-organized training data.
Likewise, [41] additionally utilized the text encoder of the pre-trained CLIP
model to exploit the prompt information by concatenating the encoded image
embeddings and text embeddings together as features. However, these CLIP-based
methods employ only one linear classifier to distinguish between real and fake
features of images. Therefore, when applied to a larger multi-generator
dataset, the simple classifier struggles to effectively fit the ID data, which
further impacts OOD performance. In this work, we further enhance the fitting
and generalization ability of CLIP-based detectors on multi-generator
datasets. We propose to learn more stable and invariant artifacts by comparing
images before and after corruption. Our approach facilitates a better
understanding of the universal differences between real and fake images.

##  3 Scaling Up Deepfake Detection

We formulate the extended _“train-on-many and test-on-many”_ setup in Sec.
3.1, and identify the key challenges for a generalized and robust detection
system in Sec. 3.2. Sec. 3.3 introduces our proposed D${}^{\textbf{3}}$
method.

###  3.1 Problem Formulation

Formally, given a generator set $\mathcal{G}$ composed of $N$ different types
of generators, $\mathcal{G}=\\{G_{i}\\}_{1}^{N}$,
$\mathcal{R}_{i}=\\{r^{(i)}_{1},r^{(i)}_{2},\ldots,r^{(i)}_{N_{i}}\\}$ and
$\mathcal{F}_{i}=\\{f^{(i)}_{1},f^{(i)}_{2},\ldots,f^{(i)}_{N_{i}}\\}$ denote
the real and fake samples of $\mathcal{G}_{i}$, where $N_{i}$ is the number of
samples in each class. Our goal is to correctly distinguish between real and
fake when given any image from the overall dataset,
$\mathcal{D}=\\{\\{\mathcal{R}_{i}\\}_{1}^{N}\cup\\{\mathcal{F}_{i}\\}_{1}^{N}\\}$.
The existing detection task setup involves training on one of the generators
$G_{i}$. In terms of evaluation, they regard the seen generator $G_{i}$ as the
ID generator and the other unseen generators $\\{G_{j}\\}_{j\neq i}$ as the
OOD generators. This setup implies learning to detect universal artifacts from
only one specific kind of artifact, which is hard and unnecessary. This
impractical setup may mislead the research community because the
generalization performance is unstable and would highly depend on the choice
of the training generator. For example, the OOD performances of UFD [32] would
drop significantly from 87.6% to 58.9% when switching the training from the
ProGAN [23] to Midjourney [2] in this scheme.

Therefore, we propose to step out of the “train-on-one and test-on-many” task
setup and leverage a new _“train-on-many and test-on-many”_ evaluation scheme.
More specifically, we add more generators into the current dataset and train
on a subset composed of $k$ $(k<N)$ generators, denoted by
$\mathcal{G}_{s}=\\{G_{s_{i}}\\}_{1}^{k}(s_{i}\in\\{1,2,\ldots,N\\})$. In this
way, we regard the $k$ generators in $\mathcal{G}_{s}$ as the ID generators
and the other $N-k$ generators as the OOD generators. In practice, we merge
two large-scale existing deepfake detection datasets, i.e., the UFD dataset
[32] and GenImage dataset [52]. Finally, there are in total 20 state-of-the-
art generators available in the scale-up setup. Considering the diversity, we
regard all the 8 generators used in different previous works, including 2 GANs
methods like ProGAN [23], and 6 DMs like ADM [13]. The rest 12 generators
(including both GANs and DMs) are used as the OOD set.

###  3.2 Challenges in the Scaled-up Setup

Based on the scaled-up setup, we gradually add the generators into the
training pool in different random orders to evaluate the existing works. As
the number of generators increased from one to eight, we found existing
methodology designs tailored for the previous setup could not learn universal
artifacts. For example, one of the state-of-the-art works UFD [32] from
CVPR’23 even decreases significantly in terms of ID testing accuracy as more
generators are used for training (Fig. 2), demonstrating the underfitting
issue. While another state-of-the-art work DIRE [46] from ICCV’23 shows
extremely good ID performance (97.6%), it suffers from the generalization
issue that the OOD testing accuracy is not promising (68.4%), despite it has
already seen eight advanced generators covering typical GANs and DMs. These
results suggest existing models could not find the invariant artifacts that
are universal in multiple generators. The main reason might be the universal
data patterns we wish the model to learn are coupled with or hidden by
semantics. It inspires us to go deeper to learn universal artifacts rather
than the superficial and generator-specific artifacts that can be directly
seen on the deepfake images.

Figure 1: The overall framework of our Discrepancy Deepfake Detector
(D${}^{\textbf{3}}$). The original image and its patch-shuffled variant are
inputted into the pre-trained CLIP model [34] to obtain two features. We then
utilize the self-attention module to encourage the learning of shared
artifacts between the features. Finally, we use a single-layer linear
classifier to get the final prediction.

###  3.3 D${}^{\textbf{3}}$: Deepfake Dectection by Learning from Discrepancy

Overall Pipeline. Our proposed framework for generative image detection is
depicted in Fig. 1. Overall, the original image and its patch-shuffled
counterpart are input into the pre-trained CLIP [34] model to obtain visual
features. A self-attention layer is then used to promote learning from the
discrepancy between the two features. Finally, a linear classifier is employed
to predict real or fake labels. The design of each module will be discussed in
the following.

Visual Backbone. Directly learning on the deepfake detection data without
pretraining can lead to overfitting on the training generators, resulting in
poor generalization to OOD generators [9; 52]. To mitigate this, we follow
previous work [32; 10] and adopt the pre-trained CLIP:ViT-L/14 as our feature
extractor. More specifically, we extract features from the penultimate layer
of the pre-trained CLIP model, which keeps more detailed visual clues of the
input.

Dual-Path Discrepancy Learning. To encourage the model to find deeper robust
artifacts, we propose to introduce an additional branch that takes a corrupted
image as input. Given an original image $x_{o}$ from the dataset
$\mathcal{D}_{s}=\\{\\{\mathcal{R}_{i}\\}_{1}^{k}\cup\\{\mathcal{F}_{i}\\}_{1}^{k}\\}$,
corresponding to the subset $\mathcal{G}_{s}$. In the new branch, we divide
the input image $x_{o}$ into patches, and then randomly shuffle them to form a
new image, where the superficial artifacts will be destroyed to avoid the
model simply learning a generator-specific shortcut. After that, we input the
corrupted data to the same visual backbone to extract features,

$e_{o}=\texttt{CLIP}^{*}(\texttt{AUG}(x_{o})),e_{s}=\texttt{CLIP}^{*}(\texttt{PatchShuffle}(\texttt{AUG}(x_{o}))),$
(1)

where $e_{o}$ and $e_{s}$ are the visual embeddings of the respective images,
$\texttt{CLIP}^{*}$ denotes the extraction of the penultimate feature using
the pre-trained CLIP model, PatchShuffle is the patch-shuffling operation, and
AUG represents the common data augmentation method used in the previous work
[45].

Self-Attention Invariance Extraction. Having obtained two features, $e_{o}$
and $e_{s}$, from the original and patch-shuffled images respectively, we
stack the two features together, $e=[e_{o},e_{s}]$. The correlation between
$e_{o}$ and $e_{s}$ is crucial for detecting fake images. To this end, we
employ a self-attention layer on the stacked feature $e$ to extract the shared
artifact feature $E$ by learning the associations between the features. In
this way, we force the model to learn from the discrepancy between the
original image and its patch-shuffled image. In experiments, we found the more
distinct between the original image and the corrupted image, the better the
deepfake detection performance the model can get.

Loss Function. Based on the fused feature $E$, we employ a single fully
connected layer, denoted as $\Phi(\cdot)$, to compute the classification
logits. We then train the model using the binary cross entropy loss by,

$\mathcal{L}=-\frac{1}{N_{B}}\sum_{i=1}^{N_{B}}(y_{i}\log(\Phi(E))+(1-y_{i})\log(1-\Phi(E))),$
(2)

where $N_{B}$ is the batch size, and $y_{i}$ is the label of the $i$-th input
sample. Note that our visual backbone is frozen, thus the learnable parameters
are only self-attention and the classification layer.

##  4 Experiments

###  4.1 Scaled-Up Datasets

To evaluate the effectiveness of our proposed method, we conducted experiments
on the UniversalFakeDetect dataset [32] and the GenImage dataset [52].
Notably, we scale up the experimental data by merging the two datasets
together.

The UniversalFakeDetect (abbreviated to “UFD”) dataset contains 720k images in
its training set, including 360k real images (drew from LSUN [48], LAION [40]
and ImageNet [12]) and 360k fake images generated by different generators
including GANs like ProGAN [23], CycleGAN [51], BigGAN [4], StyleGAN [24],
GauGAN [33], StarGAN [8], Deepfakes [37], SITD [6], SAN [11], CRN [7] and IMLE
[28], and diffusion models like Guided Diffusion [13], LDM [36], GLIDE [31]
and DALL$\cdot$E [35].

The GenImage dataset comprises 2,681,167 images, segregated into 1,331,167
real (drawn from ImageNet) and 1,350,000 fake images, with 50k images left for
testing in each generator type. Fake images in this dataset are generated by 8
generators, including 1 GAN (BigGAN [4]) and 7 diffusion models (Stable
Diffusion V1.4 [36], Stable Diffusion V1.5 [36], GLIDE [31], VQDM [20], Wukong
[3], ADM [13] and Midjourney [2]).

In our new setup, we merge the two datasets into one and regard generators
with the same architecture but different parameters (e.g. SD v1.4 and SD v1.5)
as the same type. To this end, we eliminate SD v1.5 from the merged training
set and remain SD v1.4. Besides, we also make sure each generator has an equal
number of samples to avoid imbalanced data. Finally, there are a total of 20
generators available in the scale-up setup, 8 in the training set (2 GANs and
6 Diffusions) and 12 in the OOD set. Compared with the former setup, we have
scaled up both the number of generators and the volume of samples within the
training set. Besides, we further split 10% samples in the testing set of our
merged dataset per generator as the validation and the rest as the testing
set.

###  4.2 Implementation Details

In our experiments, we employ the CLIP: VIT-L/14 model pre-trained on a large
dataset WebImageText as the backbone to extract patch tokens. We adopt its
penultimate feature before dimensionality reduction for better representation.
With the backbone frozen, the whole network is trained on a single NVIDIA RTX
4090 with a batch size of 128 within 5 epochs. We apply the Adam optimizer
with an initial learning rate of 0.0001 and a weight decay of 0. During
training, all images are resized to 256x256 resolution and then randomly
cropped to 224x224 resolution. We adopted the data augmentation scheme the
same as the works [45; 32], involving Gaussian blur and JPEG compression.
Specifically, the quality of JPEG compression randomly ranges from 30 to 100,
the sigma of Gaussian blur randomly ranges from 0 to 3, and their occurrence
probabilities are set to 0.5. During validation and testing, we directly
resize the image to the size of 224x224, without any augmentation.

###  4.3 Experimental Setup

In this subsection, we provide a detailed illustration of our experimental
setup, including the implementation of baselines and evaluation metrics.

Implementation of Baselines. In terms of baselines, we re-implement the
comparison methods according to their official codes under our problem
settings for fairness. The baselines include CNNDet [45], Patchfor [5], LNP
[29], DIRE [46], and UFD [32]. Detailedly, CNNDet finetunes a pre-trained
ResNet50 as a classifier and we use the ResNet50 version with default
parameters. Patchfor finetunes the truncated ResNet or Xception as the
classifier at the patch level and we choose their Xception version. LNP pre-
trains a reconstruction model with only real images to extract noise patterns
and then finetunes the pre-trained ResNet50 with spatial and spectrum
information. We use their pre-trained reconstruction model to preprocess all
the images and finetune the ResNet50 in the default way. DIRE first computes
the diffusion reconstruction error of all images and trains a ResNet50 on
them. We use their 256x256 unconditional DMs to preprocess all the images and
also finetune the ResNet50 in the default way. UFD extracts the features with
a frozen CLIP model and classifies them by nearest neighbors or linear
probing. We select their linear probing version with the CLIP: VIT-L/14 as the
backbone. In our experiments, all methods are re-implemented with their
default settings in our new setup, and the optimal checkpoint is selected
based on the performance on the validation set to ensure a fair comparison.

Evaluation Metrics. We employ mean classification accuracy (Mean acc.) and
average precision (AP) to evaluate the performance of detectors. When
calculating overall mean accuracy, we first average the accuracy of different
variants of the same generator (e.g. SD1.4 and SD1.5). Then, we average all
the accuracy scores across different generators. Note that we use a fixed
threshold of 0.5 when calculating the accuracy score for each detector.

AP quantifies the separability between fake and real images, neglecting the
specific threshold consideration. Previous literature [32; 29; 46; 45]
typically uses mAP to represent the overall AP score, where they calculate AP
for each generator independently and average the AP scores. However, the
direct average of AP disregards the substantial variance in the optimal
thresholds, or boundaries, between real and fake images from different
generators. In practice, it is infeasible for users to set different real/fake
thresholds for unseen generators. Therefore, we propose to calculate a global
AP rather than mAP, which shares the same standards among all testing
generators.

We first resample generators with the same architecture based on the maximum
number of samples among them and merge them as one. Then, we similarly
resample different types of generators to ensure the entire test set is evenly
distributed. Finally, we calculate the average precision (AP) of different
merged sets to represent the overall AP. For instance, in terms of ID AP, we
merge the results of all ID generators to compute an overall ID AP. This
approach ensures a balanced evaluation across different generators, providing
a more accurate measure of the detector’s performance.

Figure 2: Mean accuracy of training on scaled-up datasets on the testing set.
We design an experiment to show the performance of different methods when
gradually adding generators into the training pool. To avoid occasionality, we
add the generator in three distinct random orders and average the
corresponding test results. (a) “In-domain” means 1 shared training generator
(the first generator in any given order). (b) “Out-of-domain” means 12 shared
OOD generators.

###  4.4 Scaling Up the Training Data

In this section, we conduct experiments to show how the models perform as the
generators gradually scale up in Fig. 2. We re-train the detectors on the
training set with {1, 2, 4, 8} generators. To mitigate any potential bias, we
add the generator in three distinct random orders and average the
corresponding test results. To make all the curves comparable (in both
horizontal and vertical), all detectors are evaluated on the same ID and OOD
test sets, thus the curves are comparable in both horizontal and vertical and
we can observe how a model baheves as the number of training generators grows.
Note that only the first generator is commonly shared when gradually adding
training generators from one to eight, we thus test all models only on this
generator for the ID evaluation. For the OOD evaluation, we use the 12 common
unseen generators for all models.

According to Fig. 2, as the number of generators increases, our method shows a
slight fluctuation in ID performance and exhibits the highest performance and
stable upward trend in OOD generalization. CNNDet [45] and Patchfor [5] show
excellent ID fitting ability, but their OOD performances are inferior to the
other works. It is worth mentioning that Patchfor to some extent alleviates
the overfitting issue by focusing on artifacts of local patches instead of the
high-level features. Reconstruction-based methods like LNP and DIRE heavily
rely on their pre-trained reconstruction model, which could result in poor
generalization to unfamiliar domains. Moreover, the CLIP-based method UFD
exhibits superior OOD performance when there are few generators and maintains
a rising trend as the training set scales up. However, its ID performance
continues to decline, which indicates a limitation in its representative
capacity. In sum, these findings collectively demonstrate that our method
effectively improves the distinguishability and generalizability of features
by patch-shuffling and artifact invariance learning from the introduced
discrepancy, resulting in stable and outstanding performance of ID fitting and
OOD generalization.

Table 1: Comparisons to the SOTAs in mean accuracy (Mean acc.) and average precision (AP) on the testing set. “In-domain” includes 8 types of generators seen during training, “Out-of-domain” includes 12 types of generators, and “Total” includes all of them. Bold represents the best within the same metric. Our method achieves the best OOD and overall performance. A more detailed version of the result can be found in the Appendix 0.D. Methods | Pub. | Testing
---|---|---
In-domain | Out-of-domain | Total
Mean acc. | AP | Mean acc. | AP | Mean acc. | AP
CNNDet | CVPR20 | 0.933 | 0.906 | 0.699 | 0.679 | 0.792 | 0.770
Patchfor | ECCV20 | 0.979 | 0.997 | 0.789 | 0.880 | 0.865 | 0.942
LNP | ECCV22 | 0.881 | 0.902 | 0.719 | 0.797 | 0.784 | 0.823
DIRE | ICCV23 | 0.976 | 0.994 | 0.684 | 0.726 | 0.801 | 0.862
UFD | CVPR23 | 0.866 | 0.950 | 0.814 | 0.927 | 0.835 | 0.935
Ours | - | 0.966 | 0.995 | 0.867 | 0.943 | 0.907 | 0.968

###  4.5 Comparison with the State-of-the-Art Methods

We compare our method with other state-of-the-art methods in the scale-up
setup and report evaluation of their ID and OOD performance on the testing
set. Tab. 1 reports the results on the testing set, where “In-domain” includes
8 types of generators, and “Out-of-domain” includes 12 types of generators,
and “Total” includes all of them. A more detailed version of the result can be
found in the Appendix 0.D.

The experimental results exhibit the strengths and weaknesses of each method.
CNNDet overfits data from familiar generators and performs poorly on data from
unseen generators, consistent with the findings about ResNet in the paper
[52]. Patchfor achieves the best ID mean accuracy and AP. However, its subpar
OOD performance suggests that directly learning the specific local artifact
patterns may also lead to a higher potential for overfitting. LNP exhibits
poor performance in both ID and OOD fitting. DIRE demonstrates superior ID
performance, surpassed only by Patchfor. However, it lacks generalization
capabilities with OOD generators due to its tailored design for DMs. UFD’s
performance is second only to our method, but its ID performance is subpar.

In comparison, our method achieves the highest overall mean accuracy of 90.7%
and AP of 96.8%, exceeding the second-best method, Patchfor, in overall mean
accuracy by 4.2% and AP by 2.6%. Although our method marginally trails
Patchfor in terms of ID accuracy and AP, it exhibits a more balanced and
consistent overall performance. Furthermore, our method greatly outperforms
UFD, which also leverages the pre-training CLIP, in ID mean accuracy by 10%
and OOD mean accuracy by 5.3%. This underscores that our method further
exploits the potential of CLIP-based feature representation ability by
learning from the discrepancy of the shared artifact in the original image and
its patch-shuffled variant.

Figure 3: Total mean accuracy results of robustness to post-processing
operations on the testing set. We conducted experiments on Gaussian blurring
ranging from 0 to 2 and JPEG compression ranging from 30 to 100 to verify the
robustness of different methods to post-procession operations.

Robustness Evaluation. We evaluate the robustness of all methods to align more
closely with their practical effectiveness in real-world applications. Given
the potential for real-world images to undergo various degradations, ensuring
the robustness of detectors against unseen perturbations becomes another vital
aspect of evaluating their performance. A detector that can resist various
perturbations and attacks has higher potential and value for practical
application. Therefore, we evaluate all methods on two post-processing
operations, Gaussian blurring (sigma: 0-2) and JPEG compression (quality:
30-100). Fig. 3 presents the results of our comparison with the other methods,
demonstrating that our method exhibits the best robustness with a large margin
with the other methods. Note that except for Patchfor and DIRE, all the other
methods employ an augmentation strategy involving random Gaussian blurring and
JPEG compression during training to enhance robustness. Despite this, we can
observe that only our method and UFD exhibit general robustness, even to
extremely heavy degradations. In addition, from the perspective of accuracy
and the decay rate, our method shows obvious advantages compared to UFD.
Discrepancy learning between the original image and its patch-shuffled one can
to some extent offset the impact of these disturbances due to its
concentration on more stable and invariant artifacts instead of specific
artifact patterns.

###  4.6 Exploration of Different Disruptions

In this subsection, we investigate how different levels of disruptions
influence model performance. We conduct experiments by only replacing the
patch-shuffling operation with other disruption operations in our framework.
These operations include horizontal flipping, vertical flipping, and random
rotating (rotating an image with random degrees from 0° to 180°).

The experimental results are shown in Table 2, showing that disrupted feature
helps in promoting the model performance, with the effectiveness of patch-
shuffling, random rotating, vertical flipping, and horizontal flipping
diminishing in sequence. The two-branch model with the horizontal flipping
doesn’t show superior to the one-branch baseline, because the horizontally-
flipped image could not provide any discrepancy and its features are just the
same as the original image for most existing vision models. Comparatively,
vertical flipping has provided corruptions to the vision backbones to some
degree since most vision models could not well recognize the image after
vertical flipping. Thus the discrepancy helps the model learn better
artifacts, leading to an overall performance boost of 4% on mean accuracy
compared to the horizontal flipping. In addition, random rotation randomly
changes the angle of images, introducing a slightly stronger semantic
corruption on image semantics than the fixed 180° angle of vertical flipping,
and thus has a slightly better overall performance. In comparison to the three
operations, the patch-shuffling operation disrupts the image more completely,
leading to the maximum discrepancy between the two branches. Thus it achieves
the best performance. The comparison between different transformations clearly
illustrates that the discrepancy between the original images and corrupted
images helps the detector to learn more universal artifacts. The stronger
semantic corruption the operation causes, the better the model performs.

Table 2: Experimental results of different disruptions on the validation set.
We showed how different transformations influence model performance.

Augmentations | Validataion
---|---
In-domain | Out-of-domain | Total
Mean acc. | AP | Mean acc. | AP | Mean acc. | AP
Horizontal flipping | 0.887 | 0.964 | 0.819 | 0.934 | 0.846 | 0.945
Vertical flipping | 0.934 | 0.986 | 0.854 | 0.937 | 0.886 | 0.959
Random rotation | 0.954 | 0.991 | 0.854 | 0.928 | 0.891 | 0.958
Patch shuffling | 0.972 | 0.995 | 0.874 | 0.931 | 0.913 | 0.962

###  4.7 Ablation Studies

In this subsection, we break down our framework into several key modules and
explore the role of each module through ablation experiments. The ablation
experimental results are shown in Table 3, where a tick represents the
adoption of this module and the emptiness of Branch2 means using only one
branch. The experiments have been numbered with groups for the following
reference.

Patch-shuffling operation. A comparison between Group 2 and Group 4 reveals
that the integration of the original image and the patch-shuffled image
results in a substantial performance enhancement under the “SA”+“FC”
classifier. This is evidenced by an increase of 7.8% in ID mean accuracy, 4%
in OOD mean accuracy, and 5.5% in total mean accuracy. This effectively
substantiates that combining the information from the original image and its
patch-shuffled one can enhance the overall performance.

Discrepancy between Branches. Compared to Group 4, both Group 5 and Group 6
show an inferior performance, due to the absence of discrepancy features
between two images of the same type. According to the table, Group 5 and Group
6 are 6.0% and 13.3% lower than group 4 in average mean accuracy,
respectively. This further demonstrates that different inputs can provide
different information, and assist the model to learn more universal artifacts.

Self-Attention Layer. Comparing Group 1 with Group 2, adding a self-attention
layer to the single original feature only slightly improves the performance by
approximately 1 point. However, comparing Group 3 with Group 4, adding a self-
attention layer to two different features markedly enhances the performance
with an increase of 5.5% in ID mean accuracy and 3.9% in OOD mean accuracy.
This suggests that appropriately learning the connections between the shared
artifacts in the original image and its patch-shuffled one, rather than merely
enhancing the features, is another key to the success of our method.

In conclusion, each module is dispensable. The impact of any individual module
may not be substantial, but their collective integration significantly
enhances the performance. These findings provide robust evidence in support of
our proposition that invariance-learning from discrepancy is beneficial for
universal deepfake detection in a scaled-up scenario.

Table 3: Results of ablation study on the validation set. We break down our
framework into several key modules and explore the importance of each module
through ablation experiments. The checkmark represents the adoption of the
corresponding module. The emptiness of Branch 2 means using only one feature.

Group | Branch1 | Branch2 | CLIP | SA | FC | Validataion
---|---|---|---|---|---|---
In-domain | Out-of-domain | Average
Mean acc. | AP | Mean acc. | AP | Mean acc. | AP
1 | Original | - | ✓ |  | ✓ | 0.889 | 0.965 | 0.823 | 0.937 | 0.850 | 0.947
2 | Original | - | ✓ | ✓ | ✓ | 0.894 | 0.965 | 0.834 | 0.936 | 0.858 | 0.947
3 | Original | Shuffled | ✓ |  | ✓ | 0.918 | 0.977 | 0.835 | 0.938 | 0.868 | 0.954
4 | Original | Shuffled | ✓ | ✓ | ✓ | 0.972 | 0.995 | 0.874 | 0.931 | 0.913 | 0.962
5 | Original | Original | ✓ | ✓ | ✓ | 0.889 | 0.964 | 0.829 | 0.935 | 0.853 | 0.946
6 | Shuffled | Shuffled | ✓ | ✓ | ✓ | 0.841 | 0.914 | 0.738 | 0.800 | 0.780 | 0.849

##  5 Conclusion and Discussion

Conclusion. In this paper, we propose a novel evaluation setup _“train-on-one
and test-on-many”_ , aimed at advancing the development of a universal
deepfake detection system. To tackle the challenges in the new setup, we
introduce our Discrepancy Deepfake Detector (D${}^{\textbf{3}}$) to learn the
universal artifacts from multiple generators. We achieve this by introducing a
parallel network branch that takes a distorted image as an extra discrepancy
signal to supplement its original counterpart. Extensive experiments
demonstrate the effectiveness of our framework, achieving 5.3% accuracy
improvement in the OOD testing compared to the current SOTA methods and
maintaining excellent ID performance.

Limitations. The parallel design of D${}^{\textbf{3}}$ requires running the
CLIP twice to obtain two different features from one image, which doubles the
computation cost compared with UFD. Moreover, our proposed setup demands a
large-scale and comprehensive real-world dataset, while the merged dataset in
our research may not fully meet this criterion in the future. Collecting such
a high-quality generated image detection dataset may be expensive, however, it
is not infeasible given the fast-increasing research scale in the community.

## References

  * [1] http://www.whichfaceisreal.com/ (2019)
  * [2] https://www.midjourney.com/home/ (2022)
  * [3] https://xihe.mindspore.cn/modelzoo/wukong (2022)
  * [4] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. ICLR (2019)
  * [5] Chai, L., Bau, D., Lim, S.N., Isola, P.: What makes fake images detectable? understanding properties that generalize. In: ECCV (2020)
  * [6] Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR (2018)
  * [7] Chen, Q., Koltun, V.: Photographic image synthesis with cascaded refinement networks. In: ICCV (2017)
  * [8] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: CVPR (2018)
  * [9] Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., Verdoliva, L.: On the detection of synthetic images generated by diffusion models. In: ICASSP (2023)
  * [10] Cozzolino, D., Poggi, G., Corvi, R., Nießner, M., Verdoliva, L.: Raising the bar of ai-generated image detection with clip. arXiv (2023)
  * [11] Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: CVPR (2019)
  * [12] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)
  * [13] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. NeurIPS (2021)
  * [14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. ICLR (2021)
  * [15] Epstein, Z., Hertzmann, A., of Human Creativity, I., Akten, M., Farid, H., Fjeld, J., Frank, M.R., Groh, M., Herman, L., Leach, N., et al.: Art and the science of generative ai. Science (2023)
  * [16] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR (2023)
  * [17] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. NIPS 27 (2014)
  * [18] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. NeurIPS (2014)
  * [19] Gragnaniello, D., Cozzolino, D., Marra, F., Poggi, G., Verdoliva, L.: Are gan generated images easy to detect? a critical analysis of the state-of-the-art. In: ICME (2021)
  * [20] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector quantized diffusion model for text-to-image synthesis. In: CVPR (2022)
  * [21] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS (2020)
  * [22] Jeong, Y., Kim, D., Ro, Y., Kim, P., Choi, J.: Fingerprintnet: Synthesized fingerprints for generated image detection. In: ECCV (2022)
  * [23] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. ICLR (2018)
  * [24] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)
  * [25] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 (2020)
  * [26] Kim, G., Kwon, T., Ye, J.C.: Diffusionclip: Text-guided diffusion models for robust image manipulation. In: CVPR (2022)
  * [27] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. ICLR14 (2013)
  * [28] Li, K., Zhang, T., Malik, J.: Diverse image synthesis from semantic layouts via conditional imle. In: ICCV (2019)
  * [29] Liu, B., Yang, F., Bi, X., Xiao, B., Li, W., Gao, X.: Detecting generated images by real images. In: ECCV (2022)
  * [30] Marra, F., Gragnaniello, D., Verdoliva, L., Poggi, G.: Do gans leave artificial fingerprints? In: MIPR (2019)
  * [31] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML (2022)
  * [32] Ojha, U., Li, Y., Lee, Y.J.: Towards universal fake image detectors that generalize across generative models. In: CVPR (2023)
  * [33] Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR (2019)
  * [34] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
  * [35] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: ICCV (2021)
  * [36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [37] Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: Faceforensics++: Learning to detect manipulated facial images. In: ICCV (2019)
  * [38] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: CVPR (2023)
  * [39] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS (2022)
  * [40] Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., Komatsuzaki, A.: Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. NeurIPS Workshop (2021)
  * [41] Sha, Z., Li, Z., Yu, N., Zhang, Y.: De-fake: Detection and attribution of fake images generated by text-to-image generation models. In: CCS (2023)
  * [42] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML. PMLR (2015)
  * [43] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. ICLR (2021)
  * [44] Tan, C., Zhao, Y., Wei, S., Gu, G., Wei, Y.: Learning on gradients: Generalized artifacts representation for gan-generated images detection. In: CVPR (2023)
  * [45] Wang, S.Y., Wang, O., Zhang, R., Owens, A., Efros, A.A.: Cnn-generated images are surprisingly easy to spot… for now. In: CVPR (2020)
  * [46] Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., Li, H.: Dire for diffusion-generated image detection. In: ICCV (2023)
  * [47] Yang, Y., Wang, R., Qian, Z., Zhu, Y., Wu, Y.: Diffusion in diffusion: Cyclic one-way diffusion for text-vision-conditioned generation. ICLR (2024)
  * [48] Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., Xiao, J.: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. Arxiv (2015)
  * [49] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023)
  * [50] Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in gan fake images. In: WIFS (2019)
  * [51] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV (2017)
  * [52] Zhu, M., Chen, H., Yan, Q., Huang, X., Lin, G., Li, W., Tu, Z., Hu, H., Hu, J., Wang, Y.: Genimage: A million-scale benchmark for detecting ai-generated image. NeurIPS (2024)
  * [53] Zhu, Y., Wu, Y., Deng, Z., Russakovsky, O., Yan, Y.: Boundary guided learning-free semantic control with diffusion models. NeurIPS (2024)

In this appendix, we add some experiments to delve deeper into the robustness
of the differentiated features. We conduct a sensitivity analysis on the only
variable of the patch-shuffle operation, the patch size, in Appendix 0.A; we
use different classification heads to demonstrate the robustness of the
differentiated features to the classification heads in Appendix 0.B; we
showcase the samples that were correctly identified additionally by using
differentiated features compared to the baseline UFD in Appendix 0.C; finally,
we detailly report the test results of our method and the baseline across
various generators in Appendix 0.D.

##  Appendix 0.A Sensitivity of Discrepancy Features to Patch Size

Table 4: Results of different patch sizes on the validation set. The ablated
patch sizes range from 14 to 224 (the image size). The significant improvement
brought by the switch from 224 to 112 shows the effectiveness of introducing
discrepancy. Patch sizes 14, 28, and 56 achieve similarly high performance.
But patch size 1’s performance drops for the over-destruction of local
artifacts.

Patch size | Validataion
---|---
In-domain | Out-of-domain | Total
Mean acc. | AP | Mean acc. | AP | Mean acc. | AP
1 | 0.958 | 0.992 | 0.837 | 0.934 | 0.885 | 0.960
14 | 0.967 | 0.995 | 0.859 | 0.944 | 0.904 | 0.968
28 | 0.966 | 0.995 | 0.871 | 0.939 | 0.909 | 0.965
56 | 0.962 | 0.998 | 0.871 | 0.943 | 0.907 | 0.965
112 | 0.949 | 0.989 | 0.858 | 0.942 | 0.895 | 0.964
224 | 0.889 | 0.964 | 0.829 | 0.935 | 0.853 | 0.946

We study how the patch size affects the learning from the discrepancy
features. A tradeoff exists between increasing the discrepancy between
features and mining the universal local artifacts, i.e. the smaller patch size
offers more discrepancy but retains fewer local artifacts useful for deepfake
detection. Therefore, given the original image size of 224, we conduct
validations with different patch sizes, ranging from 1 to 224, to see the
changing trend. These experiments adhere to the proposed experimental setting,
with the patch size being the only variable adjusted. As shown in Tab. 4,
changing the patch size from 224 to 112 brings a significant improvement of
6.0 points in ID accuracy and 2.9 points in OOD accuracy, suggesting that
introducing additional discrepancy in features helps in expanding the
representation of features and extracting universal artifacts.

The patch sizes 14, 28, and 56 yield similarly high overall performance,
showing the robustness of introducing discrepancy in different patch sizes. It
is worth noting that when the patch size is 1, the local artifacts of the
patch-shuffled image are significantly affected, resulting in a drop in model
performance compared to patch size 14. In our state-of-the-art version, we
directly opt for a patch size of 14 to align with the backbone CLIP:ViT-L/14
[34] while introducing the highest discrepancy in features.

##  Appendix 0.B Different Classifier Heads

Table 5: Results of different classifier heads on the validation set. We
evaluate four classifier network architectures and find that a network that
learns the correlations between features will perform better.

Architecture | Validataion
---|---
In-domain | Out-of-domain | Total
Mean acc. | AP | Mean acc. | AP | Mean acc. | AP
FC | 0.918 | 0.977 | 0.835 | 0.938 | 0.868 | 0.954
MLP | 0.960 | 0.995 | 0.865 | 0.932 | 0.903 | 0.963
Self-Attention | 0.967 | 0.995 | 0.859 | 0.944 | 0.904 | 0.968
Transformer | 0.965 | 0.995 | 0.872 | 0.952 | 0.909 | 0.973

We investigate how different classifier heads influence the model’s
performance to verify the effectiveness of artifact invariance learning. We
evaluate four architectures: (i) FC: a single fully connected layer, (ii) MLP:
A two-layer non-linear perceptron network with ReLU activation and a hidden
layer dimension of $2\times 1024$ neurons, (iii) Self-Attention: a network
consisting of a self-attention layer [14] and a single fully connected layer,
and (iv) Transformer: A network composed of two transformer encoder layers
with 4 attention heads and a forward dimension of $4\times 1024$ [14] and one
fully connected layer.

Tab. 5 presents the results of these variants in our proposed experimental
setting. The findings show that the results of MLP, Self-Attention, and
Transformer are significantly improved compared to FC. This means establishing
the correlations between the two discrepancy features helps learn universal
artifacts. In addition, the performances of Self-Attention, MLP, and
Transformer don’t show an obvious gap, which demonstrates that our discrepancy
features are highly distinguishable for deepfake detection.

##  Appendix 0.C Comparative Analysis of Uniquely Detected Samples

We take a further step to explore how our method outperforms. We present a
selection of samples from both in-domain and out-of-domain. These samples are
accurately classified by our approach, yet erroneously classified by the UFD
[32], with a discrepancy in classification confidence exceeding 0.8. As shown
in Fig.4, these samples are challenging to discern with the naked eye. This
compellingly demonstrates that our method is capable of learning deeper and
more universal artifacts, thereby retaining its effectiveness even when
confronted with such challenging samples.

Figure 4: Visualization of uniquely detected samples. We present a selection
of samples from both in-domain and out-of-domain, that are accurately
classified by our approach, yet erroneously classified by the UFD [32], with a
discrepancy in classification confidence exceeding 0.8.

##  Appendix 0.D Detailed Mean Accuracy Results of Comparisons with the State-
of-the-arts

In this section, we report the detailed mean accuracy results of comparisons
with the state-of-the-art in Tab. 6, as a supplement to Table 1 in Sec 4.5,
including ADM [13], BigGAN [4], GLIDE [31], Midjourney [2], LDM [36], VQDM
[20], wukong [3], ProGAN [23], CycleGAN [51], StyleGAN [24], StyleGAN2 [25],
GauGAN [33], StarGAN [8], Deepfakes [37], whichfaceisreal [1], SITD [6], SAN
[11], CRN [7], IMLE [28], and DALL$\cdot$E [35]. Results of generators with
the same architecture but different parameters are averaged. For example, the
result of BigGAN [4] in this table is the average of BigGAN in UFD [32] and
BigGAN in GenImage [52].

Table 6: Detailed mean accuracy results of comparisons with the state-of-the-
art on the testing set. We report the mean accuracy per generator in the
percentage form. Results of generators with the same architecture but
different parameters are averaged.

| In-domain | Out-of-domain |  |  |
---|---|---|---|---|---
| ADM | Big | GLI | Mid | LDM | VQ | wu | Pro | Cycle | Style | Style | Gau | Star | Deep | which- | SI | SAN | CRN | IM | DAL |  |  |
methods |  | GAN | DE | jour |  | DM | kong | GAN | GAN | GAN | GAN2 | GAN | GAN | fakes | face | TD |  |  | LE | L$\cdot$E | ID | OOD | Total
|  |  |  | ney |  |  |  |  |  |  |  |  |  |  | isreal |  |  |  |  |  |  |  |
CNNDet | 89.6 | 79.1 | 98.2 | 97.3 | 93.7 | 97.0 | 95.7 | 95.6 | 72.0 | 75.7 | 80.0 | 56.4 | 98.2 | 80.2 | 47.3 | 49.7 | 72.4 | 56.0 | 56.0 | 94.7 | 93.3 | 69.9 | 79.2
Patchfor | 99.8 | 85.1 | 99.6 | 99.6 | 99.7 | 99.7 | 99.7 | 99.9 | 93.2 | 98.1 | 94.4 | 59.1 | 99.8 | 90.7 | 61.6 | 65.4 | 84.5 | 50.0 | 50.0 | 99.6 | 97.9 | 78.9 | 86.5
LNP | 91.0 | 72.5 | 95.0 | 94.4 | 92.2 | 89.4 | 92.8 | 87.3 | 71.0 | 89.9 | 85.0 | 68.8 | 83.7 | 65.5 | 53.9 | 52.0 | 55.8 | 50.1 | 62.6 | 84.6 | 89.3 | 68.6 | 76.9
DIRE | 99.9 | 82.6 | 99.9 | 99.8 | 99.9 | 100 | 100 | 98.4 | 60.3 | 94.2 | 95.1 | 55.3 | 88.6 | 67.9 | 50.0 | 50.0 | 59.7 | 50.0 | 50.0 | 99.7 | 97.6 | 68.4 | 80.1
UFD | 83.2 | 92.0 | 86.3 | 80.2 | 85.4 | 89.0 | 85.6 | 91.1 | 74.8 | 79.4 | 82.4 | 95.5 | 89.0 | 75.9 | 79.8 | 73.1 | 61.9 | 87.9 | 90.0 | 86.6 | 86.6 | 81.4 | 83.5
Ours | 94.8 | 98.5 | 95.0 | 96.8 | 94.4 | 96.7 | 97.1 | 99.4 | 92.7 | 94.9 | 95.7 | 98.1 | 96.0 | 67.7 | 83.1 | 73.8 | 62.6 | 88.1 | 95.0 | 92.8 | 96.6 | 87.6 | 90.7