#  Exceptional points in single open acoustic resonator due to the symmetry
breaking

Vladimir Igoshin  Mariia Tsimokha  Anastasia Nikitina  Mihail Petrov
<EMAIL_ADDRESS>Ivan Toftul  Kristina Frizyuk
<EMAIL_ADDRESS>The School of Physics and Engineering, ITMO
University, Saint-Petersburg 197101, Russia

###### Abstract

Exceptional points (EPs) have been widely studied in quantum mechanics,
condensed matter physics, optics and photonics. However, their potential in
acoustics has only recently been recognized due to the rapid development of
acoustic metamaterials. This paper proposes a method for achieving EP
conditions in acoustic resonators by lowering their symmetry and enabling
resonant mode interaction. The formation of EPs is predicted through direct
numerical simulation supported by coupled mode theory and resonant state
expansion. These findings have significant implications for the design and
optimization of acoustic metamaterials for applications such as acoustic
sensing and noise reduction.

##  I Introduction

Acoustic metamaterials are a promising class of materials that offer unique
capabilities for tailoring properties of sound waves [1, 2, 3] as well as for
mechanical manipulation [4, 5, 6]. While resonances play a central role in
acoustics metamaterials there are still many hindered physical effects and
mechanism. EPs are the points in the parameter space where the eigenvalues of
the system become degenerate and the eigenvectors coalesce, leading to a non-
diagonalizable Jordan block formation [7, 8, 9, 10]. The spectral
singularities related to EP are highly sensitive to the system parameters
making them very prospective for sensing applications [11, 12]. Despite the
progress in this area, the problem of EP appearance in single acoustic
resonators still requires thorough study and it is addressed in this paper.

EPs occur only in non-hermitian systems [10] and they are often mentioned in
the context of PT-symmetric systems [13, 8, 14, 15] observed in electronics
[16], optics and photonics [13, 11, 15], and recently in acoustics [17, 18].
However, reaching PT-symmetry requires particular engineering of gain and loss
in acoustical systems. Alternatively, EP can be observed in open resonators as
a special class of non-Hermitian systems, which was extensively studied in
optics and photonics [15, 19, 20]. One of the possible mechanisms of EP
formation is based on breaking the symmetry of the resonator [21]. While this
is not the unique approach we leave the other methods beyond the scope of the
current work and refer the readers to Ref. 8, 22. However, despite extensive
research on EPs in optics, there has been little work on EPs in the acoustics
domain.

Figure 1: The main idea of this work. Symmetry breaking perturbation merges
two groups of modes induced by symmetry considerations into a single group.
This is shown at the bottom of the figure. By changing amplitude of the
symmetry breaking perturbation we can tune a coupling strength. As shown in
the top of the figure, this allows us to observe the transition from crossing
of energy terms to avoided crossing. This transition can be followed by EP
characterized by the coalescence of eigenspaces.

In this work, we show that by engineering the shape of resonators and due to
related symmetry breaking one can enable mode coupling mechanisms leading to
formation of EP condition as schematically shown in Fig. 1: in a system with
perturbed symmetry two initially non-interacting modes transform into two
different modes a degenerate a state. This transitional state appears to be an
EP state.

We improve and adapt the powerful method of multipolar analysis [23, 24, 25]
to (i) predict the occurrence of EPs as a result of a particular symmetry
breaking and (ii) to understand deeper the mode interaction within a coupled
mode theory and resonant state expansion (RSE) method.

Exploring the physics behind the EPs formation in acoustic resonators may
unlock novel methods for the design and optimization of acoustic metamaterials
for a wide range of applications such as sound focusing [26], optomechanics
[27], sensing [28, 29, 30, 31], noise insulation [32], seismic cloaking [33,
34]. By addressing the important questions surrounding the physics of acoustic
metamaterials, we may open the door to new and exciting opportunities in
acoustics and materials science.

This work is organised as follows. In Section II we build a simple model based
on the linear acoustic equations and discuss the simple mechanism of the EP
appearance. In Section III we extend the approach to more complex symmetries
and generalize the method. Finally, in Section IV we conclude the obtained
results.

##  II Coupling of modes in acoustic resonators

Figure 2:  Tables of irreducible representations of symmetry groups
$\text{D}_{\infty\text{h}}$ and $\text{C}_{\infty\text{v}}$ and examples of
the eigenmodes, transforming under these irreps. Only the first few irreps are
shown. Monopole mode transforms under irreducible representation
$\text{A}_{1\text{g}}$ but omitted in the figure in favour of a more
illustrative mode. Examples of modes that transform under the particular
irreducible representations are given. The arrows show how the irreducible
representations of the two groups are related to each other.

###  II.1 Model physics and parameters

We focus on the studies of resonators in the framework of linear acoustics in
the frequency domain. The resonators are made of homogeneous acoustics
materials and placed in homogeneous environment (fluid or gas) which supports
only longitudinal waves. Both media are characterized by their compressibility
$\beta$ and mass density $\rho$ (the speed of sound is
$c=1/\sqrt{\beta\rho}$). The complex velocity $\mathbf{v}(\mathbf{r})$ and
pressure $p(\mathbf{r})$ fields satisfy the wave equations [35]

$\mathrm{i}\mkern 1.0mu\omega\beta
p=\divergence\mathbf{v},\qquad\mathrm{i}\mkern
1.0mu\omega\rho\mathbf{v}=\gradient p.$ (1)

By assuming the harmonic convention $\exp(-\mathrm{i}\mkern 1.0mu\omega t)$
and applying the Sommerfeld radiation condition, one can end up with a
generalized eigenvalue problem

$\underbrace{\begin{pmatrix}0&-\divergence\\\
\gradient&0\end{pmatrix}}_{\hat{\mathbb{D}}}\underbrace{\begin{pmatrix}p\\\
\mathrm{i}\mkern
1.0mu\mathbf{v}\end{pmatrix}}_{\vec{\mathbb{F}}}=\omega\underbrace{\begin{pmatrix}\beta(\textbf{r})&0\\\
0&\rho(\textbf{r})\end{pmatrix}}_{\hat{\mathbb{P}}}\underbrace{\begin{pmatrix}p\\\
\mathrm{i}\mkern 1.0mu\mathbf{v}\end{pmatrix}}_{\vec{\mathbb{F}}},$ (2)

which describes eigenmodes $\vec{\mathbb{F}}(\mathbf{r})$ of an open acoustic
resonator, where operators $\hat{\mathbb{D}}(\mathbf{r})$ and
$\hat{\mathbb{P}}(\textbf{r})$ are introduced for future convenience. The
formulated eigenvalue problem is in a full analogy to the one appearing in
optics [36].

Eigenmodes of this system can be classified by irreducible representations
(irreps) of a symmetry group of a resonator [37, 38, 23] similarly to the case
of optical resonators [24]. Some particular material parameters have to be
chosen to illustrate general conclusions of this paper which, in general, do
not depend on this choice. For the most acoustic experiments performed in air,
the wavelength in the scatterer is longer than in the host medium. However, we
have chosen our parameters in favour of numerical stability since all
conclusions do not depend on whether acoustic refractive index
$\sqrt{{\beta_{1}\rho_{1}}/{\beta_{0}\rho_{0}}}$ is greater or less then one.
To be specific, we use density $\rho_{0}=1\leavevmode\nobreak\
\text{kg}/\text{m}^{3}$ and speed of sound $c_{0}=1\leavevmode\nobreak\
\text{m/s}$ for the media and $\rho_{1}=\rho_{0}/2$, $c_{1}=c_{0}/2$ for the
resonator. Needless to say that similar scenario is also possible in practice:
high-index resonator material is typical for Mie-resonance acoustic meta-
atoms, which can be used to create metamaterials [39, 5, 40, 41].
Nevertheless, we stress again that the theory presented below does not depend
on the material parameters of the media and the resonator.

The eigenmodes computations were performed with help of numerical simulations
in COMSOL Multiphysics® and the method described briefly in Appendix C.

###  II.2 EP appearance

Figure 3: Real parts of eigenvalues $k$ scaled to $h_{0}$ of a cylinder with
$\alpha=1$ (a) and a conical frustum with $\alpha=0.89$ (b) versus $h_{0}/h$.
The branches of cylinder’s modes have different colors according to their
irrep. Intersections of modes that transform under different irreps are
circled. Corresponding avoided crossings and crossing with weak coupling are
marked with a square. The crossing and the avoided crossing shown in Fig. 4
are dashed. Geometry parameters used are $h_{0}=1\text{ m}$, $R_{0}=0.5\text{
m}$.

In order to clarify the connection between the symmetry and EP formation, we
firstly will elaborate on the resonator spectrum modification with change of
the resonator shape. We start with a structures of high symmetry such as
cylinders [42] and move to more complex shapes later. We fix the initial
geometry of the cylinder such that height to radius ratio is $h_{0}/R_{0}=2$
[see Fig. 3 (inset)].

Cylinder corresponds to $\text{D}_{\infty\text{h}}$ symmetry group, thus it
has an countably infinite set of finite-dimensional irreducible
representations [43, 44, 45] as shown in Fig. 2, where each representation is
demonstrated along with corresponding examples of numerically calculated
pressure fields. Each of those irreducible representations of a symmetry group
of a cylinder corresponds to a single value of the azimuthal number $m$. In
the following, we narrow the consideration to $m=0$ only, or, in more specific
terms, eigenmodes which transformed under irreducible representations
A${}_{\text{1u}}$ and A${}_{\text{1g}}$. These eigenmodes have rotational
symmetry, but different parity under horizontal reflection
($\sigma_{\text{h}}$ transformation). The obtained results can be
straightforwardly extended to other azimuthal numbers $m$. We have omitted the
monopole mode which transforms under irreducible representation
$\text{A}_{1\text{g}}$ in favour of another more illustrative example.
However, we stress that during the symmetry braking
$\text{D}_{\infty\text{h}}\to\text{C}_{\infty\text{v}}$ (e.g. cylinder to
cone), monopole and dipole modes start to transform under the same irreducible
representation $\text{A}_{1}$ in the cone symmetry group and hence interact
with each other. Resonators which exhibit this feature are conventionally
characterised by Willis coupling [6, 46, 47, 48].

The dependence of the real part of eigenfrequency of the $m=0$ modes on the
resonator’s height is shown in Fig. 3 (a), where each color of the line
corresponds to its own irreducible representation A${}_{\text{1u}}$ or
A${}_{\text{1g}}$. Hereafter, we would call eigenmodes transformed under
irreducible representations A${}_{\text{1u}}$ and A${}_{\text{1g}}$ as mode
A${}_{\text{1u}}$ and mode A${}_{\text{1g}}$ respectively. At this point, it
should be mentioned that modes transforming under different irreps are
orthogonal, therefore coupling between them does not occur [37, 38]. In Fig. 3
(a), this statement can be interpreted visually, where the crossing between
two orthogonal modes of a cylinder, A${}_{\text{1u}}$ and A${}_{\text{1g}}$,
occurs. Indeed, explicit analysis of the eigenmodes’ fields show that these
modes have different parities, ’odd’ and ’even’ correspondingly, with respect
to $\sigma_{\text{h}}$ (reflection in the horizontal plane) transformation. On
the other hand, Fig. 3 (a) shows the avoided crossing behavior of the energy
lines of the mode related to the same irrep.

Next, one can alter the symmetry of the resonator by perturbing its shape from
cylinder to cone, which breaks the mirror symmetry in horizontal plane. In
order to carefully trace out changes in the mode structure those symmetrical
changes should be made gradually, so we chose the parameter of
‘‘cylindricity’’ of the resonator’s shape $\alpha$ Fig. 3 (b), which denotes
the amplitude of the shape perturbation. Varying $\alpha$ from 1 to 0 reduces
the top radius of the cylinder, while simultaneously increasing the bottom
radius. Thus, decreasing the $\alpha$ parameter breaks the symmetry with
respect to $\sigma_{\text{h}}$ and changes the symmetry group of the system
from $\text{D}_{\infty\text{h}}$ to $\text{C}_{\infty\text{v}}$. According to
Fig. 2 previously considered A${}_{\text{1u}}$ and A${}_{\text{1g}}$ modes
lose their specific $\sigma_{\text{h}}$ parity property, and now transform
under the same irrep A${}_{\text{1}}$. Therefore, a coupling between those
modes should now appear in conical symmetry which manifests itself in
appearance of avoided crossings between the frequency lines shown in Fig. 3
(b). The transition from crossing to avoided crossing induced by symmetry
breaking is shown with arrows.

Figure 4: Maps of real and imaginary parts of eigenvalues $k$ of the acoustic
resonator (scaled to $h_{0}$). Examples of fields in resonators are given. (a)
The crossing of real parts of two non-interacting modes that transform under
different irreps $\text{A}_{\text{1u}}$ and $\text{A}_{\text{1g}}$ is observed
at ‘‘cylindricity’’ parameter $\alpha=1.000$. (e) At $\alpha=0.890$ there is
an anticrossing of real parts of modes that transform under the same irrep
$\text{A}_{\text{1}}$ (strong coupling). At $\alpha=0.902$ an EP is achieved
in which both real (c) and imaginary (d) parts of eigenvalues are crossed, and
therefore modes degenerate. A toy model fitting is discussed in Appendix B.

To this point, almost exclusively the real part of the eigenfrequency was
discussed. Imaginary parts of the eigenfrequencies are shown in Fig. 4 (b).
While a crossing behaviour in real parts [Fig. 4 (a)] can be observed,
imaginary parts are not equal, and the modes are not degenerate. However, by
varying the height of the resonator $h$ and the cylindricity parameter
$\alpha$, at a certain coupling strength, the simultaneous degeneracy of real
and imaginary parts of two modes is observed [see Fig. 4 (c,d)]. Now the
series of Fig. 4 (a, c, e) can be regarded as follows: from left to right
three different stages of the resonator’s deformation from a cylinder to a
cone are shown. At the most left (a,b), a $\text{D}_{\infty\text{h}}$ symmetry
of the resonator ensures a complete absence of any interaction between the two
considered modes; at the most right (e,f) the interaction is great enough to
keep those modes entirely apart from each other; evidently, based on the
smoothness of this particular symmetry breaking, at the intermediate
parameters shown in (c,d), a point corresponding to full degeneracy of two
those two modes can be found. In Fig. 4 (c, d) two lines merge into a single
point, both for the real and imaginary parts of the frequency, which
corresponds to formation of the EP. Note, that usual degeneracy of two modes
happens in so-called diabolic points [10], which correspond to the two- or
more dimensional irreps, but here the fact that A${}_{\text{1}}$ irrep is one-
dimensional make this even not possible. In simple words, we cannot obtain two
modes with the same frequency whose field are linearly independent [13, 49,
50, 11]. One should also note, that the crossing-to-avoided crossing mechanism
is precisely inverted for real and imaginary parts, as shown in Fig. 4 (b, d,
f), where the system goes from an anti-crossing, through a degeneracy, to a
crossing. The observed modes interaction via symmetry breaking can be extended
to other shapes of the resonators as discussed in Appendix A.

###  II.3 Coupled mode theory and perturbation of the material parameters

The appearance of EP in quantum mechanics and optics can be often interpreted
within a scope of a standard coupled mode theory, described by a
2$\crossproduct$2 Hamiltonian of a two-level system [13, 49, 50, 11, 8]. While
coupling of several acoustic resonators has already been analysed withing
standard coupled mode theory [51, 52], the mode coupling in a single acoustic
resonator upon its shape perturbation has not yet been discussed. In this
section, we introduce the phenomenological model of interaction of two
eigenmodes based on resonant-state expansion (RSE) or quasinormal mode
expansion [53, 54, 36, 55]. RSE has already proved its efficiency for
determining the eigenmodes of perturbed optical system, appearance of strong
coupling regime [56], and multipole coupling [24]. However, as far as we know,
no such analysis has yet been provided for linear acoustics described by Eq.
(2). This equations can be rewritten in a compact form similar to Ref. 36:

$\hat{\mathbb{D}}(\mathbf{r})\vec{\mathbb{F}}_{n}(\mathbf{r})=\omega_{n}\hat{\mathbb{P}}(\mathbf{r})\vec{\mathbb{F}}_{n}(\mathbf{r})$
(3)

where $\vec{\mathbb{F}}_{n}(\mathbf{r})=\begin{pmatrix}p_{n},&\mathrm{i}\mkern
1.0mu\mathbf{v}_{n}\end{pmatrix}^{\intercal}$ is an eigenmode, $\omega_{n}$ is
an eigenvalue. For the perturbed resonator there is a new set of eigenmodes
$\vec{\mathbb{F}}(\mathbf{r})$ and eigenvalues $\Omega$

$\hat{\mathbb{D}}(\mathbf{r})\vec{\mathbb{F}}(\mathbf{r})=\Omega\left(\hat{\mathbb{P}}(\mathbf{r})+\Delta\hat{\mathbb{P}}(\mathbf{r})\right)\vec{\mathbb{F}}(\mathbf{r})$
(4)

where $\Delta\hat{\mathbb{P}}(\mathbf{r})$ is the perturbation of the material
parameters, caused, for example, by changing the resonator’s shape.

It is assumed that all eigenmodes are a linear combination of eigenmodes of
the unperturbed system
$\vec{\mathbb{F}}(\mathbf{r})=\sum\limits_{n}c_{n}\vec{\mathbb{F}}_{n}(\mathbf{r})$.
Within this approach, the perturbation theory formalism [36] immediately
provides us with the coupled equations system

$\sum_{n}\left((\Omega-\omega_{n})\delta_{mn}+\Omega V_{mn}\right)c_{n}=0$ (5)

where
$V_{mn}\propto\displaystyle\int\vec{\mathbb{F}}_{m}^{\intercal}\Delta\hat{\mathbb{P}}\vec{\mathbb{F}}_{n}$
is the matrix element. In the two-mode approximation of Eq. (5), we obtain the
eigenvalue problem

$\begin{pmatrix}\omega_{1}&0\\\
0&\omega_{2}\end{pmatrix}\begin{pmatrix}c_{1}\\\
c_{2}\end{pmatrix}=\Omega\begin{pmatrix}1+V_{11}&V_{12}\\\
V_{12}&1+V_{22}\end{pmatrix}\begin{pmatrix}c_{1}\\\ c_{2}\end{pmatrix}$ (6)

The eigenvalues of this equation are

$\Omega_{\pm}=\frac{\xi\omega_{2}+\gamma\omega_{1}\pm\sqrt{(\xi\omega_{2}-\gamma\omega_{1})^{2}+4\kappa^{2}\omega_{1}\omega_{2}}}{2(\xi\gamma-\kappa^{2})}$
(7)

where $\kappa=V_{12}=V_{21}$, $\xi=1+V_{11}$, $\gamma=1+V_{22}$. It can be
seen that the non-diagonal element $\kappa$ is responsible for the coupling
between the modes.

The main condition of observation of EP can be summarized as
$\Omega_{-}=\Omega_{+}$. Or, in a more extended form
$(\xi\omega_{2}-\gamma\omega_{1})^{2}+4\kappa^{2}\omega_{1}\omega_{2}=0$. It
can be seen that if this condition is satisfied and $\kappa\neq 0$, the
eigenspaces collapse.

Here, we show that this system can be applied to describing the EP formation
in acoustic resonators with symmetry breaking. The exact identification of
governing parameters $\kappa$, $\xi$, and $\gamma$ requires introducing the
proper scalar product and normalization and stays beyond the scope of the
current work. Still, we found these parameters numerically by fitting the real
and imaginary frequency curves in Fig. 4 within Eq. (7) (see Appendix B for
the details). The obtained results are shown with dots in Fig. 4 demonstrating
excellent correspondence to the numerical results.

Within the coupled mode approach it becomes evident that two modes that
transform under the same irrep may couple to each other since the coupling
constant provided by the integral
$\kappa\propto\displaystyle\int\vec{\mathbb{F}}_{1}^{\intercal}\Delta\hat{\mathbb{P}}\vec{\mathbb{F}}_{2}\neq
0$ in full accordance with Wigner’s theorem [57]. It can also be derived from
the selection rules presented in [37, 38].

##  III Generalization of the approach to other symmetries

The illustrated mechanism of EP formation can be extended to resonators of
arbitrary shapes. The general recipe for that is as follows:

  1. 1.

Two non-interacting eigenmodes, which transform under different irreps have to
be chosen.

  2. 2.

Next, by particular symmetry breaking one needs to assure that these modes
will appear in the same irrep. This step is essential, as the chosen symmetry
will trigger the formation of EP states.

  3. 3.

Now the original symmetry has to be broken smoothly for transition from an
object of the original symmetry, to one of the chosen new symmetry. In other
words, we should have a homotopy equivalence between two shapes. Thus, the
shape change is moderated by one parameter. This way, an appearance of EP can
be traced easily.

Following this algorithm, one can observe EP formation in more complex shapes
and symmetries of the resonator. As an example, besides already considered
$\text{D}_{\infty\text{h}}$ to $\text{C}_{\infty\text{v}}$ symmetry breaking,
one can observe EP formation in $C_{4v}$ to $C_{2v}$ symmetry breaking as
discussed in details in Appendix A and supported by numerical simulations.
Formation of EP is provided, for example by the merging of $B_{1}$ and $A_{1}$
irreps into $A_{1}$.

##  IV Conclusion

In this work, a general mechanism of retrieving exceptional points in complex
acoustic systems is proposed. We identify an approach to achieving exceptional
point condition by breaking a particular symmetry of the resonator leading to
coupling of initially orthogonal eigenmodes. Smooth variation of asymmetry
parameter allows for finding the exact conditions for mode coalescence. This
mechanism was illustrated in single homogeneous acoustic resonators by
breaking their symmetry from $\text{D}_{\infty\text{h}}$ to
$\text{C}_{\infty\text{v}}$, from $C_{4\text{v}}$ to $C_{2\text{v}}$. The
results were supported by direct numerical simulations. Moreover, we proposed
a phenomenological model of interaction of two eigenmodes based on resonant
state expansion method and perturbation theory. We have verified that the
proposed description in the two-mode approximation can be used for qualitative
analysis of eigenfrequencies.

###### Acknowledgements.

The numerical and analytical simulations were supported by Russian Science
Foundation grant no. 20-72-10141. The work is financially supported by the
Program Priority 2030. MP acknowledges support from BASIS Foundation.

##  Appendix A Example of $\text{C}_{4\text{v}}\to\text{C}_{2\text{v}}$
transition

Figure 5:  Tables of irreducible representations of symmetry groups
$\text{C}_{4\text{v}}$ and $\text{C}_{2\text{v}}$. Examples of modes
transformed under particular irreducible representations are given. The arrows
show how the irreducible representations of the two groups are related to each
other.  Figure 6: Real parts of scaled with $h_{0}$ eigenvalues $k$ of a
square right pyramid ($\alpha=1$) and a rectangular right pyramid
($\alpha=0.92$) versus $h_{0}/h$. The branches of the square pyramid modes
have different colors according to their irrep. Intersections of modes that
transform under different irreps are circled. The corresponding avoided
crossings are marked by a square. Geometry parameters used are $h_{0}=1\text{
m}$, $a=1\text{ m}$.

The mechanism of EP formation described above is very general and is based on
symmetry requirements.

Having two modes which show a crossing behaviour in the unperturbed system, we
break the symmetry in the way that makes this modes to interact (i.e. merges
two irreps into a single one, so two modes, which transform by the different
irreps will be transformed by the same irrep after the perturbation). The EP
state occurrence for $\text{D}_{\infty\text{h}}\to\text{C}_{\infty\text{v}}$
symmetry breaking, which has already been considered above, can be generalized
to more complex cases such as a transition
$\text{C}_{4\text{v}}\to\text{C}_{2\text{v}}$.

The merging $\text{A}_{1}$ and $\text{B}_{1}$ modes of regular square pyramid
$\text{C}_{4\text{v}}$ to $\text{A}_{1}$ modes of rectangular right pyramid
$\text{C}_{2\text{v}}$ is shown in Fig. 5, while Fig. 6 illustrates crossing
between the two orthogonal modes $\text{A}_{1}$ and $\text{B}_{1}$ of the
regular square pyramid and avoided crossing between the two modes of the
rectangular right pyramid that transform under same irreducible representation
$\text{A}_{1}$. In this case avoided crossing and exceptional points can be
easily achieved due to the symmetry breaking and the proper selection of
parameters.

##  Appendix B Toy model fitting

In order to test the ability of the model (Section II.3) to describe the
effects presented in the Fig. 4, we fit its parameters $\xi$, $\gamma$,
$\kappa$. A range of heights $h_{0}/h$ from $1.52$ to $1.55$ is used, model
parameters are assumed to be constant. Eigenfrequencies for unperturbed system
are approximated as

$\displaystyle\omega_{1}(h_{0}/h)=(2.224724-\mathrm{i}\mkern
1.0mu0.023396)\cdot h_{0}/h$ $\displaystyle+6.846411-\mathrm{i}\mkern
1.0mu0.242355$

and

$\displaystyle\omega_{2}(h_{0}/h)=(6.302923-\mathrm{i}\mkern
1.0mu0.221363)\cdot h_{0}/h$ $\displaystyle+0.606508-\mathrm{i}\mkern
1.0mu0.31179$

The results of fitting for each $\alpha$ value are presented in Table 1. We
choose model parameters for which there is a sufficiently accurate match with
the numerical experiment.

Table 1:  Model parameters for different $\alpha$ values and their corresponding fitting relative errors. $\alpha$ | $\xi$ | $\gamma$ | $\kappa$ | max rel. error, %
---|---|---|---|---
$1.000$ | $1$ | $1$ | $0$ | $0.0044$
$0.902$ | $0.993783+\mathrm{i}\mkern 1.0mu0.000728$ | $1.000755-\mathrm{i}\mkern 1.0mu0.001037$ | $0.017148-\mathrm{i}\mkern 1.0mu0.001479$ | $0.0534$
$0.890$ | $0.992245+\mathrm{i}\mkern 1.0mu0.000903$ | $1.000909-\mathrm{i}\mkern 1.0mu0.001272$ | $0.019272-\mathrm{i}\mkern 1.0mu0.001574$ | $0.0113$

##  Appendix C Numerical modelling

All simulations were performed using the Pressure Acoustics branch of
Acoustics module of COMSOL Multiphysics® software. The simulation domain was a
sphere made of media material with the resonator inside. When searching
eigenmodes for Fig. 4, the perfectly matched layer (PML) was used to simulate
open boundary conditions on the surface of the spherical domain. To calculate
eigenmodes for Fig. 3 and Fig. 6 the radiation boundary condition for a
spherical wave was used instead of the PML in order to speed up calculations.
All numerical modes (also termed as PML modes) were removed during post-
processing [54].

## References

  * Cummer  _et al._ [2016] S. A. Cummer, J. Christensen, and A. Alù, Controlling sound with acoustic metamaterials, Nat. Rev. Mater. 1, 1 (2016).
  * Fok  _et al._ [2008] L. Fok, M. Ambati, and X. Zhang, Acoustic Metamaterials, MRS Bull. 33, 931 (2008).
  * Ma and Sheng [2016] G. Ma and P. Sheng, Acoustic metamaterials: From local resonances to broad horizons, Sci. Adv. 2, e1501595 (2016).
  * Toftul  _et al._ [2019] I. D. Toftul, K. Y. Bliokh, M. I. Petrov, and F. Nori, Acoustic Radiation Force and Torque on Small Particles as Measures of the Canonical Momentum and Spin Densities, Phys. Rev. Lett. 123, 183901 (2019).
  * Shi  _et al._ [2019] C. Shi, R. Zhao, Y. Long, S. Yang, Y. Wang, H. Chen, J. Ren, and X. Zhang, Observation of acoustic spin, Natl. Sci. Rev. 6, 707 (2019).
  * Sepehrirahnama  _et al._ [2022] S. Sepehrirahnama, S. Oberst, Y. K. Chiang, and D. A. Powell, Willis Coupling-Induced Acoustic Radiation Force and Torque Reversal, Phys. Rev. Lett. 129, 174501 (2022).
  * [7] T. Kato,  _Perturbation theory for linear operators_ (Springer Berlin, Heidelberg).
  * Heiss [2012] W. D. Heiss, The physics of exceptional points, J. Phys. A: Math. Theor. 45, 444016 (2012).
  * Schäfer  _et al._ [2022] R. Schäfer, J. C. Budich, and D. J. Luitz, Symmetry protected exceptional points of interacting fermions, Phys. Rev. Res. 4, 033181 (2022).
  * Moiseyev [2011] N. Moiseyev,  _Non-Hermitian Quantum Mechanics_ (Cambridge University Press, Cambridge, England, UK, 2011).
  * Wiersig [2020] J. Wiersig, Review of exceptional point-based sensors, Photonics Res. 8, 1457 (2020).
  * Park  _et al._ [2020] J.-H. Park, A. Ndao, W. Cai, L. Hsu, A. Kodigala, T. Lepetit, Y.-H. Lo, and B. Kanté, Symmetry-breaking-induced plasmonic exceptional points and nanoscale sensing, Nat. Phys. 16, 462 (2020).
  * Miri and Alù [2019] M.-A. Miri and A. Alù, Exceptional points in optics and photonics, Science 363, eaar7709 (2019).
  * El-Ganainy  _et al._ [2018] R. El-Ganainy, K. G. Makris, M. Khajavikhan, Z. H. Musslimani, S. Rotter, and D. N. Christodoulides, Non-Hermitian physics and PT symmetry, Nat. Phys. 14, 11 (2018).
  * Feng  _et al._ [2017] L. Feng, R. El-Ganainy, and L. Ge, Non-Hermitian photonics based on parity–time symmetry, Nat. Photonics 11, 752 (2017).
  * Schindler  _et al._ [2011] J. Schindler, A. Li, M. C. Zheng, F. M. Ellis, and T. Kottos, Experimental study of active LRC circuits with $\mathcal{PT}$ symmetries, Phys. Rev. A 84, 040101 (2011).
  * Achilleos  _et al._ [2017] V. Achilleos, G. Theocharis, O. Richoux, and V. Pagneux, Non-Hermitian acoustic metamaterials: Role of exceptional points in sound absorption, Phys. Rev. B 95, 144303 (2017).
  * Fleury  _et al._ [2015] R. Fleury, D. Sounas, and A. Alù, An invisible acoustic sensor based on parity-time symmetry, Nat. Commun. 6, 1 (2015).
  * Midya  _et al._ [2018] B. Midya, H. Zhao, and L. Feng, Non-Hermitian photonics promises exceptional topology of light, Nat. Commun. 9, 1 (2018).
  * Pan  _et al._ [2018] M. Pan, H. Zhao, P. Miao, S. Longhi, and L. Feng, Photonic zero mode in a non-Hermitian photonic lattice, Nat. Commun. 9, 1 (2018).
  * Valero  _et al._ [2022] A. C. Valero, V. Bobrovs, D. Redka, A. S. Shalin, and Y. Kivshar, Magnetoelectric Exceptional Points in Isolated All-Dielectric Nanoparticles, arXiv 10.48550/arXiv.2205.05735 (2022), 2205.05735 .
  * Cao and Wiersig [2015a] H. Cao and J. Wiersig, Dielectric microcavities: Model systems for wave chaos and non-Hermitian physics, Rev. Mod. Phys. 87, 61 (2015a).
  * Tsimokha  _et al._ [2022a] M. Tsimokha, V. Igoshin, A. Nikitina, I. Toftul, K. Frizyuk, and M. Petrov, Acoustic resonators: Symmetry classification and multipolar content of the eigenmodes, Phys. Rev. B 105, 165311 (2022a).
  * Gladyshev  _et al._ [2020] S. Gladyshev, K. Frizyuk, and A. Bogdanov, Symmetry analysis and multipole classification of eigenmodes in electromagnetic resonators for engineering their optical properties, Phys. Rev. B 102, 075103 (2020).
  * Sadrieva  _et al._ [2019] Z. Sadrieva, K. Frizyuk, M. Petrov, Y. Kivshar, and A. Bogdanov, Multipolar origin of bound states in the continuum, Phys. Rev. B 100, 115303 (2019).
  * Guenneau  _et al._ [2007] S. Guenneau, A. Movchan, G. Pétursson, and S. A. Ramakrishna, Acoustic metamaterials for sound focusing and confinement, New J. Phys. 9, 399 (2007).
  * Eichenfield  _et al._ [2009] M. Eichenfield, J. Chan, R. M. Camacho, K. J. Vahala, and O. Painter, Optomechanical crystals, Nature 462, 78 (2009).
  * Nirschl  _et al._ [2009] M. Nirschl, A. Blüher, C. Erler, B. Katzschner, I. Vikholm-Lundin, S. Auer, J. Vörös, W. Pompe, M. Schreiter, and M. Mertig, Film bulk acoustic resonators for DNA and protein detection and investigation of in vitro bacterial S-layer formation, Sens. Actuators, A 156, 180 (2009).
  * Thomas  _et al._ [2016] S. Thomas, M. Cole, F. H. Villa-López, and J. W. Gardner, High frequency surface acoustic wave resonator-based sensor for particulate matter detection, Sens. Actuators, A 244, 138 (2016).
  * Lee  _et al._ [2020] T. Lee, T. Nomura, X. Su, and H. Iizuka, Fano-Like Acoustic Resonance for Subwavelength Directional Sensing: 0–360 Degree Measurement, Adv. Sci. 7, 1903101 (2020).
  * Molerón and Daraio [2015] M. Molerón and C. Daraio, Acoustic metamaterial for subwavelength edge detection, Nat. Commun. 6, 1 (2015).
  * Krasikova  _et al._ [2023] M. Krasikova, S. Krasikov, A. Melnikov, Y. Baloshin, S. Marburg, D. A. Powell, and A. Bogdanov, Metahouse: Noise-Insulating Chamber Based on Periodic Structures, Adv. Mater. Technol. 8, 2200711 (2023).
  * Muhammad and Lim [2022] Muhammad and C. W. Lim, From Photonic Crystals to Seismic Metamaterials: A Review via Phononic Crystals and Acoustic Metamaterials, Arch. Comput. Methods Eng. 29, 1137 (2022).
  * Gupta  _et al._ [2023] A. Gupta, R. Sharma, A. Thakur, and P. Gulia, Metamaterial foundation for seismic wave attenuation for low and wide frequency band, Sci. Rep. 13, 1 (2023).
  * Landau and Lifshitz [1987] L. D. Landau and E. M. Lifshitz,  _Fluid Mechanics_ (Pergamon, Oxford, England, UK, 1987).
  * Muljarov and Weiss [2018] E. A. Muljarov and T. Weiss, Resonant-state expansion for open optical systems: generalization to magnetic, chiral, and bi-anisotropic materials, Opt. Lett. 43, 1978 (2018).
  * Zee [2016] A. Zee,  _Group Theory in a Nutshell for Physicists_ (Princeton University Press, Princeton, NJ, USA, 2016).
  * Landau and Lifshitz [1977a] L. D. Landau and E. M. Lifshitz,  _Quantum Mechanics_ (Pergamon, Oxford, England, UK, 1977).
  * Cheng  _et al._ [2015] Y. Cheng, C. Zhou, B. G. Yuan, D. J. Wu, Q. Wei, and X. J. Liu, Ultra-sparse metasurface for high reflection of low-frequency sound based on artificial Mie resonances, Nat. Mater. 14, 1013 (2015).
  * Liang and Li [2012] Z. Liang and J. Li, Extreme Acoustic Metamaterial by Coiling Up Space, Phys. Rev. Lett. 108, 114301 (2012).
  * Melnikov  _et al._ [2019] A. Melnikov, Y. K. Chiang, L. Quan, S. Oberst, A. Alù, S. Marburg, and D. Powell, Acoustic meta-atom with experimentally verified maximum Willis coupling, Nat. Commun. 10, 1 (2019).
  * Tsimokha  _et al._ [2022b] M. Tsimokha, V. Igoshin, A. Nikitina, I. Toftul, K. Frizyuk, and M. Petrov, Acoustic resonators: Symmetry classification and multipolar content of the eigenmodes, Phys. Rev. B 105, 165311 (2022b).
  * Landau and Lifshitz [1977b] L. D. Landau and E. M. Lifshitz,  _Quantum Mechanics_ (Pergamon, Oxford, England, UK, 1977).
  * Bib [2021] Character table for point group D$\infty$h (2021), [Online; accessed 21. Mar. 2021].
  * Bib [2001] Non-Crystallographic: Dinfh (2001), [Online; accessed 21. Mar. 2021].
  * Quan  _et al._ [2018] L. Quan, Y. Ra’di, D. L. Sounas, and A. Alù, Maximum Willis Coupling in Acoustic Scatterers, Phys. Rev. Lett. 120, 254301 (2018).
  * Sieck  _et al._ [2017] C. F. Sieck, A. Alù, and M. R. Haberman, Origins of Willis coupling and acoustic bianisotropy in acoustic metamaterials through source-driven homogenization, Phys. Rev. B 96, 104303 (2017).
  * Willis [1985] J. R. Willis, The nonlocal influence of density variations in a composite, Int. J. Solids Struct. 21, 805 (1985).
  * Cao and Wiersig [2015b] H. Cao and J. Wiersig, Dielectric microcavities: Model systems for wave chaos and non-Hermitian physics, Rev. Mod. Phys. 87, 61 (2015b).
  * Wiersig [2016] J. Wiersig, Sensors operating at exceptional points: General theory, Phys. Rev. A 93, 033809 (2016).
  * Maksimov  _et al._ [2015] D. N. Maksimov, A. F. Sadreev, A. A. Lyapina, and A. S. Pilipchuk, Coupled mode theory for acoustic resonators, Wave Motion 56, 52 (2015).
  * Miroliubov  _et al._ [2021] M. A. Miroliubov, A. K. Samusev, I. D. Toftul, and M. I. Petrov, Spectral Characteristics and Time Dynamics of Tunable Acoustic Resonators in the Strong Coupling Regime, JETP Lett. 113, 547 (2021).
  * Doost  _et al._ [2014] M. B. Doost, W. Langbein, and E. A. Muljarov, Resonant-state expansion applied to three-dimensional open optical systems, Phys. Rev. A 90, 013834 (2014).
  * Sauvan  _et al._ [2022] C. Sauvan, T. Wu, R. Zarouf, E. A. Muljarov, and P. Lalanne, Normalization, orthogonality, and completeness of quasinormal modes of open systems: the case of electromagnetism [Invited], Opt. Express 30, 6846 (2022).
  * Yan  _et al._ [2018] W. Yan, R. Faggiani, and P. Lalanne, Rigorous modal analysis of plasmonic nanoresonators, Phys. Rev. B 97, 205422 (2018).
  * Bogdanov  _et al._ [2019] A. A. Bogdanov, K. L. Koshelev, P. V. Kapitanova, M. V. Rybin, S. A. Gladyshev, Z. F. Sadrieva, K. B. Samusev, Y. S. Kivshar, and M. F. Limonov, Bound states in the continuum and Fano resonances in the strong mode coupling regime,  _Advanced Photonics, 1(1)_ , Adv. Photonics 1, 016001 (2019).
  * Sólyom [2007] J. Sólyom, Consequences of Symmetries, in  _Fundamentals of the Physics of Solids: Volume I Structure and Dynamics, Chapter 6_ (Springer, Berlin, Germany, 2007) pp. 171–201.


(cvpr) Package cvpr Warning: Incorrect font size specified - CVPR requires
10-point fonts. Please load document class ‘article’ with ‘10pt’ option

# Detecting and Localizing Copy-Move and Image-Splicing Forgery

Aditya Pandey
ap6624
New York University
New York, NY 10003
<EMAIL_ADDRESS>Anshuman Mitra
am11058
New York University
New York, NY 10003
<EMAIL_ADDRESS>
###### Abstract

In the world of fake news and deepfakes, there have been an alarmingly large
number of cases of images being tampered with and published in newspapers,
used in court and posted on social media for defamation purposes.

Detecting these tampered images is an important task and one we try to tackle.
In this paper, we focus on the methods to detect if an image has been tampered
with using both Deep Learning and Image transformation methods and comparing
the performances and robustness of each method. We then attempt to identify
the tampered area of the image and predict the corresponding mask. Based on
the results, suggestions and approaches are provided to achieve a more robust
framework to detect and identify the forgeries.
Keywords: Forgery Detection; Image Forgery; Splicing; Copy Move; Deep
Learning; Image Forensics;

##  1 Introduction

In today’s world, every user has access to some form of easy to use image
editing application which has made modifying images extremely easy. While the
human eye is capable of identifying obvious changes or modifications to an
image, it is not difficult to create virtually undetectable manipulations
using software such as Adobe Photoshop, MS Paint, Corel Draw, etc.
[Photoshop]. There are many different types of image forgery/manipulation such
as copy-move forgery, image splicing, cropping, image masking, blurring, etc.
For the purposes of this paper, we would be focusing on the latter two types
of image manipulation - image splicing and copy-move forgery.

Image splicing is the process of modifying an image by pasting a cropped part
of another image and pasting it into the first image. This is a very common
form of forgery and is often followed by some form of image augmentation such
as blurring or smoothing to hide the manipulation. Copy move forgery on the
other hand is when an image is tampered by copying and pasting a small region
from an image into the same image.

An example of image splicing [example1]

An example of copy-move forgery [Copy-Move].

Note how the 5 has been modified in a way that is virtually indistinguishable
to the eye.

As mentioned in the abstract, there are essentially two parts to identifying a
forged image - the first one is to classify an input image as forged/tampered
or not. This is a classification task and is what is called forgery detection.
The second part of the problem statement is to identify the ’forged’ part of
the image i.e. to predict the mask of the forged image which is a black and
white image (output) which describes the area of the image that has been
’tampered’ with.

A tampered image alongside the mask indicating the forged area.

##  2 Related Work

Most of the work done in the image tampering detection space has used image
transformations as the backbone. Images are essentially signals and
identifying patterns of contours, textures, curves, etc. give us a robust
enough feature set to classify images as tampered or authentic. The work done
by Mohammad et. al.[Mohammad], Shilpa et al.[shilpa] and Alahmadi et.
al.[alahmadi] use discrete cosine transform (DCTs) to extract features to
detect forgery. The application of DCT here is to essentially convert an image
into a feature set made of the sinusoids of varying magnitudes and
frequencies. A slightly different approach using Discrete Wavelet Transforms
(DWTs) with Local Binary Patterns was attempted by Mandeep et. al. [Mandeep].
Rao and Ni [rao] presented a convolutional neural network (CNN) based method
to detect forgeries by replacing the first layer weights of the CNN with high
pass filters to capture the features from input images. This feature set is
then fed into an SVM classifier to perform the classification task. On the
deep learning front, Jaiswal et. al. [jaiswal] proposed a Deep learning model
based on ResNet-50 for feature extraction from the CASIA2.0 dataset to extract
features from the input images and feed into different binary classifiers.
This method is devoid of any sort of image transformation or filtering and is
a pure Deep Learning approach to identifying tampered images. The work of
Sudiatmika et. al. [Sudiatmika] uses a VGG16 based model which is trained on
input images after Error Level Analysis (ELA) has been performed

To identify the areas of the image that are tampered with, most approaches use
a deep neural network to perform the task.

Davide et. al. [davide] use noiseprint, a recently proposed CNN-based camera
model fingerprint to localize image forgeries. Yao, Hongwei et al. [yao] use a
reliability fusion map to localize the forgery using fingerprints acquired by
images from different camera models with a CNN. Sri Kalyan et. al. [SriKalyan]
used a generative adversarial network (GAN) based model to learn features of
authentic images followed by SVMs to learn distribution and detect forged
images as anomalies. Convolutional Neural Networks were used by Liu et. al.
[liu] along with segmentation-based analysis to locate tampered areas in
images.

Photo Response Non-Uniformity (PRNU) analysis is one of the most popular
approaches to perform tampering localization. Paweł et. al [pawel] take a
multi-scale fusion approach by combining multiple candidate tamper maps using
sliding windows of varying sizes and extend it by introducing neighbour
interactions between maps. Chakraborty et. al. [Chakraborty] also use PRNU
analysis and treat the localization problem as a probabilistic binary labeling
task.

The UNet architecture created by Ronneberger et. al. [Ronneberger], initially
developed for biomedical image segmentation forms the basis of a number of
recent forgery localization solutions. UNets are popular due to the lack of
camera point information which prevents the application of PRNU based
approaches. Xiuli et. al. [Xiuli] attempt to combine UNets with encoders to
localize Image Splicing Forgery. Ding, Hongwei et. al. [Hongwei] proposed a
method based on dual-channel U-Net aka DCU-Net which consists of three parts:
encoder, feature fusion, and decoder.

##  3 Approach

We approach our problem statement in 2 distinct parts - first being detecting
forgeries i.e. a binary classification task to predict whether an input image
has been tampered with or not. The second part is to predict the mask of an
input image. Here, we are assuming that each input image, whether it is
authentic or tampered will have a mask present.

###  3.1 Detection of Tampered Image

The first approach we take to detect whether an image has been tampered is by
using the properties of JPEG compression. We perform Error Level Analysis
(ELA) which identifies areas in the image that are at different compression
levels. ELA is performed by compressing an image in JPEG format and finding
the differences between the original image and the compressed image. When
performing ELA, areas of the image that are smooth and uniform like sky,
walls, etc will have a lower ELA result.

Using Error Level Analysis identifies parts of the image which were forged as
those areas have a higher ELA values. Using this as our feature set, we train
our classifier models - we use a Support Vector Classifier. Once we have our
featureset, we also use different types of Neural Networks. We start off with
a simple Conv2D-block based CNN with 4 Convolutional blocks. We also try and
use ResNet architecure using Transfer Learning to form a Deep Network.
Finally, we experiment with an LSTM model to compare the results of our
approaches. We also test our dataset by performing Cosine Transformations with
Local Binary Patterns similar to [Mandeep] using a SVM and a simple neural
network (non-CNN).

To achieve this, we use the DCT formula (expanded for a 2D block which is
created by performing LBP on multiple sub blocks of the original image)

To test the effect of different image effects, we also compare the performance
of the models on the blurred version of the dataset. Blurring an image removes
some of the noise and distinct features and does indeed affect the performance
of our models.

###  3.2 Identifying Tamper Mask

We also attempt to identify the mask of a tampered image. A mask is
essentially a binary image which is capable of hiding (or selecting) parts of
an image to apply transformations to. In our problem statement, the mask
refers to the part of the image that has been tampered with.

To solve this, we base our experiments on the UNet Architecture proposed by
Ronneberger et. al. [Ronneberger]. We first convert our inputs to a feature
vector using Error Level Analysis before feeding it into our model.

Basic Structure of a UNet Architecture

We experiment with two approaches for our use case - one with using only the
tampered images and it’s corresponding mask [casia2_groundtruth] and the other
in which we incorporate a mask for each authentic image i.e. we create an
empty mask for an authentic image and train our model using the combined
dataset. For the internal architecture of the UNet, we compare the results of
using Conv2D blocks to build our model with a model based on the ResNet-101
architecture proposed by Kaiming et. al. [Kaiming]. Finally we reconstruct our
masks and analyze the results by visual inspection as well as the
corresponding loss values of the models trained on both versions of the
dataset.

##  4 Experiments and Results

###  4.1 Training and Evaluation

We trained our models on a combination of NYU Greene Cluster and Google Colab.
The Colab instances use the Tesla P100 GPU for all our experiments an 25GB of
RAM. To try and replicate the memory bottleneck (if any), our Greene Cluster
uses an NVIDIA V100 GPU with 32GB of RAM. We used Keras to create our model
with a TensorFlow backend.

For the Forgery Detection, we use the Accuracy, F-Score for each class
(Tampered/ Not Tampered) and Weighted average to compare the results and
performances of each model. Being a binary classification task, we perform
quantitative comparision between different approaches.

For the identification of the mask of a tampered image, a combination of loss
as well as qualitative comparision is used to compare the approaches. Each
model minimizes the validation loss (where the validation data is 0.2 of the
dataset). We resize all images and masks to dimensions 128X128 which is the
input size for our UNet.

###  4.2 Dataset

For the purposes of this paper, we will use the CASIA2.0 dataset [casia2]
introduced by Jing Dong et al. with its corresponding ground truth masks
[casia2_groundtruth] by Nam Thanh et al. The dataset contains 7408 authentic
images and 5123 tampered images out of which 3295 are of copy-move type and
1828 are spliced images. Each tampered image has a corresponding binary mask
indicating the area of tamper. For training purposes, we use the created masks
for authentic images (which is just an empty 2D array of 0s). Further
preprocessing steps include testing the model with black and white images, by
introducing ablations like blurred images and generating flipped and rotated
samples to increase the training size. For the forgery detection part, we also
scale the dataset in the process of performing Error Level Analysis. The
output of the DCT transform is also reduced to 0-1 scale to perform effective
SVC and NNet classification.

Table 1: Detecting Forged Images Model | Accuracy | F-Score 0 | F-Score 1 | Weighted Score
---|---|---|---|---
Digital Cosine Transform | 0.92 | 0.93 | 0.88 | 0.91
with SVC |  |  |  |
Digital Cosine Transform | 0.93 | 0.94 | 0.91 | 0.93
with Simple NNet |  |  |  |
ELA + Basic CNN | 0.88 | 0.92 | 0.76 | 0.86
ELA + ResNet CNN | 0.89 | 0.92 | 0.80 | 0.88
ELA + Basic CNN | 0.91 | 0.94 | 0.83 | 0.90
with BatchNorm |  |  |  |

###  4.3 Forgery Detection

As we can see from the results in Table 1, the Digital Cosine Transform
Feature Extraction combined with a simple neural network model gives us the
best results (in terms of accuracy as well as the Recall and Precision for
individual classes)

Where the results get interesting is detecting forgeries in Images that have
been processed further after the initial tamper. Blurring an image blends the
tampered bit into the original image. As we are dealing with Copy-Move and
Image-Splicing forgery, this blur would affect the frequency and general
properties of the image as the blurring operation sets each pixel as the
average of surrounding pixels.

As we can see from the results table, The Digital Cosine Transform comes up as
the best result out of all our approaches. Combined with a Basic Neural
network, we are able to achieve a robust F-Score for each class. Error Level
Analysis also gives us comparable results when used with a CNN which has batch
norm but favours the class 0. The story is even more distinct when working
with input images that have been augmented with shearing or blurred. The DCT
based models are still the best performing models although there is a loss in
accuracy here. The models based on CNNs perform much worse here - all but
failing to identify the tampered images which have been bblurred (identifying
the shear transform images to some extent).

###  4.4 Localization

While attempting to create a localization model, we use two approaches based
on UNETs. The first one takes only the forged images as input and tried to
predict the mask based on the calculated input feature map. As we can see from
the results, while there is some mask identified by the model, on further
visual inspection the performance of this model is not upto par.

Approach 1: The original pask followed by the predicted mask (output+binary
output)

The next approach involved passing two input images i.e. the original image
along with the ELA-transformed image. The output/target here is still the
corresponding mask of the image. The model here also uses a ResNet101 backend
as the base model. As we can see from the examples as well as the outputs of
input test images, there is a clear improvement in the mask prediction as
compared to the first attempt.

Approach 2: The original pask followed by the predicted mask (output+binary
output)

Loss Curve for CNN Based UNet

Loss Curve for ResNet Based UNet

(Orange is the validation loss and blue is the train loss)

Table 2: Detecting Forged Images with Augmentation, Ablations

Model | CNN UNet | ResNet UNet
---|---|---
Validation Loss | 0.156 | 0.26
Training Loss | 0.162 | 0.21
Time Per Epoch | 93s | 161s
Convergence Epoch | 17 |  $>$25

Table 3: Detecting Forged Images with Augmentation/Ablations Model | Accuracy | F-Score 0 | F-Score 1 | Weighted Score
---|---|---|---|---
ELA + Basic CNN |  |  |  |
\+ BatchNorm + Augmentation | 0.62 | 0.71 | 0.45 | 0.65
(Shear and Rotation) |  |  |  |
ELA + Basic CNN | 0.80 | 0.89 | 0.11 | 0.72
\+ Blur |  |  |  |
ELA + Basic CNN | 0.77 | 0.88 | 0.06 | 0.69
\+ Reduce Channels |  |  |  |
Digital Cosine Transform | 0.87 | 0.89 | 0.86 | 0.87
with SVC |  |  |  |
Digital Cosine Transform | 0.90 | 0.90 | 0.88 | 0.90
with Simple NNet |  |  |  |
\+ Blur |  |  |  |

##  5 Conclusion

This work demonstrates the effectiveness of using Deep Learning/Machine
learning models along with image transformations in identifying forgery in
images. As seen from the results, the Digital Cosine Transformation is an
extremely powerful tool in identifying images where there is an anomaly in the
frequency in the image.

We can see that introducing filters and some forms of augmentations make it
more difficult to identify tampering in images and aligns with the fact that
blurring and modifying tampered images which remain an effective way of hiding
forgeries.

On the front of identifying areas of the image that have been tampered, UNET
based models are effective in identifying pars of the tamper and output masks.
Performing some form of analysis on the raw image (ELA or DCT) remains an
effective step in making the model more robust.

##  6 Future Work

We can see that our DCT model while being relatively robust to blurring, other
ones do perform poorly when the number of added embellishments are high.
Performing further signal pre-processing on the images (such as Digital Wave
Transform, Fourier Transform, etc.) might result in an even more robust model.
The work can also be expanded to include forgeries outside of the copy-move
and image-splice types.

On the localization front, increasing our data size by introducing more
tampered images and their masks could increase the performance of the model.
We should also study the affect of introducing more information based on the
frequency and distribution of images into out UNet model which might increase
the performance of the same (this could be an interesting approach looking at
the performance of using Transforms in the detection task).