]Indo-Korea Science and Technology Center (IKST), New Airport Road, Yelahanka,
Bangalore, India- 560065.

# Inducing half metallicity with alloying in Heusler Compound CoFeMnSb

Upendra Kumar, P. V. Sreenivasa Reddy, Satadeep Bhattacharjee, Seung-Cheol Lee
[<EMAIL_ADDRESS>
(August 27, 2024)

###### Abstract

First principles studies were performed in order to find out the possibility
of inducing half-metallicity in Heusler Compound CoFeMnSb, by means of
alloying it with 3d-transition metal elements. Proper alloying element is
selected through the calculations of formation energies. These calculations
were tested with different concentrations of alloying elements at different
atomic sites. Among the selected transition metal elements Sc and Ti are
proposed to be excellent alloying elements particularly at Mn site. By using
these alloying elements complete half metallic behaviour is obtained in
$\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$, $\mathrm{CoFeMn_{0.75}Ti_{0.25}Sb}$,
$\mathrm{CoFeMn_{0.625}Ti_{0.375}Sb}$, $\mathrm{CoFeMn_{0.50}Ti_{0.50}Sb}$,
$\mathrm{CoFeMn_{0.25}Ti_{0.75}Sb}$ and CoFeTiSb alloys. Shifting of Co-Fe
d-states towards lower energy region leads to zero density of states at Fermi
level for the spin minority channel. Alloying effects on the electronic
structure and magnetization are discussed in details. Thermodynamical
stability of these new alloys are major part of this study. The Curie
temperatures of $\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$ and
$\mathrm{CoFeMn_{0.75}Ti_{0.25}Sb}$ were found to be 324.5 K and 682 K;
respectively, showing good candidature for spintronics applications. For
understanding the bonding nature of constituent atom of CoFeMnSb, crystal
orbital Hamiltonian populations have been analysed.

Keywords: CoFeMnSb Heusler Compound, Density Functional Theory, Curie
Temperature, Alloying effect

PACS numbers: 31.15.E, 75.50.Cc, 75.30.Et, 71.15.Mb

##  I  Introduction

The Heusler class of materials are well known for their plenty of applications
ranging from spintronics to thermoelectric and superconductors to topological
insulators due to their wide range of material properties that can be
engineered. These Heusler compounds seem to be the material of choice for many
applications because of their tunable electronic structure properties. This
helps in the design of materials with desirable properties from half-metallic
ferromagnet Kubler ; Felser over completely ferrimagnet Wurmehl to non-
magnetic semiconductors Jung ; Pierre . In the present days, the search of new
materials for spintronics applications was growing within the Heusler class of
compounds which was started before 30 years Kubler . These materials have
passionate applications in spintronics Groot ; Felser1 ; Felser2 ,
optoelectronics Kieven , as shape memory Christian alloys and as
superconductors Wernick ; Winterlik . Over the past few decades, new areas of
applications emerged comprising the environmental technologies such as
thermoelectric Sakurada ; Uher and solar cell applications Kieven . Recently,
a new trend in the material properties which belong to new quantum state of
matter is observed Chadov ; Lin . These materials are named as topological
insulators and are having technological importance due to the conduction of
electrons being extremely high because of its surface states and might attract
application in high performance electronic systems. This could be one of the
future research perspective of Heusler compounds enabling them to be more
productive.

Ternary intermetallic Heusler compounds can be represented with general
formula XYZ (known as Half-Heusler) or X2YZ, where X and Y elements generally
belong to transition metals (having $d$ states) and Z belongs to main group
element. In 1903 Fritz Heusler Heusler ; Heusler1 discovered these class of
materials. Important feature of these compounds is, the properties of the
total system is completely different from the properties of the individual
elements contained in this type of compounds. For example, if we consider the
compound Cu2MnAl, which is reported as magnetic none of its constitutional
elements possess magnetic order independently. Similarly, in another compound
TiNiSn, which is reported to have semiconducting nature all its constitutional
elements posses metallic character Aliev . The count on the number of these
type of compounds is endless and can be prepared by mixing all the elements in
the periodic table and leading to several applications and need to be
explored.

By doping in Heusler compounds, the Fermi level can be adjusted and half
metalicity can be increase as in the case of Co2MnGe1-xGa1-x Wu . The co-
doping effect of Co+Si on nearly half-metallic Heusler compound NiFeMnSn has
been seen and shown that by doping it becomes perfectly half-metallic Feng .
The more $d$-electrons doping effects has been seen in Ti2CoAl by Chen et al.
Chen , it was shown how V/Nb doping is more energetically favourable on
different site of Ti. Effect of Fe substitution by Co on off-stoichiometric
Ni-Fe-Co-Mn-Sn Heusler alloy ribbons has been seen and it was predicted that
Fe substitution by Co creates change in the magnetic behavior Mishra .

Considerable attention had paid in the quaternary Heusler compounds like
CoFeMnAl, CoFeMnSi, CoFeMnGa, CoFeMnGe, CoFeScSb, CoFeTiSb, NiFeMnGa and
NiCoMnGa. There are two types of half-metallic materials: the complete half-
metals with 100% spin polarization at the Fermi-level and the near half-metals
where there is some small density of states at the Fermi level so that the
spin polarization is less than 100% Kundu . The compounds like CoFeMnSi,
CoFeMnAs and CoFeTiSb Berri are belongs to the 100% spin polarization
category. But CoFeMnSb have only 85% spin polarization Elahmar and CoFeScSb is
reported to be nearly half metallic Gao . It is therefore worth to investigate
the possibility of 100% spin polarization in CoFeMnSb by means of fractional
alloying. The study of mechanical, structural, electronic, magnetic and
thermoelectric properties of CoFeMnSb ternary Heusler alloy has already been
done by spin polarized density functional and semiclassical Boltzmann
transport theory Lee . It was found, CoFeMnSb is more thermodynamically stable
in ferromagnetic phase Lee .

Calculation of formation energy plays a key role in identifying the
thermodynamic stability of a predicted or alloyed compound. The negative value
of formation energy indicates that at zero temperature the compound is more
stable than its constituent elements. By using the formation energy criteria
Ma et al. discovered 45 half-metals and 34 nearly half metals by identifying
the negative formation energies ma2017computational . Understanding the nature
of bonding is another important task, which can be performed by analysing the
Crystal Orbital Hamilton Population (COHP) scheme Dronskowski in which
partitioning of the energies allows to extract chemical bindings between the
atoms from the electronic structure calculations. This can be done with
LOBSTER package Stefan Maintz which is, in general, used to extract chemical
bonding from plane-wave based density functional theory.

In this article, we have theoretically investigated the alloying element
effect of transition metal (TM) (TM= Sc, Ti, V, Cr, Ni, Cu, Zn) on CoFeMnSb
ternary Heusler alloy by using density functional theory (DFT) and see how
half metalicity of CoFeMnSb changes with alloying element. Thermodynamic
stability of these new alloys are major part of our study. It was found Sc and
Ti alloying is more effective and change nearly nearly half-metallic nature of
CoFeMnSb to completely half-metallic. The paper is organized as follows:
section II presents computational details of the first-principles
calculations. The results and discussions of bulk ground state, electronic
structure, formation energy calculations and electronic structure details of
alloyed CoFeMnSb are presented in section III. Finally, conclusions are
presented in the section IV.

##  II  Computational details

All the ground state and electronic structure calculations are performed by
using Vienna ab-initio simulation package (VASP) vasp ; vasp1 . Firstly,
structural optimizations for both bulk and alloyed structures were performed
to obtain the minimal energy structure using the Perdew-Burke-Ernzerhof (PBE)
functional PBE , which is specifically customized for solids, that has been
shown to yield structural data in agreement with experiment. The PBE
pseudopotential was used with an energy cutoff of 600 eV. The Brillouin zone
has been sampled using a 8$\times$8$\times$8 $\Gamma$-centred grid. The
criteria for energy convergence between the two successive self consistent
loop is taken $10^{-6}$ eV/cell. We have used tetrahedron method with Blöchl
correction Bloch , along with 8$\times$8$\times$8 K-point mesh for the density
of states calculations. There is fully relaxation of atomic structures and
volume by using convergence gradient algorithm. Forces are converged to 0.01
eV/$\AA$. For study of alloying in CoFeMnSb, a super cell of
2$\times$2$\times$2 has been designed from the primitive cell which contains 8
atoms of each kind. In next section, we have discussed the effects of TM
alloying in CoFeMnSb on Co, Fe, Mn sites and how these alloying elements
affect half- metalicity and magnetism.

##  III  Results and Discussion

Present compound CoFeMnSb crystallizes in face centred cubic type with space
group $F\overline{4}3m$ (216) and the unit cell occupied with the Wyckoff
positions: Co (0 0 0), Fe (0.5 0.5 0.5), Mn (0.25 0.25 0.25) and Sb (0.75 0.75
0.75). We have created a 2$\times$2$\times$2 supercell which is consisting of
total 32 atoms. To start with, we performed complete structural optimization
for the supercell created in spin polarization case and the calculated
equilibrium lattice constant is found to be 8.44$\AA$. The calculated total
magnetic moment of the system is observed as 5.005$\mu$B per primitive cell.
The atom dependent magnetic moments are observed as 0.894$\mu$B for Co,
1.071$\mu$B for Fe, 3.077$\mu$B for Mn and -0.037$\mu$B for Sb. From this it
is observed that Mn, Fe and Co elements are contributing positively to the
total magnetic moment while Sb is providing negative contribution. It is also
observed that the primary contribution to the total magnetic moment is arrived
from the Mn and the secondary contribution is arriving from Fe and Co sites.

To know more about the electronic structure details, we have calculated total
and atom projected electronic density of states and are plotted in Fig.1.
Multiple peaks are observed in the total density of states both in majority
and minority spin cases. The peak around -4.6 eV in majority spin and -4 eV in
minority spin are due to the hybridization of all elements. Particularly at -4
eV it is due to the Sb-$p$ states. The peaks around -3.0 eV to -2.5 eV and
-1.4 eV to -1 eV are mainly due to Mn atoms in the majority spin case. The
states in the energy region form -1.0 eV to 0 eV in both majority and minority
are highly dominated by Fe atom. For Mn atom the split in the $d$ states is
more in compare to Fe and Co, which causes for the high magnetic moment in Mn
in compare to Fe and Co in the present system. From the overall plot the
system is found to nearly half metallic with large density of states in
majority spin case and very small value of density of states in minority spin
case at Fermi level (EF). So, it is quite informative to discuss about the
system near the Fermi level. From the inset of Fig.1, at EF, the decreasing
trend in the density of states observed from the contribution of Fe, Mn, Co
and Sb atoms respectively in majority spin case but in the minority spin case
we have observed a small value of density of states which are arising mainly
from Co and Fe atoms (indicated with red color circle). As the present
compound is almost having a half metallic nature it is worth to discuss about
the spin polarization, $P$, in the present compound. The well known formula to
calculate $P$ is
$P=((N(E_{F})\uparrow-N(E_{F})\downarrow)/(N(E_{F})\uparrow+N(E_{F})\downarrow))\times
100$%, where $N(E_{F})\uparrow$ is total density of states at EF in majority
spin case and $N(E_{F})\downarrow$ is total density of states at EF in
minority spin case. The calculated value of $N(E_{F})\uparrow$ is 13.69
states/eV and $N(E_{F})\downarrow$ is 1.229 states/eV. From this, the
estimated $P$ is 83.52% which is in good agreement with other studies (85%)
Elahmar . As the present compound is found to nearly half metallic, it could
be more useful if we can tune the Fermi level to get complete half metallic
nature with 100% spin polarization. The best way to tune the Fermi level is
fractional alloying with TM elements in the periodic table.

###  III.1  Proper selection of alloying element from the calculation of
formation energies

In order to know the possibility to synthesize a compound, calculation of
formation energy ($\Delta E_{f}$) is the best way. The formation energy can be
written as, $\Delta E_{f}=E^{{}^{\prime}}-E-\sum_{i}n_{i}\mu_{i}$, where
$E^{{}^{\prime}}$ is the total energy of the supercell containing the impurity
atom, $E$ represents the total energy of the perfect crystal using an
equivalent supercell, integer $n_{i}$ is the number of alloying element atoms,
which has been subtracted (negative value) or added (positive value) to form
the disorders and $\mu_{i}$ represents the chemical potential of these
species. The calculated value of $\Delta E_{f}$ for bulk CoFeMnSb is found to
be -0.68 eV/f.u. So, the alloyed compound should have the lower values of
$\Delta E_{f}$ in compare to bulk.

Selection of a proper alloying element is always a difficult task. It includes
lot of trails with different alloying elements along with different
concentrations at different atomic sites. In the present case, we have
selected several TM elements as alloying elements to alloy at Co, Fe and Mn
sites. The selected TM elements are Sc, Ti, V, Cr, Ni, Cu and Zn. We have
tested these alloying elements at Co, Fe and Mn sites with different alloying
element concentrations, $x$= 0.125, 0.25, 0.375, 0.50, 0.75 and 1.00 to check
the site preference for a particular alloying element. The calculated
formation energy values are plotted in Fig.2. From this it is observed that Sc
alloying element has lower formation energies in compare to other elements
with all alloying concentrations and all Co, Fe and Mn site preferences which
confirms that Sc could be the best alloying element among the chosen elements.
Along with Sc, Ti could be the second possible alloying element at Mn site
with all alloying concentrations. These results are highlighted with a
rectangular red coloured box for all $x$ concentrations at Mn site alloying.
It is also observed that, as the alloying elements concentration $x$ increases
from 0.125 to 0.375, the formation energy of CoFe(Mn1-xTMx)Sb is approaching
the bulk formation energy value which is indicated with red coloured dotted
line. At $x$= 0.500 alloying concentration of Sc, the formation energy of
$\mathrm{CoFeMn_{0.50}TM_{0.50}Sb}$ is lowering than the bulk indicating best
choice of alloying concentration of Sc in the formation of alloy. As $x$
concentration value increases further 0.75 and 1.00 both Sc and Ti alloyed
CoFe(Mn1-xTMx)Sb having lower $\Delta E_{f}$ values in compare to bulk
indicating Ti will be the second possible element to synthesize
CoFe(Mn1-xTMx)Sb alloy after Sc.

###  III.2  Alloyant effect on electronic structure and magnetic nature

To investigate the effect of alloying on the electronic structure of CoFeMnSb,
we have calculated total electronic density of states for $x$= 0.125, 0.250,
0.375 and 0.500 concentrations alloyed with TM and are plotted in Fig.S1. In
all the cases, majority spin states of CoFeMnSb in the presence various
alloying elements has been stride over the Fermi level, describing typical
metallic nature. On the other hand, the minority spin states, gradually
decreases and becomes zero, describing half metallic nature of CoFeMnSb in
presence various alloying elements along with alloying elements percentage.
From Fig.S1, half metallic nature is observed in (Co1-xScx)FeMnSb with
$x$=0.125, 0.25, 0.375 and 0.50, in Co(Fe1-xScx)MnSb with $x$= 0.125, 0.25 and
0.375, in $\mathrm{Co_{0.875}Ti_{0.125}FeMnSb}$, CoFe(Mn1-xScx)Sb with $x$=
0.125, 0.250, 0.375 and 0.500 is almost half metallic, in
$\mathrm{Co_{0.875}Ti_{0.125}FeMnSb}$, in CoFe(Mn1-xTix)Sb with $x$= 0.250,
0.375 and 0.500, in CoFe(Mn1-xVx)Sb with $x$= 0.125, 0.250, 0.375 and 0.500,
in Co(Fe1-xCrx)MnSb with $x$= 0.250, 0.375 and 0.500, in
$\mathrm{CoFeMn_{0.50}Cr_{0.50}Sb}$, in (Co1-xNix)FeMnSb with $x$= 0.125,
0.250, 0.375 and 0.500, in $\mathrm{CoFe_{0.875}Ni_{0.125}MnSb}$, in
CoFe(Mn1-xNix)Sb with $x$= 0.125 and 0.250, in
$\mathrm{Co_{0.875}Cu_{0.125}FeMnSb}$, in Co(Fe1-xCux)MnSb with $x$= 0.375 and
0.500, in $\mathrm{CoFeMn_{0.875}Cu_{0.125}Sb}$, in
$\mathrm{Co_{0.875}Zn_{0.125}FeMnSb}$, in Co(Fe1-xZnx)MnSb with $x$= 0.125,
0.250, 0.375 and 0.500.

Even though we have observed half metallic nature with different alloying
concentrations, their formation energies are not lower than the bulk one. From
the previous subsection we have found only Sc and Ti are the two possible
alloying elements particularly at Mn site. So it is worth to discuss about
only Sc and Ti alloying studies at Mn site. Calculated total density of states
are plotted separately for Sc and Ti alloying elementss only at Mn site with
different alloying concentrations in Fig.3, where we can observed complete
half metalicity in $\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$,
$\mathrm{CoFeMn_{0.75}Ti_{0.25}Sb}$, $\mathrm{CoFeMn_{0.625}Ti_{0.375}Sb}$,
$\mathrm{CoFeMn_{0.50}Ti_{0.50}Sb}$,
$\mathrm{CoFeMn_{0.25}Ti_{0.75}Sb}$ and CoFeTiSb. The calculated band
structures are given in Fig.S2 for bulk, $\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$,
$\mathrm{CoFeMn_{0.50}Ti_{0.50}Sb}$ and CoFeTiSb to know the type of band gap
in minority spin case. In all these cases we have observed a direct gap, where
top of the valence band and bottom of conduction band are on the $\Gamma$ high
symmetry point. This is completely different form the half metallic C1b
Heusler compounds which are normally having an indirect gap. Calculated spin
polarization values for Sc and Ti alloying element at Mn site are given in
Fig.4 along with different alloying concentrations. In Sc alloying at Mn site,
the spin polarization value is found to increase with alloying concentration
and observed a 100% spin polarization at $x$=0.75. At the higher alloying
concentrations of Sc greater than $x$=0.75, spin polarization is found to
decrease. From the literature, CoFeScSb is found to be nearly half-metallic
Gao which is again confirmed from our calculations by complete replace of Mn
with Sc. Here we can conclude one important point is that if we replace 0.25%
of Sc with Mn in CoFeScSb we will get 100% spin polarization. In the case of
Ti alloying at same Mn site, 100% spin polarization is observed at $x$= 0.25
and above concentrations. From the literature CoFeTiSb is found to have
complete half-metallic Berri which is again confirmed from our calculations.

To know the effect of magnetic nature with alloying, we have calculated total
and atom dependent magnetic moments for both Sc and Ti alloying elementss with
different concentrations and are tabulated in Table.1. It is observed that
total and atom dependent magnetic moments are found to be decreased with
increasing alloying concentration in the case of Sc at Mn site. The same
scenario is observed with total, Sb and alloying element magnetic moments with
Ti at Mn site but Co, Fe and Mn moments are found to increased a little with
increasing Ti alloying concentration at Mn site.

Change in the electronic structure with alloying can be understand through the
electronic density of states which are plotted in Fig.5 for Sc with 75%
alloying, Ti with 50% and 100% alloying along with bulk states. In all cases,
the total density of states in minority spin case are found to be shifted
towards to the lower energies which causes a gap at the Fermi level in
minority spin. The states near EF in minority spin case mainly belongs to $d$
states of Co and Fe in bulk. These states are found to be shifted towards the
lower energies and cause a 100% spin polarization with alloying. It is also
observed reduction in the height of peaks in Mn $d$ states after alloying with
Sc or Ti. After alloying, the splitting of Sc and Ti $d$ states is very less
as compare to Mn in bulk which causes a decreasing trend in the total magnetic
moment with alloying compare to bulk value.

To obtain the Curie temperature, we have used the SPR-KKR-method SPR-KKR , by
which we have calculated the inter-atomic exchange constants. The Curie
temperatures were calculated within the mean field approximation. It was found
CoFeMnSb has Curie temperature 923.1 K, which decreases by including alloying
concentration at Mn site (described in Table 1). CoFeMnSb and CoFeScSb are not
a perfect half metal, have spin polarization 83.52% and 92% respectively. But
$\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$ has complete half metallic nature and
Curie temperature 324.5 K i.e. more than room temperature. So
$\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$ is a perfect Heusler compound for
spintronics application. The Curie temperature of
$\mathrm{CoFeMn_{0.75}Ti_{0.25}Sb}$ is found 682 K, also has a possibility in
spintronics application. This shows that we can engineer half-metallic
compounds by alloying CoFeMnSb with Ti.

Above discussions intimating the importance of Co and Fe $d$ states, which are
highly effected with alloying and reason for half metallic nature after
alloying. So, it is very important to understand the bonding nature in between
Co and Fe, and how it is effected with alloying. For this we have computed Co-
Fe crystal orbital Hamiltonian populations(COHPs) Dronskowski . COHP is the
density of states weighted by the corresponding Hamiltonian matrix element.
COHPs indicate the strength as well as the nature of interaction with the
bonding or antibonding interactions. These interactions being indicated by
positive or negative values of COHP. Co-Fe COHPs are plotted in Fig.6 for
bulk, $\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$, $\mathrm{CoFeMn_{0.50}Ti_{0.50}Sb}$
and CoFeTiSb. It is observed that bonding states in both spin cases at
energies below around -1 eV. From the energy range -1 eV to 0 eV we have
antibonding states in majority spin case for both bulk and new alloys. In the
same energy range, the scenario is different in minority spin case, where we
have observed bonding states in bulk and Ti alloying but in Sc alloying we
have antibonding nature. At the EF, we have small electronic density of states
in minority spin case without alloying are belongs to Co and Fe. The inset of
the figure indicates that these states are having bonding nature. This bonding
nature still survived with Ti alloying but it changes to antibonding nature
with Sc alloying.

##  IV  Conclusions

We have investigated the TM element alloying effect on the electronic
structure and magnetic properties of nearly half metallic CoFeMnSb by using
first principles calculations within density functional theory. Calculations
on the relative formation energy indicates Sc and Ti mono alloying is more
favourable. The same calculations confirms the best site preference to alloy
with Sc and Ti is at Mn-site. Therefore, the mono alloyed CoFe(Mn1-xScx)Sb and
CoFe(Mn1-xTix)Sb alloys could be easily obtained. The calculated electronic
structures reveals 100% spin polarization when alloying concentration, $x$,
reaching to $x$=0.75 with Sc and $x$= 0.25, 0.375, 0.50, 0.75, 1.0 with Ti at
Mn site. $\mathrm{CoFeMn_{0.25}Sc_{0.75}Sb}$ and
$\mathrm{CoFeMn_{0.75}Ti_{0.25}Sb}$ are perfectly half metallic with 100% spin
polarization, have Curie temperature 324.5 K and 682 K, respectively.
Therefore, both become good candidature in spintronics application. Density of
states and COHP calculations confirms that both Co and Fe states are shifted
towards to lower energy region and causes to open a gap at EF in the minority
states. This leads to direct band gap in the minority spin states in these
materials.

## References

  * (1) J. Kübler, A. R. Williams and C. B. Sommers, Phys. Rev. B 28 1745-1755 (1983) .
  * (2) C. Felser, G. H. Fecher. and B. Balke, Angew. Chem. Int. Ed. 46 668-699 (2007).
  * (3) S. Wurmehl, H. C. Kandpal, G. H. Fecher and C. Felser, J. Phys.: Condens. Matter 18 6171-6181 (2006).
  * (4) D. Jung, H.-J. Koo and M.-H. Whangbo, J. Mol. Struct. (Theochem) 527 113-119 (2000).
  * (5) J. Pierre, R. V. Skolozdra, J. Tobola, S. Kaprzyk, C. Hordequin, M. A. Kouacou, I. Karla, R. Currat and E. Lelievre-Berna, J. Alloys. Comp. 262-263 101-107 (1997) .
  * (6) R. A. De Groot, F. M. Mueller, P. G. Van Engen and K. H. J. Buschow, Phys. Rev.Lett. 50 2024 (1983).
  * (7) C. Felser, B. Heitkamp, F. Kronast, D. Schmitz, St Cramm, H. A. Dürr, H. J. Elmers, G. H. Fecher, S. Wurmehl, T. Block, D. Valdaitsev, S. A. Nepijko, A. Gloskovskii, G. Jakob, G. Schönhense and W. Eberhardt, J. Phys.: Condens.Matter 15 7019-7027 (2003).
  * (8) C. Felser, Gerhard H. Fecher and Benjamin Balke, Angew. Chem. Int. Ed. 46 668-699 (2007).
  * (9) David Kieven, Reiner Klenk, Shahab Naghavi, Claudia Felser and Thomas Gruhn, Phys. Rev. B 81 075208 (2010).
  * (10) Christian G. F. Blum, Siham Ouardi, Gerhard H. Fecher, Benjamin Balke, Xeniya Kozina, Gregory Stryganyuk, Shigenori Ueda, Keisuke Kobayashi, Claudia Felser, Sabine Wurmehl and Bernd Büchner, Appl. Phys. Lett. 98 252501 (2011).
  * (11) J. H. Wernick, G. W. Hull, T. H. Geballe, J. E. Bernadini and J. V. Waszczak, Mater. Lett. 2 90-92 (1983).
  * (12) J. Winterlik, G. H. Fecher, A. Thomas and C. Felser, Phys. Rev. B 79 064 508-064 517 (2009) .
  * (13) S. Sakurada and N. Shutoh, Appl. Phys. Lett. 86 082105 (2005).
  * (14) C. Uher, J. Yang, S. Hu, D. T. Morelli and G. P. Meisner, Phys. Rev. B 59 8615-8621 (1999).
  * (15) S. Chadov, X. Qi, J. Kübler, G. H. Fecher, C. Felser and S. C. Zhang, Nat. Mat. 9 541-545 (2010).
  * (16) H. Lin, A. Wray, Y. Xia, S. Xu, S. Jia, R. J. Cava, A. Bansil and M. Z. Hasan, Nat. Mat. 9 546549 (2010).
  * (17) F. Heusler, W. Starck and E. Haupt, Verh. d. DPG5, 220-223 (1903).
  * (18) F. Heusler, Verh. d. DPG 5, 219 (1903).
  * (19) F. G. Aliev, N. B. Brandt, V. V. Moshchalkov, V. V. Kozyrkov, R. V. Skolozdra and A. I. Belogorokho, Z. Phys. B-Condens. Matter 75 167-171 (1989).
  * (20) Bo Wu, Hongkuan Yuan, Anlong Kuang, Yu Feng, and Hong Chen, J. of Phys. D: Appl. Phys. 44(40) 405301 (2011).
  * (21) Yu Feng, Xiaohui Xu, Wenxi Cao, and Ting Zhou, Comp. Mat. Sci. 147 251-257 (2018).
  * (22) Ying Chen, Bo Wu, Yu Feng, Hong-Kuan Yuan, and Hong Chen, The European Physical Journal B, 87(1) 24 (2014).
  * (23) S. S Mishra, Semanti Mukhopadhyay, T. P. Yadav, R. M. Yadav, Sruthi Radhakrishnan, R. Vajtai, P. M. Ajayan, N. K. Mukhopadhyay, H. K. Singh, and O. N. Srivastava, Mat. Res. Exp. 4(8) 086507 (2017).
  * (24) Qiang Gao, Lei Li, Gang Lei, Jian-Bo Deng, Xian-Ru Hu, J. Magn. Magn. Mater. 379 288 (2015).
  * (25) Saadi Berri, D. Maouche, M. Ibrir and F. Zerarga, J. Magn. Magn. Mater. 354 65 (2014).
  * (26) Ashis Kundu, Srikrishna Ghosh, Rudra Banerjee, Subhradip Ghosh and Biplab Sanyal, Scientific Reports 7 1803 (2017).
  * (27) M. H. Elahmar, H. Rached, D. Rached, R. Khenata, G. Murtaza, S. Bin Omran and W. K. Ahmed, J. Mag. Magn. Mater. 393 165-174 (2015).
  * (28) Seung-Cheol Lee et al. J. Alloy. Comp. 765 1055-1060 (2018).
  * (29) Jianhua Ma, Vinay I Hegde, Kamaram Munira, Yunkun Xie, Sahar Keshavarz, David T Mildebrath, C Wolverton, Avik W Ghosh, and WH Butler, Physical Review B 95 024411 (2017).
  * (30) H. Ebert et al., The Munich SPR-KKR Package, Version 6.3, (2012).
  * (31) R. Dronskowski and P. E. Blochl, J. Phys. Chem. 97 8617 (1993).
  * (32) Stefan Maintz, Volker L. Deringer, Andrei L. Tchougréeff and Richard Dronskowski. J. Comp. Chem. 37, 1030-1035 (2016)
  * (33) G. Kresse and J. Hafner, Phys. Rev. B 47 558 (1993).
  * (34) G. Kresse and J. Furthmüller, Phys. Rev. B 54 11169 (1996).
  * (35) J. P. Perdew. et. al, Phys. Rev. Lett. 100 136406 (2008).
  * (36) Peter E Blöchl, Ove Jepsen, and Ole Krogh Andersen, Phys. Rev. B, 49(23) 16223 (1994).

Table 1: Magnetic moments and Curie temperature for different alloying
concentrations of Sc and Ti at Mn site.

alloying elements $x$ $\mu$B(Co) $\mu$${}_{B(\mathrm{alloying\;element})}$
$\mu$B(Fe) $\mu$B(Mn) $\mu$B(Sb) $\mu$B(Total) Tc(K) Sc 0.000 0.894 0.0 1.071
3.077 -0.037 5.005 923.1 0.125 0.869 -0.194 1.018 3.054 -0.031 4.504 - 0.250
0.839 -0.185 0.960 3.049 -0.026 4.013 - 0.375 0.815 -0.168 0.889 3.048 -0.020
3.526 - 0.500 0.796 -0.162 0.823 3.032 -0.015 3.039 - 0.750 0.719 -0.153 0.768
2.792 -0.006 2.064 324.5 1.00 0.514 -0.143 0.738 0.00 -0.010 1.099 222.0 Ti
0.00 0.894 0.0 1.071 3.077 -0.037 5.005 923.1 0.125 0.896 -0.362 1.094 3.084
-0.031 4.612 - 0.250 0.898 -0.330 1.110 3.114 -0.025 4.236 682.0 0.375 0.905
-0.289 1.118 3.150 -0.019 3.864 - 0.500 0.908 -0.246 1.137 3.166 -0.012 3.493
- 0.750 0.924 -0.176 1.213 3.211 -0.002 2.805 - 1.00 1.017 -0.203 1.204 0.00
0.009 2.027 -

Figure 1: (a) Total and atom projected electronic density of states for bulk
CoFeMnSb in 2$\times$2$\times$2 supercell case. (b) Orbital projected density
of states for atom.

Figure 2: Calculated formation energy (in eV) with alloying concentration (a)
$x$=0.125, (b) $x$= 0.250, (c) $x$= 0.375, (d) $x$= 0.500, (e) $x$= 0.750 and
(f) $x$= 1.00 at Co, Fe and Mn sites.

Figure 3: Doping effect on electronic structure of CoFeMnSb at Mn site with Sc
and Ti.

Figure 4: Spin polarization for Sc and Ti alloying at Mn site

Figure 5: Total and $d$ orbital projected density for (a) without alloying and
with alloying of 0.75% Sc at Mn site, (b) without alloying and with alloying
of 0.50% Ti at Mn site and (c) without alloying and with alloying of 100% Ti
at Mn site. In all plots black color data indicates without alloying and red
color data indicates with alloying at Mn site.  Figure 6: Crystal orbital
Hamiltonian population analysis for Co-Fe.

## Supplementary Material

---
---
Figure S1: Doping effect on electronic structure of CoFeMnSb at varies
transitional metal sites. In each row left, middle and right plots show TM-
alloyed at Co, Fe and Mn sites in CoFeMnSb respectively.

Figure S2: Band structure of bulk (a) spin up (b) spin dn,
CoFe(Mn0.25Sc0.75)Sb (c) spin up (d) spin dn, CoFe(Mn0.50Ti0.50)Sb (e) spin up
(f) spin dn, CoFeTiSb (g) spin up (h) spin dn.


# RGB-X Classification for Electronics Sorting
††thanks: 1Robotics Institute, Carnegie Mellon University, 5000 Forbes Avenue,
Pittsburgh, USA<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>††thanks: 2Apple Inc.

FNU Abhimanyu1, Tejas Zodage1, Umesh Thillaivasan2, Xinyue Lai1, Rahul
Chakwate1,
Javier Santillan2, Emma Oti2, Ming Zhao2, Ralph Boirum1 Howie Choset1, Matthew
Travers1

###### Abstract

Effectively disassembling and recovering materials from waste electrical and
electronic equipment (WEEE) is a critical step in moving global supply chains
from carbon-intensive, mined materials to recycled and renewable ones.
Conventional recycling processes rely on shredding and sorting waste streams,
but for WEEE, which is comprised of numerous dissimilar materials, we explore
targeted disassembly of numerous objects for improved material recovery. Many
WEEE objects share many key features and therefore can look quite similar, but
their material composition and internal component layout can vary, and thus it
is critical to have an accurate classifier for subsequent disassembly steps
for accurate material separation and recovery. This work introduces RGB-X, a
multi-modal image classification approach, that utilizes key features from
external RGB images with those generated from X-ray images to accurately
classify electronic objects. More specifically, this work develops Iterative
Class Activation Mapping (iCAM), a novel network architecture that explicitly
focuses on the finer-details in the multi-modal feature maps that are needed
for accurate electronic object classification. In order to train a classifier,
electronic objects lack large and well annotated X-ray datasets due to expense
and need of expert guidance. To overcome this issue, we present a novel way of
creating a synthetic dataset using domain randomization applied to the X-ray
domain. The combined RGB-X approach gives us an accuracy of 98.6$\%$ on 10
generations of modern smartphones, which is greater than their individual
accuracies of 89.1$\%$ (RGB) and 97.9$\%$ (X-ray) independently. We provide
experimental results333Experimental work done at Biorobotics Lab, Robotics
Institute, Carnegie Mellon University to corroborate our results.

##  I Introduction

Daisy[1],[2] and Dave[3],[4], Apple’s existing disassembly robots, disassemble
electronic devices and components to enable the recovery of precious materials
like rare earth elements, steel, and tungsten. Diverse object disassembly
involves hundreds of decisions throughout the process, and object
classification is the first major decision in the pipeline. In practice,
precisely classifying electronics objects is challenging as different objects
of similar product line often have subtle visible differences leading to
incorrect classifications that can cause sub-optimal material recovery if
material compositions differ between the actual and predicted class.
Conventional machine vision has to be highly accurate for recycling, however
to classify all desired WEEE objects for disassembly using one line also
requires that the system be capable of handling high variations seen in each
class of multi-class WEEE such as cracks, bends, missing or replaced
components, or other deformations that are difficult to predict and that
inhibit conventional classification methods (e.g., RGB-based methods). These
variations eliminate methods like weighing when numerous classes differ by
only a few grams. Thus, this paper addresses challenges in classifying WEEE as
they have subtle differences on the outside forcing a classifier to focus on
extremely specific details on the outside or look inside for other
distinguishing features.

Figure 1: Our approach fuses feature maps of RGB ${I_{R}}$ and X-ray image
${I_{X}}$ in order to capture both internal and external features of
electronics devices. We develop a novel architecture, $iCAM$ for suitable
feature maps extraction of electronics images and show that the RGB-X fusion
along with $iCAM$ gives classification accuracy as high as 98.8%. Figure 2:
Examples of the training and the testing data for the RGB and the X-ray
images. From left to right: synthetic RGB images (training), synthetic X-ray
images (training), real RGB images (testing), real X-ray images (testing).

In this work, we present a novel attention based iterative strategy, Iterative
Class Activation Mapping (iCAM), to localize and then extract key features of
relevant components from inside and outside of the devices for classification.
This effectively guides the models to focus on the appropriate features for
better decision making.

Also, to leverage both outside and inside features of electronic objects, we
propose RGB and X-ray classification modalities individually and then uniquely
combine them for their classification. Though there has been previous work on
multi-domain classification, to the best of our knowledge, this work is the
first to combine RGB and X-ray domains to perform classification for recycling
purposes. This combination helps find common key features among the internal
and external parts of the objects, making the classification robust to
conditions like lighting, or wear and tear of the device.

The single-mode (RGB mode or the X-ray mode), as well as the joint training
phase, needs an annotated dataset for training. Unlike other areas of image
classification, used electronics lack a well-annotated dataset for training an
image classifier robustly, especially in the X-ray domain. The creation of
annotated X-ray data is tedious and requires expert guidance. Thus, in this
work, we propose a data randomization pipeline to generate synthetic X-ray
data for training. We create synthetic X-ray images by projecting 3D-Computed
Tomography (CT) scans into a 2D space and randomizing the relevant parameters
like orientation, intensity settings, noise and background.

The paper is organized as follows: Section II introduces the prior work
reported for electronic objects’ image classification. Section III provides a
detailed description on the dataset, iterative class activation mapping (iCAM)
architecture and the training strategy. Section IV describes the experimental
results on RGB-X classification. Section V discusses the conclusion and the
future work.

##  II Prior Work

This section discusses the existing WEEE classification methods. It also
discusses the prior work regarding the dataset generation process, fine-
grained image classification and multi-modal classification.

###  II-A Classification for WEEE sorting

WEEE is an existing problem where both classical techniques as well as deep
learning architectures have been used recognize objects. UNU-KEYS [5] is a
classification method that is being used to classify WEEE by using attributes
such as average weight, material compositions and end-of-life characteristics.
This works well for coarse level classification, but does not work well for
similar looking devices which have subtle feature differences. To overcome
that, state of the art deep learning methods have also being used to
accurately classify WEEE items. Standard network architectures like Faster
Squeeze-Net [6] and YOLOv3 [7] were used to sort electronics. The Swedish
company, Refind, in collaboration with the Danish Institute of Technology has
also demonstrated the feasibility of sorting different types of printed
circuit boards, mixed electronic scrap, and batteries by capturing color
images and applying a deep learning architecture [8],[9]. Use cases include
applying deep learning to sort different types of coin cells [10], and
additional imaging inputs such as thermal and X-ray imaging are being explored
for WEEE recognition [11], [12]. While the prevalence of deep learning has
allowed for advances in classification, these methods are still primarily used
for component level classification and they use a single mode for
classification, therefore making models sensitive to real world scenarios like
lighting, noise, and deformations.

Figure 3: Classification network architecture with SP1 as the input image
$I_{0}$. For iteration $i$, an image $I_{i}$ is passed as an input to the
$iCAM$ module. Internally, the $iCAM$ module extracts feature maps ($FM_{i}$)
of image $I_{i}$ and extracts the region of activation (i.e., the region most
important for classification) using the $CAM$ module. Based on the importance
regions, an image $I_{i+1}$ is cropped from $I_{i}$, which is used for the
iteration $i+1$. For the final prediction, we aggregate the feature maps in
the feature aggregator ($FA$) and pass it through a classifier block to
generate a prediction. Subimages (a), (b) and (c) shows the heat map, the most
important region of the image given the heat map, and the cropped image from
the heat map.

###  II-B Domain Randomization

One of the obstacles of using a machine learning algorithm is the lack of
readily available, large, annotated datasets. The machine learning community
has developed the concept of domain randomization (DR) to synthetically
generate large datasets instead of collecting and labeling data manually. DR
aims to add large variability to the synthetic data generation process,
attempting to capture the real-world variations.

Svetozar and Jianhong [13] show a useful analysis of important parameters to
generate synthetic data for classification for RGB images. DR has also been
used in non-classification tasks. Tobin _et. al._ [14] used DR to
synthetically generate images of geometrical shapes for an autonomous robotic
picking task. Additional applications include autonomous drone flight [15],
object detection [16], viewpoint estimation [17], and human pose estimation
[18]. These works demonstrate the application of DR in the RGB domain, but
they do not explore their use in other domains like X-rays.

###  II-C Fine-grained image classification

Fine-grained image classification captures discriminative features of fine-
grained objects and uses those features in image classification. In this
paradigm, a localization sub-network is designed for locating the key regions
of interest followed by a classification network used for recognition.
Previous works on this paradigm depended on adding dense part annotations for
accurate part localization and then using them for image recognition [19],
[20], [21]. Recent works require only image labels for accurate part
localization for training the recognition sub-network [22],[23],[24]. Finally,
additional techniques such as attention mechanisms [25] and multi-stage
strategies [26] perform the joint training of the integrated localization-
classification sub-networks.

###  II-D Multi-domain classification

Multi-domain learning aims to improve the classification performance in
general domains by making full use of the information in each domain. Prior
work has focused on extracting hand-crafted features such as scale-invariant
feature transform (SIFT) from the multi-domain (RGB-D) image to get a combined
image descriptor [27]. Blum et al. proposed an RGB-D descriptor that relies on
a k-means-based feature learning approach[28]. However, the hand-crafted
features are often dataset-specific and require a strong understanding of
domain-specific knowledge [29]. To reduce the dependency on hand-crafted
features, machine learning techniques are explored, and one of the most common
techniques is using network parameter sharing [30].

##  III Approach

This section discusses the X-ray dataset generation process, the network
architecture used for classification, and the joint training strategy used for
combining the X-ray mode with the existing RGB mode for a combined RGB-X
classification. In both these domains, we conduct experiments on 10 classes of
modern smart phones (SP) classes.

###  III-A Datasets

The 10 classes of modern smart phones will be referred to as SP1, SP2, SP3,
SP4, SP5, SP6, SP7, SP8, SP9, SP10 in this work. For the RGB domain, the test
dataset consists of 799 real RGB images. RGB training set includes a
collection of synthetic and real images. The training dataset consists of a
combination of 1000 synthetic and 250 real images per class. We collect RGB
images using a Microsoft Azure Kinect camera mounted over a conveyor belt. We
load the mesh of the phones in Blender [31] and use DR to create synthetic
dataset by varying color, texture, and camera intrinsic and extrinsic
parameters as suggested in [14].

For the entire X-ray domain, the test dataset consists of 749 real X-ray
images and around $\approx$25000 synthetic X-ray images for training. Only
synthetic X-ray images are used for training to demonstrate the efficacy of
domain randomization and because of the lack of enough real X-ray present for
each class. Test images are collected using an X-ray system equipped with
300kV micro focus and 180kV nano focus tubes, and a cesium iodide (CsI)
detector. Both tubes are used to generate 2D and 3D images with resolutions
ranging from 57.5 $\mu$m to 100 $\mu$m. The 2D and 3D images taken with the
300 kV tube use a peak energy of 220 kV with a current ranging from 200 to 300
$\mu$A, while images acquired with the 180 kV tube use 160 kV and 280 $\mu$A.
For training, we utilize DR in order to generate a large scaled labelled
dataset. To generate X-ray training data, we initially generate 3D voxel
models of 2-3 devices per class using off the shelf CT machines. CT scans
consist of a scalar intensity assigned to each voxel based on the structure of
the device. The voxel intensities are mapped to an opacity value and projected
onto a plane using a simulated projector model to render X-ray images. The
voxel intensity to voxel opacity mapping is known as a transfer function. A
variety of X-ray machine settings can be simulated by the shape of the
transfer function as shown in Fig 4. We use VTK library [32] to randomize the
transfer function, and the intrinsic and extrinsic parameters of simulated
projectors.

Figure 4: X-ray Domain Randomization Setup. To generate synthetic X-ray
images, we project CT scans onto a plane, simulating the process of taking an
X-ray image of an electronic object. We show our setup on a simple electronic
device due to privacy reasons tied to the phone image.

Randomizing the transfer function arbitrarily can undesirably produce
completely saturated images by assigning the same value to each pixel. To
avoid this, we seed the randomization with 5 manually selected transfer
function parameters. These manual seeds were selected using the visualization
tool 3D Slicer [33]. We add randomness to these parameters and only select
rendered images that have more than 100 SuperPoint [34] keypoints. Sample
training and testing images from the RGB and X-ray dataset are shown in Fig 2.

###  III-B Network architecture and training strategy

This section describes the fine-grained classification network used for
training the RGB and the X-ray mode, and the training strategy used in the
single training and joint training phases. Let RGB image ($I_{R}$) or X-ray
image ($I_{X}$) be of dimension $H\times W\times C$ in the labeled dataset
available for training the classification model, and $y_{i}$ be the image
label in one-hot encoding - _i.e._ , $y_{i}\in R^{m}$ is a vector of
dimensionality m (the number of classes) with $y_{i}^{j=l}$ = 1 and
$y_{i}^{j\neq l}$ = 0 for the position l denoting the image label. We train
the model using a three-stage approach: first training the 2 stream networks
individually, followed by a joint fine-tuning stage.

####  III-B1 Iterative class activation mapping (iCAM)

As shown in Fig 3, our network has 4 modules: feature extractor module ($E$),
class activation mapping module ($CAM$), feature aggregator module ($FA$), and
classifier module ($C$). The feature extractor module ($E$) and the $CAM$
module are stacked together in an iterative manner followed by the feature
aggregator module and classifier module.

In our method, we use class activation mapping (CAM) [35] using the global
average pooling (GAP) to indicate the discriminative image regions used by
CNNs to identify the respective classes. For a given image
$I\in\mathbb{R}^{H\times W\times C}$, let $FM(x,y)=\sum_{j=1}^{k}fm_{j}(x,y)$
represent the activation of the last convolutional layer at spatial location
$(x,y)$ with $k$ channels. $\theta_{C}^{c}$ denotes the weight corresponding
to class c in the classifier module C. We define $m_{c}$ as the class
activation map for class c as:

$\begin{split}&fm_{0\dots k}(x,y)=f_{\theta_{E}}(I),\\\
&m_{c}(x,y)=\sum_{j=1}^{k}{\theta_{C}^{c}fm_{i}(x,y)}.\end{split}$ (1)

Thus, $m_{c}(x,y)$ is an unnormalized probability distribution directly
indicating the importance of the activation at spatial location $(x,y)$. More
details on CAM can be found in [35].

In our experiments, InceptionNet-v3 [36] up to the Mixed7c layer is chosen as
the feature extraction module with $\theta_{E}$ as a learnable parameter. The
feature map ($FM_{i}$) and the class activation map ($m_{c,i}$), for every
iteration $i$, is calculated using Eq.1. The activation map is upsampled to
the original image size $H\times W\times C$, and then a square region around
the most dominant pixel of the map is selected resulting in image $I_{i+1}$.
The cropped image is then upsampled to the original image dimension $H\times
W\times C$ using bilinear upsampling function in Pytorch. After running the
feature extraction and the CAM module for a pre-defined number of iterations
$n$, $F_{0..n}$ is passed to the feature aggregator module followed by the
classifier module to get the final classification score $S$:

$\begin{split}&FM(x,y)=\sum_{i=1}^{n}{\theta_{FA}FM_{i}(x,y)},\\\
&S=\mbox{softmax}(f_{\theta_{C}}(FM)),\end{split}$ (2)

where $FM$ is the aggregated feature map, and $\theta_{FA}$ and $\theta_{C}$
are the learnable parameters of the feature aggregator and the classifier
module. For our experiments, we choose n=3.

####  III-B2 Training strategy

We proceed by training the iCAM network to minimize the negative log
likelihood, $\mathcal{L}$, of the training data. While training we solve for:

$\theta_{E}^{*},\theta_{FA}^{*},\theta_{C}^{*}=\\\
\mbox{argmin}_{\theta_{E},\theta_{FA},\theta_{C}}\sum_{i=1}^{N}\mathcal{L}(f_{iCAM}(I,y_{i};\theta_{E},\theta_{FA},\theta_{C}),y_{i}),$
(3)

We individually train the $iCAM_{R}$ and $iCAM_{X}$ for $I_{R}$ and $I_{X}$
images on RGB and X-ray datasets. For both the streams, the $\theta_{E}$ is
initialized with InceptionNet-v3 weight pretrained on ImageNet and were
trained for 40 epochs each. The implementation is highly parallelized and
performs full-batch gradient descent using the Stochastic Gradient Descent
[37] optimizer in the Pytorch Autograd library [38], with a batch size of 16
with a learning rate of 0.001. At the end of this individual training stage,
$iCAM_{R}$ and $iCAM_{X}$ have different sets of weights which are later fused
in the joint training stage.

Once both domains are trained individually, we use the parameter sharing
method to combine the information of both the domains as shown in Fig1. During
the joint-training, the softmax activation is discarded and the output from
the classification layers are concatenated. Their individual responses
$FM_{R}$ from $iCAM_{R}$, and $FM_{X}$ from $iCAM_{X}$, are fused and fed
through an additional stream $FS([g_{R},g_{X}])$ with parameters
$\theta_{FS}$. This fusion network again ends in a softmax classification
layer. During the joint-training phase, $\theta_{E,R}$, $\theta_{FA,R}$,
$\theta_{E,X}$ and $\theta_{FA,X}$ are frozen and only $\theta_{C,R}$,
$\theta_{C,X}$ and $\theta_{F}$ are open to training. In this stage of
training, we solve:

$\theta_{C,R}^{*},\theta_{C,X}^{*},\theta_{FS}^{*}=\\\
\mbox{argmin}_{\theta_{C,R},\theta_{C,X},\theta_{FS}}\sum_{i=1}^{N}\mathcal{L}(f_{FS}([g_{R}(I_{R}),g_{X}(I_{X})],y_{i}).$
(4)

During the joint-training phase, only the weights of the fusion layer and the
weights of individual classification layers are optimized, keeping all other
weights from individual mode intact.

##  IV Analysis and Results

###  IV-A Domain randomization

In this set of experiments, we evaluate the effect of domain randomization
parameters on test accuracy. Our current implementation randomizes image white
noise, background color, transfer function parameters, and rendering projector
position and orientation. As shown in Table I in the first experiment, we
introduce noise of increasing variance in the rendered image. We observe that
the best accuracy of $98.1\%$ is obtained when $\sigma=1000$ with varying
background, transfer function, and projector position and orientation. Also,
we see a drop in the accuracy with the drop in the noise variance. The
presence of noise improves convergence and makes training less susceptible to
local minima as shown in [39]. We also observe that fixing the projector
position and orientation affects accuracy the most, lowering it down to only
$31.5\%$. This result emphasizes the use of DR as this projector position
simulates different views of a 3D object, which can not be simulated by simple
2D data augmentation. Another important factor is the transfer function, which
as discussed in Sec. III, simulates different X-ray machines and X-ray machine
settings. Varying the transfer function, generalizes our data over different
machine distributions. Unlike the RGB domain, we avoid pattern randomization
since such variations do not exist in our test data.

###  IV-B iCAM

TABLE I: Effect of DR parameters on accuracy. Current DR has non-fixed
background, non-fixed transfer function, non-fixed simulated projector
position and orientation and $\sigma=1000$ as the variance for the white
noise.

Parameters Status Accuracy Current DR Sec. III 0.981 Noise $\sigma$=10 0.934
$\sigma$=100 0.923 $\sigma$=500 0.957 Background Fixed 0.922 Transfer function
Fixed 0.655 Simulated projector position and orientation Fixed 0.315

iCAM is a fine grained classification method. For the sake of result section,
we show the performance of iCAM on the modern smart phones (SP) classes
mentioned in Section III and also CUB-200[40] dataset. For comparisons, we
evaluate the results with baseline classification networks like VGG-19[41],
ResNet-101 [42], ResNeXt-101[43] and InceptionNet-v3[44]. We also compare it
with the SqueezeExcitation[45] with ResNet101 as the base module. All these
experiments are performed on the X-ray dataset. We use average accuracy, and
inference time as the metrics to compare the results for different networks as
suggested in [46]. We also compare average precision and average recall of all
these network architecture to our method as suggested in [47]. These results
are presented in Table II. Additionally, we compare the iCAM+InceptionNet-v3
to these baseline classification networks on the CUB-200 dataset [40], a
standard fine-grained classification dataset used for bench-marking image
classification algorithms.

Table II shows that the iCAM block when combined with the InceptionNet-v3 base
module outperforms the baseline image classification networks. The high
accuracy value is attributed to the hierarchical approach iCAM takes in order
to extract and combine key features from an image. Intuitively this approach
is similar to looking and extracting key features at different levels of
magnification and then using all the features for classifying that image. For
further analysis we calculate the heat-map using CAM as seen in Fig 6, and use
them to visualize the CNN’s area of interest for similar looking classes
(e.g., SP2 vs SP3). It is seen that iCAM block indeed helps in localizing key
features in an X-ray image. Additionally, Table II presents that
iCAM+InceptionNet-v3 has an inference time of $\approx$30fps, which allows the
model to run in real-time with a 30fps industrial camera. Table III, shows
that when iCAM is combined with InceptionNet-v3, the network outperforms
InceptionNet-v3 by 10.5$\%$. We also combine the iCAM block with the
ResNet-101 feature extractor, and the result outperforms ResNet-101[42], and
other networks like ResNeXt-101[43], SE-ResNet-101[45], and VGG-19[41]. This
shows the efficacy of iCAM block with other extraction modules.

TABLE II: Comparison of baseline classification networks to the iCAM network
on the X-ray dataset. Learning rate: 0.001, Batch Size: 16, Weight decay: 0.1
every 7 epochs, Epochs: 40. The bolded row highlights the network architecture
with the best average accuracy.

Network Architecture Average accuracy Precision Recall Infer time (ms)
InceptionNet-v3 0.933 0.907 0.935 12.3 ResNet-101 0.953 0.899 0.956 11.1
VGG-19 0.956 0.898 0.961 7.53 ResNeXt-101 (64x4d) 0.929 0.869 0.948 20.15 SE-
ResNet-101 0.891 0.841 0.926 14.01 iCAM+ InceptionNet-v3 0.979 0.942 0.969
30.86

TABLE III: Comparison of baseline classification networks to the iCAM network
on the CUB-200[40] dataset. Same parameters as Table 2. The bolded row
highlights the network architecture with the best average accuracy for
InceptionNet-v3 and ResNet-101 feature extractor.

Network Architecture Accuracy Precision Recall InceptionNet-v3 0.692 0.697
0.693 iCAM+ InceptionNet-v3 0.797 0.791 0.801 ResNet-101 0.814 0.818 0.815
ResNeXt-101 (64x4d) 0.829 0.835 0.830 SE-ResNet-101 0.818 0.826 0.820 iCAM+
ResNet-101 0.834 0.831 0.838 VGG-19 0.778 0.781 0.779

Figure 5: Examples of training and testing images from the CUB-200[40]
dataset. Small inter-class variance and large intra-class variance between
species motivates us to use this dataset for benchmarking iCAM. Figure 6:
Heatmap for SP2 (left) and SP3 (right) generated using CAM. SP2 has a
prominent logo visible in the X-ray image, that is the dominant region for
classification. SP3 has a logo and a different component, that helps identify
SP3.

We also conduct a study on the number of iterations the iCAM module is run
during the training time. Table IV shows the accuracy, the time taken to train
40 epochs, and the inference time for $n=2,3,5$. As shown in the table, $n=3$
shows a considerable increase in accuracy from using $n=2$ for far less
training and inference time that $n=5$.

TABLE IV: Comparison of accuracy, training time and inference time of the
iCAM+InceptionNet-v3 network for different values of iteration N. Training was
done for 40 epochs. The bolded row highlights the current $n$ chosen for all
the experiments.

No. of CAM iteration (N) Average accuracy Training time (mins) Infer-time(ms)
n=2 0.962 325 20.11 n=3 0.979 490 30.86 n=5 0.981 723 50.96

A comparative study is done to select the best feature aggregator strategy
where 3 different feature aggregator strategies are tested. The most common
strategies used are mean, sum, and weighted mean of the available feature
maps. All of these experiments are done for $n=3$. We achieve an accuracy of
$0.941$ for the mean, $0.768$ for the sum, and $0.979$ for the weighted mean
version of the feature aggregator. Based on this study, we conclude that
weighted mean is the best feature aggregating strategy because it weighs
different level of magnification differently .

###  IV-C RGB-X classification

We evaluate the multi-modal classification approach with both the single-mode
iCAM+InceptionNet-v3 architectures. For this experiment, an RGB image of a
particular class is paired with a random X-ray image sampled from the same
class. This is done because of the lack of a streamlined RGB + X-ray sensor
setup. While pairing the images, we randomly re-orient the X-ray images to
prevent any bias induced due to similar orientation of the RGB and X-ray
image. The total training time for RGB-X mode is $520$ mins as the RGB and
X-ray modes are trained in parallel. Table V shows the average accuracy,
precision, recall and the inference time of each of the mode (RGB and X-ray)
as well as the combined RGB-X mode.  In context of a recycling plant that
recycles around 200 phones per hour [2], the 0.6 percentage gain in accuracy
from RGB-X ($n=3$) over X-ray ($n=5$) is critical as it saves 12 phones per
hour. Although this adds $~{}11ms$ to the inference time, the total training
time is lowered to $520$ mins (RGB-X) from $723$ mins (X-ray). The lower
training time is crucial while expanding to newer products without
compromising on the recycling rate of 200 phones per hour.

TABLE V: Comparison of single (RGB, X-ray) mode accuracy to the multi-modal
(RGB-X mode) accuracy. The bolded row highlights the mode with the best
average accuracy.

Mode Average accuracy Average precision Average recall Infer time (ms) RGB
0.891 0.932 0.863 30.86 X-ray 0.979 0.942 0.969 30.86 RGB-X 0.987 0.987 0.995
62.4

As mentioned in Sec.III-B2, for joint training we combine the deepest layers
and train for them, by keeping the higher CNN layers frozen. This fusion
approach is termed as “fast fusion” [48]. We evaluate this approach with the
“slow fusion” approach also mentioned in [48]. For slow fusion, the network
architecture is similar to fast fusion, except for the fact that the weights
of the higher CNN layers are also trainable. This allows the higher layers to
have access to more global information.

TABLE VI: Comparison of different fusion techniques. The bolded row highlights
the mode with the best average accuracy.

Mode Average accuracy Average precison Average recall Slow Fusion 0.952 0.951
0.958 Fast Fusion 0.987 0.987 0.995

Table VI, shows the comparisons of 2 different fusion techniques,
demonstrating that fast fusion performs better than the slow fusion, as fast
fusion doesn’t affect the lower level activation learned by the iCAM layers.

##  V Conclusion and Future work

We introduce a useful multi-modal neural network architecture for the RGB-X
object for the task of WEEE classification. Our method consists of a two-
stream convolutional neural network that learns to fuse information from both
RGB and X-ray domains automatically before classification. We also present a
novel fine-grained network architecture, which is used to train the individual
streams of data. This method iteratively localizes key features in an object
and uses them for classification. Our experiments explore how the various
components of this algorithm achieve better classification than the baseline
algorithms when tested on 10 classes of modern smart phones. For this paper,
we use iPhone as our modern smart phones, as the devices and the dataset were
readily available. We also present a novel domain randomization pipeline for
X-ray images using CT scans, a method that reduces the burden of annotating
thousands of X-ray images. As our future work, we plan to extend the work to
classify smart phones from different manufacturers like Samsung, Motorola and
also between inter-manufacturer models. Additionally, we plan to extend the
approach to attend to multiple important regions in an activation map. We also
plan to generalize the hierarchical attention approach to other interpretable
methods like Grad-CAM[49].

The high accuracy as well as high inference rate of this approach make it well
suited for integration into electronics sorting processes at material recovery
facilities.

## References

  * [1] Apple. Apple adds earth day donations to trade-in and recycling program. [Online]. Available: https://www.apple.com/newsroom/2018/04/apple-adds-earth-day-donations-to-trade-in-and-recycling-program
  * [2] Apple. Apple expands global recycling programs. [Online]. Available: https://www.apple.com/newsroom/2019/04/apple-expands-global-recycling-programs
  * [3] Apple. Apple commits to be 100 percent carbon neutral for its supply chain and products by 2030.
  * [4] Apple. Environmental progress report. [Online]. Available: https://www.apple.com/environment/pdf/Apple$\\_$Environmental$\\_$Progress-$\\_$Report$\\_$2021.pdf
  * [5] K. R. Forti V., Balde C.P. E-waste statistics guidelines on classification reporting and indicators. [Online]. Available: http://collections.unu.edu/view/UNU:6477
  * [6] Y. Xu, G. ke Yang, J. Luo, and J. He, “An electronic component recognition algorithm based on deep learning with a faster squeezenet,” _Mathematical Problems in Engineering_ , vol. 2020, pp. 1–11, 2020.
  * [7] S. Kumar, D. Yadav, H. Gupta, O. Verma, I. Ansari, and C. W. Ahn, “A novel yolov3 algorithm-based deep learning approach for waste segregation: Towards smart waste management,” _Electronics_ , vol. 10, p. 14, 12 2020.
  * [8] DTI. Refind, $\&$ stena. march 23 new robot system extracts dangerous and valuable items from waste using artificial intelligence.
  * [9] . Refind. Optical battery sorter 500. [Online]. Available: https://www.refind.se/optical-battery-s orter-500
  * [10] H. Karbasi, A. Sanderson, A. Sharifi, and C. Pop, “Robotic sorting of used button cell batteries: Utilizing deep learning,” in _2018 IEEE Conference on Technologies for Sustainability (SusTech)_ , 2018, pp. 1–6.
  * [11] S. Gundupalli, S. Hait, A. Thakur, and A. Trivedi, _Classification of Recyclables from E-Waste Stream Using Thermal Imaging-Based Technique_ , 12 2018, pp. 67–78.
  * [12] W. Sterkens, D. Diaz-Romero, T. Goedemé, W. Dewulf, and J. R. Peeters, “Detection and recognition of batteries on x-ray images of waste electrical and electronic equipment using deep learning,” _Resources, Conservation and Recycling_ , vol. 168, p. 105246, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0921344920305619
  * [13] S. Z. Valtchev and J. Wu, “Domain randomization for neural network classification,” _Journal of Big Data_ , vol. 8, no. 1, pp. 1–12, 2021.
  * [14] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 23–30, 2017.
  * [15] A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: From simulation to reality with domain randomization,” _IEEE Transactions on Robotics_ , vol. 36, no. 1, pp. 1–14, 2020.
  * [16] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 1082–10 828, 2018.
  * [17] Y. Movshovitz-Attias, T. Kanade, and Y. Sheikh, “How useful is photo-realistic rendering for visual learning?” in _Computer Vision – ECCV 2016 Workshops_ , G. Hua and H. Jégou, Eds. Cham: Springer International Publishing, 2016, pp. 202–217.
  * [18] W. Chen, H. Wang, Y. Li, H. Su, Z. Wang, C. Tu, D. Lischinski, D. Cohen-Or, and B. Chen, “Synthesizing training images for boosting human 3d pose estimation,” in _2016 Fourth International Conference on 3D Vision (3DV)_ , 2016, pp. 479–488.
  * [19] N. Zhang, J. Donahue, R. B. Girshick, and T. Darrell, “Part-based r-cnns for fine-grained category detection,” _ArXiv_ , vol. abs/1407.3867, 2014.
  * [20] D. Lin, X. Shen, C. Lu, and J. Jia, “Deep lac: Deep localization, alignment and classification for fine-grained recognition,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1666–1674.
  * [21] X.-S. Wei, C.-W. Xie, J. Wu, and C. Shen, “Mask-cnn: Localizing parts and selecting descriptors for fine-grained bird species categorization,” _Pattern Recognition_ , vol. 76, pp. 704–714, 2018.
  * [22] M. Jaderberg, K. Simonyan, A. Zisserman, _et al._ , “Spatial transformer networks,” _Advances in neural information processing systems_ , vol. 28, pp. 2017–2025, 2015.
  * [23] J. Fu, H. Zheng, and T. Mei, “Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4438–4446.
  * [24] H. Zheng, J. Fu, T. Mei, and J. Luo, “Learning multi-attention convolutional neural network for fine-grained image recognition,” 10 2017, pp. 5219–5227.
  * [25] Z. Yang, T. Luo, D. Wang, Z. Hu, J. Gao, and L. Wang, “Learning to navigate for fine-grained classification,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 420–435.
  * [26] X. He and Y. Peng, “Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification,” in _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * [27] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 1817–1824.
  * [28] M. Blum, J. T. Springenberg, J. Wülfing, and M. Riedmiller, “A learned feature descriptor for object recognition in rgb-d data,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 1298–1303.
  * [29] L. Zheng, Y. Yang, and Q. Tian, “Sift meets cnn: A decade survey of instance retrieval,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 5, pp. 1224–1244, 2017.
  * [30] A. Eitel, J. T. Springenberg, L. Spinello, M. Riedmiller, and W. Burgard, “Multimodal deep learning for robust rgb-d object recognition,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 681–687.
  * [31] B. O. Community, _Blender - a 3D modelling and rendering package_ , Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. [Online]. Available: http://www.blender.org
  * [32] W. Schroeder, K. M. Martin, and W. E. Lorensen, _The visualization toolkit an object-oriented approach to 3D graphics_. Prentice-Hall, Inc., 1998.
  * [33] A. Fedorov, R. Beichel, J. Kalpathy-Cramer, J. Finet, J.-C. Fillion-Robin, S. Pujol, C. Bauer, D. Jennings, F. Fennessy, M. Sonka, J. Buatti, S. Aylward, J. Miller, S. Pieper, and R. Kikinis, “3d slicer as an image computing platform for the quantitative imaging network,” _Magnetic resonance imaging_ , vol. 30, pp. 1323–41, 07 2012.
  * [34] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 224–236.
  * [35] B. Zhou, A. Khosla, L. A., A. Oliva, and A. Torralba, “Learning Deep Features for Discriminative Localization.” _CVPR_ , 2016.
  * [36] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [37] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT’2010_. Springer, 2010, pp. 177–186.
  * [38] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in _NIPS-W_ , 2017.
  * [39] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” _CoRR_ , vol. abs/1703.06907, 2017. [Online]. Available: http://arxiv.org/abs/1703.06907
  * [40] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [41] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [42] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [43] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1492–1500.
  * [44] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision. 2015,” _arXiv preprint arXiv:1512.00567_ , 2015.
  * [45] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [46] S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” _IEEE Access_ , vol. 6, p. 64270–64277, 2018. [Online]. Available: http://dx.doi.org/10.1109/ACCESS.2018.2877890
  * [47] M. Grandini, E. Bagli, and G. Visani, “Metrics for multi-class classification: an overview,” 2020.
  * [48] M. B. Shaikh and D. Chai, “Rgb-d data-based action recognition: A review,” _Sensors_ , vol. 21, no. 12, p. 4246, 2021.
  * [49] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.