# Proximitized insulators from disordered superconductors

Moshe Haim  Department of Physics, Jack and Pearl Resnick Institute and the
Institute of Nanotechnology and Advanced Materials, Bar-Ilan University,
Ramat-Gan 52900, Israel  Department of Condensed Matter Physics, Weizmann
Institute of Science, Rehovot 76100, Israel  David Dentelski  Department of
Physics, Jack and Pearl Resnick Institute and the Institute of Nanotechnology
and Advanced Materials, Bar-Ilan University, Ramat-Gan 52900, Israel  Aviad
Frydman  Department of Physics, Jack and Pearl Resnick Institute and the
Institute of Nanotechnology and Advanced Materials, Bar-Ilan University,
Ramat-Gan 52900, Israel

###### Abstract

We present an experimental study of bilayers of a disordered $Ag$ metal layer
close to the metal-insulator transition and an Indium Oxide film which is on
the insulating side of the superconductor-insulator-transition. Our results
show that superconducting fluctuations within the indium-oxide film, that
proximitize the underlying metal layer, induce insulating rather than
superconducting behavior. This is ascribed to suppression of density of states
(due to the superconducting energy gap) for quasiparticles in the proximitized
regions. Our results present a novel manifestation of the proximity effect
phenomenon and provide important insight into the nature of the insulating
phase of the disorder driven superconductor-insulator-transition.

The interplay between superconductivity and disorder is a very active topic of
investigation. It was recognized decades ago that s-wave superconductivity is
remarkably robust against weak disorder [1]. The situation is different for
strong disorder. Experiments show that superconductivity in 2D films can be
destroyed by strong enough disorder as well as other non-thermal tuning
parameters, g, such as magnetic field, thickness, chemical composition, gate
voltage and pressure [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. Once
superconductivity is destroyed, the system undergoes a transition to an
insulating state (for reviews see [34, 35]). This superconductor-insulator-
transition (SIT) is a paradigmatic example for a quantum phase transition that
can occur in systems driven by a non-thermal tuning parameter [36].

One of the ongoing deliberations in the field of the SIT is the nature of the
insulating phase, $I_{S}$. It has been shown both theoretically [37, 38, 39,
40] and experimentally [41] that in some nominally s-wave BCS superconductors,
the presence of disorder can separate the temperature $T^{*}$ where pairing
occurs accompanied by the development of an energy gap in the local density of
states [16, 42], and the actual $T_{c}$ where the superfluid density becomes
finite. The pseudogap region between these temperatures grows with disorder as
the SIT is approached and eventually, in the insulating regime, a finite
superconducting gap, $\Delta$, exists even in the absence of superfluid
density. Indeed, similar energy gap [41] (as well as vortex motion [17, 43]
and Nernst signals [18]) were measured in both the superconductor and $I_{S}$
phases of disordered films. These results led to the realization that a BCS
superconductor can undergo a quantum phase transition to an insulating phase
of bosons rather than unpaired electrons. Hence, $I_{S}$, which is composed of
uncorrelated superconducting fluctuations, shows a number of properties
similar to those of a bulk superconductor despite being an electrical
insulator. In this letter, we study a special aspect of the superconducting
nature of $I_{S}$, i.e. the proximity effect to a normal metal.

The classic proximity effect describes the mutual influence of two "clean"
layers, one is a superconductor, $SC$, and one is a normal metal, $N$, placed
in a good electric contact, resulting in the induction of finite
superconductivity into the $N$ and suppression of the superconducting order
parameter in the $SC$ [44, 45]. Experiments have shown [46, 47] that, when the
superconductor is highly disordered, superconducting quantum fluctuations
(even within the $I_{S}$) can induce superconductivity into a proximitized
$N$. In this letter we report on a more exotic effect which occurs when the
$N$ is highly disordered so that it is close to the metal-insulator-
transition. Our main results are the following:

  1. 1.

Placing a highly disordered metal in proximity to an $I_{S}$ can induce
$\it{insulating}$ behavior in the $N$.

  2. 2.

This effect becomes more prominent as the $I_{S}$ is driven towards the SIT.

  3. 3.

The effect is larger the more disordered is the $N$ layer.

We provide a simple model to explain these results based on enhanced
electronic localization in the disordered metal due to proximitized
superconducting fluctuations in the normal region.

The samples for this study were prepared using the following scheme. Six $Au$
leads were deposited on an insulating $SiO$ substrate (gold pads in Fig 1a).
Then, two $10nm$ thick $Ag$ strips were deposited between two sets of leads,
to be used as the $N$ proximity layer (grey strips in Fig. 1a). In order to
increase the disorder of the silver, the samples were thinned in an $Ar$
plasma chamber in short pulses for different amounts of time. Here we present
results for three highly disordered $Ag$ films, $S1$, $S2$ and $S3$, having
decreasing $Ag$ room temperature sheet resistances of 250, 220 and 150
$\Omega_{\square}$, respectively. Finally, a $30nm$ thick layer of amorphous
Indium Oxide ($InO$) was e-beam deposited in a $10^{-4}mbar$ partial oxygen
pressure resulting in a highly disordered, insulating film (purple layer in
Fig 1a). The resistance of the $InO$ film was sequentially reduced via low
temperature thermal annealing, thus driving the film through the insulator to
superconductor transition [48, 18]. This setup takes advantage of the fact the
$Ag$ layer is significantly more conductive than the $InO$ layer and allows to
use the Ag layer as a voltage terminal for four-probe resistance measurements
of the bare $InO$ film and two-probe measurements of the resistance of the
$Ag/InO$ bilayer as the $InO$ is driven through the SIT.

Figure 1: a: A sketch of the device containing six leads (gold), two silver
strips (gray) and the $InO$ layer (purple). b: Resistance vs temperature of
the $InO$ layer of sample $S1$, for different stages of annealing (as noted in
the legend), measured between the silver strips. c: $T_{0}$ (extracted from
the $R=R_{0}e^{\frac{T_{0}}{T}}$ dependency) vs the 1K $InO$ sheet resistance.
The dashed line is a guide to the eye.

$InO$ films, despite being morphologically uniform [48, 49], have been shown
to include emergent granularity in the form of superconducting puddles
embedded in an insulating matrix [50, 41, 43, 18]. Hence, local
superconductivity is present even in the insulating phase of the SIT. Fig. 1b
shows the resistance vs temperature curves of the bare $InO$ of $S1$ for
different annealing stages (sequentially reducing $R_{\square}$). The
insulating curves are found to follow $R=R_{0}e^{T_{0}/T}$ behavior. This is a
typical feature of $I_{S}$ insulators which are characterized by emergent
granularity [15, 51, 52, 53]. Fig. 1c presents $T_{0}$ versus $R$ of the $InO$
film, which is found to decreases as the sample approaches the SIT and
extrapolates to zero close (but beyond) to it, in consistence with previously
reported works [15].

Figure 2:  Resistance vs temperature of the three $Ag/InO$ bilayer of samples
$S1$, $S2$ and $S3$ (panels a, b and c, respectively), for different stages of
annealing. The colors signaling the annealing stage (see legend) apply for S2
and S3 as well, however, the sheet resistances of the InO film may differ
between the samples. Black lines are plots for the bare $Ag$ films. For
clarity, the curves are normalized to the resistance at 10K. d: Resistance
maximum vs the $InO$ Resistance at 1K of samples S1 (black), S2 (red) and S3
(blue).

The key result of this work is presented in Fig. 2 which shows the resistance
versus temperature curves for the three $Ag/InO$ bilayers. As long as the
$InO$ is in the insulating phase, the resistance of such a bilayer is governed
by the $Ag$ layer which has a much lower sheet resistance ($\approx
200\Omega_{\square}$) than that of the $InO$ ($\approx
100k\Omega_{\square}-100M\Omega_{\square}$) for $T\leq 10K$. Thus, when
measuring the bilayer we are, in fact, measuring the Ag layer almost strictly.
We note that the annealing process may slightly affect the Ag resistance as
well.

We start with considering the results of sample $S1$ (having the highest $Ag$
resistance) represented in Fig. 2a. Surprisingly, the addition of $InO$ causes
the resistance to increase with decreasing temperature below $\sim 10K$. This
is very counter-intuitive since one naively expects that the $InO$ would add
conductivity in parallel and, if anything, would reduce the total resistance.
Instead, the $InO$ overlayer seems to be inducing insulating behavior in the
underlying $Ag$ film. This effect gets larger as the $InO$ is driven towards
the SIT, eventually exceeding a $50\%$ amplitude increase before reversing the
trend at low temperatures where the resistance starts decreasing with lowering
temperature.

A similar effect, though with smaller magnitude, was seen for samples $S2$ and
$S3$, having decreasing disorder respectively (Fig. 2b and c). Fig 2d shows
the peak amplitude versus $R_{\square}$ for the three samples, indicating that
the resistance increase depends on two parameters. It is larger the more
disordered the $Ag$ film is and also the closer the $InO$ is to the SIT.

Figure 3: Illustration of the current paths (in orange) through the $Ag$ film.
a: The 2D conductivity map of the $Ag$ film prior to the $InO$ deposition. The
current is carried by the most conductive parts (peaks in the 2D map). b:
Adding an $InO$ layer proximitizes different parts of $Ag$ film and induces
superconductivity, represented in purple. The current bypasses the SC regions
due to suppression of the DOS in these locations. c: Annealing the $InO$ leads
to more sections of the $Ag$ being proximitized thus further limiting the
current paths and forcing them to choose less conductive routes, thus further
increasing the resistance. d: Differential conductivity vs bias voltage of
sample $S1$ for the last insulating stage, normalized to the data at 150 meV.
The inset shows the relative peak height and energy separation for different
disorder degrees.

In order to understand how inducing superconducting fluctuations in a $N$
layer results in an increase of resistance we recall that our $Ag$ films are
highly disordered, close to being insulating themselves. The conductivity is
thus inhomogeneous due to strong spatial fluctuations of the underlying
electronic potential. The current does not flow uniformly through the sample
but rather through preferred high conductance trajectories as illustrated in
the conductivity map of Fig. 3a.

Adding a disordered superconducting overlayer induces islands of $SC$ regions
in the $Ag$ film (purple dots in Fig 3b) at temperatures below $T_{c}$. As the
temperature is lowered, the density of superconducting regions increases.
However, when global phase coherence is absent, the insertion of
superconducting islands into a highly disorder metal can actually increase the
resistivity. This is due to the formation of a local energy gap, $\Delta$,
within each island, which suppresses the density of states for quasiparticles
and thus limiting the current flow through these islands. Because the regions
that are more prone to the proximity process are, naturally, those with higher
conductivity, the current is restricted to one of two options: flowing through
the high resistance trajectories (Fig. 3b) or through the puddles of zero
resistance, but at an energy ’cost’ of $2\Delta$. Therefore, the sample can be
viewed as a network of SIS junctions where the experimentally observed gap
overlays the individual local ones which the current must tunnel through. A
similar process was suggested as the origin for the giant magnetoresistance
peak observed in these materials at high fields and low temperatures [54, 55].

Lowering the disorder of the $SC$ and pushing it towards the SIT (e.g. by
annealing the $InO$ layer) increases the density of superconducting puddles
(Fig. 3c), thus further limiting the current carrying network and forcing the
current to flow through higher resistance trajectories. This results in
increasing the bilayer resistivity as the $InO$ film is pushed towards the SIT
as indeed seen in the experiment (Fig. 2d). In addition, reducing the $N$
disorder smooths the potential background thus suppressing the above process.

The temperature onset of this unique proximity effect is $\sim 10K$ which is
significantly larger than the maximal $T_{c}$ measured in $InO$ films ($\sim
3.5K$ [18]). However, STM measurements on a film of $InO$ with $T_{C}\approx
3K$ have detected a finite $\Delta$ up to temperatures of $\approx 6.5$ K
[16]. In the insulator, $\Delta$ is predicted to grow further and increase as
disorder increases [39]. The real pairing critical temperature, $T^{*}$, of
the $I_{s}$ phase of $InO$, is yet unknown, but the results presented here
show signs for superconductivity up to $T\approx 10K$.

Figure 4: Normalized sheet resistance $R_{\square}(t)$ of an $L=50$ squared
lattice normalized by its value at $T_{c}$, as a function of the reduced
temperature $t=T/T_{c}$. a: Fixed disorder density $N_{d}=0.05$ for different
values of $Z$. As the temperature decreases, there is an interplay between the
gain of a current passing through a zero-resistance superconducting island and
the cost, $Z$, to enter and exit the island. b: Fixed $Z=3$ for samples with
different disorder density, $N_{d}$. The dashed lines mark the sheet
resistance at $T=T_{c}$.

The schematic representation of the current flow through the $Ag$ layer in
Fig. 3a-c is modeled here by considering an $L\times L$ squared lattice, where
each site can be either a superconductor or a normal metal. We consider only
nearest neighbors connections and assign a resistance $r=1$ for each bond
between two metallic sites. The total sheet resistance, $R_{\square}$, is then
calculated by the minimal resistance path required for the current to flow
from one side of the bilayer to the other normalized by the size of the
lattice. This is to say, that for $t=\frac{T}{T_{c}}\geq 1$, where all sites
are metallic, $R_{\square}=1$.

With decreasing temperature, proximitized superconducting islands start to
form in the $Ag$ layer. This is manifested by re-weighting all the bonds’
resistances connecting a superconducting site to a metallic one by a factor
$Z\geq 1$ while all bonds between two neighbouring superconducting sites
(within a SC island) are assigned a resistance $r=0$, such that
$R_{\square}\rightarrow 0$ as the superconducting density, $n_{sc}$, becomes
large. The different sites are chosen randomly to be metallic or
superconductors, depending on the value of $n_{sc}(t)\in[0,1]$.

We include the effect of disorder by introducing sites in the normal metal
that prevent the formation of superconductivity. We assign a resistance
$r(t)=r_{0}e^{T_{0}/T}$ between a site within a superconducting or a metal
region to a disordered one. Here, we use $r_{0}=\exp(-1)$ and $T_{0}=1.05\
T_{c}$, such that the high-temperature resistivity of the different samples is
$\approx 5\%$ higher than the clean one. The strength of the disorder is
defined by the density of these sites, $N_{d}$.

Fig. 4a shows the resistance as a function of the reduced temperature $t$ for
a fixed value of $N_{d}$ and different values of $Z$, where we used the
empirical approximate temperature dependence of the superconducting fraction:
$n_{sc}(t)=1-t^{0.4}$. For all values of disorder, decreasing the temperature
increases the density of superconducting islands and hence increasing
$R_{\square}$. This trend continues down to a disorder-dependent temperature,
which marks the onset of global superconductivity, thus leading to the peak in
the R-T curve as is found in the experiments. Note that the broad transition
is also consistent with the experimental results. Fig. 4b shows the results
for constant $Z=3$ and different degree of disorder, $N_{d}$. It is seen that
as the disorder increases, the resistivity peak is found to be higher as
indeed observed in the experiments.

The above picture is strongly supported by the differential conductance curve
shown in Fig 3d which resembles an insulating stage of $S1$. The overall
$dI/dV$ versus $V$ curve exhibits a suppression of conductance as the bias
voltage is lowered, due to the Altshuler-Aronov (AA) mechanism of electron-
electron interactions [56] in the disordered $Ag$ film (dashed curve). At low
bias, the curve exhibits a superimposed structure which includes two
symmetrical maxima which resemble the coherence peaks of a superconducting gap
structure. The peak amplitude and energy scale grow as the sample is further
annealed and pushed towards the SIT, however they are only observed in the
most disordered $Ag$ film ($S1$), and when the overlayer $InO$ film is close
to the SIT, as seen in the inset. The energy scales extracted from these
features are 19.6 and 30.8 meV for the two last insulating $InO$ stages.
Interestingly, these are integer multiples (14 and 22 respectively) of
$1.4meV$, which is the value of $2\Delta$ for amorphous $InO$ [41]. This is
consistent with the suggested model of current flowing through a series of
$SIS$ junctions giving rise to a global gap-like structure, which is the sum
of the individual local gaps on each of the SC island and is superimposed on
the AA trend.

The results presented in this paper demonstrate a new type of proximity
effect. We show that inducing superconductivity into an $N$ layer is not
limited to ’clean’ superconductors but can also be extended to an $I_{S}$
phase. Moreover, in the case of a highly disordered $N$, the proximity of a
disordered metal to the $I_{S}$ can induce insulating-like behavior thus
reducing its conductance. This effect becomes more prominent, the larger the N
disorder. Such a proximitized bilayer can also offer a useful tool to study
the $I_{S}$ deep into the insulating phase. Attempting to measure
superconducting fluctuations in an insulating sample by transport is
ineffective, since the exponentially increasing resistance screens local
superconductivity. Tunneling measurement require a barrier that is much more
resistive than the sample itself, limiting the measurement to samples that are
close to the transition and at relatively high temperatures. In contrary, by
coupling a film that is well within the insulating phase of the SIT to a
normal metal, one can access transport and tunneling measurement of the
coupled metal, regardless of how insulating the superconductor is. The
interplay between the metal and the superconductor is quantified in our
numerical model as a single parameter denoted as $Z$, which can be extracted
directly from simple tunneling measurements and can be studied for different
samples and materials.

We are grateful for technical help from I. Volotsenko, R. Cohen, Y. Stein, A.
Fried, A. Roy and M. Laav and useful discussions with J. Ruhman, N. Trivedi
and T. Baturina. This work was supported by the US-Israel Bi-national Science
Foundation (BSF) grant No. 2020331.

## References

  * Anderson [1958] P. W. Anderson, Absence of diffusion in certain random lattices, Physical review 109, 1492 (1958).
  * Strongin and Thompson [1970] R. Strongin and O. Thompson, Kammerer, and j.e crow, Physical Review B 1, 1078 (1970).
  * Dynes  _et al._ [1986] R. Dynes, A. White, J. Graybeal, and J. Garno, Breakdown of eliashberg theory for two-dimensional superconductivity in the presence of disorder, Physical Review Letters 57, 2195 (1986).
  * Haviland  _et al._ [1989] D. Haviland, Y. Liu, and A. M. Goldman, Onset of superconductivity in the two-dimensional limit, Physical Review Letters 62, 2180 (1989).
  * Valles Jr  _et al._ [1992] J. Valles Jr, R. Dynes, and J. Garno, Electron tunneling determination of the order-parameter amplitude at the superconductor-insulator transition in 2d, Physical Review Letters 69, 3567 (1992).
  * Frydman  _et al._ [2002] A. Frydman, O. Naaman, and R. Dynes, Universal transport in two-dimensional granular superconductors, Physical Review B 66, 052509 (2002).
  * Aubin  _et al._ [2006] H. Aubin, C. Marrache-Kikuchi, A. Pourret, K. Behnia, L. Bergé, L. Dumoulin, and J. Lesueur, Magnetic-field-induced quantum superconductor-insulator transition in nb 0.15 si 0.85, Physical Review B 73, 094521 (2006).
  * Stewart  _et al._ [2007] M. Stewart, A. Yin, J. Xu, and J. M. Valles, Superconducting pair correlations in an amorphous insulating nanohoneycomb film, Science 318, 1273 (2007).
  * Sacépé  _et al._ [2008] B. Sacépé, C. Chapelier, T. Baturina, V. Vinokur, M. Baklanov, and M. Sanquer, Disorder-induced inhomogeneities of the superconducting state close to the superconductor-insulator transition, Physical Review Letters 101, 157006 (2008).
  * Marrache-Kikuchi  _et al._ [2008] C. Marrache-Kikuchi, H. Aubin, A. Pourret, K. Behnia, J. Lesueur, L. Bergé, and L. Dumoulin, Thickness-tuned superconductor-insulator transitions under magnetic field in a-nbsi, Physical Review B 78, 144520 (2008).
  * Hollen  _et al._ [2011] S. M. Hollen, H. Nguyen, E. Rudisaile, M. Stewart Jr, J. Shainline, J. Xu, and J. Valles Jr, Cooper-pair insulator phase in superconducting amorphous bi films induced by nanometer-scale thickness variations, Physical Review B 84, 064528 (2011).
  * Postolova  _et al._ [2017] S. V. Postolova, A. Y. Mironov, M. R. Baklanov, V. M. Vinokur, and T. I. Baturina, Reentrant resistive behavior and dimensional crossover in disordered superconducting tin films, Scientific Reports 7, 1 (2017).
  * Baturina  _et al._ [2011] T. I. Baturina, V. Vinokur, A. Y. Mironov, N. Chtchelkatchev, D. Nasimov, and A. Latyshev, Nanopattern-stimulated superconductor-insulator transition in thin tin films, EPL (Europhysics Letters) 93, 47002 (2011).
  * Poran  _et al._ [2017] S. Poran, T. Nguyen-Duc, A. Auerbach, N. Dupuis, A. Frydman, and O. Bourgeois, Quantum criticality at the superconductor-insulator transition revealed by specific heat measurements, Nature Communications 8, 1 (2017).
  * Shahar and Ovadyahu [1992] D. Shahar and Z. Ovadyahu, Superconductivity near the mobility edge, Physical Review B 46, 10917 (1992).
  * Sacépé  _et al._ [2011] B. Sacépé, T. Dubouchet, C. Chapelier, M. Sanquer, M. Ovadia, D. Shahar, M. Feigel’Man, and L. Ioffe, Localization of preformed cooper pairs in disordered superconductors, Nature Physics 7, 239 (2011).
  * Poran  _et al._ [2011] S. Poran, E. Shimshoni, and A. Frydman, Disorder-induced superconducting ratchet effect in nanowires, Physical Review B 84, 014529 (2011).
  * Roy  _et al._ [2018] A. Roy, E. Shimshoni, and A. Frydman, Quantum criticality at the superconductor-insulator transition probed by the nernst effect, Physical review letters 121, 047003 (2018).
  * Paalanen  _et al._ [1992] M. Paalanen, A. Hebard, and R. Ruel, Low-temperature insulating phases of uniformly disordered two-dimensional superconductors, Physical review letters 69, 1604 (1992).
  * Yazdani and Kapitulnik [1995] A. Yazdani and A. Kapitulnik, Superconducting-insulating transition in two-dimensional a-moge thin films, Physical review letters 74, 3037 (1995).
  * Gantmakher  _et al._ [1998] V. Gantmakher, M. Golubkov, V. Dolgopolov, G. Tsydynzhapov, and A. Shashkin, Destruction of localized electron pairs above the magnetic-field-driven superconductor-insulator transition in amorphous in-o films, Journal of Experimental and Theoretical Physics Letters 68, 363 (1998).
  * Sambandamurthy  _et al._ [2004] G. Sambandamurthy, L. Engel, A. Johansson, and D. Shahar, Superconductivity-related insulating behavior, Physical review letters 92, 107005 (2004).
  * Sambandamurthy  _et al._ [2005] G. Sambandamurthy, L. Engel, A. Johansson, E. Peled, and D. Shahar, Experimental evidence for a collective insulating state in two-dimensional superconductors, Physical Review Letters 94, 017003 (2005).
  * Steiner  _et al._ [2005] M. A. Steiner, G. Boebinger, and A. Kapitulnik, Possible field-tuned superconductor-insulator transition in high-t c superconductors: Implications for pairing at high magnetic fields, Physical Review Letters 94, 107008 (2005).
  * Baturina  _et al._ [2005] T. Baturina, J. Bentner, C. Strunk, M. Baklanov, and A. Satta, From quantum corrections to magnetic-field-tuned superconductor–insulator quantum phase transition in tin films, Physica B: Condensed Matter 359, 500 (2005).
  * Baturina  _et al._ [2007a] T. Baturina, C. Strunk, M. Baklanov, and A. Satta, Quantum metallicity on the high-field side of the superconductor-insulator transition, Physical review letters 98, 127003 (2007a).
  * Crane  _et al._ [2007] R. Crane, N. Armitage, A. Johansson, G. Sambandamurthy, D. Shahar, and G. Grüner, Fluctuations, dissipation, and nonuniversal superfluid jumps in two-dimensional superconductors, Physical Review B 75, 094506 (2007).
  * Vinokur  _et al._ [2008] V. M. Vinokur, T. I. Baturina, M. V. Fistul, A. Y. Mironov, M. R. Baklanov, and C. Strunk, Superinsulator and quantum synchronization, Nature 452, 613 (2008).
  * Ganguly  _et al._ [2017] R. Ganguly, I. Roy, A. Banerjee, H. Singh, A. Ghosal, and P. Raychaudhuri, Magnetic field induced emergent inhomogeneity in a superconducting film with weak and homogeneous disorder, Physical Review B 96, 054509 (2017).
  * Mondal  _et al._ [2011] M. Mondal, A. Kamlapure, M. Chand, G. Saraswat, S. Kumar, J. Jesudasan, L. Benfatto, V. Tripathi, and P. Raychaudhuri, Phase fluctuations in a strongly disordered s-wave nbn superconductor close to the metal-insulator transition, Physical Review Letters 106, 047001 (2011).
  * Parendo  _et al._ [2005] K. A. Parendo, K. S. B. Tan, A. Bhattacharya, M. Eblen-Zayas, N. Staley, and A. Goldman, Electrostatic tuning of the superconductor-insulator transition in two dimensions, Physical Review Letters 94, 197004 (2005).
  * Caviglia  _et al._ [2008] A. Caviglia, S. Gariglio, N. Reyren, D. Jaccard, T. Schneider, M. Gabay, S. Thiel, G. Hammerl, J. Mannhart, and J.-M. Triscone, Electric field control of the laalo 3/srtio 3 interface ground state, Nature 456, 624 (2008).
  * Cohen  _et al._ [2021] R. Cohen, M. Nikolaevsky, R. Salem, and A. Frydman, Setup for pressurizing thin films through the superconductor–insulator transition, Review of Scientific Instruments 92, 083903 (2021).
  * Goldman and Markovic [1998] A. M. Goldman and N. Markovic, Superconductor-insulator transitions in the two-dimensional limit, PhT 51, 39 (1998).
  * Lin  _et al._ [2015] Y.-H. Lin, J. Nelson, and A. Goldman, Superconductivity of very thin films: The superconductor–insulator transition, Physica C: Superconductivity and its Applications 514, 130 (2015).
  * Sachdev [2007] S. Sachdev, Quantum phase transitions, Handbook of Magnetism and Advanced Magnetic Materials (2007).
  * Ghosal  _et al._ [1998] A. Ghosal, M. Randeria, and N. Trivedi, Role of spatial amplitude fluctuations in highly disordered s-wave superconductors, Physical Review Letters 81, 3940 (1998).
  * Ghosal  _et al._ [2001] A. Ghosal, M. Randeria, and N. Trivedi, Inhomogeneous pairing in highly disordered s-wave superconductors, Physical Review B 65, 014501 (2001).
  * Bouadim  _et al._ [2011] K. Bouadim, Y. L. Loh, M. Randeria, and N. Trivedi, Single-and two-particle energy gaps across the disorder-driven superconductor–insulator transition, Nature Physics 7, 884 (2011).
  * Dentelski  _et al._ [2018] D. Dentelski, A. Frydman, E. Shimshoni, and E. G. Dalla Torre, Tunneling probe of fluctuating superconductivity in disordered thin films, Physical Review B 97, 100503 (2018).
  * Sherman  _et al._ [2012] D. Sherman, G. Kopnov, D. Shahar, and A. Frydman, Measurement of a superconducting energy gap in a homogeneously amorphous insulator, Physical Review Letters 108, 177006 (2012).
  * Kamlapure  _et al._ [2013] A. Kamlapure, T. Das, S. C. Ganguli, J. B. Parmar, S. Bhattacharyya, and P. Raychaudhuri, Emergence of nanoscale inhomogeneity in the superconducting state of a homogeneously disordered conventional superconductor, Scientific reports 3, 1 (2013).
  * Kopnov  _et al._ [2012] G. Kopnov, O. Cohen, M. Ovadia, K. H. Lee, C. C. Wong, and D. Shahar, Little-parks oscillations in an insulator, Physical review letters 109, 167002 (2012).
  * de Gennes [1964] P. de Gennes, Boundary effects in superconductors, Reviews of Modern Physics 36, 225 (1964).
  * Deutscher [1969] G. Deutscher, Pg de gennes in superconductivity, rd parks, ed (1969).
  * Merchant  _et al._ [2001] L. Merchant, J. Ostrick, R. Barber Jr, and R. C. Dynes, Crossover from phase fluctuation to amplitude-dominated superconductivity: A model system, Physical Review B 63, 134508 (2001).
  * Frydman and Dynes [2001] A. Frydman and R. Dynes, Granular superconductors and ferromagnets: a proximity-effect-based analogue, Philosophical Magazine B 81, 1153 (2001).
  * Ovadyahu [1986] Z. Ovadyahu, Some finite temperature aspects of the anderson transition, Journal of Physics C: Solid State Physics 19, 5187 (1986).
  * Givan and Ovadyahu [2012] U. Givan and Z. Ovadyahu, Compositional disorder and transport peculiarities in the amorphous indium oxides, Physical Review B 86, 165101 (2012).
  * Kowal and Ovadyahu [1994] D. Kowal and Z. Ovadyahu, Disorder induced granularity in an amorphous superconductor, Solid State Communications 90, 783 (1994).
  * Baturina  _et al._ [2007b] T. I. Baturina, A. Y. Mironov, V. Vinokur, M. Baklanov, and C. Strunk, Localized superconductivity in the quantum-critical region of the disorder-driven superconductor-insulator transition in tin thin films, Physical review letters 99, 257003 (2007b).
  * Baturina  _et al._ [2008] T. I. Baturina, A. Y. Mironov, V. Vinokur, M. Baklanov, and C. Strunk, Hyperactivated resistance in tin films on the insulating side of the disorder-driven superconductor-insulator transition, JETP letters 88, 752 (2008).
  * Humbert  _et al._ [2021] V. Humbert, M. Ortuño, A. Somoza, L. Bergé, L. Dumoulin, and C. Marrache-Kikuchi, Overactivated transport in the localized phase of the superconductor-insulator transition, Nature communications 12, 1 (2021).
  * Dubi  _et al._ [2006] Y. Dubi, Y. Meir, and Y. Avishai, Theory of the magnetoresistance of disordered superconducting films, Physical Review B 73, 054509 (2006).
  * Dubi  _et al._ [2007] Y. Dubi, Y. Meir, and Y. Avishai, Nature of the superconductor–insulator transition in disordered superconductors, Nature 449, 876 (2007).
  * Altshuler and Aronov [1985] B. L. Altshuler and A. G. Aronov, Electron–electron interaction in disordered conductors, in  _Modern Problems in condensed matter sciences_ , Vol. 10 (Elsevier, 1985) pp. 1–153.


# Intelligent Blockchain-based Edge Computing
via Deep Reinforcement Learning:
Solutions and Challenges

Dinh C. Nguyen, Van-Dinh Nguyen, Ming Ding, Symeon Chatzinotas, Pubudu N.
Pathirana,
Aruna Seneviratne, Octavia Dobre, , and Albert Y. Zomaya, Dinh C. Nguyen is
with the School of Electrical and Computer Engineering, Purdue University,
USA.Van-Dinh Nguyen and Symeon Chatzinotas are with the Interdisciplinary
Centre for Security, Reliability and Trust (SnT), University of
Luxembourg.Ming Ding is with Data61, CSIRO, Australia.Pubudu N. Pathirana is
with the School of Engineering, Deakin University, Australia.Aruna Seneviratne
is with School of Electrical Engineering and Telecommunications, UNSW,
Australia.Octavia Dobre is with the Faculty of Engineering and Applied
Science, Memorial University, Canada.Albert Y. Zomaya is with the School of
Computer Science, The University of Sydney, Australia.

###### Abstract

The convergence of mobile edge computing (MEC) and blockchain is transforming
the current computing services in wireless Internet-of-Things networks, by
enabling task offloading with security enhancement based on blockchain mining.
Yet the existing approaches for these enabling technologies are isolated,
providing only tailored solutions for specific services and scenarios. To fill
this gap, we propose a novel cooperative task offloading and blockchain mining
(TOBM) scheme for a blockchain-based MEC system, where each edge device not
only handles computation tasks but also deals with block mining for improving
system utility. To address the latency issues caused by the blockchain
operation in MEC, we develop a new Proof-of-Reputation consensus mechanism
based on a lightweight block verification strategy. To accommodate the highly
dynamic environment and high-dimensional system state space, we apply a novel
distributed deep reinforcement learning-based approach by using a multi-agent
deep deterministic policy gradient algorithm. Experimental results demonstrate
the superior performance of the proposed TOBM scheme in terms of enhanced
system reward, improved offloading utility with lower blockchain mining
latency, and better system utility, compared to the existing cooperative and
non-cooperative schemes. The paper concludes with key technical challenges and
possible directions for future blockchain-based MEC research.

###### Index Terms:

Mobile edge computing, blockchain, deep reinforcement learning.

##  I Introduction

Recent advances in wireless Internet-of-Things (IoT) have promoted the
proliferation of mission-critical applications, e.g., augmented reality and
autonomous driving, which rely heavily on edge devices (EDs) to collect data
from IoT sensors to serve end users. To meet the ever-growing computation
demands of EDs, mobile edge computing (MEC) has been proposed as a promising
technique to improve the computation experience of EDs, by offloading
computationally-intensive IoT tasks to a nearby MEC server located at a base
station (BS) [1]. Multiple EDs can share computation and communication
resources of the BS to handle data tasks without device’s battery depletion.
Task offloading with MEC thus becomes a viable solution to satisfy various
EDs’ computation demands, thus enhancing the quality-of-experience (QoE) of
end users. Furthermore, to provide security in MEC systems, blockchain [2] has
emerged as a strong candidate due to its decentralization, immutability, and
traceability, which forms blockchain-based MEC (B-MEC) paradigms [3]. Also,
blockchain can build trusted B-MEC schemes by employing community verification
among network entities (e.g., EDs) via mining mechanisms such as Delegated
Proof of Stake (DPoS) [4] without requiring a central authority.

Figure 1: The proposed cooperative task offloading and block mining
architecture in the B-MEC system.

In this context, how to ensure high performance, e.g., system utility, for the
B-MEC system is a critical challenge. The task offloading process between EDs
and the MEC server consumes much energy and latency, while the operation of
blockchain results in delays in the offloading due to mining task execution,
which would degrade the overall system utility. Hence, it is paramount to
simultaneously consider both task offloading and blockchain mining via a joint
design and optimization solution, aiming to enhance the system utility of the
B-MEC system.

###  I-A Existing Solutions for Intelligent Performance Optimization in B-MEC

To achieve intelligent performance optimization in B-MEC systems, different
solutions have been proposed in the open literature. The authors in [3]
considered a blockchain-empowered computation offloading scheme where smart
devices can offload their computing tasks to the MEC server under the control
of blockchain mining for data integrity. Another work in [5] suggested an
online computation offloading approach for both data processing and mining
tasks in blockchain-empowered MEC with a deep reinforcement learning (DRL)
algorithm [6], [7]. An intelligent offloading framework with actor-critic DRL
was also proposed in [8], while the study in [9] focused on joint optimization
of computation offloading and resource allocation using a double-dueling deep
Q-network (DQN) for blockchain-enabled MEC systems. [10] developed a
computation offloading framework for blockchain-based IoT networks with a
multi-agent DRL algorithm [11].

Despite these research efforts, there are still several urgent issues to be
addressed:

  * •

Non-cooperative Offloading: Most of the existing B-MEC schemes use traditional
single-agent DRL algorithms [3, 5, 8] which exhibit critical design challenges
caused by diversified learning environments. Indeed, each agent only observes
its local information in the training without updating the policies of other
agents, which makes the learning environment nonstationary [12]. Moreover,
non-cooperative multi-agent DRL solutions [9] are not able to learn the mutual
offloading policy, which limits the resource utilization for task offloading.

  * •

High Blockchain Latency: The integration of blockchain in MEC potentially
results in unnecessary network latency due to block mining [8, 9] (i.e., block
verification and consensus) which can degrade the overall performance of the
B-MEC system.

  * •

Lack of Joint Offloading and Mining Design: In most current B-MEC schemes [8,
9, 10], the design and optimization of task offloading and blockchain mining
are done separately, leading to a suboptimal performance. To improve the
overall performance of the B-MEC system, a joint design of task offloading and
blockchain mining is highly needed.

###  I-B Our Key Contributions

Motivated by the aforementioned limitations, we propose a novel cooperative
DRL solution for joint task offloading and blockchain mining (TOBM), aiming to
maximize the overall B-MEC system utility as a sum of the offloading utility
and the mining utility. Our main contributions are highlighted as follows:

  * •

We propose a novel cooperative TOBM scheme for B-MEC to enable a joint design
of task offloading and blockchain mining in Section II. To reduce the
blockchain network latency, we develop a new Proof-of-Reputation (PoR) mining
mechanism via a lightweight block verification solution.

  * •

We propose a novel cooperative DRL solution in Section III using a multi-agent
deep deterministic policy gradient (MA-DDPG) approach [13] to optimize the
system utility.

  * •

We implement simulations to verify the effectiveness of our proposal in
Section IV. We highlight the technical challenges in B-MEC research and
discuss several directions for future works in Section V.

##  II Blockchain-Empowered MEC System

###  II-A Overview of Network Architecture

We consider a cooperative TOBM architecture in a B-MEC system, as illustrated
in Fig. 1. An MEC server co-located at a BS provides computation services for
EDs. We assume that each ED has an IoT data task to be executed locally or
offloaded to the MEC server. Furthermore, each ED participates in the block
mining by using a PoR consensus mechanism. The key network components of the
B-MEC system are described as follows.

  * •

IoT Sensors: IoT sensors such as cameras, smart meters, and wearables are
responsible for sensing physical environments and generating data which need
to be processed to serve end users. IoT sensors also act as lightweight
blockchain nodes to transmit data to EDs.

  * •

Edge Devices: Each ED such as a laptop or a powerful smartphone manages a
group of IoT sensors under its coverage. Based on the QoE requirements, EDs
can use their computational capability to process data tasks locally or
offload to a nearby MEC server via wireless links. EDs also work as miners to
perform block consensus where IoT sensors’ users vote to select representative
EDs for mining.

  * •

MEC Server: In our considered B-MEC system, there is a single MEC sever to
handle computationally extensive data tasks offloaded from EDs. By analyzing
the task profile such as task sizes, channel conditions, and available
resources, EDs can make offloading decisions so that the MEC server allocates
its resources to execute data tasks under QoE requirements.

  * •

Blockchain: A blockchain network is deployed over the MEC system where each ED
acts as a blockchain miner [9]. In this paper, we pay attention to a PoR
mining design to solve blockchain latency issues. The proposed PoR scheme
allows EDs to join the block mining with mining utility enhancement which
helps improve the overall performance of the B-MEC system.

###  II-B Task Offloading Model

We consider a B-MEC system with the set of EDs and the available sub-channels
of the BS denoted by $\mathcal{N}$ and $\mathcal{K}$, respectively. It is
assumed that each ED $n\in\mathcal{N}$ has an IoT data task including input
data and required CPU workload to be executed locally or offloaded to the BS
via one of the sub-channels $k\in\mathcal{K}$. Here, the offloading policy is
scheduled by a binary variable, which equals 1 (offloading to the MEC server)
or 0 (local execution). Each ED $n$ makes offloading decisions based on three
main factors: task data size, channel condition, and transmit power level.
Moreover, each ED needs to allocate portion of its computation resource to
execute the task locally. Accordingly, we define four policies to schedule the
offloading process of each ED, including offloading decision, channel
allocation, transmit power, and computation resource allocation.

We here formulate an offloading utility function from the QoE perspective,
which is characterized by the task computation time (including local time and
offloading time) and energy consumption (including local energy and offloading
energy). We define a QoE-aware offloading utility function $J_{n}^{off}$ to
provide a trade-off between the time and energy consumption of the task
offloading compared with the local execution at each ED. The offloading
utility function reflects the offloading improvement in QoE over the local
execution. If the offloading computation cost is lower than the local
execution cost, the user utility can be positive, implying the user’s QoE
improvement. However, if offloading too many tasks, EDs may suffer from higher
latency due to the traffic congestion, which reduces the user’s QoE. As a
result, the user offloading utility can be negative.

###  II-C Blockchain Mining Protocol

Figure 2: The proposed PoR consensus in our B-MEC system.

(a) The proposed MA-DDPG architecture.

(b) The MA-DDPG training procedure.

Figure 3: The design of learning framework for the TOBM system.

In the B-MEC system, a crucial component is blockchain consensus that aims to
mine the blocks of transactions (i.e., IoT data records) and add them to the
blockchain. In traditional consensus mechanisms, e.g., DPoS [4],[9], each
miner node must implement a repeated verification process across the miner
network, which results in unnecessary blockchain latency. Therefore, we here
propose a new PoR consensus to solve mining latency issues, including two main
parts: miner node formulation and block verification, as illustrated in Fig.
2.

####  II-C1 Miner Node Formulation

In our B-MEC system, IoT sensors’ users participate in the delegate selection
process to vote the mining candidates among EDs. In this regard, each IoT user
votes for its preferred ED with the most reputation based on its mining
latency. Specifically, an ED that exhibits a lower mining latency will have a
better reputation via a predefined mining utility function. Based on the
calculated reputation score, each IoT user votes for ED candidates based on
their reputation ranking. The top EDs with highest reputation scores are
selected to become miners to perform consensus. During its time slot of the
consensus process, each miner acts as a block manager which is responsible for
performing block generation, verification, and aggregating blocks after being
verified.

####  II-C2 Lightweight Block Verification

The block manager first produces an unverified block that contains
transactions collected by EDs in a given time. Then, the manager broadcasts
this block to all other miners within the miner network for verification.
Different from the traditional DPoS scheme which relies on a repeated
verification process among miners, here we implement a lightweight
verification solution. That is, a miner only verifies once with another node
during the consensus process, which significantly reduces the verification
latency. Specifically, the block manager first divides the block into a set of
equal transaction parts that are assigned to each miner within the miner group
along with a unique random number. Next, the miner chooses to associate with
one of the miners within its group to implement the verification for its
assigned transaction part by allocating its CPU resource. If 51% of miners
respond with positive verification, and the sum of random numbers calculated
by all miners is equal to a pre-defined number, the block manager accepts the
verified block and adds it to blockchain. In summary, the mining procedure
includes four stages: (1) transmitting unverified block from the block manager
to the EMs, (2) verifying the local block at each EM, (3) sharing the
verification result among two EMs, and (4) transmitting the verification
result back to the manager.

To this end, we build a mining utility function to characterize the mining
efficiency of the proposed scheme. Motivated by [14], we characterize the
mining utility $J_{n}^{mine}$ of each ED $n$ via an exponential function where
the mining utility is inversely proportional to the mining latency.
Accordingly, an ED that exhibits a lower mining latency has a better mining
utility with respect to a CPU resource allocation policy.

##  III Proposed Cooperative DRL Solution for System Utility Optimization

###  III-A System Utility Formulation

Here, we formulate the system utility for the proposed TOBM scheme by taking
both offloading utility and mining utility into account. As explained
previously, the offloading utility $J_{n}^{off}$ reflects the efficiency of
task offloading over local execution from the QoE perspective. Meanwhile, the
mining utility $J_{n}^{mine}$ reflects the efficiency of mining blocks in the
B-MEC network via the mining latency metric. Therefore, our key objective is
to maximize the total system utility, i.e., $J$, as the sum of the offloading
utility $J_{n}^{off}$ and the mining utility $J_{n}^{mine}$ of all EDs
$n\in\mathcal{N}$, with respect to the offloading policies including
offloading decision, channel allocation, transmit power, computation resource
allocation, and the mining policy with CPU resource allocation.

To apply DRL to the formulated TOBM problem, we need to convert the objective
function from a system utility maximization problem to a reward maximization
problem. To do this, we formulate the proposed problem using a multi-agent
version of the Markov decision process, also known as a Markov game. This is
represented by a tuple of agent set $\mathcal{N}$, state set $\mathcal{S}$,
action set $\mathcal{A}$, and observation set $\mathcal{O}$ of all agents
[11]. In fact, at each time slot, an MD does not always have access to all
states of the environment, but only observes certain states, which is called
observations. While states are complete and detailed information that are
relevant to the current task (e.g., which specific wireless channel is
occupied by a certain MD for offloading), observations are general information
received by an MD (e.g., how many wireless channels are occupied).

Each ED $n$ is considered as an intelligent agent to learn its optimal policy
by observing the local environment formed by the cooperation of EDs and the
MEC server, as shown in Fig. 3(a). We assume that the considered collaborative
task offloading and block mining scheme operates on discrete-time horizon with
each time slot $t$ equal and non-overlapping, and the communication parameters
remain unchanged during each time slot. Now, we define each item in the tuple
at each time slot $t$ as follows:

  * •

State: The environment state $\mathcal{S}(t)$ at time slot $t$ in the proposed
TOBM scheme includes five components: task state, channel state, power state,
resource state, and transaction state. Here, the task state is defined as the
matrix of input data and required CPU workload of all EDs. The channel state
is defined via a matrix of channel condition variables of all BSs at EDs,
where each variable equals 1 (occupied channel) or 0 (available channel). The
power state consists of transmit power levels of EDs in each sub-channel.
Moreover, the resource state contains the states of available computation
resource for data task execution, and transaction state includes transaction
data size.

  * •

Action: By observing the environment states, each ED takes an action according
to the offloading decision, channel selection, transmit power selection,
computation resource allocation, and CPU resource allocation, to complete task
execution and block mining at each time slot $t$. Therefore, the action space
of each ED is the combination of the above action sets. Accordingly, the
action space $\mathcal{A}(t)$ of the cooperative game can be defined as a
matrix of action sets of all agents.

  * •

System Reward Function: The system reward at one time slot $t$ is the sum of
the rewards of all EDs. The objective of our formulated TOBM problem is to
maximize the overall system utility $J$ as the sum of the offloading utility
and the mining utility. Therefore, we define $J$ as our system reward
function.

(a) Average system rewards with different algorithms.

(b) Average offloading utility with different numbers of EDs.

(c) Comparison of block verification latency.

Figure 4: Evaluation of system reward, task offloading, and blockchain
performance.

###  III-B Proposed Cooperative DRL Algorithm

In the literature, most existing schemes have focused on a conventional
single-agent [3, 5, 8] or an independent multi-agent [11] setting; however,
these solutions are unable to obtain the cooperative policies of EDs because
of nonstationary and partially observable environments. Indeed, when the
policies of other agents change due to computation mode preference, the ED’s
observation can be changed (nonstationary) Moreover, in independent multi-
agent learning schemes, an agent only has the local information and cannot
know the updates from other agents due to non-collaboration. This makes the
agents’ learning algorithm hard to ensure stability and convergence [12].
Thus, we propose a novel cooperative multi-agent DRL scheme using MA-DDPG [13]
for our proposed TOBM scheme. The key reason behind the adaption of DDPG to
optimize the system utility is that with this scheme, the actor can directly
map states to actions instead of outputting the probability distribution
across a discrete action space like DQN, which greatly reduces action sampling
complexity. Moreover, given the stochasticity of the policy, there exists high
variance of the obtained system reward between different training episodes,
where DDPG can come as an efficient solution, by enabling off-policy learning
via the joint use of behavior network and target network.

The MA-DDPG algorithm employs a deep neural network (DNN) as the non-linear
approximator to obtain the optimal policies for agents. Each agent updates its
parameters to obtain a optimal policy for maximizing its utility. MA-DDPG is a
policy gradient-based off-policy actor-critic method where each agent
comprises the actor to make decisions over time slots with a behavior network
and the critic to evaluate the behavior of the actor, which helps improve its
performance. Specifically, given an episode sample from the memory buffer, the
actor at each agent updates the behavior network by computing its gradient
based on a centralized action-value Q-function. Moreover, the critic updates
the behavior Q-function for the state-action pair of the actor network by
minimizing the loss function with the inputs including both local agent’s
observation and the observations of all other agents. The cooperation of
multi-agents helps improve robustness against malicious attacks since this
allows multiple agents to monitor the shared policy update via observations
[15].

To reduce the computational complexity caused by online training at EDs and
solve the nonstationary issues from the concurrently learning process of all
EDs, we adopt a centralized learning and decentralized execution solution. In
the centralized training step, the information of state-action of all EDs is
aggregated by the MEC server to train the DRL model, where each agent can
achieve the global view of the learning environment to obtain the observations
of other agents for building the collaborative offloading policy. After
training at the MEC server, the learned parameters (i.e. neural weights of
DNNs of the actor and critic) are downloaded to each ED to execute the model
for decision making based on its own locally observed information.
Specifically, given the downloaded neural weights, each agent can easily
compute the policy via its DNN with system utility used as the objective
function, and then sample an action. Subsequently, each agent executes the
sampled action, i.e. making decision to offload the data or not, in the
defined TOBM environment given its states to obtain a reward. The proposed
algorithm is illustrated in Fig. 3(b).

##  IV Performance Evaluation

In this section, we conduct numerical simulations to evaluate the performance
of the proposed TOBM scheme in a B-MEC system, employing the widely used
Shanghai Telecom dataset111http://www.sguangwang.com/dataset/telecom.zip.
Here, we consider an MEC network with an MEC server and a maximum of 500
mobile phones as EDs distributed over a 1 km $\times$ 1 km area in Shanghai
city. The number of channel sub-bands of the BS is set to 30, and each ED has
task CPU workloads of [0.8-1.5] Gcyles and transmit power range of [0-24] dBm.
A DNN structure with three hidden layers (64, 32 and 32 neurons) [9] is
employed with the Adam optimizer for learning simulation. To prove the
advantages of the proposed cooperative MA-DDPG scheme, we compare its
performance with the state-of-the-art non-cooperative schemes, including DDPG,
actor-critic [8] and DQN [11].

Evaluation of Training Performance: Fig. 4(a) shows the learning curves of the
average system reward with the increase of learning episodes for the B-MEC
system with 50 EDs. It is clear that our MA-DDPG scheme is more robust and
yields the best performance in terms of average system reward, compared to
baseline schemes. This is because the proposed scheme allows EDs to learn
mutually the cooperative offloading policy which helps reduce the channel
congestion and user interference, and enhance computation resource efficiency.
Meanwhile, in the DQN and actor-critic schemes, EDs greedily access the
wireless channel spectrum to maximize their own utility without collaboration,
which increases the possibility of channel collision and thus results in
higher offloading latency. Also, DDPG scheme still remains a non-stationary
learning issue and its average reward is lower than that of the MA-DDPG
scheme.

(a) Comparison of system utility with non-cooperative schemes.

(b) Comparison of system utility with cooperative schemes.

Figure 5: Evaluation of overall system utility performance.

Evaluation of Task Offloading and Blockchain Performance: Fig. 4(b) indicates
the performance of the average offloading utility versus different numbers of
EDs. As can be seen when the number of EDs is small, the average offloading
utility increases with the number of EDs because in this case, the MEC system
can support sufficient spectrum and computing resources for handling all tasks
of EDs. However, when exceeding a certain threshold (i.e., 60 EDs), the
offloading utility decreases because the higher the number of offloaded EDs,
the higher the competition of resource usage (i.e., channel spectrum). This in
turn increases the offloading latency, and thus degrades the overall
offloading utility. Nevertheless, our MA-DDPG scheme still achieves the best
utility performance due to its collaborative offloading policies among EDs
compared to other schemes with selfish learning.

Next, we evaluate our proposed PoR consensus scheme and compare it with the
traditional DPoS scheme [9] via the verification block latency metric. We set
up 10 transactions per block and vary the numbers of mining nodes from 2 to
200. As shown in Fig. 4(c), our proposed PoR scheme requires significantly
less time for mining blocks compared to DPoS, due to the optimized block
verification procedure. Although the block verification latency increases with
increasing the number of miners, our scheme still achieves much better
performance than DPoS, which verifies the effectiveness of our lightweight
blockchain consensus design.

Evaluation of the Overall System Utility Performance: We evaluate the
performance of our proposed TOBM scheme in terms of the overall system utility
as the sum of offloading utility and mining utility. The performances of our
cooperative TOBM scheme with our PoR mining design and other non-cooperative
schemes with PoR and DPoS mining are illustrated in Fig. 5(a). Unsurprisingly,
our cooperative scheme with a PoR mining design achieves the best overall
system utility. The reasons for this observation are two-fold. First, our
offloading scheme with a cooperative MA-DDPG algorithm outperforms other non-
cooperative offloading schemes in terms of a better offloading utility, as
evidenced in Fig. 4(a). Second, our PoR design yields a lower mining latency
which consequently improves the mining utility. Moreover, due to better mining
utility, our PoR design contributes to better overall system utilities in each
non-cooperative offloading scheme, compared to the use of DPoS design.

We compare the system utility performance of our proposed TOBM scheme with the
cooperative scheme without mining design [8] and the cooperative scheme with
DPoS design [9]. As shown in Fig. 5(b), our TOBM scheme with PoR design
achieves higher system utility than the cooperative scheme with DPoS design,
thanks to the better mining utility of our proposed PoR framework. Moreover,
compared with our approach, the cooperative scheme in [8] has the lowest
system utility due to the lack of mining design. This simulation result also
reveals that a joint design of offloading and mining is of paramount
importance to improving the overall system performance in B-MEC systems.

(a) Comparison of mining utility with cooperative schemes.

(b) Comparison of blockchain throughput with mining schemes.

Figure 6: Evaluation of blockchain performance.

We compare the mining utility of our scheme with other cooperative schemes in
Fig. 6(a). Due to a lower mining latency achieved by the proposed consensus
design, our scheme yields a better utility compared with its counterparts. The
traditional approach without mining design has the highest mining latency,
resulting in the lowest utility. Moreover, we investigate the throughput in
Fig. 6(b) which is defined as the number of successful transactions per
second. We set five transactions per block and make offloading requests
ranging from 10 to 200. Compared with DPoS and Proof-of-Work (PoW) schemes,
the fast block verification rate of our scheme significantly enhances the
throughput, before its performance decreases when the number of requests is
higher than 120 since the system cannot handle excessive requests under this
configuration.

##  V Research Challenges and Future Perspectives

###  V-A User Mobility

In realistic B-MEC systems, EDs can move with high speed in wireless networks
(e.g., in on-vehicle applications). This makes their location highly dynamic
which has direct impacts on offloading decision making. For example, an ED is
likely to execute locally if it moves out of BS coverage, which makes the
offloading design ineffective. Therefore, the user mobility needs to be
considered in the offloading formulation in future B-MEC systems, where
dynamic characteristics such as velocity and channel conditions should be
taken into account to come up with a robust mobility-aware offloading policy.

###  V-B Data Privacy

In this work, the DRL training performed at the MEC server potentially raises
data privacy leakage due to the data exchange during the training. Federated
learning [2] can be an attractive solution to perform collaborative training,
where only trained parameters are shared with the MEC server while actual data
and user information are stored at local EDs for privacy enhancement. In the
B-MEC context, for example, each ED can run a DRL function to learn its
offloading and mining policy based on its own local observation. Then, the
participating EDs can communicate with the MEC server for DRL model
aggregation (e.g., model averaging) to create a new global DRL model without
sharing their private data.

###  V-C Incentive Issues

In practice, how to encourage EDs, which serve as both offloading and mining
nodes, to join computation and data mining for the long term is a critical
challenge for B-MEC systems. Although blockchain is able to incentivize EDs
via coin payment based on their mining effort, it is still not enough to
compensate for the energy resources consumed for computation task offloading.
Thus, it is important to jointly consider incentives, offloading and mining in
B-MEC system optimization. A possible direction is to design a smart contract-
inspired incentive mechanism to accelerate the data offloading and block
mining. Another interesting area is to jointly optimize utility with respect
to monetary benefits and resource usage of offloading and mining.

##  VI Conclusions

This article proposed the novel concept of TOBM to assist B-MEC systems. A
joint design of offloading and mining was considered, where a PoR consensus
mechanism was proposed. Then, a cooperative DRL approach was proposed to solve
the TOBM problem, showing a huge system utility improvement over the existing
cooperative and non-cooperative schemes. Finally, we highlighted the key
research challenges and promising directions for future B-MEC research.

## References

  * [1] Y. Dai, D. Xu, S. Maharjan, Z. Chen, Q. He, and Y. Zhang, “Blockchain and Deep Reinforcement Learning Empowered Intelligent 5G Beyond,” _IEEE Network_ , vol. 33, no. 3, pp. 10–17, May 2019.
  * [2] H. Yao, T. Mai, J. Wang, Z. Ji, C. Jiang, and Y. Qian, “Resource Trading in Blockchain-based Industrial Internet of Things,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 6, pp. 3602–3609, 2019\.
  * [3] X. Xu, X. Zhang, H. Gao, Y. Xue, L. Qi, and W. Dou, “BeCome: Blockchain-Enabled Computation Offloading for IoT in Mobile Edge Computing,” _IEEE Trans. Industr. Infor._ , vol. 16, no. 6, pp. 4187–4195, Jun. 2020.
  * [4] J. Kang, Z. Xiong, D. Niyato, D. Ye, D. I. Kim, and J. Zhao, “Toward Secure Blockchain-Enabled Internet of Vehicles: Optimizing Consensus Management Using Reputation and Contract Theory,” _IEEE Trans. Veh. Tech._ , vol. 68, no. 3, pp. 2906–2920, Mar. 2019.
  * [5] X. Qiu, L. Liu, W. Chen, Z. Hong, and Z. Zheng, “Online Deep Reinforcement Learning for Computation Offloading in Blockchain-Empowered Mobile Edge Computing,” _IEEE Trans. Veh. Tech._ , vol. 68, no. 8, pp. 8050–8062, Aug. 2019.
  * [6] J. Wang, C. Jiang, H. Zhang, Y. Ren, K.-C. Chen, and L. Hanzo, “Thirty Years of Machine Learning: The Road to Pareto-optimal Wireless Networks,” _IEEE Communications Surveys and Tutorials_ , vol. 22, no. 3, pp. 1472–1514, 2020.
  * [7] L. Xiao, Y. Ding, D. Jiang, J. Huang, D. Wang, J. Li, and H. V. Poor, “A Reinforcement Learning and Blockchain-based Trust Mechanism for Edge Networks,” _IEEE Transactions on Communications_ , vol. 68, no. 9, pp. 5460–5470, 2020.
  * [8] J. Feng, F. R. Yu, Q. Pei, X. Chu, J. Du, and L. Zhu, “Cooperative Computation Offloading and Resource Allocation for Blockchain-Enabled Mobile-Edge Computing: A Deep Reinforcement Learning Approach,” _IEEE Inter. Things J._ , vol. 7, no. 7, pp. 6214–6228, Jul. 2020.
  * [9] F. Guo, F. R. Yu, H. Zhang, H. Ji, M. Liu, and V. C. Leung, “Adaptive Resource Allocation in Future Wireless Networks With Blockchain and Mobile Edge Computing,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 3, pp. 1689–1703, Mar. 2020.
  * [10] Z. Li, M. Xu, J. Nie, J. Kang, W. Chen, and S. Xie, “NOMA-Enabled Cooperative Computation Offloading for Blockchain-Empowered Internet of Things: A Learning Approach,” _IEEE Inter. Things J._ , pp. 1–1, Aug. 2020.
  * [11] J. Heydari, V. Ganapathy, and M. Shah, “Dynamic Task Offloading in Multi-Agent Mobile Edge Computing Networks,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019, pp. 1–6.
  * [12] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments,” in _Proc. Advances in Neural Infor. Process Sys._ , 2017, pp. 6379–6390.
  * [13] D. Kwon, J. Jeon, S. Park, J. Kim, and S. Cho, “Multi-Agent DDPG-based Deep Learning for Smart Ocean Federated Learning IoT Networks,” _IEEE Inter. Things J._ , pp. 1–1, Apr. 2020.
  * [14] Y. Dai, D. Xu, K. Zhang, S. Maharjan, and Y. Zhang, “Deep Reinforcement Learning and Permissioned Blockchain for Content Caching in Vehicular Edge Computing and Networks,” _IEEE Trans. Veh. Tech._ , vol. 69, no. 4, pp. 4312–4324, Apr. 2020.
  * [15] W. Xue, W. Qiu, B. An, Z. Rabinovich, S. Obraztsova, and C. K. Yeo, “Mis-spoke or Mis-lead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning,” Jan. 2022, arXiv: 2108.03803.

Dinh C. Nguyen is a Postdoctoral Associate at the School of Electrical and
Computer Engineering, Purdue University, USA. He obtained the Ph.D. degree at
Deakin University, Australia in 2022. His research focuses on federated
learning, blockchain, and edge computing. He is an Associate Editor of the
IEEE Open Journal of the Communications Society.
---
Van-Dinh Nguyen received Ph.D. degree in electronic engineering from Soongsil
University, South Korea 2018. He is currently a Research Associate with the
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University
of Luxembourg. His current research activity is focused on the mathematical
modeling of 5G cellular networks and machine learning for wireless
communications. Dr. Nguyen received several best conference paper awards, IEEE
Communications Letters Exemplary Editor 2019 and 2021.
---
Ming Ding is currently a Senior Research Scientist with the CSIRO Data61,
Sydney, NSW, Australia. His research interests include information technology,
data privacy and security, machine learning and AI. He has authored over 100
articles in IEEE journals and conferences. He is an Editor of the IEEE
Transactions on Wireless Communications and the IEEE Wireless Communications
Letters.
---
Symeon Chatzinotas is Full Professor/Chief Scientist I and Co-Head of the
SIGCOM Research Group at SnT, University of Luxembourg. He was a co-recipient
of the 2014 IEEE Distinguished Contributions to Satellite Communications
Award, the CROWNCOM 2015 Best Paper Award and the 2018 EURASIC JWCN Best Paper
Award. He has (co-)authored more than 400 technical papers in refereed
international journals, conferences and scientific books. He is currently in
the editorial board of the IEEE Open Journal of Vehicular Technology and the
International Journal of Satellite Communications and Networking.
---
Pubudu N. Pathirana is a full Professor and the Director of Networked Sensing
and Control group at the School of Engineering, Deakin University, Geelong,
Australia. He was a visiting professor at Yale University in 2009. His current
research interests include Bio-Medical assistive device design,
mobile/wireless networks, and rehabilitation robotics.
---
Aruna Seneviratne is currently a Foundation Professor of telecommunications
with the University of New South Wales, Australia, where he holds the
Mahanakorn Chair of telecommunications. His current research interests are in
physical analytics: technologies that enable applications to interact
intelligently and securely with their environment in real time.
---
Octavia A Dobre is a Professor and Research Chair at Memorial University,
Canada. Her research interests include technologies for beyond 5G, as well as
optical and underwater communications. She published over 300 referred papers
in these areas. Dr. Dobre serves as the Editor-in-Chief (EiC) of the IEEE Open
Journal of the Communications Society. She was the EiC of the IEEE
Communications Letters, a senior editor and an editor with prestigious
journals, as well as General Chair and Technical Co-Chair of flagship
conferences in her area of expertise. She is Fellow of the IEEE and the
Engineering Institute of Canada.
---
Albert Y. Zomaya is Peter Nicol Russell Chair Professor of Computer Science
and Director of the Centre for Distributed and High Performance Computing at
Sydney University. He has published more than 600 scientific papers and is an
author, co-author, or editor of more than 30 books. He is the Editor-in-Chief
of the ACM Computing Surveys and served in the past as the Editor-in-Chief of
the IEEE Transactions on Computers and the IEEE Transactions on Sustainable
Computing. He is a Fellow of the AAAS, the IEEE, the IET (UK), and was
inducted in 2022 into the Australian Academy of Science.
---