11institutetext: Tsinghua University 22institutetext: Zhejiang Future
Technology Institute (Jiaxing)
22email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# A Novel Convolutional Neural Network Architecture with a Continuous Symmetry

Yao Liu 11 Hang Shao 22 Bing Bai 11

###### Abstract

This paper introduces a new Convolutional Neural Network (ConvNet)
architecture inspired by a class of partial differential equations (PDEs)
called quasi-linear hyperbolic systems. With comparable performance on the
image classification task, it allows for the modification of the weights via a
continuous group of symmetry. This is a significant shift from traditional
models where the architecture and weights are essentially fixed. We wish to
promote the (internal) symmetry as a new desirable property for a neural
network, and to draw attention to the PDE perspective in analyzing and
interpreting ConvNets in the broader Deep Learning community.

###### Keywords:

Convolutional Neural Networks Partial Differential Equations Continuous
Symmetry

##  1 Introduction

With the tremendous success of Deep Learning in diverse fields from computer
vision [14] to natural language processing [28], the model invariably acts as
a black box of numerical computation [2], with the architecture and the
weights largely fixed, i.e., they can not be modified without changing the
output (for a fixed input), except by permuting neurons or units from the same
layer. One would say that the symmetry of the neural network, the set of
transformations that do not affect the model’s prediction on any input, is

$\operatorname{Sym}(\mathrm{model})=\prod_{i}S_{n_{i}}\,,$

the product of symmetric groups on $n_{i}$ “letters,” where $n_{i}$ is the
number of _interchangeable_ neurons in the $i$-th layer.

Although quite large as a group, it does not permit us to modify the model in
any meaningful way. In the case of Convolutional Neural Networks (ConvNets),
the channels are essentially fixed and frozen in place, due to the presence of
coordinate-wise activation functions [31] (such as ReLU), which arguably build
certain semantic contents in the channels [21], thus mixing them would destroy
the model. The _nonlinear_ activation unit is generally thought to be an
essential component for the neural network to fit arbitrary _nonlinear_
functions.

Figure 1: Schematic of a single block of our ConvNet architecture based on Eq.
(3), to replace the bottleneck block of ResNet50. The trapezoidal shapes
represent the increase/decrease in the number of channels. The corresponding
components of the equation are color-coded.

With inspiration and guidance from partial differential equations (PDEs),
specifically _first-order quasi-linear hyperbolic systems_ [1], we introduce a
new architecture of ConvNet with a different type of nonlinearity, which
allows us to remove the activation functions without degrading performance. As
a result, the new architecture admits a _continuous_ group of symmetry (i.e.,
a Lie group, as opposed to a discrete group) that allows mixing of the
channels; in one version, it is the full _general linear_ (GL) _group_ , the
set of all invertible $n_{i}\times n_{i}$ matrices:

$\operatorname{Sym}(\mathrm{model})=\prod_{i}GL(n_{i},\mathbb{R})\,.$

With a judicious choice of such transformations, one may alter the weights so
that the connections become more sparse, resulting in a smaller model (a kind
of lossless pruning). Since the group is continuous, one might use the method
of gradient descent to search for it. In addition, it may also lead to a
better understanding of the inner workings of the neural network, much like
how matrix diagonalization leads to the decoupling of a system of (linear)
differential equations into different “modes”, which are easier to interpret.

We primarily present the simplest version of our model based on ResNet50,
illustrated in Fig. 1 alongside the corresponding PDE. The nonlinearity is at
the element-wise multiplication of the two branches (3x3 and 1x1 convs), and
we apply activation functions _only_ at the end of 4 of the 16 blocks. See
§5.1 for details.

This is a preliminary report on our new architecture, and the relevant parts
of partial differential equations behind it. It is our hope that the research
community builds upon and analyzes the properties of this new architecture,
and to take the PDE perspective seriously in designing, analyzing, or simply
describing _all_ aspects of ConvNets. Moreover, given that the Transformer
architecture [28] also involves a similar kind of nonlinearity apart from
softmax and activation functions, it could potentially be made to admit a
continuous symmetry as well.

##  2 Related Work

The link with differential equations has been recognized, and well exploited
[11, 20, 24, 4, 13], soon after the introduction of ResNet [14] by K. He _et
al._ in 2015, if not known implicitly before; see also [25] for a more recent
analysis. With a few exceptions [20, 24], most discussions do not make an
emphasis on _partial_ differential equations, and to the best of our
knowledge, little activity has been devoted to _designing_ new architecture
from this perspective. Even though a PDE can be regarded as an ODE in which
the state space is “infinite-dimensional”, or upon discretization, a finite
but large system with interactions only between neighboring “pixels,” we find
the PDE perspective, specifically of hyperbolic systems, more illuminating and
_fruitful_ (albeit limited to ConvNets), and deserves more attention and
further study in the broader Deep Learning community.

A related but distinct field of research is using Deep Learning methods to
solve various PDEs of interests to physicists, applied mathematicians, and
engineers. We shall only mention two pioneering works that have attracted the
most attention: Fourier Neural Operators (FNO) [19] and Physics-informed
Neural Networks (PINN) [23].

Symmetry and equivariance often appear in the theoretical discussions of
ConvNets and Graph Neural Networks (GNN) [8, 3], though it is worth pointing
out the distinction from our usage: More often, one says that a neural network
has a (translational or permutation) _symmetry_ , or is _equivariant_ or
_invariant_ (under translations or permutations), if when we transform the
input in a certain way, the output is also transformed accordingly, or does
not change at all; the model itself remains fixed. In our scenario, we are
directly transforming the weights of the model, which incidentally does not
need to be trained. Nevertheless, much of our work involves finding a good
enough model that achieves comparable performance on standard training sets.
(It shall be apparent that, as with conventional ConvNets, our model is also
equivariant under translations.)

One type of operation, known under the term structural reparametrization [10],
can claim to modify the model after training. However, it can only merge
consecutive layers or operations that are both linear; the basic example is
conv followed by batchnorm. As such, it is better regarded as a trick in
training: for whatever reason, it is better to train with a deeper and more
complicated architecture than is necessary for the model, and is fundamentally
different from the kind of symmetry that our model has.

##  3 Designing ConvNets from the PDE perspective

Given that ResNet is numerically solving a particular system of PDEs,

$\displaystyle\frac{\partial u_{i}}{\partial t}\,=$
$\displaystyle\,\sigma\biggl{(}\sum_{j}L_{ij}u_{j}\biggr{)}\,,\qquad\qquad
i=1,\ldots,n$ $\displaystyle L_{ij}\,:=$
$\displaystyle\,\alpha_{ij}\frac{\partial}{\partial
x}+\beta_{ij}\frac{\partial}{\partial
y}+\gamma_{ij}\frac{\partial^{2}}{\partial x\partial y}+\cdots$

of $n$ unknowns $u_{i}\equiv u_{i}(x,y,t)$, with initial condition at $t=0$,
in which the coefficients $\alpha_{ij},\ldots$ are the parameters to be
learned (for background, see Appendix), it is natural to take inspiration from
other PDEs as found in mathematics and physics, and to see what new ConvNet
architecture would come out. Here are some natural changes that one could
make:

  * $\bullet$

Make the coefficients _variables_ (of $x$ and $y$), e.g., simply as linear or
polynomial functions. The equation would still be linear, but now the space of
PDEs would include this special equation: ($n=1$)

$\frac{\partial u}{\partial t}=-y\,\frac{\partial u}{\partial
x}+x\,\frac{\partial u}{\partial y}\,,$

which is solved by simply _rotating_ the initial data $f(x,y)$ by angle $t$
(around the origin):

$u(x,y,t)=f(x\cos t-y\sin t,x\sin t+y\cos t)\,,$

as can be readily verified. It is reasonable to expect that such a variation
on ResNet would allow the model to make _rotations_ and _dilations_ — in
addition to translations — on the input image.

  * $\bullet$

The standard “zero padding” of conv layer seems to correspond to the so-called
_Dirichlet_ boundary condition: the values of $u$ on the boundary are
prescribed for all $t>0$. On closer inspection, it is slightly different.
Furthermore, one could also experiment with other boundary conditions; the
other natural one in physics is the _Neumann_ condition, that the _normal
derivative_ of $u$ on the boundary is prescribed. The different conditions
have the effect that the signals would “bounce back” off the boundary
differently.

  * $\bullet$

In a typical PDE, the matrix of coefficients is constant or slowly varying
with time $t$, while in neural networks the weights from different layers are
initialized independently, drawn from a (normal) distribution. One could try
to force the weights from neighboring layers to correlate, either by weight-
sharing or by introducing a term in the loss function that penalizes large
variations between layers.

Having experimented with some of these ideas on small datasets, we did not
find a specific variation that yields convincing results on the full ImageNet.
We then looked into ways that the coefficients may depend on $u$ itself, which
makes the equation nonlinear (apart from the activation functions). It may be
viewed as a kind of “dynamic kernel,” but we draw inspiration from a class of
PDEs called _quasi-linear hyperbolic systems_ , which may be the simplest,
well-studied nonlinear systems in which the number of equations can be
arbitrary.

In two spatial and one time dimensions, a first-order quasi-linear system is
(typically) of the form

$\frac{\partial u_{i}}{\partial t}=\sum_{j}\mathcal{A}_{ij}(u)\frac{\partial
u_{j}}{\partial x}+\sum_{j}\mathcal{B}_{ij}(u)\frac{\partial u_{j}}{\partial
y}\,,$ (1)

where the coefficient matrices may depend on $u$ (but not derivatives of $u$),
and it is hyperbolic111The designation may sound cryptic. It originates from
the classic wave equation, which has some semblance in form with the equation
of a hyperbola or hyperboloid; see Appendix §A.3. To avoid any confusion, it
is _not_ related to hyperbolic plane/space/geometry that has made into machine
learning if any linear combination of $\mathcal{A}$ and $\mathcal{B}$ is
diagonalizable with only _real_ eigenvalues (e.g., $\mathcal{A}$ and
$\mathcal{B}$ are symmetric) for _any_ value of $u$. Leaving aside the latter
condition, the simplest example is to make each entry a linear function of
$u$:

$\mathcal{A}_{ij}(u)=\sum_{k}\mathcal{A}_{ijk}u_{k}\,,$ (2)

and similarly for $\mathcal{B}$. By dimension count, such a tensor would be
very large (for large $n$), and it would deviate too much from typical
ConvNets. Instead, we shall restrict to

$\frac{\partial u_{i}}{\partial
t}=\sum_{j}A_{ij}\sum_{k}C_{jk}u_{k}\frac{\partial u_{j}}{\partial
x}+\sum_{j}B_{ij}\sum_{k}D_{jk}u_{k}\frac{\partial u_{j}}{\partial y}\,,$ (3)

which is straightforward to turn into a ConvNet (see §5.1 and Fig. 1 for
details), and the number of parameters is kept at a reasonable level. Since
nonlinearity is already built-in, we thought it would not be necessary to add
activation functions, at least not at every turn; and much to our surprise,
the model trains just as well, if not better. With this simple change, the
model now has a _continuous_ symmetry, from mixing of the channels, that is
not present in conventional ConvNets with coordinate-wise activation functions
at every conv layer, and we believe it is a more significant contribution than
matching or breaking the state of the art (SOTA). It is likely that, with
enough compute, ingenuity, and perhaps techniques from Neural Architecture
Search, variations of this architecture could compete with the best image
models of comparable size. (We have not tried to incorporate the modifications
listed earlier, as they are all linear, and we wish to explore the new
nonlinearity on its own.)

It is observed that, once we remove all the activation functions, or use the
standard ReLU, the training is prone to breaking down: it would fail at a
particular epoch, with one or more samples (either from the training or
validation sets) causing the network to output NaN, and the model could not
recover from it. To mitigate this, we add activation functions such as
hardtanh that clip off large values, only once every few blocks. It is also
observed that resuming training with a smaller learning rate may get around
the “bad regions” of the parameter space. More analyses are needed to
determine the precise nature and cause of this phenomenon (it might be related
to the formation of “shock waves” in nonlinear hyperbolic equations [1]), and
perhaps other ways to circumvent it.

We provide here the details of the activation functions that we experimented
with. In standard PyTorch [22], nn.Hardtanh is implemented as

$\operatorname{hardtanh}(x):=\begin{cases}\mathrm{max\\_val}&\text{if
}x>\mathrm{max\\_val}\\\ \mathrm{min\\_val}&\text{if }x<\mathrm{min\\_val}\\\
x&\mathrm{otherwise}\,.\end{cases}$

We typically use $\pm 1$ as the clip-off values. We also introduce two multi-
dimensional variants that we call “hardball” and “softball,” which appear to
give better performance. Hardball is so defined that it takes a vector
$\mathbf{x}\in\mathbb{R}^{n}$ and maps it into the ball of radius $R$,

$\operatorname{hardball}(\mathbf{x}):=\begin{cases}\mathbf{x}&\text{if
}|\mathbf{x}|<R\\\ R\mathbf{x}/|\mathbf{x}|&\text{if }|\mathbf{x}|\geq
R\,,\end{cases}$

where $|\mathbf{x}|$ is the Euclidean norm. We set $R$ to be the square root
of $n$ (the number of channels), though other choices may be better. Softball
is a soft version,

$\operatorname{softball}(\mathbf{x}):=\frac{\mathbf{x}}{\sqrt{1+|\mathbf{x}|^{2}/R^{2}}}\,,$

and they are both spherically symmetric. (One may perhaps regard them as
normalization layers rather than activation functions.)

##  4 Symmetry of the model

The symmetry of our model would depend on the specific implementation, and may
be more complicated than one would naively expect. We shall first consider it
on the level of the PDE.

With a change of coordinates $\tilde{u}_{i}=\sum_{j}T_{ij}u_{j}$ for an
invertible matrix $T$, the general equation (1) with (2)

$\frac{\partial u_{i}}{\partial
t}=\sum_{j,k}\mathcal{A}_{ijk}u_{k}\frac{\partial u_{j}}{\partial
x}+\sum_{j,k}\mathcal{B}_{ijk}u_{k}\frac{\partial u_{j}}{\partial y}$

would transform _only_ in the coefficient tensors $\mathcal{A}_{ijk}$ and
$\mathcal{B}_{ijk}$. Indeed,

(For clarity, we omit the second half involving $\mathcal{B}$ and
$\frac{\partial}{\partial y}$.)

$\displaystyle\frac{\partial\tilde{u}_{i}}{\partial t}$
$\displaystyle=\sum_{j}T_{ij}\frac{\partial u_{j}}{\partial t}$
$\displaystyle=\sum_{j}T_{ij}\sum_{k,l}\mathcal{A}_{jkl}u_{l}\frac{\partial
u_{k}}{\partial x}$
$\displaystyle=\sum_{j}T_{ij}\sum_{k,l}\mathcal{A}_{jkl}\sum_{m}T^{-1}_{lm}\tilde{u}_{m}\sum_{r}T^{-1}_{kr}\frac{\partial\tilde{u}_{r}}{\partial
x}$
$\displaystyle=\sum_{m,r}\Biggl{(}\underbrace{\sum_{j,k,l}T_{ij}\mathcal{A}_{jkl}T^{-1}_{lm}T^{-1}_{kr}}_{\begin{subarray}{c}\tilde{\mathcal{A}}_{irm}\end{subarray}}\Biggr{)}\tilde{u}_{m}\frac{\partial\tilde{u}_{r}}{\partial
x}\,.$

We note in passing that similar calculations are commonplace in _classical_
differential geometry when making a change of coordinates _on the base space_
; here, we are making a change of coordinates on the “dependent” variables.
From a more abstract point of view, this is the induced representation of
$GL(V)$ on the tensor product $V\otimes V^{*}\otimes
V^{*}\cong\operatorname{Hom}(V\otimes V,\,V)$ for a vector space
$V\cong\mathbb{R}^{n}$.

On the level of the neural network, we only need to make sure that the
$T^{-1}$ comes from the previous layer, i.e., it is the inverse of the $T$
that appears in transforming the previous block (if different). With such a
transformation at each block, we find that the overall symmetry of the model
is

$\operatorname{Sym}(\text{model})=\prod_{i}G_{i}\,,$

with

$G_{i}=\begin{cases}S_{n}&\sigma=\text{relu, or any element-wise activation
function}\\\ O(n)&\sigma=\text{hardball, softball, etc.}\\\
GL(n,\mathbb{R})&\sigma=\text{identity}\,.\end{cases}$

As noted before, this “fully connected” block would be too costly to train, if
we are to match ResNet in which the last stage uses as many as $n=512$
channels. One simple way to reduce the number of parameters is to make the
tensor “block diagonal”, and the transformation would only mix channels from
the same block. The bottleneck block of ResNet achieves this by shrinking the
number of channels before applying the 3x3 conv, but a similar approach would
introduce additional layers that shield the main operations that we would like
to transform.

If we are to take $\mathcal{A}_{ijk}$ to be of the special form as in Eq. (3),
i.e., as the product of two matrices $A_{ij}C_{jk}$ (no summation), then it is
not guaranteed that the transformed tensor would still factorize in the same
way, for a generic $T$. By simple dimension count, the set of tensors that are
factorizable in the prescribed way is a subvariety (submanifold) of dimension
at most $2n^{2}$, and one wishes to find a $T\in GL(n,\mathbb{R})$ that keeps
the resulting $\tilde{\mathcal{A}}_{ijk}$ on the subvariety. Given that the
dimension of $GL(n,\mathbb{R})$ is $n^{2}$ and that $2n^{2}+n^{2}\ll n^{3}$,
it is not _a priori_ obvious that such transformations exist, apart from
simple scalings and $S_{n}$, that are universal for all $\mathcal{A}_{ijk}$.

What one may hope for is that, for a specific $\mathcal{A}_{ijk}$ that
factors, we can find such a $T$. For example, if for some pair of indices
$j,j^{\prime}$, we have $A_{ij}C_{jk}=A_{ij^{\prime}}C_{j^{\prime}k}$ for all
$i,k$, then we can perform a rotation in the plane of the $j$ and $j^{\prime}$
directions:

$\begin{cases}\tilde{u}_{j}=\alpha u_{j}+\beta u_{j^{\prime}}\\\
\tilde{u}_{j^{\prime}}=\gamma u_{j}+\delta
u_{j^{\prime}}\end{cases}\qquad\alpha\delta-\beta\gamma\neq 0\,.$

Further investigation, either theoretical or numerical, may be needed to
answer this question satisfactorily. It may be the case that there exists a
symmetry that preserves the output _not_ for all inputs, but only those inputs
that are “similar” to the dataset (“in distribution”). It would be a weaker
form of symmetry, but no less useful in practice.

Lastly, it should be remarked that there is a trivial “symmetry” in the tensor
$\mathcal{A}_{ijk}$ in the last two indices, i.e., $\mathcal{A}_{ijk}$ and
$\mathcal{A}_{ikj}$ can be interchanged (so long as their sum is fixed). One
may regard this as a redundancy in the parameter space, for we can force
$\mathcal{A}_{ijk}=\mathcal{A}_{ikj}$, or $\mathcal{A}_{ijk}=0$ for $j<k$ (and
reduce the dimension roughly by half), and not due to mixing of the channels.
We have not exploited this in the present work.

##  5 Experimental Results

###  5.1 Details of the Architecture

How do we turn Eq. (3) into a ConvNet? We first make the differential
operators $\partial/\partial x$ and $\partial/\partial y$ into 3x3
convolutional kernels, but we allow the weights to be trainable instead of
fixed. Each incoming channel (each $u_{j}$) would split into 2 — or better, 4
— channels, and this is conveniently implemented in nn.Conv2d by setting
groups to equal the number of input channels, as in “depthwise convolution”
[5]. The matrices are simply 1x1 conv layers, with $A$ and $B$ stacked into
one, and $C$ and $D$ stacked into one. Batchnorm is applied after the 3x3 and
at the end. As with standard ResNet, the time derivative turns into the skip
connection, and we arrive at the architecture of a single block as illustrated
in Fig. 1. We do not enforce symmetry of these matrices to make the equation
hyperbolic in the technical sense. As a general rule, we need not strictly
follow the equation, but take the liberty in relaxing the weights whenever
convenient.

One novelty is to make the weights of the 3x3 conv shared across the groups,
which would make the claimed symmetry easier to implement. One may achieve
this by making an nn.Conv2d with 1 input channel and 4 output channels, and at
forward pass, we “repeat” the $4\times 1\times 3\times 3$ weights in the
zeroth dimension before acting on the input tensor. Most of our experiments
are with this “minimalist” 3x3 conv, except the ones marked “no ws” (no
weight-sharing) in Table 1.

It may be possible to implement this kind of “variable-coefficient
convolution” _natively_ , instead of using the existing torch.nn layer which
is tailored to conventional convolutions.

For the full design of the neural network, we simply take the classic ResNet50
with [3,4,6,3] as the numbers of blocks in the four stages. No activation
function is applied except once only in each stage (e.g., at each
downsampling), and we use nn.Hardtanh or our variants, hardball and softball
(see §3 for definitions), instead of ReLU, lest the training would fail
completely or the resulting model would not work as well.

###  5.2 Experiments

As is standard in computer vision since the 2012 Deep Learning revolution, we
have trained our model as an image classification task on the ImageNet
dataset. The ImageNet-1k contains 1000 classes of labeled images, and for the
sake of faster iterations, we primarily trained on a 100-class subset on a
single GPU, while maintaining the standard image size of 224x224.

We use the timm library of PyTorch image models [29] for best practices in
implementing ResNet and its training [30]. We took the official training
script for ResNeXt-50 (SGD, cosine learning rate, with a warmup of 5 epochs,
batch size of 192, etc.) except that the peak learning rate is set to 0.3
instead of 0.6, and the total number of epochs is set to 50. The results are
in Table 1, where we mainly record two ways of modifying the model: changing
only the activation function, and altering the placements of the conv layers
within the block, corresponding to modifying Eq. (3) into Eqs. (4)–(7):

$\displaystyle\frac{\partial u_{i}}{\partial t}$
$\displaystyle=\sum_{k}C_{ik}u_{k}\sum_{j}A_{ij}\frac{\partial u_{j}}{\partial
x}+\sum_{k}D_{ik}u_{k}\sum_{j}B_{ij}\frac{\partial u_{j}}{\partial y}$ (4)
$\displaystyle\frac{\partial u_{i}}{\partial t}$
$\displaystyle=\sum_{k}C_{ik}u_{k}\frac{\partial u_{i}}{\partial
x}+\sum_{k}D_{ik}u_{k}\frac{\partial u_{i}}{\partial y}$ (5)
$\displaystyle\frac{\partial u_{i}}{\partial t}$
$\displaystyle=\sum_{j}A_{ij}\frac{\partial}{\partial
x}\sum_{k}C_{jk}u_{j}u_{k}+\sum_{j}B_{ij}\frac{\partial}{\partial
y}\sum_{k}C_{jk}u_{j}u_{k}$ (6) $\displaystyle\frac{\partial u_{i}}{\partial
t}$ $\displaystyle=\sum_{j}A_{ij}\frac{\partial}{\partial
x}\sum_{k}C_{jk}u_{j}u_{k}+\sum_{j}B_{ij}\frac{\partial}{\partial
y}\sum_{k}D_{jk}u_{j}u_{k}$ (7)

Table 1: Performance on a 100-class subset of ImageNet-1k, trained for 50 epochs with identical training strategy. For our model, activation is applied either at the end of each block (@all), or only at downsampling (@ds). Inside the block, the number of channels increases by a factor of 4, except when indicated with “x6”. model | #parameters | top1-acc | activation
---|---|---|---
ResNet50 | 23.7M | 84.52 |
MobileNet_v3_large | 4.33M | 82.91 |
Eq. (3) | 8.61M | 82.06 | relu@all
Eq. (3) | 8.61M | 82.34 | hardtanh@ds
Eq. (3) | 8.61M | 83.50 | hardball@ds
Eq. (3) | 8.61M | 83.66 | softball@ds
Eq. (3) (no ws) | 8.73M | 84.24 | softball@ds
Eq. (3) (no ws, x6) | 13.0M | 84.58 | softball@ds
Eq. (4) | 5.70M | 81.88 | softball@ds
Eq. (5) | 4.26M | 78.64
Eq. (6) | 5.61M | 82.52
Eq. (7) (no ws, x6) | 13.0M | 84.96

Note that Eq. (5) only has each $u_{i}$ depending on its own derivatives, and
the model is thus smaller and limited in expressivity or capacity. Eqs. (6)
and (7) have the derivative acting on the product $u_{j}u_{k}$, and it is
often called a system of _conservation laws_. They can easily be rewritten in
the form of Eq. (3), and the difference in performance may be attributable
simply to the model size.

It is expected that, when going to the full ImageNet-1k, and allowing for
longer training and hyperparameter tuning, the best-performing model may be
different from the ones we found in Table 1.

We refrain from making assertions on _why_ — or _if_ — this class of PDEs is
superior as a base model for ConvNets, or _how_ the theory of PDE can provide
the ultimate answer to the _effectiveness_ of neural networks as universal
function approximators by means of gradient descent. Whatever mechanisms that
make ResNet work, also make our model work.

##  6 Conclusion

We present a new ConvNet architecture inspired by a class of PDEs called
quasi-linear hyperbolic systems, and with preliminary experiments, we found a
simple implementation that showed promising results. Even though it is known,
within small circles, the close connection between PDE and ConvNet, we made
the first architecture design directly based on a nonlinear PDE, and as a
result, we are able to remove most of the activation functions which are
generally regarded as indispensable. The new architecture admits a continuous
symmetry that could be exploited, hopefully in future works. We expect that
this work opens up a new direction in neural architectural design,
demonstrates the power of the PDE perspective for ConvNets, and opens the door
for other concepts and techniques in nonlinear PDE, both theoretical and
numerical, for improved understanding of neural networks.

## Acknowledgements

This work is supported in part by Provincial Key R&D Program of Zhejiang under
contract No. 2021C01016, in part by Young Elite Scientists Sponsorship Program
by CAST under contract No. 2022QNRC001.

## Addendum

It has come to our attention that the recent work [32], &c., are highly
relevant though largely complementary to ours, in terms of exploiting the kind
of continuous symmetry presented here. Specifically, our activation functions,
hardball and softball, are examples of what they call _radial_ activation
functions, and the symmetries are more aptly referred to as _symmetries in the
parameter space_.

Not surprisingly, the operation of element-wise multiplying two branches of
the input had been proposed independently by several authors: polynomial
networks or $\Pi$-Nets [7, 6] in which the output is a polynomial function of
the input, PDO-eConvs [15] that comes from the equivariance perspective, and
what is called the _gated linear unit_ (GLU) [9, 26] that has made its
appearance in recent language models, though not without antecedents in the
winters of neural networks. We apologize for our omissions in the published
version.

## References

  * [1] Alinhac, S.: Hyperbolic partial differential equations. Springer Science & Business Media (2009)
  * [2] Bai, B., Liang, J., Zhang, G., Li, H., Bai, K., Wang, F.: Why attentions may not be interpretable? In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. pp. 25–34 (2021)
  * [3] Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P.: Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021)
  * [4] Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. Advances in Neural Information Processing Systems 31 (2018)
  * [5] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 1251–1258 (2017)
  * [6] Chrysos, G.G., Moschoglou, S., Bouritsas, G., Deng, J., Panagakis, Y., Zafeiriou, S.: Deep polynomial neural networks. IEEE transactions on pattern analysis and machine intelligence 44(8), 4021–4034 (2021)
  * [7] Chrysos, G.G., Moschoglou, S., Bouritsas, G., Panagakis, Y., Deng, J., Zafeiriou, S.: P-nets: Deep polynomial neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7325–7335 (2020)
  * [8] Cohen, T., Welling, M.: Group equivariant convolutional networks. In: International Conference on Machine Learning. pp. 2990–2999. PMLR (2016)
  * [9] Dauphin, Y.N., Fan, A., Auli, M., Grangier, D.: Language modeling with gated convolutional networks. In: International conference on machine learning. pp. 933–941. PMLR (2017)
  * [10] Ding, X., Zhang, X., Han, J., Ding, G.: Diverse branch block: Building a convolution as an Inception-like unit. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10886–10895 (2021)
  * [11] E, W.: A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics 1(5), 1–11 (2017)
  * [12] E, W., Han, J., Jentzen, A.: Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning. Nonlinearity 35(1), 278 (2021)
  * [13] E, W., Han, J., Li, Q.: Dynamical systems and optimal control approach to deep learning. Mathematical Aspects of Deep Learning pp. 422–438 (2022)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [15] He, L., Chen, Y., Shen, Z., Yang, Y., Lin, Z.: Neural ePDOs: Spatially adaptive equivariant partial differential operator based networks. In: The Eleventh International Conference on Learning Representations (2022)
  * [16] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  * [17] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 4700–4708 (2017)
  * [18] Li, Q.: Dynamical systems and machine learning. Summer School, Peking University: Beijing, China (2020)
  * [19] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.: Fourier neural operator for parametric partial differential equations. In: International Conference on Learning Representations (2021)
  * [20] Long, Z., Lu, Y., Ma, X., Dong, B.: PDE-Net: Learning PDEs from data. In: International Conference on Machine Learning. pp. 3208–3216. PMLR (2018)
  * [21] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., Carter, S.: Zoom in: An introduction to circuits. Distill 5(3), e00024–001 (2020)
  * [22] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019)
  * [23] Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686–707 (2019)
  * [24] Ruthotto, L., Haber, E.: Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision 62, 352–364 (2020), available as arXiv preprint since 2018
  * [25] Sander, M., Ablin, P., Peyré, G.: Do residual neural networks discretize neural ordinary differential equations? Advances in Neural Information Processing Systems 35, 36520–36532 (2022)
  * [26] Shazeer, N.: GLU variants improve transformer. arXiv preprint arXiv:2002.05202 (2020)
  * [27] Strichartz, R.: A Guide to Distribution Theory and Fourier Transforms. Studies in Advanced Mathematics, CRC-Press (1994)
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in Neural Information Processing Systems 30 (2017)
  * [29] Wightman, R.: Pytorch image models. https://github.com/rwightman/pytorch-image-models (2019). https://doi.org/10.5281/zenodo.4414861
  * [30] Wightman, R., Touvron, H., Jégou, H.: ResNet strikes back: An improved training procedure in timm. In: NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future (2021)
  * [31] Zhang, H., Weng, T.W., Chen, P.Y., Hsieh, C.J., Daniel, L.: Efficient neural network robustness certification with general activation functions. Advances in Neural Information Processing Systems 31 (2018)
  * [32] Zhao, B., Ganev, I., Walters, R., Yu, R., Dehmamy, N.: Symmetries, flat minima, and the conserved quantities of gradient flow. In: International Conference on Learning Representations (2023), to appear

##  Appendix 0.A Appendix: A Crash Course on PDE

We shall give a short, gentle, yet (mostly) self-contained primer on the
elementary facts of PDEs that are most relevant to Deep Learning. There is no
original content, except for the overall scope, choice of presentation, and
certain points which may not be easy to find in one place. The reader who is
familiar may skip §A.1, with a glance at Footnote 5. The mathematical field of
PDE is staggeringly vast, and any account is by necessity limited, sometimes
with the same term (e.g., hyperbolicity) appearing to have very different
definitions in different accounts. As with any area of mathematics, it may
appear to the uninitiated more complicated than it actually is, with the level
of generality and confusing (even conflicting) notations that it operates in.
Luckily, we shall not need any of the heavy technical machineries; the key is
to form the bridge, and lots of engineering and experimentation with training
neural networks.

We shall take advantage of a small simplification: we will exclusively be
dealing with _two_ spatial dimensions ($x$ and $y$), for a ConvNet takes a 2D
image as input, and each subsequent layer (before the penultimate layer) has a
2D structure to it.

###  0.A.1 First example: the heat equation

It is instructive, perhaps even necessary, to begin with a consideration of
the _heat equation_ on, say, a two-dimensional metal plate:

$\frac{\partial u}{\partial t}=\mathrm{\Delta}u\,.$

Here $u$ is the temperature at a point in space _and_ time, i.e., $u\equiv
u(x,y,t)$, and $\mathrm{\Delta}$ is the Laplacian operator:

$\mathrm{\Delta}u:=\frac{\partial^{2}u}{\partial
x^{2}}+\frac{\partial^{2}u}{\partial y^{2}}\,.$

The $x$ and $y$ coordinates extend over the domain, say from $[0,1]$ for a
square-shaped plate, and $t$ goes from $0$ to $\infty$. The equation of the
unknown function $u$ is called a _partial differential equation_ , since it
involves the partial derivatives of $u$; and by a _solution_ we simply mean a
specific function $u$ that _solves_ or satisfies the equation, for all values
of $x,y$, and $t$.

The heat equation describes (or models) how temperature evolves: if you are at
the bottom of a valley, your temperature should go up, and if you are at the
top of a hill, your temperature should come down. It is found that, up to some
proportionality constant, this captures well how heat diffuses in the real
world.

As with _any_ partial differential equation, one would also need to impose
some _initial_ and/or _boundary conditions_. In this case, we specify the
temperature over the whole plate at time $t=0$, and at each point on the
boundary for all $t>0$; solving the equation _along with_ the appropriate
initial (and boundary) conditions is often called the Cauchy problem, or
Initial(-Boundary) Value Problem. Our physical intuition tells us that the
temperature is then determined inside for all $t>0$, and it is formalized
mathematically that the Cauchy problem is _well-posed_ (in the sense of J.
Hadamard): the _existence_ and _uniqueness_ of solution which depends
_continuously_ on the initial and boundary data.

Figure 2: The time-lapse of a solution to the heat equation, over a square
domain. The temperature is represented in the vertical direction and by the
coloring. Credits: Oleg Alexandrov via Wikimedia Commons.

How does one go about solving the heat equation? There are elaborate tools for
solving various, though rather limited, PDEs, starting with the Fourier series
(first introduced by J. Fourier precisely to solve the heat equation); but for
us it is more important to walk through the process of solving it
_numerically_. First, imagine we discretize the square plate into a grid, so
that the temperature $u_{i}$ at each (discrete) time $t_{i}$ is represented by
a 2D array of numbers. Starting with the initial data $u_{0}$ at $t_{0}=0$, we
can approximate its Laplacian by

$\displaystyle\mathrm{\Delta}_{h}u_{0}(x,y):=$
$\displaystyle\frac{u_{0}(x+h,y)-2u_{0}(x,y)+u_{0}(x-h,y)}{h^{2}}+\frac{u_{0}(x,y+h)-2u_{0}(x,y)+u_{0}(x,y-h)}{h^{2}}$
$\displaystyle=$
$\displaystyle\frac{u_{0}(x+h,y)+u_{0}(x-h,y)+u_{0}(x,y+h)+u_{0}(x,y-h)-4u_{0}(x,y)}{h^{2}}\,,$

which is used to find the temperature $u_{1}$ at time $t_{1}=\tau$:

$\frac{u_{1}-u_{0}}{\tau}\approx\left.\frac{\partial u}{\partial
t}\right|_{t=0}=\mathrm{\Delta}u_{0}\,.$

Iteratively, we may find the temperature $u_{i}$ at time $t_{i}$ from
$u_{i-1}$,333Note that the subscript in $u_{i}$ throughout the Appendix
represents the “time slice” at $t=t_{i}$, _not_ a component of the system of
equations as in the main paper. namely,

$u_{i}\approx u_{i-1}+\tau\mathrm{\Delta}_{h}u_{i-1}\,.$

More sophisticated numerical schemes may be used instead of this simplest
approach, which is usually called the (forward) Euler method.

One may have noticed that the whole process is very similar to a ResNet (with
a single channel throughout) with a single kernel:

$k=\frac{\tau}{h^{2}}\begin{bmatrix}0&1&0\\\ 1&-4&1\\\ 0&1&0\end{bmatrix}\,,$
(8)

and that the boundary condition is needed to compute the convolution $k*u_{i}$
along the boundary. One can hardly escape the conclusion that ResNet (i.e.,
the feedforward of ResNet) is simply a discretization of a PDE of the form

$\frac{\partial u}{\partial t}=\sigma(Lu)\,,$

where $L$ is some (linear combination of) differential operators

$L=\alpha\frac{\partial}{\partial x}+\beta\frac{\partial}{\partial
y}+\gamma\mathrm{\Delta}+\cdots\,,$

and $\sigma$ is the nonlinear activation function. Indeed,
$\frac{\partial}{\partial x}$ and $\frac{\partial}{\partial y}$ can be
realized by the so-called _Sobel operators_ , and the space of differential
operators of order $\leq 2$ is six-dimensional, well represented in the
9-dimensional space of kernels. If $u$ is a vector-valued function (for ResNet
with multiple channels), then $L$ would be a matrix of differential operators,
i.e., a square matrix where each entry takes the form above; more on systems
later. The coefficients $\alpha,\beta,\ldots$ may depend on $t$, to account
for the fact that the weights in a ConvNet differ from one layer to the next.

We shall summarize in the form of a dictionary, which hopefully suffices for
understanding Fig. 1 and the main thesis of the paper.

Table 2: Dictionary of terminologies between ConvNet and PDE555Many items in the table apply more broadly, between general neural networks and ODEs or dynamical systems. There are also notable omissions from this table, such as batchnorm and max/avg pooling.. ConvNet | PDE
---|---
input | initial condition
feed forward | solving the equation _numerically_
hidden layers/feature maps | solution at intermediate times
final layer | solution at final time
3x3 convolution | differential operator of order $\leq 2$
weights/parameters | coefficients
padding | boundary condition
groups | matrix is block diagonal
residual connection | time derivative

It is unfortunate that ConvNets historically evolved mainly without much
direct input from PDEs, resulting in some conflicts in terminology with common
mathematical usage (notably, block and group). It is likely that Y. LeCun was
well aware of the connection, yet he chose to present the first ConvNet using
convolutions, familiar as _filters_ in computer vision and image processing,
in lieu of differential operators.

Now, the obvious question is: What does the training of a neural network via
gradient descent (the _backpropagation_ algorithm) correspond to in
mathematics? For many applied fields that are governed or modeled by PDEs, one
often does not have the full information of the underlying equation or the
initial/boundary data, which one needs to _infer_ from the solutions that
_can_ be observed. The problem of finding the equation from its solutions is
generally known as the _inverse problem_ , and is a large field in and of
itself. We shall, however, briefly turn to a somewhat different topic, with
certain points of general interest.

###  0.A.2 Optimal control and the calculus of variations

A slight change of terminology (and a shift in goal) in certain applied fields
has given rise to Control Theory and Optimal Control, where the variables $u$
are sometimes called the _state variables_ , and the parameters that can be
tweaked (or learned) are called the _control variables_. One may see the
equation written with the explicit mention of the control variables $\theta$:

$\frac{\partial u}{\partial t}=F(u,\nabla u,\ldots;x,y,t;\theta)$

where one is supposed to optimize $\theta$ for a certain _loss function_ ,
possibly subject to additional constraints as the case may be. See [18] for an
introduction to Optimal Control theory in connection with Deep Learning.

In the case that the control variables form a continuum (that is,
$\theta:S\to\mathbb{R}$ is a function over some _continuous_ space $S$) and
the loss function takes the form of an integral

$\mathrm{Loss}(\theta)=\int_{S}\ell(\theta(s))\,ds\,,$

it is a classical _calculus of variations_ problem for which one may derive
the _exact_ equation (the Euler-Lagrange equation) that the optimal $\theta$
needs to satisfy, or there is an algorithm to approximate the minimum —
literally the method of gradient descent. The classic example is when
$\ell(\theta)=|\nabla\theta|^{2}$: the exact solution is given by Laplace’s
equation $\mathrm{\Delta}\theta=0$ on $S$ (with suitable boundary conditions),
and the approximation is nothing but solving the heat equation
$\frac{\partial\theta}{\partial t}=\mathrm{\Delta}\theta$ on $S$ — one may
refer to this as _gradient flow_ , in contrast to the discrete steps of
gradient descent.

More commonly, however, the loss function actually depends on the solution $u$
of the state equation (or the solution at the final time, as in ConvNets),
thus the dependence on $\theta$ is indirect or _implicit_. A loss term of
explicit $\theta$ dependence would correspond to the _regularization_ of the
neural network model.

Much has been expounded on the _curse of dimensionality_ by experts in applied
mathematics and scientific computing, and how neural networks seem to defy the
curse [12]. It might be helpful to take the view that, much like how calculus
provides an efficient way to optimize a function over a _large_ number of
points, the calculus of variations is a sort of simplification, at least on a
psychological level, on optimizing a function over a _large_ number of
dimensions:

Table 3: Finding the minimum of a function $f:X\to\mathbb{R}$, for various types of $X$. | $X=\\{x_{1},\ldots,x_{n}\\}$ | $X=\mathbb{R}^{n}$ | $X=\\{\theta:S\to\mathbb{R}\\}$
---|---|---|---
$X$ is | a finite set | a finite-dimensional | an infinite-dimensional
space or manifold | vector space
exact solution | comparison | calculus: | calculus of variations:
(pairwise) | $\nabla f=0$ | $\delta f=0$
approximation | – | gradient descent | successive approximation

###  0.A.3 The wave equation and hyperbolic systems

It can be argued that the standard ResNet is most similar to a (first-order)
hyperbolic system. Even though the concept of hyperbolic systems or PDEs may
sound esoteric, some of the most famous equations in physics fall in this
“category”: Maxwell and Dirac are linear, while Einstein and Navier-Stokes are
nonlinear examples that have been the driving forces behind a substantial
amount of current research in _both_ theoretical and numerical PDE.

We shall begin with the prototype of hyperbolic equations: the _wave equation_

$\frac{\partial^{2}u}{\partial t^{2}}=\mathrm{\Delta}u\,,$

which models the up-and-down movement of a two-dimensional rubber sheet or
membrane that is fixed along the boundary. To solve it, just imposing an
initial condition on $u$ is not enough; we also need to specify
$\frac{\partial u}{\partial t}$ at time $t=0$. Together, they give the “first-
order” approximation to $u_{1}$ at time $t_{1}=\tau$. To find $u_{2}$, we note
that

$\frac{u_{2}-2u_{1}+u_{0}}{\tau^{2}}\approx\left.\frac{\partial^{2}u}{\partial
t^{2}}\right|_{t=t_{1}}=\mathrm{\Delta}u_{1}\,,$

which gives $u_{2}$ in terms of $u_{1}$ and $u_{0}$, and so on for all
$u_{i}$:

$u_{i}\approx 2u_{i-1}-u_{i-2}+\tau^{2}\mathrm{\Delta}_{h}u_{i-1}\,.$

This would be a ResNet with two skip connections at each layer, or a kind of
DenseNet [17] (of a single channel throughout).

We can rewrite the wave equation as a _system_ of two equations:

$\left\\{\begin{aligned} \tfrac{\partial u}{\partial t}&=v\\\ \tfrac{\partial
v}{\partial
t}&=\mathrm{\Delta}u\end{aligned}\right.\qquad\text{or}\qquad\tfrac{\partial}{\partial
t}\begin{pmatrix}u\\\ v\end{pmatrix}=\begin{pmatrix}0&\phantom{.}1\\\
\mathrm{\Delta}&\phantom{.}0\end{pmatrix}\begin{pmatrix}u\\\
v\end{pmatrix}\,,$

which makes clear that we need to specify both $u$ and $v=\frac{\partial
u}{\partial t}$ at time $t=0$. Solving this system of PDEs numerically is
nothing but a ResNet with two channels, with the nontrivial 3x3 kernel given
by the same $k$ as in Eq. (8).

It turns out that we can rewrite the wave equation in yet another way: with
$w=\big{(}\tfrac{\partial u}{\partial x},\tfrac{\partial u}{\partial
y},\tfrac{\partial u}{\partial t}\big{)}^{T}$, one can easily verify that $w$
satisfies

$\frac{\partial w}{\partial
t}=\begin{pmatrix}0\phantom{.}&\phantom{.}0\phantom{.}&\phantom{.}1\\\
0\phantom{.}&0&\phantom{.}0\\\
1\phantom{.}&0&\phantom{.}0\end{pmatrix}\frac{\partial w}{\partial
x}+\begin{pmatrix}0\phantom{.}&\phantom{.}0\phantom{.}&\phantom{.}0\\\
0\phantom{.}&0&\phantom{.}1\\\
0\phantom{.}&1&\phantom{.}0\end{pmatrix}\frac{\partial w}{\partial y}\,,$

which is a system of _first-order_ equations (only the first-order derivatives
of $w$ appear) in which the coefficient matrices are _symmetric_. As such, it
is an example of a _first-order linear hyperbolic system_ :

$\frac{\partial w}{\partial t}=A\,\frac{\partial w}{\partial
x}+B\,\frac{\partial w}{\partial y}\,,$ (9)

where any linear combination of $A$ and $B$ is diagonalizable with only _real_
eigenvalues. Maxwell’s equations (in vacuum) and the Dirac equation (with zero
mass) are of this form, but in _three_ spatial dimensions; and it is this form
that leads to the generalization of _quasilinear_ hyperbolic systems of Eq.
(1), by allowing the matrices $A$ and $B$ to depend on $w$. It is also
referred to as a system of _conservation laws_ , especially when written in
_divergence form_ :

$\frac{\partial w}{\partial t}=\frac{\partial}{\partial
x}F(w)+\frac{\partial}{\partial y}G(w)$

for some nonlinear (vector-valued) functions $F$ and $G$, for which the
Jacobian matrices $A(w)=\big{(}\frac{\partial F}{\partial w}\big{)}$ and
$B(w)=\big{(}\frac{\partial G}{\partial w}\big{)}$ satisfy the diagonalization
condition above, for any value of $w$. The prototypical examples are the Euler
and Navier-Stokes equations in fluid mechanics.

The resemblance of the _linear_ case of Eq. (9) to existing ConvNet
architectures is in fact more pronounced than Table 2 suggests, even in the
absence of second-order derivatives: rewriting the equation of $u$ in terms of
$w$ corresponds to the “stem” that converts the RGB channels of the input into
as many as 64 channels, and the separation of the coefficient matrices from
the differential operators $\frac{\partial}{\partial x}$ and
$\frac{\partial}{\partial y}$ corresponds to the so-called _depthwise
separable convolution_ [16, 5]. It should also be remarked that diagonalizing
$A$ and $B$ would effect a change of variables on $w$, which offers a curious
interpretation of the 1x1 conv layers in the standard bottleneck block of
ResNet50.

###  0.A.4 General theory of linear PDEs

> This section may be read for any $d$ of spatial dimensions, and we shall
> employ $\vec{x}$ and $\vec{y}$ to denote points in $\mathbb{R}^{d}$.

There is a general and elegant theory of _linear_ PDEs, which is simpler in
the absence of boundary conditions. For a linear (evolution) equation,
$\frac{\partial u}{\partial t}=Lu$, (i.e., first-order in time) of _constant_
coefficients, the solution takes the form

$u(\vec{x},t)=\int_{\mathbb{R}^{d}}K(\vec{x}-\vec{y},t)u_{0}(\vec{y})\,d\vec{y}\,,$

where $K$ is a function666known variously as the _Green’s function_ , the
_propagator_ , or the _fundamental solution_ of the linear operator
$\frac{\partial}{\partial t}-L$. By definition, it satisfies
$(\frac{\partial}{\partial t}-L)K=\delta$, where $\delta$ is the Dirac delta
function, and $K$ may be thought of as the “inverse” to
$\frac{\partial}{\partial t}-L$., or a _generalized function_ (Schwartz
distribution), on $\mathbb{R}^{d+1}$, that depends only on the equation, not
the initial data. The integral may be viewed as the convolution of $u_{0}$
with kernel $K(\cdot,t)$, for each time $t$. If $u$ is vector-valued, then $K$
is a _matrix_ -valued function, or a matrix of convolution operators. While
for both the wave and the heat equations, the kernel $K$ can be written down
explicitly, for any $d$ (see, for example, [27]), we shall only note the
important difference: for the wave equation, $K(\vec{x},t)$ is zero outside
the cone777Such a region is known as the _support_ of the function or
distribution, and corresponds to the so-called _receptive field_ in ConvNet.
$|\vec{x}|\leq|t|$, while for the heat equation, $K(\vec{x},t)$ is always
nonzero (only defined for $t>0$), although vanishingly small as
$|\vec{x}|\to\infty$. In other words, the wave equation has _finite speeds of
propagation_ in all directions, while the heat equation has an _infinite_
speed — physically impossible, and should be taken as a mathematical
idealization. The distinction gives rise to two separate “classes” of PDEs:
the _hyperbolic_ and _parabolic_ equations, respectively.

Another characteristic difference is that, if the initial data is not smooth
(infinitely differentiable), the heat equation will _smooth out_ any kinks or
jumps immediately (see Fig. 2), while the wave equation preserves and spreads
the discontinuities for all time. (This, too, can be “read off” from the
kernel $K$.) A ConvNet is likely to involve both processes, even though we
have emphasized the hyperbolic aspect in this paper.

The general formula also offers a (somewhat) different interpretation of the
convolution operation in ConvNets that also applies to large kernels: instead
of being a finite-difference approximation to _differential_ operators, the
conv layers are _integral_ operators with these $K(\cdot,t)$ kernels, for
discrete values of $t$. What is being learned is not so much the underlying
equation, but the kernel $K$ that solves the equation.

If the coefficients in the operator $L$ are not constants, but _variables_
(i.e., functions of $\vec{x}$ and $t$), the equation is still _linear_ but the
kernel $K$ would _not_ be a function of a single point
$(\vec{x},t)\in\mathbb{R}^{d+1}$, but of _two points_ : $K\equiv
K(\vec{x},t,\vec{y},s)$, which quite literally encodes how much the solution
or the initial data at the point $(\vec{y},s)$ _contributes_ to the solution
at the point $(\vec{x},t)$. In the case that $L$ is constant-coefficients, the
kernel must then only depend on the difference $(\vec{x}-\vec{y},t-s)$ by a
simple argument with translation equivariance.

It is, after all, not a coincidence that convolutions and kernels feature so
prominently in the languages in both fields.

###  0.A.5 Nonlinear PDEs

Once we introduce nonlinearities into the equation or system, not only is
there rarely a closed formula for the solution, there also does not exist
anything close to a general theory, even just for hyperbolic type. Many
aspects of the solution need to be considered on a case-by-case basis (not
only on the equation, but also the function class to which the initial data
belongs):

  * –

local existence (for a short time)

  * –

global existence (for all time)

  * –

uniqueness

  * –

regularity (smoothness)

  * –

continuous dependence on initial/boundary data

  * –

stability under small perturbations

  * –

…

It is impossible to do justice to nonlinear PDEs in a few pages. One of the
most celebrated phenomena, not confined to PDEs but shared with other
nonlinear systems (of ODEs, or even discrete systems), is that the solution
may be _so_ sensitive to the initial data that one can _not_ make accurate
long-term predictions, even qualitatively. It has grown into whole fields of
research, under the names of _nonlinear dynamics_ , _chaos theory_ , or
_complex systems_ , though the parts that directly concern PDE are relatively
nascent and less known. Naturally, one would like to bring the large body of
work, both results and methods, to bear on the theory of neural networks, to
have something useful to say, i.e., more than just importing terms such as
“emergence.”

On the opposite end of the spectrum, there is the (very rare) phenomenon of
_integrable systems_ , exemplified by the KdV equation, that are characterized
by exact solutions and rich algebraic structures. Would it be sensible to view
the training of neural networks as finding the “integrable” ones within the
large sea of chaotic systems?


# The Ground-state Inter-plane Superconducting Coherence Length of
La1.875Sr0.125CuO4 Measured by a “Xiometer”

Itay Mangel<EMAIL_ADDRESS>Amit Keren
<EMAIL_ADDRESS>Department of Physics, Technion-Israel Institute of
Technology, Haifa, 3200003, Israel

###### Abstract

A long excitation coil piercing a superconducting (SC) ring is used to
generate ever increasing persistent current in the ring, until the current
destroys the order parameter. Given that the penetration depth $\lambda$ is
known, this experiment measures, hypothetically, the coherence length $\xi$,
hence the name ”Xiometer”. We examine various aspects of this theoretically
driven hypothesis by testing niobium rings with different dimensions, and by
comparing the results to the known values of $\xi$. We then apply the method
to two La1.875Sr0.125CuO4 rings at $T\rightarrow 0$. In one, the current flows
in the CuO2 planes hence it is set by $\xi_{ab}$. In the other, the current
must cross planes and is determined by $\xi_{c}$. We find that $\xi_{c}=1.3\pm
0.1$ nm, and $\xi_{ab}<2.3~{}$nm indicating that at low temperatures the
Cooper pairs are three dimensional.

In the world of superconductivity there are two important length scales: the
penetration depth $\lambda$, and the coherence length $\xi$. From an
application point of view, $\lambda^{-2}$ is a measure of superconducting
carrier density and determines the current carrying capabilities of a
superconductor, while $\xi$ sets the dimensions of Josephson junctions. There
is an arsenal of experimental methods to directly measure the superconducting
stiffness $\rho_{s}$ and find $\lambda$ via the relation
$\rho_{s}=\frac{1}{\mu_{0}\lambda^{2}}$ in different crystal orientations.
However, methods of measuring $\xi$ are limited. In the cuprates the coherence
length in the $ab$ plane ($\xi_{ab}$) was measured by: electron–boson spectral
density function [1] and vortex diameter via scanning tunneling microscopy
(STM) [2]. More commonly, one finds $H_{c2}$ near $T_{c}$ using one of various
methods: resistivity [3] vortex-Nernst effect [4], specific heat [5], or
thermal conductivity [6], extrapolating to $T=0$ using theories that are not
necessarily accurate over the whole temperature range, and uses the relation
$\xi=\sqrt{\Phi_{0}/2\pi H_{c2}}$ [3, 7]. All methods find $\xi_{ab}$ on the
scale of $1.5-3$ nm.

Measuring coherence length in the $c$ direction ($\xi_{c}$) in cuprates is
more difficult since it is smaller and because cleaving in a plane including
the $c$ axis is challenging, making scanning techniques nearly impossible.
$\xi_{c}$ is bound by $1.5$ nm, the thickness of a superconducting
Bi2Sr2CaCu2O8+x (Bi2212) monolayer [8]. Extrapolations from high temperatures
lead to $\xi\sim 1$ Å [7, 3]. The only measured value of $\xi_{c}=0.86$ nm at
$T\rightarrow 0$, as far as we know, is from an $H_{c2}$ of $250$ T obtained
using the electromagnetic flux compression method [9]. This number has been
questioned due to the transient nature of the magnetic field. Exact
determination of $\xi_{c}$ is becoming exceedingly important due to Josephson
junctions created by twisted Bi2212 crystals [10, 11, 12, 13, 14] showing
fractional Shapiro steps [15, 16], and as a challenge for cuprates theory.

A new approach for measuring $\xi$ was suggested, and a very simple analysis
formula was given, in Ref. [17]. We name this approach “Xiometer”. Here we
briefly present the approach, justify the formula intuitively, and test it on
Nb. Then, we apply it to $\xi$ measurements in La1.875Sr0.125CuO4 (LSCO-1/8).
We find that the $\xi_{c}=1.3\pm 0.1$ nm determined by the Xiometer at
$T\rightarrow 0$ is similar to the one obtained from $H_{c2}=250$ T of Ref.
[9] and calculation in Ref. [7]. The implication of this finding is that the
Cooper pairs are more spherical than previously thought.

Figure 1:  Experimental setup: (a) A niobium ring on a SC excitation coil
(photo), an illustration of the gradiometer, and the external coil that serves
as a shim. (b) Niobium rings with different inner radii (up), outer radii
(middle), and different height (down).

The Xiometer, like the Stiffnessometer [18], is based on a long, current
carrying, excitation coil (EC) piercing a superconducting (SC) ring and a
measurement of the ring’s magnetic moment. The measurement is done by moving
both ring and EC rigidly relative to a gradiometer type pick-up loops as
demonstrated in Fig. 1(a). The gradiometer is connected to a superconducting
quantum interference device (SQUID). The ring and EC are cooled to $T<T_{c}$,
only then the current in the EC is turned on, and the magnetic moment of the
ring is measured. The gauge invariant London equation states that

$\bm{j}=-\rho_{s}\left(\bm{A}_{tot}-\frac{\Phi_{0}}{2\pi}\bm{\nabla}\varphi\right)$
(1)

where $\bm{j}$ is the superconducting current density, $\bm{A}_{tot}$ is the
total vector potential, $\Phi_{0}$ is the flux quanta, and $\varphi$ is the
phase of the superconductor’s order parameter. $\bm{A}_{tot}$ is a combination
of the EC vector potential $\bm{A}_{ec}=\frac{\Phi_{ec}}{2\pi
r}\hat{\varphi}$, and the SC ring self induced vector potential $\bm{A}_{sc}$.

To minimize the kinetic energy after cooling, the superconductor sets its own
phase gradient to zero. Since this phase is quantized, turning the EC current
on, gently, conserve the phase. Therefore, at the start of a measurement, the
London equation $\bm{j}=-\rho_{s}\bm{A}_{tot}$ is valid. In principle, this
relation can be used to determine the stiffness and in this case the apparatus
works as a Stiffnessometer. However, for bulk crystals at low temperatures,
when $\lambda$ is mach smaller than all dimensions of the sample, the SC ring
exactly expels the applied flux, to keep $A_{tot}=0$ deep inside the ring,
regardless of $\lambda$. In this case, the apparatus cannot be used to
properly determine the stiffness, but only the break point of the London
equation, which is set by $\lambda\xi$ as we explain below. For crystals at
$T\ll T_{c}$ we use literature values of $\lambda$ and determine $\xi$. Hence,
the name Xiometer.

The EC is home made from a NbTi SC wire with: $8$ layers of $600$ windings
each ($4800$ in total), a wire diameter $0.106$ mm, core diameter $0.35$ mm,
outer diameter $1.95$ mm, coil length $60$ mm, and flux to current ratio of
$1.21\cdot 10^{-7}~{}Tm^{2}/A$. The apparatus is an add-on to a Cryogenics
S700X SQUID magnetometer. External magnetic fields can be canceled by an
external coil shown in Fig. 1(a), with a resolution of $5\times 10^{-7}$ T.
The gradiometer radius is $R_{pl}=13$ mm, its total height is $14.0$ mm, and
it is made of: two windings clockwise, four anticlockwise, and two clockwise
also depicted in Fig. 1(a). The set of Nb rings with different dimensions used
in the first part of this experiment is shown in Fig. 1(b).

A typical data set of the SQUID output voltage $V$ as a function of ring
position $z$, is depicted in Fig. 2 with and without a ring. In these
measurements the EC was allowed to transverse the gradiometer from one side to
the next. The peaks and valleys away from $z=0$ are due to the ends of the
coil moving through the different winding groups of the gradiometer. Without
the ring (red symbols) a moderately concave signal is observed around $z=0$.
This occurs when the center of the gradiometer and the center of the coil are
at the same height. In this situation, flux through the gradiometer due to the
EC barely changes, therefore the measurement is sensitive mostly to
$\bm{A}_{sc}$. With the ring, a new signal (blue symbols) appears around
$z=0$. The difference is the net SC ring’s signal (inset) and its amplitude is
proportional to the magnetic moment $m$ (or $A_{sc}$) of the ring. We note
that a linear base-line was subtracted from both data sets due to the EC
asymmetry (wires enter and exit from one side only). Linear base-line
subtraction is irrelevant for the data analysis (see below).

As the EC current increases, the signal from the EC traversing the gradiometer
overwhelms the ring’s signal, as is clear from Fig. 3. Therefore, we limit the
motion to a small region around $z=0$. Consequently it is impossible to detect
the bottom of the signal and evaluate its amplitude. However, it is clear that
the top of the peak at $z=0$ becomes sharper with increasing current.
Therefore, we use the second derivative of the SQUID’s output voltage
$V^{\prime\prime}_{0}=\frac{d^{2}V}{d^{2}z}(z=0)$ as a measure of $m$. This
method also eliminates the undesired linear contribution of the coil’s
asymmetry. The conversion from $V^{\prime\prime}_{0}$ to $m$ is explained
shortly.

Finally, to keep the leads, coil, and ring cooled, liquid helium is sprayed
via a diffuser from the bottom of the sample chamber on the EC and sample, and
pumped along the current leads all the way to the top of the cryostat just
before thicker leads are connected to the power supply. This way the EC
remains cold even when currents of more than $10$ A are applied.

Figure 2: Raw data. Main: The squid output voltage $V$ as a function of the
relative position $z$ of a Nb ring and coil to the gradiometer center. The EC
current is $10.0$ mA, $T=1.6$ K, and the ring dimensions are: $r_{in}=1.0$ mm,
$r_{out}=1.75$ mm, and $h=1.0$ mm. Red spheres is the EC signal without the
ring. Blue spheres is the combined signal of the ring and coil. Dashed black
line show a numerical fit used to determine the conversion factor between the
output voltage of the SQUID and the magnetic moment of a sample. Inset: The
subtraction of the two measurements giving the ring’s signal. The data
presented is after subtraction of a linear component. Figure 3: Raw data in a
limited scan length. SQUID output voltage as a function of the ring’s position
$z$ for different applied EC currents. For low currents ($0-4~{}A$) it is
possible to measure the full signal of the coil and the ring (see Fig. 2). For
higher currents the coil’s signal exceeds the detector’s dynamic range and a
measurement on a shorter range is needed. Consequently, the edges of the coil
are not observed and the signal amplitude cannot be determined. Therefore, the
second derivative of the signal $V^{\prime\prime}_{0}$ is used to determine
the moment as explained in the text.

The measured samples are $99.9\%$ pure Nb rings, and two LSCO-$1/8$ single
crystal rings, grown, using a traveling solvent floating zone technique and
cut with a laser cutter. Laue x-ray diffraction is used to identify the
orientation of the crystals and the two rings are cut in different
orientations of the CuO2 planes relative to ring’s symmetry axis. In one ring
the CuO2 planes are parallel to the symmetry axis. In the other ring the
planes are perpendicular to this axis. We address the two rings as $a$ and $c$
rings, respectively. In the $c$ ring, current can flow around the ring on CuO2
planes without crossing planes. In order to flow around the $a$ ring current
must cross CuO2 planes.

A detailed derivation of the Xiometer analysis theory can be found in Ref.
[17]. Here we provide a back of the envelope calculation that gives the same
answer up to a numerical factor. Since, as mentioned before, deep in the
superconductor there are no currents (and no fields), $A_{tot}=0$. Therefore,
the magnetic flux of the EC, $\Phi_{ec}$, is perfectly matched by the magnetic
flux from the super-current $\Phi_{sc}$ in the ring, namely,
$\Phi_{sc}=-\Phi_{ec}$. For low $\Phi_{ec}$ we assume that the supper current
density $j$ is uniform along a cylinder of width $\lambda$ attached to the
inner rim of the ring at $r_{in}$ as in Fig. 4(a). The magnetic flux of such a
current is $\Phi_{sc}=\mu_{0}\lambda j\pi r_{in}^{2}$. When $\Phi_{ec}$
increases, $j$ will also increases until it reaches the critical current
density $j_{c}$ of the SC. When this happens the order parameter is destroyed
next to the inner rim, and the current has to retreat from the inner rim to an
effective radius $r_{eff}$ demonstrated in Fig. 4(b). This process continues
until $r_{eff}=r_{out}$ as shown in Fig. 4(c). At this situation vortices
start entering the sample, and the applied flux is named critical flux; it is
given by $\Phi_{c}=\mu_{0}\lambda j_{c}\pi r_{out}^{2}$. Using the definition
$j=e^{*}nv=\frac{m^{*}v}{\mu_{0}\lambda^{2}e^{*}}$ where $n$, $v$, $e^{*}$,
and $m^{*}$ are the carrier’s density, velocity, charge, and mass
respectively, and the relations of the critical momentum
$m^{*}v_{c}=\frac{\hbar}{\sqrt{3}\xi}$ [19], and the flux quanta
$\Phi_{0}=2\pi\hbar/e^{*}$ one finds

$\frac{\Phi_{c}}{\Phi_{0}}=\frac{r_{out}^{2}}{2\sqrt{\alpha}\lambda\xi}$ (2)

where $\alpha=3$. In the exact derivation [17] $\alpha=2$.

It should be pointed out that the SC produce a field in the volume where the
order parameter is destroyed. One might wonder if this field penetrates as
vortices into the SC when it is of Type-II. It was found in Ref. [20], using a
scanning SQUID, that in ultra thin film that show vortices due to sporadic
magnetic field, the current in the coil does not add new vortices. This is not
surprising since there is no pressure from twisted field lines bypassing the
sample to penetrate into the sample. In our case the field lines are strait
and in the center of the sample. They are in the most convenient place to be
and do not need to penetrate the sample as vortices.

Figure 4: Schematic description of the superconductor screening currents
evolution for increasing flux in the EC. Each sketch shows cross sections of
the EC and a cylindrical ring. Yellow and gray regions represent the normal
and SC states, respectively. The current flows in the red region. The flux
current relations are given at the top of each panel. (a) Low flux in the EC
resulting in screening current along the inner rim of the cylinder. (b)
Stronger flux in the EC forces the screening current to move outwards while
the inner region of the cylinder becomes normal. (c) The critical flux is
reached once the screening current reaches the outer radius of the cylinder
and its bulk is no longer SC. Figure 5: Critical current of different rings.
Nb ring’s magnetic moment $m$ obtained using Eq. 4 as function of current in
the excitation coil. The measurements are arranged in three sets. In each set
only one parameter of the rings is changing. (a) Only the inner radius of the
rings varies, (b) Only the outer radius of the rings differs, (c) Only the
height of the rings changes. The experiment was done more then once with each
ring for statistical purpose. The signal grow linearly with the EC current
until it drops at some critical current $I_{c}$. The drop is due to a phase
slip. The critical current varies between runs of the same ring due to thermal
instability.

The output voltage of the SQUID is related to the flux through the gradiometer
by $V=K\Phi$ where $K$ is a conversion factor. When a sample with magnetic
moment $m$ is located at height $z$ from the center of a single pickup-loop
with radius $R_{pl}$, it’s flux through the loop is
$\Phi=\frac{\mu_{0}R_{pl}^{2}m}{2(R_{pl}^{2}+z^{2})^{3/2}}$. To calibrate $K$
we measured our coil for which we calculate the magnetic moment as a bundle of
current loops with moment $m=\pi r^{2}I$ each, where $r$ is the loop radius.
The voltage output in this case is:

$\begin{split}&V(z)=\frac{\mu_{0}R_{pl}^{2}\pi
IK}{2}\sum_{i}r_{i}^{2}\left[\frac{2}{(R_{pl}^{2}+(z+z_{i}-\Delta_{pl})^{2})^{3/2}}+\right.\\\
&\left.\frac{2}{(R_{pl}^{2}+(z+z_{i}+\Delta_{pl})^{2})^{3/2}}-\frac{4}{(R_{pl}^{2}+(z+z_{i})^{2})^{3/2}}\right]\end{split}$
(3)

where $z$ is the distance between the center of the coil and the center of the
gradiometer, $\Delta_{pl}$ is the difference between the gradiometer’s
bundles, $r_{i}$ is the radius of the $i$’th layer, and $z_{i}$ the height of
the $i$’th loop. By fitting Eq. 3 to a measurement of the coil with current of
$10.0~{}mA$, as shown in Fig. 2, we find the conversion factor to be
$K=63.035$ V/Tmm2. This procedure gives

$m=4.95*10^{-10}\cdot V^{\prime\prime}_{0}$ (4)

where $V^{\prime\prime}_{0}$ is in units of mV/mm2, and $m$ in units of Am2.

To test Eq. 2 we measure a set of Nb rings with different inner and outer
radii, and different height. Fig. 5 presents $m$ as a function of the applied
EC current $I$. $m(I)$ is linear for low currents. At some high current
$I_{c}$, a jump in $m$ is observed indicating that the critical flux in the
coil $\Phi_{c}$ is reached, and that a phase slip has taken place. This
process was done more than once for each ring for better statistics.

The results of the experiment with the Nb rings is separated to three sets. In
each set only one parameter is changing: inner radius $r_{in}$ panel (a),
outer radius $r_{out}$ panel (b), and the ring’s height $h$ panel (c).
Variation of $r_{in}$ is limited because of the coil. Nevertheless, it does
not seem to impact the $\Phi_{c}$. Variation of $r_{out}$ has noticeable and
systematic influence on the critical flux as expected from Eq. 2. Finally,
between the smallest $h=0.5$ mm and all other values of $h$ a variation in
$\Phi_{c}$ is detected. This is not expected from Eq. 2. We ascribe this
exception to the fact that Eq. 2 is derived in the limit of a tall, cylinder
like, ring and the $h=0.5$ mm is not in this limit. A summary of $I_{c}$ as a
function of $L$ ($L=r_{in},r_{out},h$) is depicted in the inset of Fig. 6
where it is clear that $r_{out}$ is the most important parameter. The error
bars are statistical.

In Fig. 6 we present the $r_{out}$ dependence of $I_{c}$ on a full scale
including the origin. When taking $\lambda$ and $\xi$ of Nb as $38$ nm and
$39$ nm respectively from Ref. [21] and applying those in Eq. 2 (black line),
we find reasonable agreement between the measurements and theory. When fitting
Eq. 2 to the data (red line), we find $\lambda\xi=1267\pm 32.5$ nm2 while the
literature value is $1482$ nm2.

Figure 6: Critical current as function of different Nb ring sets. Main, the
set with changing outer radius. The black line follows Eq. 2 for $\lambda=38$
nm and $\xi=39$ nm [21]. The red line is a parabolic fit to the data points
(through the origin). The fit parameter mounts to $\lambda\xi=1267\pm 32.5$
nm2. Inset: black squares - rings with different inner radius, red circles -
rings with varying outer radius, blue triangles - rings with changing height.
The dashed lines are a guide to the eye. Figure 7: Critical flux determination
of LSCO-$1/8$. $m(I)$ for two rings. The insets show a pictures of the rings
and the orientation of the planes. Their parameters are $r^{a}_{in}=1.05$ mm,
$r^{a}_{out}=1.42$ mm, and $h_{a}=1.02$ mm, and $r^{c}_{in}=1.05$ mm,
$r^{c}_{out}=1.35$ mm, and $h^{c}=0.89$ mm. $I_{c}$ of the a-ring is $2.2$ A
and $I_{c}$ of the c-ring is higher than the maximum available current of
$12.4$ A.

Having established Eq. 2 we use it to measure coherence length of the two
LSCO-$1/8$ rings shown in the insets of Fig. 7 with orientation and planes
illustrated in the main panel. Previously, it was found that the persistent
current critical temperature for the c-ring is higher than for the a-ring
[22]. Figure. 7 shows $m$ at $T=1.6$ K as function of EC current for both the
a-, and c-rings. At low currents, $m(I)$ is linear for both rings. But at
about $2.2$ A, the a-ring moment $m_{a}$ has a break point where its behaviour
changes. The c-ring moment $m_{c}(I)$ stays linear all the way up to our
maximum current of $12.4$ A. We interpret this break point as the critical
flux where vortices start entering the sample. [floatfix] At the experiment’s
temperature, the penetration depths of LSCO with $x=0.125$ are:
$\lambda_{c}=4500$ nm [23], and $\lambda_{ab}=350$ nm [22]. The outer radii of
the a- and c-rings are $r^{a}_{out}=1.42$ mm, and $r^{c}_{out}=1.35$ mm.
Following Eq. 2 with $\lambda_{c}$ we find $\xi_{c}=1.3\pm 0.1$ nm. The
assumption here is that the bottleneck for current in the a-ring is the flow
between planes (c-direction), and that the order parameter is destroyed first
on the planes perpendicular to the flow. We can also place an upper bound on
$\xi_{ab}<2.3$ nm using $\lambda_{ab}$.

The Ginzburg–Landau $\xi$ at $T=0$, which we measure with the Xiometer, is
related to the Cooper pair size $\xi_{0}$ by a factor of 0.74 [19]. $\xi_{0}$,
in turn, is set by $\hbar v_{F}/\Delta$ where $v_{F}$ is the Fermi velocity
and delta is the superconducting gap. However, in the cuprates $\Delta$ varies
along the Fermi surface, $v_{F}$ in the $c$ direction is not known, and there
is no theory that can be used to extract more fundamental properties or be
contrasted with our findings. A derivation of such a theory could be useful.

To summarize, the relation between the dimensions of a superconducting ring,
pierced by a long coil, and the critical flux in the coil, is tested. It is
demonstrated that if the ring’s height is similar to or bigger than its radii,
only the outer radius of the ring is relevant, and the critical flux depends
quadratically on this radius. Using this observation we measured $\xi_{ab}$
and $\xi_{c}$ of LSCO-$1/8$ at $T=1.6$ K. Despite the cuprates being very
unisotropic systems, we found that $\xi_{c}$ is similar to the literature
value of $\xi_{ab}$, indicating a 3D Cooper pair.

## Acknowledgements

This research was funded by the Israeli Science foundation grant number
3875/21.

## References

  * Hwang [2021] J. Hwang, Scientific Reports — 11, 11668 (2021).
  * Pan  _et al._ [2000] S. H. Pan, E. W. Hudson, A. K. Gupta, K.-W. Ng, H. Eisaki, S. Uchida, and J. C. Davis, Phys. Rev. Lett. 85, 1536 (2000).
  * Oh  _et al._ [1988] B. Oh, K. Char, A. D. Kent, M. Naito, M. R. Beasley, T. H. Geballe, R. H. Hammond, A. Kapitulnik, and J. M. Graybeal, Phys. Rev. B 37, 7861 (1988).
  * Wang  _et al._ [2003] Y. Wang, S. Ono, Y. Onose, G. Gu, Y. Ando, Y. Tokura, S. Uchida, and N. P. Ong, Science 299, 86 (2003).
  * Wang and Wen [2008] Y. Wang and H.-H. Wen, EPL (Europhysics Letters) 81, 57007 (2008).
  * Grissonnanche  _et al._ [2014] G. Grissonnanche, O. Cyr-Choinière, F. Laliberté, S. R. de Cotret, A. Juneau-Fecteau, S. Dufour-Beauséjour, M. Ã. Delage, D. LeBoeuf, J. Chang, B. J. Ramshaw, D. A. Bonn, W. N. Hardy, R. Liang, S. Adachi, N. E. Hussey, B. Vignolle, C. Proust, M. Sutherland, S. Krï¿½mer, J. H. Park, D. Graf, N. Doiron-Leyraud, and L. Taillefer, Nat. Commun. 5 (2014).
  * Petrenko  _et al._ [2022] Â. E. V. Petrenko, L. V. Omelchenko, A. V. Terekhov, Y. A. Kolesnichenko, K. Rogacki, D. M. Sergeyev, A. L. Solovjov, and E. V. Petrenko, Low Temperature Physics 48, 755 (2022).
  * Yu  _et al._ [2019] Y. Yu, L. Ma, P. Cai, R. Zhong, C. Ye, J. Shen, G. D. Gu, X. H. Chen, and Y. Zhang, Nature 2019 575:7781 575, 156 (2019).
  * Sekitani  _et al._ [2004] T. Sekitani, N. Miura, S. Ikeda, Y. H. Matsuda, and Y. Shiohara, Physica B: Condensed Matter 346-347, 319 (2004).
  * Lee  _et al._ [2022] Y. Lee, M. Martini, T. Confalone, S. Shokri, C. N. Saggau, G. Gu, K. Watanabe, T. Taniguchi, D. Montemurro, V. M. Vinokur, K. Nielsch, and N. Poccia 10.48550/arxiv.2210.02124 (2022).
  * Klemm [2005] R. A. Klemm, Philosophical Magazine 85, 801 (2005).
  * Song  _et al._ [2022] X.-Y. Song, Y.-H. Zhang, and A. Vishwanath, Physical Review B 105, L201102 (2022).
  * Lee  _et al._ [2021] J. Lee, W. Lee, G.-Y. Kim, Y.-B. Choi, J. Park, S. Jang, G. Gu, S.-Y. Choi, G. Y. Cho, G.-H. Lee, and H.-J. Lee, Nano Letters 21, 10469 (2021).
  * Takano  _et al._ [2002] Y. Takano, T. Hatano, A. Fukuyo, A. Ishii, M. Ohmori, S. Arisawa, K. Togano, and M. Tachiki, Physical Review B 65, 140513 (2002).
  * Zhao  _et al._ [2021] S. Y. F. Zhao, N. Poccia, X. Cui, P. A. Volkov, H. Yoo, R. Engelke, Y. Ronen, R. Zhong, G. Gu, S. Plugge, T. Tummuru, M. Franz, J. H. Pixley, and P. Kim 10.48550/arxiv.2108.13455 (2021).
  * Tummuru  _et al._ [2022] T. Tummuru, S. Plugge, and M. Franz, Physical Review B 105, 064501 (2022).
  * Gavish  _et al._ [2021] N. Gavish, O. Kenneth, and A. Keren, Physica D: Nonlinear Phenomena 415, 132767 1 (2021).
  * Mangel  _et al._ [2020] I. Mangel, I. Kapon, N. Blau, K. Golubkov, N. Gavish, and A. Keren, Phys. Rev. B 102, 024502 1 (2020).
  * Tinkham [2004] M. Tinkham,  _Introduction to Superconductivity: v. 1_ (Dover Publications Inc., 2004).
  * Khanukov  _et al._ [2022] A. Khanukov, I. Mangel, S. Wissberg, A. Keren, and B. Kalisky, PHYSICAL REVIEW B 106, 144510 (2022).
  * Maxfield and Mclean [1965] B. W. Maxfield and W. L. Mclean, Phys. Rev. 139, A1515 (1965).
  * Kapon  _et al._ [2019] I. Kapon, Z. Salman, I. Mangel, T. Prokscha, N. Gavish, and A. Keren, Nat. Commun. 10 (2019).
  * Panagopoulos  _et al._ [2000] C. Panagopoulos, J. R. Cooper, T. Xiang, Y. S. Wang, and C. W. Chu, Phys. Rev. B 61, 163 (2000).

*


# Classical discrete operators on variable $\ell^{p(\cdot)}(\mathbb{Z})$
spaces

Pablo Rocha

###### Abstract

We show, by applying discrete weighted norm inequalities and the Rubio de
Francia algorithm, that the discrete Hilbert transform and discrete Riesz
potential are bounded on variable $\ell^{p(\cdot)}(\mathbb{Z})$ spaces
whenever the discrete Hardy-Littlewood maximal is bounded on
$\ell^{p^{\prime}(\cdot)}(\mathbb{Z})$. We also obtain vector-valued
inequalities for the discrete fractional maximal operator.

†† 2020 Mathematics Subject Classification: 46B45, 44A15, 47B06, 42B25,
26D15†† Key words and phrases: variable sequence spaces, discrete Hilbert
transform, discrete Riesz potential, discrete weights, Rubio de Francia
algorithm

##  1 Introduction

Given a sequence $p(\cdot):\mathbb{Z}\to[1,\infty)$, we define the variable
$\ell^{p(\cdot)}=\ell^{p(\cdot)}(\mathbb{Z})$ space to be the set of all
complex sequences $a=\\{a(i)\\}_{i\in\mathbb{Z}}$ such that

$\|a\|_{\ell^{p(\cdot)}}=\inf\left\\{\lambda>0:\sum_{i\in\mathbb{Z}}|a(i)/\lambda|^{p(i)}\leq
1\right\\}<\infty.$

It is well known that the couple
$(\ell^{p(\cdot)},\|\cdot\|_{\ell^{p(\cdot)}})$ results a Banach space. W.
Orlicz [13] and H. Nakano [10] were the first in studying these kind of spaces
(see also [4], [11], [12]). P. Hästö [8] showed that the variable
$\ell^{p(\cdot)}$ spaces have applications to the study of operators on
variable Lebesgue spaces in $\mathbb{R}^{n}$.

Let $0\leq\alpha<1$, for a sequence $a=\\{a(i)\\}_{i\in\mathbb{Z}}$, define
the fractional Hardy-Littlewood maximal sequence $M_{\alpha}a$ by

(1)
$(M_{\alpha}a)(j)=\sup\frac{1}{(n-m+1)^{1-\alpha}}\sum_{i=m}^{n}|a(i)|,\,\,\,\,j\in\mathbb{Z},$

where the supremum is taken over all $m,n\in\mathbb{Z}$ such that $m\leq j\leq
n$. When $\alpha=0$, we have that $M_{0}=M$, where $M$ is the discrete Hardy-
Littlewood maximal operator. Recently, A. Swarup and A. Alphonse [16] proved
that the discrete fractional maximal $M_{\alpha}$ is a bounded operator
$\ell^{p(\cdot)}\to\ell^{q(\cdot)}$, for
$\frac{1}{p(\cdot)}-\frac{1}{q(\cdot)}=\alpha$, under the assumption that

$1<p_{-}:=\inf\\{p(i):i\in\mathbb{Z}\\}\leq
p_{+}:=\sup\\{p(i):i\in\mathbb{Z}\\}<\alpha^{-1}\,\,\,\text{and}\,\,\,p(\cdot)\in
LH_{\infty}(\mathbb{Z}).$

We say that $p(\cdot)\in LH_{\infty}(\mathbb{Z})$, if there exist positive
real constants $p_{\infty}$ and $C_{\infty}$ such that

$|p(i)-p_{\infty}|\leq\frac{C_{\infty}}{\log(e+|i|)},\,\,\,\,\text{for
all}\,\,i\in\mathbb{Z}.$

This property is known as the log-Hölder continuity at infinity for an
exponent $p(\cdot)$. We set

$\mathcal{P}=\\{p(\cdot):\mathbb{Z}\to[1,\infty):1<p_{-}\leq p_{+}<\infty\\},$

and

$\mathcal{B}=\\{p(\cdot)\in\mathcal{P}:M\,\,\text{is bounded
on}\,\,\ell^{p(\cdot)}\\}.$

Then, $LH_{\infty}\cap\mathcal{P}\subset\mathcal{B}$. Such inclusion also
follows from a result of A. Nekvinda; indeed, he proved in [11] that if
$p(\cdot)\in LH_{\infty}\cap\mathcal{P}$, then
$\ell^{p(\cdot)}\equiv\ell^{p_{\infty}}$ with equivalent norms (see also [3,
Corollary 4.1.9]). Then, this equivalence and [15, Theorem 2.3 and Proposition
2.4] give that $M_{\alpha}$ is a bounded operator
$\ell^{p(\cdot)}\to\ell^{q(\cdot)}$, where
$\frac{1}{q(\cdot)}=\frac{1}{p(\cdot)}-\alpha$ and $0\leq\alpha<1$, when
$p(\cdot)\in LH_{\infty}\cap\mathcal{P}$ and $p_{+}<\alpha^{-1}$.

By [12, Theorem 5.10], one has that
$LH_{\infty}\cap\mathcal{P}\subsetneqq\mathcal{B}$.

Given an exponent sequence $p(\cdot):\mathbb{Z}\to[1,\infty)$, its conjugate
exponent $p^{\prime}(\cdot)$ is defined by
$\frac{1}{p(i)}+\frac{1}{p^{\prime}(i)}=1$, for all $i\in\mathbb{Z}$.

We point out that in our main results, we will only assume that the exponent
$(p(\cdot)/r)^{\prime}\in\mathcal{B}$ for some $r>1$.

######  Remark 1.1.

One can see that
$\ell^{p(\cdot)}=\\{a:\sum_{i\in\mathbb{Z}}|a(i)|^{p(i)}<\infty\\}$ and
$\ell^{p(\cdot)}\subset\ell^{p_{+}}$, when $p_{+}<\infty$.

Next, we introduce two classical discrete operators. Given a sequence
$a=\\{a(i)\\}_{i\in\mathbb{Z}}$, define the discrete Hilbert transform by

(2) $(Ha)(j)=\sum_{i\neq j}\frac{a(i)}{i-j},\,\,\,\,j\in\mathbb{Z}.$

In [14], M. Riesz proved that $H$ is a bounded operator $\ell^{r}\to\ell^{r}$
for $1<r<\infty$. Thus, by Remark 1.1, the operator $H$ is well defined on
$\ell^{p(\cdot)}$ for every $p(\cdot)\in\mathcal{P}$.

For $0<\alpha<1$, the discrete Riesz potential is defined by

(3) $(I_{\alpha}a)(j)=\sum_{i\neq
j}\frac{a(i)}{|i-j|^{1-\alpha}},\,\,\,\,j\in\mathbb{Z}.$

Since $I_{\alpha}$ is a bounded operator $\ell^{r}\to\ell^{s}$ for
$1<r<\alpha^{-1}$ and $\frac{1}{s}=\frac{1}{r}-\alpha$ (see [7], p. 288), by
Remark 1.1, it follows that the operator $I_{\alpha}$ is well defined on
$\ell^{p(\cdot)}$ for every $p(\cdot)\in\mathcal{P}$, with
$p_{+}<\alpha^{-1}$.

The purpose of this note, which was inspired by the work [1], is to prove the
following three results.

######  Theorem 1.2.

Let $H$ be the discrete Hilbert transform given by (2) and $1<r<p_{-}$. If
$(p(\cdot)/r)^{\prime}\in\mathcal{B}$, then there exists a positive constant
$C$ such that

$\|Ha\|_{\ell^{p(\cdot)}}\leq C\|a\|_{\ell^{p(\cdot)}},$

for all $a\in\ell^{p(\cdot)}$.

######  Theorem 1.3.

Let $I_{\alpha}$ be the discrete Riesz potential given by (3) and
$\frac{1}{1-\alpha}<s<q_{-}$. If $(q(\cdot)/s)^{\prime}\in\mathcal{B}$ and
$\frac{1}{p(\cdot)}=\frac{1}{q(\cdot)}+\alpha$, then there exists a positive
constant $C$ such that

$\|I_{\alpha}a\|_{\ell^{p(\cdot)}}\leq C\|a\|_{\ell^{q(\cdot)}},$

for all $a\in\ell^{p(\cdot)}$.

######  Theorem 1.4.

Given $0\leq\alpha<1$, let $M_{\alpha}$ be the discrete fractional maximal
given by (1) and $\frac{1}{1-\alpha}<s<q_{-}$. If
$(q(\cdot)/s)^{\prime}\in\mathcal{B}$,
$\frac{1}{p(\cdot)}=\frac{1}{q(\cdot)}+\alpha$ and $\theta\in(1,\infty)$, then
there exists a positive constant $C$ such that

(4)
$\left\|\left\\{\sum_{k=1}^{\infty}(M_{\alpha}a_{k})^{\theta}\right\\}^{1/\theta}\right\|_{\ell^{p(\cdot)}}\leq
C\left\|\left\\{\sum_{k=1}^{\infty}|a_{k}|^{\theta}\right\\}^{1/\theta}\right\|_{\ell^{q(\cdot)}},$

for all sequences of functions
$\\{a_{k}:\mathbb{Z}\to\mathbb{C}:k\in\mathbb{N}\\}$ with finite support.

This paper is organized as follows. Section 2 presents some facts about the
$\ell^{p(\cdot)}$ spaces and it also introduces the discrete weights. In
Section 3, we prove our main results by applying discrete weighted norm
inequalities and the Rubio de Francia algorithm.

Notation. Given a sequence $a=\\{a(i)\\}$ and a real number $s>0$, we put
$|a|^{s}=\\{|a(i)|^{s}\\}$. The symbol $A\approx B$ stands for the inequality
$c_{1}B\leq A\leq c_{2}B$, where $c_{1}$ and $c_{2}$ are positive constants.
Throughout this paper, $C$ will denote a positive real constant not
necessarily the same at each occurrence.

##  2 Preliminaries

In this section, we give some basic results about the variable
$\ell^{p(\cdot)}$ spaces and also introduce the discrete Muckenhoupt classes.

######  Lemma 2.1.

Let $p(\cdot)\in\mathcal{P}$, then for every $r\in(0,p_{-})$

$\|a\|_{\ell^{p(\cdot)}}^{r}=\||a|^{r}\|_{\ell^{p(\cdot)/r}}.$

###### Proof..

It is follows from the definition of the $\ell^{p(\cdot)}$-norm. ∎

######  Proposition 2.2.

(Hölder’s inequality) Let $p(\cdot)\in\mathcal{P}$, then there exists a
constant $C>0$ such that

$\sum_{i\in\mathbb{Z}}|a(i)b(i)|dx\leq
C\|a\|_{\ell^{p(\cdot)}}\|b\|_{\ell^{p^{\prime}(\cdot)}}.$

###### Proof..

The proposition follows from [3, Lemma 3.2.20] considering there
$A=\mathbb{Z}$ and $\mu$ being the counting measure. ∎

######  Proposition 2.3.

Let $p(\cdot)\in\mathcal{P}$, then

$\|a\|_{\ell^{p(\cdot)}}\approx\sup\left\\{\sum_{i\in\mathbb{Z}}|a(i)b(i)|:\|b\|_{\ell^{p^{\prime}(\cdot)}}\leq
1\right\\}.$

###### Proof..

The proposition follows from [3, Corollary 3.2.14]. ∎

A discrete weight is a positive real sequence. For a constant $1<r<\infty$, we
say that a weight $w=\\{w(i)\\}_{i\in\mathbb{Z}}$ belongs to $\mathcal{A}_{r}$
if there exists a positive constant $C$ such that

$\left(\sum_{i=m}^{n}w(i)\right)\left(\sum_{i=m}^{n}w(i)^{-1/(r-1)}\right)^{r-1}\leq
C(n-m+1)^{r},$

for all $m,n\in\mathbb{Z}$, with $m\leq n$. For $r=1$, we say that a weight
$w=\\{w(i)\\}_{i\in\mathbb{Z}}$ belongs to $\mathcal{A}_{1}$ if there exists a
positive constant $C$ such that

$\frac{1}{n-m+1}\sum_{i=m}^{n}w(i)\leq C\inf\\{w(i):i\in\mathbb{Z}\\},$

for all $m,n\in\mathbb{Z}$, with $m\leq n$. Equivalently, a weight
$w=\\{w(i)\\}_{i\in\mathbb{Z}}$ belongs to $\mathcal{A}_{1}$ if there exists a
positive constant $C$ such that

$(Mw)(j)\leq Cw(j),\,\,\,\,\text{for all}\,j\in\mathbb{Z}.$

Given $1<r\leq s<\infty$, a weight $w=\\{w(i)\\}_{i\in\mathbb{Z}}$ is said to
belong to $\mathcal{A}_{r,s}$ if there exists a positive constant $C$ such
that

$\left(\frac{1}{n-m+1}\sum_{i=m}^{n}w(i)^{s}\right)^{1/s}\left(\frac{1}{n-m+1}\sum_{i=m}^{n}w(i)^{-r^{\prime}}\right)^{1/r^{\prime}}\leq
C,$

for all $m,n\in\mathbb{Z}$, with $m\leq n$.

######  Remark 2.4.

It is clear that $\mathcal{A}_{r}\subset\mathcal{A}_{s}$, when $1\leq
r<s<\infty$. Moreover, if $w\in\mathcal{A}_{1}$ then
$w^{1/s}\in\mathcal{A}_{r,s}$ for every $1<r\leq s<\infty$.

##  3 Main results

In this section we prove our main results by adapting the techniques used in
[1] to our discrete setting.

Proof of Theorem 1.2. Let $p(\cdot)\in LH_{\infty}\cap\mathcal{P}$. We choose
a real number $r$ such that $1<r<p_{-}$. By [9, Theorem 10] we have, for every
sequence $a=\\{a(i)\\}$ such that $|(Ha)(j)|<\infty$ for all $j\in\mathbb{Z}$
and every weight $w=\\{w(j)\\}\in\mathcal{A}_{r}$, that

(5) $\sum_{j\in\mathbb{Z}}|(Ha)(j)|^{r}w(j)\leq
C\sum_{j\in\mathbb{Z}}|a(j)|^{r}w(j).$

In particular, since the operator $H$ is well defined on $\ell^{p(\cdot)}$,
the inequality (5) holds for $a\in\ell^{p(\cdot)}$.

On the other hand, Lemma 2.1 and Proposition 2.3 give

(6) $\|Ha\|_{\ell^{p(\cdot)}}^{r}=\||Ha|^{r}\|_{\ell^{p(\cdot)/r}}\leq
C\sup\sum_{j\in\mathbb{Z}}|(Ha)(j)|^{r}b(j),$

where the supremum is taken over all non-negative sequences
$b=\\{b(j)\\}\in\ell^{(p(\cdot)/r)^{\prime}}$ with
$\|b\|_{\ell^{(p(\cdot)/r)^{\prime}}}\leq 1$. Since, by hypothesis, the
discrete maximal $M$ is bounded on $\ell^{(p(\cdot)/r)^{\prime}}$, we define
the operator $\mathcal{R}$ on $\ell^{(p(\cdot)/r)^{\prime}}$ by

$(\mathcal{R}b)(j)=\sum_{k=0}^{\infty}\frac{(M^{k}b)(j)}{2^{k}A},$

where $A=\|M\|_{\ell^{(p(\cdot)/r)^{\prime}}\to\ell^{(p(\cdot)/r)^{\prime}}}$,
for $k\geq 1$, $M^{k}=M\circ\cdot\cdot\cdot\circ M$ denotes $k$ iterations of
the discrete maximal, and $M^{0}$ is the identity operator. It is easy to
check that

$(i)$ if $b(j)\geq 0$, then $b(j)\leq(\mathcal{R}b)(j)$ for all
$j\in\mathbb{Z}$;

$(ii)$ $\|\mathcal{R}b\|_{\ell^{(p(\cdot)/r)^{\prime}}}\leq
2\|b\|_{\ell^{(p(\cdot)/r)^{\prime}}}$;

$(iii)$ $M(\mathcal{R}b)(j)\leq 2A(\mathcal{R}b)(j)$, for all
$j\in\mathbb{Z}$, so $\mathcal{R}b\in\mathcal{A}_{1}$.
Now, from $(i)$, it follows that

$\sum_{j\in\mathbb{Z}}|(Ha)(j)|^{r}b(j)\leq\sum_{j\in\mathbb{Z}}|(Ha)(j)|^{r}(\mathcal{R}b)(j),$

since $\mathcal{R}b\in\mathcal{A}_{1}$ (see $(iii)$), by Remark 2.4 and (5),
we obtain

$\leq\sum_{j\in\mathbb{Z}}|a(j)|^{r}(\mathcal{R}b)(j),$

to apply Proposition 2.2 on this expression, we get

$\leq
C\||a|^{r}\|_{\ell^{p(\cdot)/r}}\|\mathcal{R}b\|_{\ell^{(p(\cdot)/r)^{\prime}}},$

the property $(ii)$ gives

$\leq C\||a|^{r}\|_{\ell^{p(\cdot)/r}}\|b\|_{\ell^{(p(\cdot)/r)^{\prime}}}.$

The above string of inequalities leads to

(7) $\sum_{j\in\mathbb{Z}}|(Ha)(j)|^{r}b(j)\leq
C\||a|^{r}\|_{\ell^{p(\cdot)/r}}\|b\|_{\ell^{(p(\cdot)/r)^{\prime}}}.$

Finally, by taking the supremum over all non-negative sequences
$b=\\{b(j)\\}\in\ell^{(p(\cdot)/r)^{\prime}}$ such that
$\|b\|_{\ell^{(p(\cdot)/r)^{\prime}}}\leq 1$ in (7), from (6) and Lemma 2.1,
we conclude that

$\|Ha\|_{\ell^{p(\cdot)}}\leq C\|a\|_{\ell^{p(\cdot)}}.$

This completes the proof. $\square$

Proof of Theorem 1.3. Given $s$ sucht that $\frac{1}{1-\alpha}<s<q_{-}$, we
define $\frac{1}{r}=\frac{1}{s}+\alpha$, so $1<r<p_{-}$. Since the operator
$I_{\alpha}$ is well defined on $\ell^{p(\cdot)}$, by Remark 2.4 and [6,
Theorem 4.4], the inequality

(8) $\sum_{j\in\mathbb{Z}}|(I_{\alpha}a)(j)|^{s}w(j)\leq
C\left(\sum_{j\in\mathbb{Z}}|a(j)|^{r}w(j)^{r/s}\right)^{s/r}$

holds for all $a=\\{a(i)\\}\in\ell^{p(\cdot)}$ and all
$w=\\{w(i)\\}\in\mathcal{A}_{1}$. By Lemma 2.1 and Proposition 2.3, we have

(9)
$\|I_{\alpha}a\|_{\ell^{q(\cdot)}}^{s}=\||I_{\alpha}a|^{s}\|_{\ell^{q(\cdot)/s}}\leq
C\sup\sum_{j\in\mathbb{Z}}|(I_{\alpha}a)(j)|^{s}b(j),$

where the supremum is taken over all non-negative sequences
$b=\\{b(j)\\}\in\ell^{(q(\cdot)/s)^{\prime}}$ with
$\|b\|_{\ell^{(q(\cdot)/s)^{\prime}}}\leq 1$. By our assumption on $q(\cdot)$,
the discrete maximal $M$ is bounded on $\ell^{(q(\cdot)/s)^{\prime}}$. Then,
we define the operator $\mathcal{R}$ on $\ell^{(q(\cdot)/s)^{\prime}}$ by

$(\mathcal{R}b)(j)=\sum_{k=0}^{\infty}\frac{(M^{k}b)(j)}{2^{k}A},$

where $A=\|M\|_{\ell^{(q(\cdot)/s)^{\prime}}\to\ell^{(q(\cdot)/s)^{\prime}}}$.
We observe that the properties $(i)$ and $(iii)$ above hold and $(ii)$ holds
with $q(\cdot)/s$ instead of $p(\cdot)/r$. Proceeding as in the proof of
Theorem 1.2, but now considering (8) instead of (5), we obtain

(10) $\sum_{j\in\mathbb{Z}}|(I_{\alpha}a)(j)|^{s}b(j)\leq
C\||a|^{r}\|_{\ell^{p(\cdot)/r}}^{s/r}\|(\mathcal{R}b)^{r/s}\|_{\ell^{(p(\cdot)/r)^{\prime}}}^{s/r}.$

A computation gives
$\|(\mathcal{R}b)^{r/s}\|_{\ell^{(p(\cdot)/r)^{\prime}}}^{s/r}=\|\mathcal{R}b\|_{\ell^{(q(\cdot)/s)^{\prime}}}\leq
C\|b\|_{\ell^{(q(\cdot)/s)^{\prime}}}$. So, the theorem follows from this
inequality, (10) and (9). $\square$

Proof of Theorem 1.4. We first consider the case $0<\alpha<1$. We define

$\mathcal{F}_{\alpha}=\left\\{\left(\left\\{\sum_{k=1}^{N}(M_{\alpha}a_{k})^{\theta}\right\\}^{1/\theta},\left\\{\sum_{k=1}^{N}|a_{k}|^{\theta}\right\\}^{1/\theta}\right):N\in\mathbb{N},\\{a_{k}\\}_{k=1}^{N}\subset\ell^{\infty}_{comp}\right\\},$

where $\theta\in(1,\infty)$ and $\ell^{\infty}_{comp}$ denotes the set of
sequences with finite support on $\mathbb{Z}$.

Given $\frac{1}{1-\alpha}<s<q_{-}$ fixed, let $r$ be defined by
$\frac{1}{r}=\frac{1}{s}+\alpha$. So, $1<r<p_{-}$. By [6, Theorem 3.3], [2,
Theorem 3.23] (which also holds in the discrete setting), and Remark 2.4,
there exists an universal constant $C>0$ such that for any
$(F,G)\in\mathcal{F}_{\alpha}$ and any $w=\\{w(i)\\}\in\mathcal{A}_{1}$

(11) $\sum_{j\in\mathbb{Z}}[F(j)]^{s}w(j)\leq
C\left(\sum_{j\in\mathbb{Z}}[G(j)]^{r}w(j)^{r/s}\right)^{s/r}.$

Now, proceeding as in the proof of Theorem 1.3, we apply the Rubio de Francia
iteration algorithm with respect to $\ell^{(q(\cdot)/s)^{\prime}}$ to obtain

$\|F\|_{\ell^{q(\cdot)}}\leq C\|G\|_{\ell^{p(\cdot)}},$

for all pair $(F,G)\in\mathcal{F}_{\alpha}$. This is

(12)
$\left\|\left\\{\sum_{k=1}^{N}(M_{\alpha}a_{k})^{\theta}\right\\}^{1/\theta}\right\|_{\ell^{q(\cdot)}}\leq
C\left\|\left\\{\sum_{k=1}^{N}|a_{k}|^{\theta}\right\\}^{1/\theta}\right\|_{\ell^{p(\cdot)}}.$

Then, letting $N\to\infty$ in (12), we obtain (4) for the case $0<\alpha<1$.

Finally, by [5, Proposition 2.9] and Remark 2.4, we have that (11) holds with
$\alpha=0$ and $r=s$. Thus, by reasoning as above, the theorem follows for the
case $\alpha=0$. $\square$

## References

  * [1] D. Cruz-Uribe, A. Fiorenza, J. M. Martell and C. Pérez, The boundedness of classical operators on variable $L^{p}$ spaces, Ann. Acad. Sci. Fenn., Math. 31, No. 1 (2006), 239-264.
  * [2] D. V. Cruz-Uribe, J. M. Martell and C. Pérez, Weights, Extrapolation and the Theory of Rubio de Francia, Birkhäuser. Operator theory: advances and applications, Vol. 215 (2011).
  * [3] L. Diening, P. Harjulehto, P. Hästö and M. Rů$\check{\text{z}}$i$\check{\text{c}}$ka, Lebesgue and Sobolev spaces with Variable Exponents, Springer, 2011.
  * [4] D. E. Edmunds and A. Nekvinda, Averaging operators on on $\ell^{\\{p_{n}\\}}$ and $L^{p(x)}$, Math. inequal. Appl. 5 (2), (2002), 235-246.
  * [5] X. Hao, B. Li and S. Yang, The Hardy-Littlewood maximal operator on discrete weighted Morrey spaces, Acta Math. Hung. 172, (2) (2024), 445-469.
  * [6] X. Hao, B. Li and S. Yang, Estimates of Discrete Riesz Potentials on Discrete Weighted Lebesgue Spaces, Ann. Funct. Anal. 15 (3), Paper No. 51 (2024), 24 p.
  * [7] G. H. Hardy, J. E. Littlewood and G. Pólya, Inequalities, 2nd ed., Cambridge Univ. Press, London and new York, 1952.
  * [8] P. Hästö, Local-to-global results in variable exponent spaces, Math. Res. Letters, 16 (2) (2009), 263-278.
  * [9] R. Hunt, B. Muckenhoupt and R. Wheeden, Weighted norm inequalities for the conjugate function and Hilbert transform, Trans. Amer. Math. Soc. 176 (1973), 227-251.
  * [10] H. Nakano, Modulared sequence spaces, Proc. Japan Acad., 27 (1951), 508-512.
  * [11] A. Nekvinda, Equivalence of $\ell^{\\{p_{n}\\}}$ norms and shift operators, Math. Inequal. Appl. 5 (4), (2002), 711-723.
  * [12] A. Nekvinda, A note on maximal operator on $\ell^{\\{p_{n}\\}}$ and $L^{p(x)}(\mathbb{R})$, J. Funct. Spaces Appl. 5 (1), (2007), 49-88.
  * [13] W. Orlicz, Über konjugierte Exponentenfolgen, Studia Math., 3 (1931), 200-211.
  * [14] M. Riesz, Sur les fonctions conjuguées, Math. Z., 27 (1928), 218-244.
  * [15] P. Rocha, A note about discrete Riesz potential on $\mathbb{Z}^{n}$, Preprint 2024: https://arxiv.org/abs/2407.15262
  * [16] A. S. S. Swarup and A. M. Alphonse, The boundedness of Fractional Hardy-Littlewood maximal operator on variable $\ell^{p(\cdot)}(\mathbb{Z})$ spaces using Calderón-Zygmund decomposition. J. Indian Math. Soc., New Ser. 91, No. 1-2 (2024), 237-252.

Departamento de Matemática
Universidad Nacional del Sur
Bahía Blanca, 8000 Buenos Aires
Argentina

E-mail address<EMAIL_ADDRESS>