# A note on Gauss operators
and quantizations of Yang-Mills theories

Bruno Bucciotti
Scuola Normale Superiore, Piazza dei Cavalieri 7 Pisa, Italy
INFN, Sezione di Pisa, Largo B. Pontecorvo, 3, I-56127 Pisa, Italy
Franco Strocchi
Dipartimento di Fisica, Università di Pisa, Largo B. Pontecorvo, 3, I-56127
Pisa, Italy Bruno Bucciotti(1,2), Franco Strocchi(3)
(1)Scuola Normale Superiore, Piazza dei Cavalieri 7 Pisa, Italy
(2)INFN, Sezione di Pisa, Largo Pontecorvo, 3, Ed. C, 56127 Pisa, Italy
(3)Dipartimento di Fisica, Università di Pisa, Largo B. Pontecorvo, 3, I-56127
Pisa, Italy

The quantization of Yang-Mills field theories requires the introduction of a
gauge fixing leading to a violation of the Local Gauss Law described by the
so-called Gauss operator. We discuss the local quantizations of Yang-Mills
theories preserving the invariance under the global gauge group, in terms of a
characterization of the possible Gauss operators, which appear to have a more
physical meaning than the gauge fixings. We show that it is impossible to have
a quantization of non-abelian Yang-Mills field theories, which is Lorentz
covariant and preserves a residual subgroup of local gauge transformations.
Thus, quite generally, one cannot have the properties of Feynman quantization
of Quantum Electrodynamics. By relaxing the covariance condition, we
characterize the Gauss operators which satisfy gauge-covariant conservation
laws and generate non-trivial subgroups of local gauge transformations.

Mathematical Subject Classification: 81T13, 81T10, 81T99

Keywords: Yang-Mills gauge theories, Local gauge transformations, Local Gauss
Laws

##  1 Introduction

Given a Lie (global) gauge group $G$ the correspondent local gauge group
${\cal G}$ is conventionally obtained by replacing the constant group
parameters of $G$ by infinitely differentiable functions of compact support in
space.111 Regularity conditions on the gauge functions are needed since the
quantum fields are operator valued distributions, so that the transformations
induced by the action of ${\cal G}$ should have the property of multipliers of
the related test functions. The requirement of compact support in space
guarantees the very useful property of space localization of the gauge
transformations and it also provides the needed clear cut distinction between
the global and the local gauge group. The condition of compact support in time
would exclude the local gauge transformations in the temporal gauge and it is
not required. Very problematic is the generalization of the local gauge
transformations with gauge functions replaced by field operators, since it
involves the singular (actually divergent) multiplication of operator valued
distributions at the same point. For more detailed motivations and useful
consequences of such choices see [1, 2].

A classical gauge field theory is defined by a Lagrangian ${\cal L}_{inv}$
invariant under a local gauge group ${\cal G}$, with the fields (which enter
in the definition of ${\cal L}_{inv}$) transforming as a representation of
${\cal G}$. In order to have a well posed Cauchy problem for the time
evolution of the fields one must add a gauge fixing ${\cal L}_{GF}$, which
need not breaking ${\cal G}$ to the identity, but only to the extent of
allowing a deterministic field dynamics [2, 3].

The introduction of a gauge fixing is crucial for a quantization of the field
algebra since, according to the principles of quantum mechanics the dynamics
is described by a unitary operator $U(t)=e^{-iHt}$, and therefore the dynamics
of the quantum field operators must be deterministic.

Thus, a quantization of a gauge field theory is defined by the choice of a
gauge fixing, different gauge fixing identifying different quantum field
algebras ${\cal F}$, even if the resulting representations of the observable
field subalgebra ${\cal F}_{obs}$ are all equivalent (one of the roles of
${\cal F}$ being the construction of the representations of ${\cal F}_{obs}$
beyond the vacuum one [3]).

We consider quantizations of Yang-Mills field theories (with $G$ a compact Lie
group) defined by a vacuum representation of a local field algebra ${\cal F}$,
obtained by gauge fixings which do not break the global gauge group $G$,
satisfy locality and leave a residual local gauge group ${\cal G}_{r}$
unbroken (very helpful for renormalization). The motivations for such a choice
are discussed in [3]. Additionally, the unbroken residual group allows to
exploit its topology, which helps in codifying the interplay between gauge
invariance and other symmetries, in particular chiral symmetries. Indeed, the
representation of the topology of the residual group has been shown to yield a
non-perturbative solution of the $U(1)$ problem in Quantum Chromodynamics,
without relying on the semi-classical instanton approximation, which is not
free of mathematical problems [1, 4].

In general, the presence of the gauge fixing invalidates the conclusion of the
second Noether theorem, namely the validity of  Local Gauss Laws

$\displaystyle{J_{\mu}^{a}=\partial^{\nu}F^{a}_{\nu\,\mu}.}$ (1.1)

satisfied by the currents $J_{\mu}^{a}$, ($a=1,...n,n=$ the dimension of $G$),
which generate the infinitesimal transformations of $G$ on the field algebra
${\cal F}$:

$\displaystyle{\delta^{a}F=\lim_{R\rightarrow\infty}\,i\,[Q^{a}_{R},\,F\,],\,\,\,\,\,\forall
F\in{\cal F},}$ (1.2)

where $Q^{a}_{R}$ is a suitably regularized integral of $J^{a}_{0}$ over a
sphere of radius $R$. [1]

Instead of eq. 1.1 one rather gets

$\displaystyle{J_{\mu}^{a}=\partial^{\nu}F^{a}_{\nu\,\mu}+G_{\mu}^{a},}$ (1.3)

where $G_{\mu}^{a}$, henceforth called the Gauss operator, depends on the
choice of the gauge fixing.

The unbroken global group $G$ implies that $\partial^{\mu}J^{a}_{\mu}=0$ and
therefore eq. 1.3 and the antisymmetry of $F_{\nu\,\mu}$ yield

$\displaystyle{\partial^{\mu}G^{a}_{\mu}=0,}$ (1.4)

i.e. the Gauss operator satisfies a continuity equation. Thus , it plays the
role of an "external" conserved current.

On the other hand, for any physical state $\Psi$ (typically obtained by a non-
local construction through the vacuum representation of ${\cal F}$), the Local
Gauss Law holds:222For more details and proofs, see [1, 2, 3]

$\displaystyle{(\Psi,(J_{\mu}^{a}-\partial^{\nu}F^{a}_{\nu\,\mu})\,\Psi)=(\Psi,G^{a}_{\mu}\,\Psi)=0.}$
(1.5)

Actually, such a property qualifies the physical states (in Quantum
Electrodynamics amounts to the validity of the Maxwell equations on the
physical states) to the extent that the Gauss operator provides the
characterization of the physical states.

Moreover, by the locality of the field algebra the Gauss operator generates
the infinitesimal transformations of the residual local gauge group in the
given representation of ${\cal F}$, so that it belongs to the commutant of
${\cal F}_{obs}$ [2, 3] and therefore it classifies the representations of
${\cal F}_{obs}$ contained in the representation space of ${\cal F}$. In the
abelian case, the Gauss operator itself belongs to ${\cal F}_{obs}$ and
therefore to its center.

The meaning of the Gauss operator in such a construction of the physical
states becomes more evident in connection with locality.

In fact, the Gauss law eq. 1.1, yields that the flux of $F^{a}_{i\,0}$ at
infinity, briefly called Gauss charge and denoted by $Q^{a}_{G}$, coincides
with the total charge, a suitable integral of $J^{a}_{0}$, called current
charge and denoted by $Q^{a}_{J}$. Then, since by locality the Gauss charge
vanishes on the local states, so does the charge $Q^{a}_{J}$ and the global
gauge group $G$ acts trivially in the representation space of ${\cal F}$,
${\cal V}={\cal F}{\Psi_{0}}$.

The role of the Gauss operator is to screen/compensate the contribution of the
current charge to the flux of $F^{a}_{i\,0}$ at infinity allowing for local
states with non-zero charge and, consequently, for a non-trivial action of $G$
in the representation space of ${\cal F}$.

Furthermore, a clever choice of the Gauss operator may make more explicit the
hyperbolic and elliptic contents of eq. 1.3, which in the abelian case is
equivalent to:

$\displaystyle{\square
F_{\mu\,\nu}=\partial_{\mu}J_{\nu}-\partial_{\nu}J_{\mu}-\partial_{\mu}G_{\nu}+\partial_{\nu}G_{\mu},\qquad\partial^{i}F_{i\,0}=J_{0}-G_{0}.}$
(1.6)

In fact, in the abelian case the choice of the temporal gauge: $G_{i}=0$,
which implies $\partial_{0}G_{0}=0$, leads to a standard hyperbolic equation
for $F_{i\,j}$,

$\displaystyle{\square F_{i\,j}=\partial_{i}J_{j}-\partial_{j}J_{i},}$ (1.7)

without involving the Gauss operator, which only enters in the "elliptic"
constraint involving $F_{i\,0}$ and $J_{0}$, consistent with the time
evolution of $F_{i\,0}$

$\displaystyle{\partial^{0}F_{0\,i}=\partial^{j}F_{i\,j}+J_{i}.}$ (1.8)

Hence, the role of $G_{0}$ is that of a screening static charge density which
is seen by the local states but not by the non-local physical charged states.

Furthermore, in the temporal gauge the Gauss operator satisfies a covariant
conservation law, namely $D^{\mu}G_{\mu}=0$, in agreement with its
interpretation as the current of a (fictitious) charge distribution.

A similar role and meaning of the Gauss operator occurs also in the temporal
gauge of the non-abelian case.

For these reasons, it has been argued [3] that for the characterization of the
possible quantizations of a gauge field theory, the Gauss operator is better
qualified than the gauge fixing, which does not have such properties.

The aim of this note is to discuss the constraints on the Gauss operators in
the non-abelian case, and consequently the constraints on the possible
quantizations.

We show that it is impossible to have a quantization of non-abelian Yang-Mills
field theories, i.e. with a Gauss operator, which is local , Lorentz covariant
and leaves a subgroup of local gauge transformations unbroken. Thus, quite
generally, it is impossible to have quantization, equivalently a Gauss
operator, with the properties of Feynman quantization of Quantum
Electrodynamics.

In addition, we obtain a classification of the Gauss operators which satisfy a
(gauge) covariant conservation law. This property yields that the gauge fixing
and the corresponding Gauss operator minimally affect the gauge group geometry
of the Yang-Mills equations, being in particular required if the current which
generates the global group has to be left unchanged. As a consequence, one has
the desirable property that the action on the matter fields by the Gauss
operator and by the topological operators (namely, the operators which
describe the topology of the residual group) is the same as that of the
current in the absence of gauge fixing.

This pattern is not available in the BRST quantization [5], because it does
not leave a subgroup of local gauge transformations unbroken. In addition, as
discussed above, the mathematical control of transformations with parameters
which are field operators is rather problematic, beyond the perturbative
approach. Finally, the singular (distributional) properties of the fields
which parametrize such transformations preclude the possibility of rigorously
considering their topological properties.

##  2 Gauss operators in Yang-Mills gauge theories

By the reasons discussed above, in the following we consider the possible
realization of a Gauss operators $G_{\mu}$ which has the following properties:

  1. i)

it leaves the global gauge group $G$ unbroken,

  2. ii)

it generates derivations on the field algebra which correspond to the
infinitesimal transformations of a residual subgroup ${\cal G}_{r}$ of local
gauge transformations. This allows the exploitation of the corresponding Ward
identities, and in the non-abelian case also the effect of the topology of
${\cal G}_{r}$ on the properties of the physical states,

  3. iii)

it defines a local field, in the representation space ${\cal V}$ of the local
field algebra,

  4. iv)

it allows a simple characterization of the physical states in terms of the
subspace of the representation space of ${\cal F}$, or of a suitable
completion of it, with vanishing expectation of $G_{\mu}$.333 For examples
which realize such a structure see [1, 2].

###  2.a Group theoretical constraints on Gauss operators

Since by ii) the Gauss operators under our consideration are related to
residual subgroups of local gauge transformations, the characterization of
such subgroups provides group theoretical constraints on the analysis of
possible Gauss operators.

As mentioned above, the local gauge group ${\cal G}$ is parametrized by the
set of infinitely differentiable functions of compact support in space,
${\varepsilon}^{a}(x)$, $a=1,...n$, which form not only a vector space ${\cal
V}$ with the sum defined by

$({\varepsilon}_{1}+{\varepsilon}_{2})^{a}(x)\equiv{\varepsilon}_{1}^{a}(x)+{\varepsilon}_{2}^{a}(x),$

but also an algebra ${\cal E}$ with product

$({\varepsilon}_{1}\wedge{\varepsilon}_{2})^{a}(x)\equiv
f^{a}_{b\,c}{\varepsilon}_{1}^{b}(x)\,{\varepsilon}_{2}^{c}(x).$

The abelian subgroups ${\cal G}_{i}$ of ${\cal G}$, stable under $G$ (by i)),
are identified by subspaces ${\cal V}_{i}$ of ${\cal V}$, whereas the non-
abelian subgroups are identified by subalgebras ${\cal E}_{i}$ of ${\cal E}$,
since in the non-abelian case the group composition law requires stability
under the above product (see the Zassenhaus’ formula below).

The stability of ${\cal G}_{i}$ under the global group $G$ requires that if
${\varepsilon}^{a}(x)T^{a}$ belongs to the Lie algebra of ${\cal G}_{i}$ so
does $[\,{\varepsilon}^{b}\,T^{b},{\varepsilon}^{a}(x)T^{a}\,],\forall b,$
i.e. if ${\varepsilon}^{a}$ belongs to the parameter space of ${\cal G}_{i}$
so does ${\varepsilon}^{c}(x)=f^{c}_{a\,b}{\varepsilon}^{a}(x),\,\forall b$.

The above properties must be satisfied by the local gauge subgroups related to
the derivations defined by the Gauss operators and therefore provide a first
classification of them.

Another restriction comes from the condition that the Gauss operators admits a
derivation from a Lagrangian.

The invariance of the gauge fixing under the residual group ${\cal G}_{r}$,
with gauge parameters ${\varepsilon}_{r}$ is codified by the following
equations in terms of the gauge fixing ${\cal L}_{GF}$:

$\displaystyle
L_{a}^{\mu}\,\partial_{\mu}{\varepsilon}^{a}_{r}\equiv\left\\{\frac{\delta{\cal
L}_{GF}}{\delta A^{a}_{\mu}}+T^{a}_{bc}\left[\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}A^{b}_{\nu}}A^{c}_{\nu}+\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}B^{b}}B^{c}\right]\right\\}\,\partial_{\mu}{\varepsilon}_{r}^{a}=0,$
(2.1) $\displaystyle
L_{a}^{\mu\,\nu}\,\partial_{\mu}\partial_{\nu}{\varepsilon}^{a}_{r}\equiv\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}A^{a}_{\nu}}\partial_{\mu}\partial_{\nu}{\varepsilon}^{a}_{r}=0,$
(2.2)

where for simplicity, we omitted the presence of matter fields and considered
gauge fixings ${\cal L}_{GF}$ which are functions of $A^{a}_{\mu}$ and of
fields $B^{a}$, of the Nakanishi-Lautrup type, which transform according to
irreducible adjoint representations of $G$, taken to be a simple group.

In conclusion, in the case of an abelian global group $G$ the possible
residual groups, equivalently the possible Gauss operators, are identified by
$G$ invariant subspaces of ${\cal V}$ with gauge parameters
${\varepsilon}_{r}(x)$ satisfying eqs. 2.1 and 2.2.

For non-abelian $G$ the characterizazion is provided by $G$ invariant
subalgebras of ${\cal E}$ with gauge parameters ${\varepsilon}_{r}(x)$
satisfying eqs. 2.1 and 2.2.

Explicit cases are discussed below.

###  2.b Feynman Gauss operator

A distinguished choice of the Gauss operator is that of the Feynman gauge in
Quantum Electrodynamics, which preserves locality and covariance :
$G_{\mu}=\partial_{\mu}L$, with $L$ a massless longitudinal field.

In addition to the properties i)-iv), the Feynman Gauss operator in QED
satisfies

  1. v)

it is Lorentz covariant, allowing for a Lorentz covariant field algebra,

  2. vi)

it provides a simple subsidiary condition, which linearizes the constraint
(1.5) for the identification of the physical states,

  3. vii)

it does not require fancy ghost fields, since it involves only the vector
potential and possibly the Nakanishi-Lautrup field $B$.

The technical advantages of such a choice are well known, in particular for
the control of renormalization, the explicit construction of the physical
charged states etc.444See, e.g. [1]. It is then natural to ask whether a Gauss
operator with such properties may be obtained also in the non-abelian case.

A negative answer has been presented in the literature and it is part of the
standard textbook presentation of the need of BRST quantization. However, such
a negative conclusion has been reached for the choice of a gauge fixing which
is a literal transcription of the Feynman choice, namely for the gauge fixing
${\scriptstyle{\frac{1}{2}}}\xi(\partial^{\mu}A^{a}_{\mu})^{2}$ or for its
Nakanishi-Lautrup variations, e.g.
$B^{a}\partial^{\mu}A^{a}_{\mu}+{\scriptstyle{\frac{1}{2}}}\xi(\partial^{\mu}A^{a}_{\mu})^{2}$.

One may reasonably ask whether in the non-abelian case a more clever choice of
the gauge fixing may work. Quite generally this leads to investigate the
possible Lorentz covariant solutions of the equation
$\partial^{\mu}G_{\mu}=0.$

For simplicity, we discuss the problem in the absence of matter fields and
consider gauge fixings ${\cal L}_{GF}$ which are functions of $A^{a}_{\mu}$
and of fields $B^{a}$, of the Nakanishi-Lautrup type, which transform
according to irreducible adjoint representations of $G$, for simplicity taken
to be a simple group. Then, the invariance of the gauge fixing ${\cal L}_{GF}$
under the global gauge group $G$ requires

$T^{a}_{b\,c}\,\left[\frac{\delta{\cal L}_{GF}}{\delta
A^{b}_{\mu}}A^{c}_{\mu}+\frac{\delta{\cal L}_{GF}}{\delta
B^{b}}B^{c}+\frac{\delta{\cal
L}_{GF}}{\delta\partial_{\nu}A^{b}_{\mu}}\partial_{\nu}A^{c}_{\mu}+\frac{\delta{\cal
L}_{GF}}{\delta\partial_{\nu}B^{b}}\partial_{\nu}B^{c}\right]=0.$

The additional restriction of Lorentz covariance plays a crucial role,
requiring the Lorentz covariance of the subspaces of gauge parametrs in the
abelian case and of the subalgebras of ${\cal E}$ in the non-abelian case.
Then, the group parameters of ${\cal G}_{r}$ should define a vector space
stable under $G$ and under Lorentz transformations. Now, since $G$ does not
have invariant subgroups, eq. 2.1 cannot provide restrictions on the index
$a$, and similarly there is no linear subspace invariant under the Lorentz
transformations to be spanned by the four vectors
$\partial_{\mu}{\varepsilon}^{a}$.

Then, in the absence of restrictions on the group parameters, in order to
satisfy eq. 2.1 the expression in curly brackets should vanish

$\displaystyle{\left\\{\frac{\delta{\cal L}_{GF}}{\delta
A^{a}_{\mu}}+T^{a}_{bc}\left[\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}A^{b}_{\nu}}A^{c}_{\nu}+\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}B^{b}}B^{c}\right]\right\\}=0.}$ (2.3)

It is instructive to check that eq. 2.1 provides restrictions on the gauge
parameter ${\varepsilon}_{r}$ if Lorentz covariance is not required, as it
happens for the temporal gauge. In fact, in this case the term in square
brackets and $\delta{\cal L}_{GF}/\delta A^{a}_{i},i=1,2,3$ vanish,(see
below), and eq. 2.1 gives the restriction $\partial_{0}{\varepsilon}_{r}=0$,
i.e. the residual group of the temporal gauge.

It remains to exploit eq. 2.2, which yields the following characterization

$\displaystyle{\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}A^{a}_{\nu}}+\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\nu}A^{a}_{\mu}}=g_{\mu\,\nu}L^{a},\qquad\square\,{\varepsilon}^{a}_{r}(x)=0,}$
(2.4)

with $L^{a}\neq 0,$ a scalar function of the fields. Clearly, if $L^{a}=0$,
the condition $\square\,{\varepsilon}^{a}=0$ is not required, but then the
absence of conditions on the ${\varepsilon}^{a}$ would imply ${\cal
G}_{r}={\cal G}$, i.e. no effective gauge fixing.

Hence, the possible residual subgroup should be characterized by gauge
parameters ${\varepsilon}^{a}(x)$ satisfying the (free) wave equation. Indeed,
as we shall see, this works in the abelian case, but not in the non-abelian
case.

In fact, in the non-abelian case there cannot be a residual group ${\cal
G}_{r}$ characterized by gauge parameters ${\varepsilon}_{r}$ satisfying
$\square{\varepsilon}_{r}=0$, since this condition keeps being satisfied by
linear combination of such functions (vector space structure), but not by
their products in conflict with the group composition rule.

This is displayed by the Zassenhaus’ formula, [6, 7], which is known to
converge for small values of the parameters; in fact, it gives

$e^{i{\varepsilon}^{a}_{1}(x)t^{a}}\,e^{i{\varepsilon}^{b}_{2}(x)t^{b}}=e^{i{\varepsilon}^{a}_{1}(x)t^{a}+i{\varepsilon}^{b}_{2}(x)t^{b}}\,e^{{\varepsilon}^{a}_{1}(x){\varepsilon}^{b}_{2}(x)[t^{a},\,t^{b}]}...$

The first term on the right hand side is the exponential of an element of the
Lie algebra, and therefore an element of the group, but the other factors,
which do not vanish in the non-abelian case, do not define elements of a gauge
group characterized by the above gauge parameters, because the product of
gauge functions which satisfy the wave equation does not satisfy the wave
equation.

In conclusion, quite generally  in the non-abelian case it is impossible to
have a covariant Gauss operator satisfying i)-ii), as a function of
$A^{a}_{\mu}$ and possible Nakanishi-Lautrup fields; hence the properties of
Feynman Gauss operator are in general excluded. Lorentz covariance can be
rescued by relaxing the condition of a nontrivial residual gauge group, as is
the case in BRST quantization.

###  2.c Gauss operator and covariant conservation law

By the defining eq. 1.3, where, thanks to Noether theorem, $J^{a}_{\mu}$ is
the current which generate the unbroken global gauge group $G$,
$\partial^{\mu}G_{\mu}=0$. Now, in the absence of a gauge fixing, the Gauss
law may also be written as

$\displaystyle{D^{\nu}F^{a}_{\nu\,\mu}=j^{a}_{\mu}(\psi),}$ (2.5)

where $D^{\mu}$ denotes the covariant derivative and $j^{a}_{\mu}(\psi)$ is
the contribution to the conserved current by the matter fields (the so-called
matter current). Both terms of this equation satisfy a covariant conservation
law

$\displaystyle{D^{\mu}D^{\nu}F^{a}_{\nu\,\mu}=D^{\mu}j^{a}_{\mu}(\psi)=0.}$
(2.6)

One may then ask whether such covariant conservation laws, strictly related to
the gauge group geometry, keep holding also in the presence of a gauge fixing.
Thus we look for Gauss operators which satisfy a covariant conservation law
(besides the ordinary conservation law), i.e.

$\displaystyle{D^{\mu}G^{a}_{\mu}=0.}$ (2.7)

We allow for gauge fixings which are functions of $A^{a}_{\mu}$ and of
auxiliary fields of the Nakanishi-Lautrup type $B^{a}_{i},i=1,...k$, which
transform as the adjoint representation of $G$, and possibly $G$ invariant
fields $B_{i}$.

We shall see that if

a) the gauge fixing does not depend on the derivatives of $A^{a}_{\mu}$, and

b) the current

$j^{a}_{\mu}(B)=-\frac{\delta{\cal L}}{\delta\partial_{\mu}B^{a}_{i}}\delta
B^{a}_{i}=-\frac{\delta{\cal L}_{GF}}{\delta\partial_{\mu}B^{a}_{i}}\delta
B^{a}_{i}$

satisfies a covariant conservation law (in particular $j^{a}_{\mu}(B)=0$ if
${\cal L}_{GF}$ does not depend on the derivatives of the $B^{a}_{i}$)

then, the corresponding Gauss operator satisfies a covariant conservation law.
Indeed we shall see that, contrary to what implicitly argued in the
literature, there are solutions of eq. 2.7 and of the continuity equation.

We denote by ${\cal L}_{0}$ the gauge invariant Lagrangian , so that ${\cal
L}={\cal L}_{0}+{\cal L}_{GF}$. By Noether theorem, the invariance of ${\cal
L}$ under the global gauge group $G$ determines the conserved current
$J^{a}_{\mu}$ which generates the transformations of $G$:

$\displaystyle J^{a}_{\mu}=-i\,T^{a}_{bc}\left[\frac{\delta{\cal
L}}{\delta\,\partial_{\mu}A^{b}_{\nu}}A^{c}_{\nu}+\frac{\delta{\cal
L}}{\delta\,\partial_{\mu}B^{b}_{i}}B^{c}_{i}\right]+j^{a}_{\mu}(\psi)=$ (2.8)
$\displaystyle=-i\,T^{a}_{bc}\left[\frac{\delta{\cal
L}_{0}}{\delta\,\partial_{\mu}A^{b}_{\nu}}A^{c}_{\nu}+\frac{\delta{\cal
L}_{GF}}{\delta\,\partial_{\mu}B^{b}_{i}}B^{c}_{i}\right]+j^{a}_{\mu}(\psi)=j^{a}_{\mu}(A)+j^{a}_{\mu}(B_{i})+j^{a}_{\mu}(\psi),$
(2.9)

where $j^{a}_{\mu}(\psi)$ denotes the contribution of the matter fields, the
so-called matter current, and
$j^{a}_{\mu}(A)=-iT^{a}_{b\,c}\,(A^{b})^{\nu}F^{c}_{\nu\mu}$.

The second Noether theorem applied to ${\cal L}-{\cal L}_{GF}$ provides
information on the effect of ${\cal L}_{GF}$ in yielding eq. 1.3.

The coefficient of $\partial_{\mu}{\varepsilon}$ in the infinitesimal
variation of ${\cal L}_{0}={\cal L}-{\cal L}_{GF}$ must vanish and one gets

$\displaystyle-J^{a}_{\mu}+\frac{\delta{\cal L}}{\delta
A^{a}_{\mu}}-\frac{\delta{\cal L}_{GF}}{\delta
A^{a}_{\mu}}+j^{a}_{\mu}(B_{i})=-J^{a}_{\mu}+\partial^{\nu}\frac{\delta{\cal
L}_{0}}{\delta\partial_{\nu}A^{a}_{\mu}}-\frac{\delta{\cal L}_{GF}}{\delta
A^{a}_{\mu}}+j^{a}_{\mu}(B_{i})=$
$\displaystyle=-J^{a}_{\mu}+\partial^{\nu}F^{a}_{\nu\,\mu}-\frac{\delta{\cal
L}_{GF}}{\delta A^{a}_{\mu}}+j^{a}_{\mu}(B_{i})=0,$ (2.10)

where we have used that

1) ${\cal L}_{GF}$ does not depend on the derivatives of $A^{a}_{\mu}$,

2) ${\cal L}_{0}$ does not depend on the auxiliary fields $B^{a}_{i}$, so that
the derivatives of ${\cal L}_{GF}$ with respect to the fields $B^{a}_{i}$ and
to their derivatives may be replaced by the corresponding derivatives of
${\cal L}$, allowing to use the equations of motion for the fields
$B^{a}_{i}$.

Hence, the Gauss operator is given by

$\displaystyle{G^{a}_{\mu}=-\frac{\delta{\cal L}_{GF}}{\delta
A^{a}_{\mu}}+j^{a}_{\mu}(B_{i}).}$ (2.11)

and eq. 2.10 gives

$\displaystyle{D^{\nu}F^{a}_{\nu\mu}=j^{a}_{\mu}(\psi)+j^{a}_{\mu}(B)-G^{a}_{\mu}.}$
(2.12)

Now, the matter current satisfies a covariant conservation law and, by
assumption so does also $j^{a}_{\mu}(B_{i})$:

$D^{\mu}j^{a}_{\mu}(B_{i})=0.$

Then, since $D^{\nu}D^{\mu}F^{a}_{\nu\,\mu}=0$, eq. 2.10 gives

$\displaystyle{D^{\mu}G^{a}_{\mu}=0.}$ (2.13)

###  2.d Solutions of the equations
$\partial^{\mu}G^{a}_{\mu}=0,\;D^{\mu}G^{a}_{\mu}=0$

Given a solution $G^{a}_{\mu}$ of the conservation law
$\partial^{\mu}G^{a}_{\mu}=0$, the additional covariant conservation law
requires

$\displaystyle{T^{a}_{b\,c}\,(A^{b})^{\mu}G^{c}_{\mu}=0.}$ (2.14)

The vanishing of the left hand side may be due either to the summation over
the Lorentz indices or to the summation over the $G$-group indices $b,c$ (for
each Lorentz index $\mu$).

We have already shown that in the non-abelian case the equation
$\partial^{\mu}G^{a}_{\mu}=0$ does not have Lorentz covariant solutions
fulfilling our requirements. One can see that in the non-abelian case also eq.
2.14 does not admit Lorentz covariant solutions.

In fact, the invariance of the solutions under the corresponding residual
group requires

$T^{a}_{b\,c}\,\partial^{\mu}{\varepsilon}^{b}_{r}(x)\,G^{c}_{\mu}=0.$

Since the parameters $\partial^{\mu}{\varepsilon}_{r}(x)$ span a Lorentz
covariant vector subspace and no such subspace exits except the whole Lorentz
covariant vector space,

$\displaystyle(v^{b})^{\mu}\,T^{a}_{b\,c}\,G^{c}_{\mu}=0,$ (2.15)

for any Lorentz vector $v^{b}_{\mu}$. Therefore $G^{c}_{\mu}=0$ and ${\cal
G}_{r}=\text{1}$.

Thus, one must look for solutions which violate Lorentz covariance, still
keeping the conditions i)-iv).

Now, the covariance of eq. 1.3 under the unbroken global group requires that
the Gauss operator must transform according to the adjoint representation.

Therefore, from a group theoretical point of view, in terms of the
(irreducible) adjoint representation of $SU(N)$, eq. 2.13 may be written in
the form of scalar product

$(G,T^{a}A)=0,$

where $A$ and $G$ are vectors of the (irreducible) adjoint representation.

Now, in an irreducible representation given any vector by applying the
generators $T^{a}$ and the identity one reaches any other vector; therefore
the above equation requires $G\sim A$.

Thus, the solutions of eq. 2.13 are $G^{a}_{\mu}=A^{a}_{\mu}B$, with $B$ an
operator invariant under $G$ transformations.555 This is easily obtained in
the case $G=SU(2)$, since then eq. 2.13 requires the vanishing of the vector
product $\bf{A}_{\mu}\wedge\bf{G}_{\mu}$.

A very relevant example is provided by the Gauss operator of the temporal
gauge, characterized by $G^{a}_{i}=0,\,\partial_{0}G^{a}_{0}=0$. It is
uniquely selected by the requirement of rotational covariance of
$G^{a}_{\mu}$, which implies that ${\varepsilon}(x)\in{\cal
G}_{r}\Rightarrow{\varepsilon}_{R}(x)={\varepsilon}(Rx)$ is also in ${\cal
G}_{r}$ for any rotation $R$. Then eq. 2.15 is true for any Lorentz vector
$v^{b}_{\mu}$ with fixed $v^{b}_{0}$, hence $G^{a}_{i}=0$ concluding the
proof.

Its explicit form is obtainable by the gauge fixing term ${\cal
L}_{GF}=A^{a}_{0}\,B^{a}$, with $B^{a}$ a Nakanishi-Lautrup field which plays
the role of a Lagrange multiplier leading to $A^{a}_{0}=0$.

Then, one has

$\displaystyle{G^{a}_{\mu}=-\delta{\cal L}_{GF}/\delta
A^{a}_{\mu}=-\delta_{\mu,0}\,B^{a}_{\mu},}$ (2.16)

and eq. 2.13 is obviously satisfied since $G^{a}_{i}=0$ and $A^{a}_{0}=0$.

In this case, the residual local covariance group of eq. 1.3 is given by time
independent gauge parameters, which characterize the  temporal gauge. Indeed,
the constraint of eq. 2.1 is satisfied by unrestricted
$\partial_{i}{\varepsilon}^{a}$, which require $\delta{\cal L}_{GF}/\delta
A^{a}_{i}=0$, and by $\partial_{0}{\varepsilon}^{a}=0$, which allows for
$\delta{\cal L}_{GF}/\delta A^{a}_{0}\neq 0$.

This realization of the temporal gauge is particularly convenient because it
directly provides the violation of the Local Gauss Law in terms of the time
independent Nakanishi-Lautrup local field $B^{a}$. It is one of the basic
fields which generate the local field algebra and it does not arise as a
composite field with the related problem of point splitting regularization;
thus, its algebraic properties are derivable from canonical quantization.

A similar pattern is realized by the axial gauge or by variations of it, like
e.g. the $n^{\mu}A^{a}_{\mu}=0$ gauges, with $n_{\mu}$ timelike, spacelike or
lightlike. In this case, rotational covariance is replaced by the covariance
under the stability group of $n_{\mu}$.

##  3 Conclusion

We have shown that it is impossible to quantize non-abelian Yang-Mills gauge
theories with a Gauss operator which is Lorentz covariant and generates a
residual subgroup of local gauge transformations. This excludes the
possibility of having the properties of Feynman quantization of Quantum
Electrodynamics. In BRST quantization, Lorentz convariance is preserved by
breaking ${\cal G}$ to the identity, with BRST transformations as the _new_
invariance group. If the covariance condition is relaxed, one obtains a
characterization of the Gauss operators which satisfy (gauge) covariant
conservation laws and generate non-trivial subgroups of local gauge
transformations.

It is worthwhile to remark that most of the non-perturbative results on gauge
field theories666See the discussion in [1, 2, 4]. have been obtained by using
such a class of Gauss operators, thanks to their properties of a) preserving
locality, b) leaving a residual local gauge group unbroken (helpful for
renormalization), whose topology plays a crucial role in solving the $U(1)$
problem and c) satisfying a covariant conservation law, which allows for a
simple control of its action on the matter fields.

## References

  * [1] F. Strocchi, An introduction to non-perturbative foundations of quantum field theory, Oxford University Press 2013, 2016.
  * [2] F. Strocchi, Symmetry Breaking in the Standard Model. A non-perturbative outlook, Scuola Normale Superiore, Pisa, 2019.
  * [3] F. Strocchi, Local Gauss law and local gauge symmetries in QFT, arXiv:2301.06543v2 [hep-th].
  * [4] F. Strocchi, Symmetry Breaking, third Edition, Springer Theoretical and Mathematical Physics, 2021
  * [5] S. Weinberg, The Quantum Theory of Fields, Vol. II, Cambridge University Press 1996.
  * [6] W. Magnus, Commun. Pure Appl. Math. 7, 649, (1954).
  * [7] R.M. Wilcox, J. Math. Phys., 8, 962 (1967).


# On groups with BFC-covered word values

Eloisa Detomi  Dipartimento di Matematica “Tullio Levi-Civita”, Università di
Padova, Via Trieste 63, 35121 Padova, Italy<EMAIL_ADDRESS>,  Marta
Morigi  Dipartimento di Matematica, Università di Bologna
Piazza di Porta San Donato 5
40126 Bologna
Italy<EMAIL_ADDRESS>and  Pavel Shumyatsky  Department of Mathematics,
University of Brasilia
Brasilia-DF
70910-900 Brazil<EMAIL_ADDRESS>
###### Abstract.

For a group $G$ and a positive integer $n$ write $B_{n}(G)=\\{x\in
G:|x^{G}|\leq n\\}$. If $s\geq 1$ and $w$ is a group word, say that $G$
satisfies the $(n,s)$-covering condition with respect to the word $w$ if there
exists a subset $S\subseteq G$ such that $|S|\leq s$ and all $w$-values of $G$
are contained in $B_{n}(G)S$. In a natural way, this condition emerged in the
study of probabilistically nilpotent groups of class two. In this paper we
obtain the following results.

Let $w$ be a multilinear commutator word on $k$ variables and let $G$ be a
group satisfying the $(n,s)$-covering condition with respect to the word $w$.
Then $G$ has a soluble subgroup $T$ such that $[G:T]$ and the derived length
of $T$ are both $(k,n,s)$-bounded. (Theorem 1.1.)

Let $k\geq 1$ and $G$ be a group satisfying the $(n,s)$-covering condition
with respect to the word $\gamma_{k}$. Then (1) $\gamma_{2k-1}(G)$ has a
subgroup $T$ such that $[\gamma_{2k-1}(G):T]$ and $|T^{\prime}|$ are both
$(k,n,s)$-bounded; and (2) $G$ has a nilpotent subgroup $U$ such that $[G:U]$
and the nilpotency class of $U$ are both $(k,n,s)$-bounded. (Theorem 1.2.)

###### Key words and phrases:

conjugacy classes, word values, nilpotent groups

###### 2020 Mathematics Subject Classification:

20E45,20F24,20F19

The first and second authors are members of GNSAGA (INDAM), and the third
author was supported by FAPDF and CNPq.

##  1\. Introduction

An interesting covering condition has emerged in the study of
probabilistically nilpotent groups of class two [8]. If $G$ is a finite group,
we set

$d_{k}(G)=|\\{(x_{1},...,x_{k+1})\in
G^{k+1}:[x_{1},...,x_{k+1}]=1\\}|/|G|^{k+1},$

and we say that $G$ is probabilistically nilpotent of class $k$ if $d_{k}(G)$
is bounded away from zero.

The well-known theorem of P. M. Neumann [12] says that if $G$ is
probabilistically nilpotent of class $1,$ i.e. $d_{1}(G)\geq\epsilon>0$, then
$G$ has a subgroup $H$ such that $[G:H]$ and $|H^{\prime}|$ are both
$\epsilon$-bounded, that is, $G$ is bounded-by-abelian-by-bounded. Throughout,
we say a group $G$ is $X$-by-$Y$ to mean that there is a normal $X$-subgroup
$N$ in $G$ such that $G/N$ is a $Y$-group. We say “$(a,b,c,\dots)$-bounded” to
mean “bounded by a function of $a,b,c,\dots$ only. Where the parameters
$a,b,c,\dots$ are clear from the context we often just say “bounded”. The main
result of [8] states that if $d_{2}(G)\geq\epsilon>0$, then $G$ has a subgroup
$H$ such that $[G:H]$ and $|\gamma_{4}(H)|$ are both $\epsilon$-bounded. The
structure of finite groups $G$ with $d_{k}(G)\geq\epsilon>0$, where $k\geq 3$
remains a mystery.

Much of the paper [8] consists of a study of groups satisfying a certain
commutator covering condition. For a group $G$ and a positive integer $n$
write $B_{n}(G)=\\{x\in G:|x^{G}|\leq n\\}$. It was shown in [8] that if
$d_{2}(G)\geq\epsilon>0$, then there are $\epsilon$-bounded positive integers
$n$ and $s$ with the property that $G$ has a bounded index subgroup $K$ and a
subset $S\subseteq K$ such that $|S|\leq s$ and $[x,y]\in B_{n}(K)S$ for any
$x,y\in K$. Note that the condition makes sense for infinite as well as finite
groups, and is satisfied by groups where all commutators have conjugacy
classes of bounded size (the case S = {1}), which were studied in [5]. A
theorem obtained in [8] says that any group $K$ satisfying the above condition
has a subgroup $T$ such that $[K:T]$ and $|\gamma_{4}(T)|$ are both finite and
$(n,s)$-bounded.

Given a group word $w=w(x_{1},\dots,x_{k})$, write $G_{w}$ for the set of all
values $w(g_{1},\ldots,g_{k})$, where $g_{1},\ldots,g_{k}$ are elements of
$G$. We say that the group $G$ satisfies the $(n,s)$-covering condition with
respect to the word $w$ if there exists a subset $S\subseteq G$ such that
$|S|\leq s$ and

(1) $G_{w}\subseteq B_{n}(G)S.$

In the present paper we study groups $G$ admitting a word $w$ for which the
set of $w$-values satisfies the above covering condition. In particular, we
work with multilinear commutator words, sometimes also called in the
literature outer commutator words. Examples of multilinear commutators include
the lower central words $\gamma_{k}(x_{1},\dots,x_{k})=[x_{1},\dots,x_{k}]$.
Of course, $\gamma_{k}(G)=\langle G_{\gamma_{k}}\rangle$ is the $k$-th term of
the lower central series of $G$.

We obtain the following results.

######  Theorem 1.1.

Let $w$ be a multilinear commutator word on $k$ variables and let $G$ be a
group satisfying the $(n,s)$-covering condition with respect to the word $w$.
Then $G$ has a soluble subgroup $T$ such that $[G:T]$ and the derived length
of $T$ are both $(k,n,s)$-bounded.

######  Theorem 1.2.

Let $k\geq 1$ and $G$ be a group satisfying the $(n,s)$-covering condition
with respect to the word $\gamma_{k}$. Then

  1. (1)

$\gamma_{2k-1}(G)$ has a subgroup $T$ such that $[\gamma_{2k-1}(G):T]$ and
$|T^{\prime}|$ are both $(k,n,s)$-bounded.

  2. (2)

$G$ has a nilpotent subgroup $U$ such that $[G:U]$ and the nilpotency class of
$U$ are both $(k,n,s)$-bounded.

Note that in the cases where $G_{w}\subseteq B_{n}(G)$ the above theorems have
been established in [4] and [14]. All these results are natural
generalizations of the theorem of B. H. Neumann saying that if $G$ is an
$n$-BFC-group, that is $G=B_{n}(G)$, then $G^{\prime}$ has finite $n$-bounded
order [11].

##  2\. A technical result

We recall that multilinear commutator words (multilinear commutators, for
short), are recursively defined as follows: The word $w(x)=x$ in one variable
is a multilinear commutator; if $u$ and $v$ are multilinear commutators
involving disjoint sets of variables, then the word $w=[u,v]$ is a multilinear
commutator, and all multilinear commutators are obtained in this way. Well-
known examples of multilinear commutators are the aforementioned lower central
words $\gamma_{k}$ and the derived words $\delta_{i}(x_{1},\dots,x_{2^{i}}),$
defined by $\delta_{0}=x_{1}$ and

$\delta_{i+1}(x_{1},\dots,x_{2^{i+1}})=[\delta_{i}(x_{1},\dots,x_{2^{i}}),\delta_{i}(x_{2^{i}+1},\dots,x_{2^{i+1}})].$

The main result of this section is the following quite technical proposition,
which is a stronger form of Theorem 1.1 in [4]. As usual, $\langle X\rangle$
denotes the (sub)group generated by a set $X$.

######  Proposition 2.1.

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator, $G$ a group and
$A_{1}$,…,$A_{k}$ normal subgroups of $G$. Let $B=B_{n}(G)$, $H=\langle
B\rangle$, and $Y=\\{w(g_{1},\dots,g_{n})\,:\,g_{i}\in A_{i}\\}$. Assume that
$Y\subseteq B.$ Then the commutator subgroup $[H,\langle Y\rangle]$ has finite
$(n,k)$-bounded order.

We will require combinatorial techniques developed in [3].

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator. If
$A_{1},\dots,A_{k}$ are subsets of a group $G$, we write
$w(A_{1},\dots,A_{k})$ for the subgroup generated by the $w$-values
$w(a_{1},\dots,a_{k})$ with $a_{i}\in A_{i}$.

Let $I$ be a subset of $\\{1,\dots,k\\}$. Suppose that we have a family
$A_{i_{1}},\dots,A_{i_{s}}$ of subsets of $G$ with indices running over $I$
and another family $B_{l_{1}},\dots,B_{l_{t}}$ of subsets with indices running
over $\\{1,\dots,k\\}\setminus I.$ We write

$w_{I}(A_{i};B_{i})$

for $w(X_{1},\dots,X_{k})$, where $X_{j}=A_{j}$ if $j\in I$, and $X_{j}=B_{j}$
otherwise. On the other hand, whenever $a_{i}\in A_{i}$ for $i\in I$ and
$b_{i}\in B_{i}$ for $i\in\\{1,\dots,k\\}\setminus I$, the symbol
$w_{I}(a_{i};b_{i})$ stands for the element $w(g_{1},\dots,g_{k})$, where
$g_{j}=a_{j}$ if $j\in I$, and $g_{j}=b_{j}$ otherwise.

The following results are Lemma 2.4, Lemma 2.5 and Lemma 4.1 of [3],
respectively.

######  Lemma 2.2.

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator. Assume that $M$ is a
normal subgroup of a group $G$. Let $g_{1},\dots,g_{k}\in G$, $h\in M$ and fix
$l\in\\{1,\dots,k\\}$. Then there exist $y_{j}\in g_{j}^{M}$, for
$j=1,\dots,k$, such that

$\displaystyle
w_{\\{l\\}}(g_{l}h;g_{i})=w(y_{1},\dots,y_{k})w_{\\{l\\}}(h;g_{i}).$

######  Lemma 2.3.

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator. Let $G$ be a group
and let $A_{1},\dots,A_{k},M$ be normal subgroups of $G$. Assume that

$w(y_{1}(A_{1}\cap M),\dots,y_{k}(A_{k}\cap M))=1$

for some elements $y_{i}\in A_{i}$. Then

$w_{I}(y_{i}(A_{i}\cap M);A_{i}\cap M)=1,$

for any subset $I$ of $\\{1,\dots,k\\}$.

######  Lemma 2.4.

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator and let $I$ be a
subset of $\\{1,\dots,k\\}$. Let $M$ be a normal subgroup of a group $G$.
Assume that

$w_{J}(A_{j};A_{j}\cap M)=1\quad\textrm{for every}\ J\subsetneq I.$

Suppose we are given elements $y_{i}\in A_{i}$ with $i\in I$ and elements
$m_{r}\in A_{r}\cap M$ with $r\in\\{1,\dots,k\\}$. Then we have

$w_{I}(y_{i}m_{i};m_{i})=w_{I}(y_{i};m_{i}).$

Let $G$ be a group generated by a set $X$ such that $X=X^{-1}$. Given an
element $g\in G$, we write $l_{X}(g)$ for the minimal number $l$ with the
property that $g$ can be written as a product of $l$ elements of $X$. Clearly,
$l_{X}(g)=0$ if and only if $g=1$. We call $l_{X}(g)$ the length of $g$ with
respect to $X$. The following result is Lemma 2.1 in [5].

######  Lemma 2.5.

Let $T$ be a group generated by a set $X=X^{-1}$ and let $M$ be a subgroup of
finite index $n$ in $T$. Then each coset $Mb$ contains an element $g$ such
that $l_{X}(g)\leq n-1$.

Recall that if $G$ is a group, $a\in G$ and $H$ is a subgroup of $G$, then
$[H,a]$ denotes the subgroup of $G$ generated by all commutators of the form
$[h,a]$, where $h\in H$. It is well-known that $[H,a]$ is normalized by $a$
and $H$.

In the sequel we will assume the hypotheses of Proposition 2.1; therefore
$w=w(x_{1},\dots,x_{k})$ is a multilinear commutator, $A_{1}$,…,$A_{k}$ are
normal subgroups of a group $G$, $H=\langle B\rangle$, where $B=B_{n}(G)$, and
every element of the set

$Y=\\{w(g_{1},\dots,g_{n})\,:\,g_{i}\in A_{i}\\}$

has at most $n$ conjugates in $G$.

######  Lemma 2.6.

The subgroup $[H,y]^{G}$ has $n$-bounded order for every $y\in Y$.

###### Proof.

Choose $y\in Y$. Since $C_{H}(y)$ has index at most $n$ in $H=\langle
B\rangle$, by Lemma 2.5 we can choose elements $h_{1},\dots,h_{n}$ such that
$l_{B}(h_{i})\leq n-1$ and $[H,y]$ is generated by the commutators
$[h_{i},y]$. Note that, as $l_{B}(h_{i})\leq n-1$, each $h_{i}$ has at most
$n^{n-1}$ conjugates in $G$, so $C_{H}(h_{i})$ has finite $n$-bounded index in
$H$. As $C_{H}(y)$ has index at most $n$ in $H,$ the centralizer $C$ of
$T=\langle y,h_{1},\dots,h_{n}\rangle$ in $H$ has finite $n$-bounded index in
$H$. Since $C\cap T$ is an $n$-bounded index subgroup contained in the center
of $T$, we deduce that $Z(T)$ has $n$-bounded index in $T$. Thus, Schur’s
theorem [13, 10.1.4] tells us that $T^{\prime}$ has finite $n$-bounded order.
As $[H,y]\leq T^{\prime}$, it follows that $[H,y]$ has $n$-bounded order.

Observe that $[H,y]$ has at most $n$ conjugates in $G$ and the conjugates
normalize each other. Thus, $[H,y]^{G}$ is a product of at most $n$ subgroups
that normalize each other and have $n$-bounded order. The lemma follows. ∎

The following lemma can be seen as a generalization of Lemma 2.4 in [4]. It
plays a central role in our arguments.

######  Lemma 2.7.

Let $M$ be a normal subgroup of $G$ of finite index $j$ and choose $a_{i}\in
A_{i}$ for $i=1,\dots,k$. Then there exist elements $\tilde{a}_{i}\in
a_{i}(M\cap A_{i})$, for $i=1,\dots,k$, and a normal subgroup $\tilde{M}$ of
$M$ of finite $(j,n)$-bounded index, such that the order of
$[H,w(\tilde{a}_{1}(A_{1}\cap\tilde{M}),\dots,\tilde{a}_{k}(A_{k}\cap\tilde{M}))]^{G}$
is finite and $n$-bounded.

###### Proof.

Consider the set

$S=\\{w(a_{1}u_{1},\dots,a_{k}u_{k})\,:\,u_{i}\in A_{i}\cap M\textrm{ for
every }i\\}.$

Choose $\tilde{a}_{i}\in a_{i}(M\cap A_{i})$, for every $i$, such that
$a=w(\tilde{a}_{1},\dots,\tilde{a}_{k})$ has the maximum number of conjugates
in $H$ among all the elements of $S$, that is $|a^{H}|\geq|g^{H}|$ for all
$g\in S$.

By Lemma 2.5 we can choose $b_{1},\dots,b_{r}\in H$ such that
$l_{B}(b_{i})\leq n-1$ and $a^{H}=\\{a^{b_{i}}\,:\,i=1,\dots,r\\}$. Let
$\tilde{M}$ be the intersection of $M$ and all conjugates of $C_{G}(\langle
b_{1},\dots,b_{r}\rangle)$. Since $l_{B}(b_{i})\leq n-1$ and $C_{G}(x)$ has
index at most $n$ in $G$ for each $x\in B$, the subgroup $C_{G}(\langle
b_{1},\dots,b_{r}\rangle)$ has $n$-bounded index in $G$, and so $\tilde{M}$
has $(j,n)$-bounded index.

Consider the element $w(\tilde{a}_{1}v_{1},\dots,\tilde{a}_{k}v_{k})\in S$
where $v_{i}\in(A_{i}\cap\tilde{M})$, for $i=1,\dots,k$. We have

$w(\tilde{a}_{1}v_{1},\dots,\tilde{a}_{k}v_{k})=va,$

for some $v\in\tilde{M}\leq C_{G}(b_{1},\dots,b_{r})$. It follows that
$(va)^{b_{i}}=va^{b_{i}}$ for each $i=1,\dots,r$. Therefore the elements
$va^{b_{i}}$ form the conjugacy class $(va)^{H}$ because they are all
different and their number is the allowed maximum. So, for an arbitrary
element $h\in H$ there exists $b\in\\{b_{1},\dots,b_{r}\\}$ such that
$(va)^{h}=va^{b}$ and hence $v^{h}a^{h}=va^{b}$. Therefore
$[h,v]=v^{-h}v=a^{h}a^{-b}$ and so
$[h,v]^{a}=a^{-1}a^{h}a^{-b}a=[a,h][b,a]\in[H,a].$ Thus $[H,v]^{a}\leq[H,a]$
and

$[H,va]=[H,a][H,v]^{a}\leq[H,a].$

This shows that
$[H,w(\tilde{a}_{1}(A_{1}\cap\tilde{M}),\dots,\tilde{a}_{k}(A_{k}\cap\tilde{M}))]\leq[H,a]$.
Lemma 2.6 states that $[H,a]^{G}$ has $n$-bounded order. The result follows. ∎

For the reader’s convenience, the most technical part of our argument is
isolated in the following proposition.

######  Proposition 2.8.

Let $I\subseteq\\{1,\dots,k\\}$ with $|I|=s$. Under the hypotheses of
Proposition 2.1, assume that $G$ contains a normal subgroup $R$ of finite
order $r$ and a normal subgroup $M$ of finite index $j$ such that

$[H,w_{J}(A_{i};A_{i}\cap M)]\leq R\quad\textrm{for every}\ J\subsetneq I.$

Then there exists a finite normal subgroup $R_{I}$ of $G$ of
$(r,j,n,s)$-bounded order with $R\leq R_{I}$ and a normal subgroup $M_{I}$ of
$G$ of $(j,n,s)$-bounded index with $M_{I}\leq M$ such that

$[H,w_{I}(A_{i};A_{i}\cap M_{I})]\leq R_{I}.$

###### Proof.

For every $i=1,\dots,k$ choose a left transversal $C_{i}$ of $A_{i}\cap M$ in
$A_{i}$. Observe that $|C_{i}|\leq j$. Let $\Omega$ be the set of $k$-tuples
$\underline{c}=(c_{1},\dots,c_{k})$ where $c_{i}\in C_{i}$ if $i\in I$ and
$c_{i}=1$ otherwise. Notice that the size of $\Omega$ is at most $j^{s}$. For
any $k$-tuple $\underline{c}=(c_{1},\dots,c_{k})\in\Omega$, by Lemma 2.7,
there exist elements $d_{i}\in c_{i}(A_{i}\cap M)$, with $i=1,\dots,k$, and a
normal subgroup $M_{\underline{c}}$ of $(j,n)$-bounded index in $G$ such that
the order of

$[H,w(d_{1}(A_{1}\cap M_{\underline{c}}),\dots,d_{k}(A_{k}\cap
M_{\underline{c}}))]^{G}$

is $n$-bounded. Let

$\displaystyle M_{I}$ $\displaystyle=$ $\displaystyle
M\cap\bigg{(}\bigcap_{\underline{c}\in\Omega}M_{\underline{c}}\bigg{)},$
$\displaystyle R_{I}$ $\displaystyle=$ $\displaystyle
R\,\prod_{\underline{c}\in\Omega}[H,w(d_{1}(A_{1}\cap
M_{\underline{c}}),\dots,d_{k}(A_{k}\cap M_{\underline{c}}))]^{G}.$

As $|\Omega|\leq j^{s}$, it follows that $M_{I}$ has $(j,n,s)$-bounded index
in $G$ and $R_{I}$ has $(r,j,n,s)$-bounded order.

Let $Z/R_{I}$ be the center of $HR_{I}/R_{I}$ in the quotient group $G/R_{I}$
and let $\bar{G}=G/Z$. The image of a subgroup $U$ of $G$ in $\bar{G}$ will be
denoted by $\bar{U}$, and similarly for the image of an element.

Pick an arbitrary element $w_{I}(a_{i},h_{i})\in w_{I}(A_{i};A_{i}\cap M_{i})$
(where $a_{i}\in A_{i}$ for $i\in I$, and $h_{i}\in A_{i}\cap M_{I}$
otherwise). Define the $k$-tuple $\underline{c}=(c_{1},\dots,c_{k})\in\Omega$
by $a_{i}\in c_{i}(A_{i}\cap M)$ if $i\in I$ and $c_{i}=1$ otherwise. Let
$d_{1},\dots,d_{k}$ be the elements as above, corresponding to the $k$-tuple
$\underline{c}$. Then

$[H,w(d_{1}(A_{1}\cap M_{I}),\dots,d_{k}(A_{k}\cap M_{I}))]\leq R_{I},$

that is

$\overline{w(d_{1}(A_{1}\cap M_{I}),\dots,d_{k}(A_{k}\cap M_{I}))}=1,$

in the quotient group $\bar{G}=G/Z$. By Lemma 2.3, we deduce that

(2) $\overline{w_{I}(d_{i}(A_{i}\cap M_{I});A_{i}\cap M_{i})}=1.$

Moreover, as $c_{i}(A_{i}\cap M)=d_{i}(A_{i}\cap M)$, we have that
$a_{i}=d_{i}v_{i}$ for some $v_{i}\in A_{i}\cap M$. It also follows from our
assumptions that

$\overline{w_{J}(A_{i};A_{i}\cap M)}=1$

for every proper subset $J$ of $I$. Thus we can apply Lemma 2.4 obtaining that

$w_{I}(\overline{a}_{i};\overline{h}_{i})=w_{I}(\overline{d}_{i}\overline{v}_{i};\overline{h}_{i})=w_{I}(\overline{d}_{i};\overline{h}_{i})=1,$

where in the last equality we have used (2). Since $w_{I}(a_{i},h_{i})$ was an
arbitrary element of $w_{I}(A_{i};A_{i}\cap M_{i})$, it follows that

$\overline{w_{I}(A_{i};A_{i}\cap M_{i})}=1,$

that is

$[H,w(A_{i};A_{i}\cap M_{i})]\leq R_{I},$

as desired. ∎

Now the proof of Proposition 2.1 is an easy induction.

###### Proof of Proposition 2.1..

We will prove that for every $s=0,\dots,k$ the group $G$ contains a finite
normal subgroup $R_{s}$ of $(n,k)$-bounded order and a normal subgroup $M_{s}$
of $(n,k)$-bounded index such that

$[H,w_{I}(A_{i};A_{i}\cap M_{s})]\leq R_{s}$

for every subset $I$ of $\\{1,\dots,k\\}$ with $|I|\leq s$. Once this is done,
the theorem will follow taking $s=k$.

Assume that $s=0$. We apply Lemma 2.7 with $M=G$ and $a_{i}=1$ for every
$i=1,\dots,k$. Thus there exist elements $a_{i}\in A_{i}$ for each
$i=1,\dots,k$ and a normal subgroup of $n$-bounded index $M_{0}$ of $G$, such
that the order of

$R_{0}=[H,w(a_{1}(A_{1}\cap M_{0}),\dots,a_{k}(A_{k}\cap M_{0}))]^{G}$

is $n$-bounded.

Let $Z/R_{0}$ be the center of $HR_{0}/R_{0}$ in the quotient group $G/R_{0}$
and let $\bar{G}=G/Z$. Since

$\overline{w(a_{1}(A_{1}\cap M_{0}),\dots,a_{k}(A_{k}\cap M_{0}))}=1,$

it follows from Lemma 2.3 that

$\overline{w(A_{1}\cap M_{0},\dots,A_{k}\cap M_{0})}=1,$

that is, $[H,w(A_{1}\cap M_{0},\dots,A_{k}\cap M_{0})]\leq R_{0}$. This proves
the result in the case $s=0$.

Now assume $s\geq 1$. Choose $I\subseteq\\{1,\dots,k\\}$ with $|I|=s$. By
induction, the hypotheses of Proposition 2.8 are satisfied with $R=R_{s-1}$
and $M=M_{s-1}$, so there exists a finite normal subgroup $R_{I}$ of $G$ of
$(n,k)$-bounded order with $R_{s-1}\leq R_{I}$ and a normal subgroup $M_{I}$
of $G$ of $(n,k)$-bounded index with $M_{I}\leq M_{s-1}$ such that

$[H,w_{I}(A_{i};(A_{i}\cap M_{I})]\leq R_{I}.$

Let

$M_{s}=\bigcap_{|I|=s}M_{I},\quad R_{s}=\prod_{|I|=s}R_{I},$

where the intersection (resp. the product) ranges over all subsets $I$ of
$\\{1,\dots,k\\}$ of size $s$.

As there is a $k$-bounded number of choices for $I$, it follows that $R_{s}$
(resp. $M_{s}$) has $(n,k)$-bounded order (resp. $(n,k)$-bounded index in
$G$). Note that $M_{s}\leq M_{s-1}$ and $R_{s-1}\leq R_{s}$. Therefore

$[H,w_{I}(A_{i};A_{i}\cap M_{s})]\leq R_{s}$

for every $I\subseteq\\{1,\dots,k\\}$ with $|I|\leq s$. This completes the
induction step. The proposition is established. ∎

##  3\. Proof of Theorem 1.1

Throughout this section we will assume that $w=w(x_{1},\dots,x_{k})$ is a
multilinear commutator word and the group $G$ satisfies the $(n,s)$-covering
condition:

(3) $G_{w}\subseteq BS,$

where $B=B_{n}(G)=\\{x\in G\,:\,|x^{G}|\leq n\\}$, and $S$ is a subset of $G$
of size $s$. We will show that $G$ has a soluble subgroup $T$ such that
$[G:T]$ and the derived length of $T$ are both $(k,n,s)$-bounded. Actually,
$T$ can be chosen of derived length at most $2k+1$.

We will use the following key property of multilinear commutators.

######  Lemma 3.1.

Let $w=w(x_{1},\dots,x_{k})$ be a multilinear commutator and let
$\delta_{k-1}$ be the $(k-1)$-th derived word. Then $G_{\delta_{k-1}}\subseteq
G_{w}$ for every group $G$.

###### Proof.

The proof is by induction on $k$. If $k=1$ then $w=\delta_{0}=x$ and the
result is true. So now assume that $k\geq 2$. Then $w=[v_{1},v_{2}]$ where
$v_{1},v_{2}$ are multilinear commutators on $k_{1}$ and $k_{2}$ variables
respectively, where clearly $k_{1},k_{2}\leq k-1$. By induction, every
$\delta_{k_{i}-1}$-value is a $v_{i}$-value. In particular, every
$\delta_{k-2}$-value is a $v_{i}$-value, for $i=1,2$, and the result follows.
∎

At some point, without loss of generality, we will be able to assume that $G$
satisfies the following “stability condition”.

(4) Any $w$-value $x$ such that $|x^{G}|\leq n^{2^{k}}$ is contained in $B$.

######  Lemma 3.2.

Suppose $G$ satisfies the condition (4). Let $H=\langle B\rangle$. Then
$w(H,G,\dots,G)$ has derived subgroup of finite $(n,k)$-bounded order.

###### Proof.

For any element $h\in H$, let $l_{B}(h)$ be the minimal number $j$ such that
$h$ can be written as a product of $j$ elements from $B$. Clearly $|h^{G}|\leq
n^{l_{B}(h)}$.

Assume there is an element $h\in H$ such that $w(h,g_{2},\dots,g_{k})\notin B$
for some $g_{i}\in G$ and choose $h$ in such a way that $l_{B}(h)$ is minimal.
Note that, if $h\in B$, then the element $w(h,g_{2},\dots,g_{k})$ can be
written as a product of $2^{k-1}$ conjugates of the elements $h$, $h^{-1}$,
whence, by the stability condition (4), $w(h,g_{2},\dots,g_{k})\in B$, a
contradiction.

Thus, $l_{B}(h)\geq 2$ and we can write $h=h_{1}h_{2}$ where
$l_{B}(h_{1}),l_{B}(h_{2})<l_{B}(h)$. By Lemma 2.2 there exists $y_{1}\in
h_{1}^{G}$ and $y_{j}\in g_{j}^{G}$, for $j=2,\dots,k$, such that

$w(h_{1}h_{2},g_{2},\dots,g_{k})=w(y_{1},y_{2},\dots,y_{k})w(h_{2},g_{2},\dots,g_{k}).$

By minimality, both factors on the right hand side of the above equality
belong to $B$. So $w(h,g_{2},\dots,g_{k})$ has at most $n^{2}$ conjugates, and
therefore, by the stability condition (4), it belongs to $B$, a contradiction.

This proves that for every $h\in H$ and every $g_{i}\in G$ the element
$w(h,g_{2},\dots,g_{k})$ belongs to $B$. So we can apply Proposition 2.1 to
deduce that $w(H,G,\dots,G)^{\prime}$ has finite $(n,k)$-bounded order. ∎

###### Proof of Theorem 1.1.

Note that it is sufficient to prove that $G$ has a subgroup $\tilde{T}$ such
that $[G:\tilde{T}]$ and $|\tilde{T}^{(2k)}|$ are both $(k,n,s)$-bounded,
since then $C_{\tilde{T}}(\tilde{T}^{(2k)})$ will be a soluble subgroup of
derived length at most $2k+1$ and $(k,n,s)$-bounded index in $G$.

Without loss of generality we may assume that $1\in S$. In what follows we
argue by induction on $s$. If $s=1$, then $G_{w}\subseteq B$ and it follows
from Theorem 1.2 in [4] that $w(G)^{\prime}$ has finite $(k,n)$-bounded order.
Since $G^{(k-1)}\leq w(G)$ by Lemma 3.1, we conclude that $G^{(k)}$ has finite
$(k,n)$-bounded order.

So assume $s\geq 2$ and let $A=2^{k}$. Suppose there is some commutator $x\in
G_{w}$ such that $n<|x^{G}|\leq n^{A}$. By the covering condition $x\in Ba$
for some nontrivial $a\in S$. This implies that $|a^{G}|\leq n^{A+1}$. Then
for any $g\in Ba$ we have $|g^{G}|\leq n^{A+2}$. Hence we can remove $a$ from
$S$ at the cost of increasing $n$ to $n^{A+2}$ and then apply induction on
$s$. Hence we may assume $B$ satisfies the stability condition (4): Any
$w$-value $x$ such that $|x^{G}|\leq n^{2^{k}}$ is contained in $B$.

Let $H=\langle B\rangle$. It follows from Lemma 3.2 that $w(H,G,\dots,G)$ has
commutator subgroup of finite $(k,n,s)$-bounded order. Note that
$H^{(k-1)}\leq w(H)$ by Lemma 3.1. Hence, $H^{(k)}\leq w(H)^{\prime}\leq
w(H,G,\dots,G)^{\prime}$ and we deduce that $H^{(k)}$ has finite
$(k,n,s)$-bounded order as well.

Note that $H$ is a normal subgroup of $G$. Taking into account the covering
condition observe that $w$ takes at most $s$ values in $G/H$. As $w$ is a
boundedly concise word (see [7]), it follows that the verbal subgroup $w(G/H)$
has $(s,k)$-bounded order. So the centralizer $C/H=C_{G/H}(w(G/H))$ has
$(s,k)$-bounded index. Moreover $G^{(k-1)}\leq w(G)$ by Lemma 3.1, whence
$C^{(k)}\leq H$. Therefore $C^{(2k)}\leq H^{(k)}$ has finite $(k,n,s)$-bounded
order. This completes the proof. ∎

##  4\. Proof of Theorem 1.2

The goal of this section is to furnish a proof of Theorem 1.2. We start with
some general facts that will be used in the proof.

If $K$ is a subgroup of a finite group $G$, the commuting probability
$\mathrm{Pr}(K,G)$ of $K$ in $G$ is the probability that a random element of
$K$ commutes with a random element of $G$. Thus,

$\mathrm{Pr}(K,G)=\frac{|\\{(x,y)\in K\times G\,:\,xy=yx\\}|}{|K|\,|G|}.$

Note that if $|x^{G}|\leq n$ for every $x\in K$, then
$\mathrm{Pr}(K,G)\geq{1}/{n}$. It was shown in [2] that if
$\mathrm{Pr}(K,G)\geq\epsilon>0$, then there is a normal subgroup $T\leq G$
and a subgroup $B\leq K$ such that the indices $[G:T]$ and $[K:B]$ and the
order of the commutator subgroup $[T,B]$ are $\epsilon$-bounded. Moreover, if
$K$ is normal in $G$, then we can assume that $B$ is normal in $G$ and $K\cap
T\leq Z_{3}(T)$ (see [2, Remark 2.7]).

We recall the following well-known criterion for nilpotency of a group. The
result is due to P. Hall [9].

######  Theorem 4.1.

Let $N$ be a nilpotent normal subgroup of a group $G$ such that $N$ and
$G/N^{\prime}$ are both nilpotent. Then $G$ is nilpotent of class bounded in
terms of the classes of $N$ and $G/N^{\prime}$.

The following results are Lemma 2.2 and Lemma 2.4 in [14].

######  Lemma 4.2.

Let $G$ be a group and $k,m$ positive integers.

  1. (1)

$|\gamma_{k}(G)|\leq m$ if and only if $|\gamma_{k}(H)|\leq m$ for any
finitely generated subgroup $H\leq G$.

  2. (2)

If $G$ is residually finite, $|\gamma_{k}(G)|\leq m$ if and only if
$|\gamma_{k}(Q)|\leq m$ for any finite quotient $Q$ of $G$.

######  Lemma 4.3.

Let $G$ be a metabelian group and suppose that $a,b\in G$ are $l$-Engel
elements. If $x\in\langle a,b\rangle$, then $x$ is $(2l+1)$-Engel.

We say for short that a group $A$, acting on a group $G$ by automorphisms,
acts coprimely on $G$ if the orders of $A$ and $G$ are finite and coprime,
that is, $(|A|,|G|)=1$. The following lemma records a standard result on
coprime action (see [1, (24.5)]).

######  Lemma 4.4.

Let $A$ be a finite group acting coprimely on a finite group $G$ such that
$(|A|,|G|)=1$. Then $[G,A,A]=[G,A]$.

Throughout the remaining part of the paper we assume the group $G$ satisfies
the $(n,s)$-covering condition with respect to the word $\gamma_{k}$:

$G_{\gamma_{k}}\subseteq BS,$

where $B=B_{n}(G)=\\{x\in G\,:\,|x^{G}|\leq n\\}$ and $S$ is a finite subset
of $G$ of size $s$.

First, we prove that, under the above hypothesis, $\gamma_{2k-1}(G)$ has a
subgroup $T$ such that $[\gamma_{2k-1}(G):T]$ and $|T^{\prime}|$ are both
$(k,n,s)$-bounded.

###### Proof of Theorem 1.2 (1)..

We may assume $1\in S$. If $s=1$, then the result holds by Theorem 1.2 in [4],
since $\gamma_{k}(G)^{\prime}$ turns out to have finite $(n,k)$-bounded order.

Assume $s\geq 2$ and argue by induction on $s$. Suppose there is a commutator
$x\in G_{\gamma_{k}}$ such that $n<|x^{G}|\leq n^{2^{k}}$. By the covering
condition $x\in Ba$ for some nontrivial $a\in S$. This implies that
$|a^{G}|\leq n^{2^{k}+1}$. Then for any $g\in Ba$ we have $|g^{G}|\leq
n^{2^{k}+2}$. Hence we can remove $a$ from $S$ at the cost of increasing $n$
to $n^{2^{k}+2}$ and then apply induction on $s$. Hence, we may assume $B$
satisfies the stability condition (4):

Any $\gamma_{k}$-value $x$ such that $|x^{G}|\leq n^{2^{k}}$ is contained in
$B$.

Let $H=\langle B\rangle$. It follows from Lemma 3.2 that
$[H,G,\dots,G]^{\prime}$ has finite $(k,n)$-bounded order.

Now it is sufficient to prove that $[H,G,\dots,G]$ has bounded index in
$\gamma_{2k-1}(G)$.

Note that in $G/H$ there are at most $s$ values of the word $\gamma_{k}$. As
$\gamma_{k}$ is boundedly concise (see [7]), we deduce that $\gamma_{k}(G/H)$
has finite bounded order. Therefore the index of $H\cap\gamma_{k}(G)$ in
$\gamma_{k}(G)$ is bounded.

Let $\overline{G}$ be the quotient group of $G$ over $[H,G,\dots,G]$. The
image $\overline{H}$ of $H$ in $\overline{G}$ is contained in
$Z_{k-1}(\overline{G}).$ Here and throughout $Z_{i}(M)$ denotes the $i$th term
of the upper central series of a group $M$. Since the index of
$\overline{H}\cap\gamma_{k}(\overline{G})$ in $\gamma_{k}(\overline{G})$ is
bounded, we deduce that $Z_{k-1}(\overline{G})\cap\gamma_{k}(\overline{G})$
has bounded index in $\gamma_{k}(\overline{G})$.

It follows from Theorem B in [6] that $Z_{2{k-1}}(\overline{G})$ has bounded
index in $\overline{G}$. Hence, we deduce from Baer’s theorem (see [13,
4.5.1]) that $\gamma_{2(k-1)+1}(\overline{G})$ has finite bounded order.
Therefore $[H,G,\dots,G]$ has bounded index in $\gamma_{2k-1}(G)$, as claimed,
and the proof is complete. ∎

Now we proceed with the proof of the second part of Theorem 1.2. We need to
show that $G$ is nilpotent-of-bounded-class-by-bounded, with bounds depending
only on $s$, $n$ and $k$.

###### Proof of Theorem 1.2 (2).

By Theorem 1.1, we can assume that $G$ is soluble of bounded derived length.

By induction on the derived length, $G^{\prime}$ has a nilpotent subgroup $A$
of bounded index and class. In view of [10] we can assume that $A$ is
characteristic in $G^{\prime}$. It follows that $A$ is normal in $G$. If
$A^{\prime}\neq 1$ then by induction on the class of $A$ there is a nilpotent
subgroup of $G/A^{\prime}$ of bounded index and class. By Hall’s criterion we
are done in this case, so we may assume that $A$ is abelian. Since
$G^{\prime}/A$ is bounded, we may replace $G$ with a bounded-index subgroup
whose image in $G/A$ has class at most $2$. Thus, without loss of generality
we assume that $G$ is abelian-by-class-2. By another classical result of Hall,
any abelian-by-nilpotent group is locally residually finite (see [13,
15.4.1]).Thus, by Lemma 4.2, it is sufficient to prove the result for finite
quotients of finitely generated subgroups of $G$. Therefore without loss of
generality we may assume that $G$ is finite and $\gamma_{3}(G)$ is abelian.

Assume first that $G$ is metabelian. Let $N=G^{\prime}$. Set

${\bf G}=G\times\dots\times G\ \ (k-1\text{ factors}).$

If ${\bf g}=(g_{1},\dots,g_{k-1})\in{\bf G}$, let $N_{\bf
g}=[N,g_{1},\dots,g_{k-1}]$. As $N$ is abelian, the map
$a\mapsto[a,g_{1},\dots,g_{k-1}]$ is a homomorphism $N\to N_{g},$ thus the
subgroup $N_{\bf g}$ consists of $\gamma_{k}$-values, hence $N_{\bf
g}\subseteq BS$.

Suppose that $[a_{1},g_{1},\dots,g_{k-1}],[a_{2},g_{1},\dots,g_{k-1}]\in Bx$
for some $a_{1},a_{2}\in N$ and $x\in S$. Then

$[a_{1}a_{2}^{-1},g_{1},\dots,g_{k-1}]=[a_{1},g_{1},\dots,g_{k-1}][a_{2},g_{1},\dots,g_{k-1}]^{-1}\in
B^{2}.$

As $|S|\leq s$, it follows that

$\mathrm{Pr}_{a\in N}([a,g_{1},\dots,g_{k-1}]\in B^{2})\geq 1/s,$

so

$\mathrm{Pr}_{a\in N\,h\in
G}([a,g_{1},\dots,g_{k-1},h]=1)\geq\frac{1}{sn^{2}},$

that is, $\mathrm{Pr}(N_{\bf g},G)\geq\frac{1}{sn^{2}}$. As $N=G^{\prime}$ is
abelian and $g_{i}^{g}\in Ng_{i}$ for each $g\in G$ and for each $i$, it
follows that $N_{\bf g}$ is a normal subgroup of $G$. By Proposition 1.2 and
Remarks 2.6 - 2.7 in [2] there are two normal subgroups $T_{\bf g}$ and
$B_{\bf g}\leq N_{\bf g}$ such that the indices $[G:T_{\bf g}]$ and $[N_{\bf
g}:B_{\bf g}]$ and the order of $[T_{\bf g},B_{\bf g}]$ are $(n,s)$-bounded.
Moreover, the subgroup $T_{\bf g}$ can be chosen in such a way that $N_{\bf
g}\cap T_{\bf g}\leq Z_{3}(T_{\bf g})$.

We now will handle the particular case where $G$ is a metabelian $p$-group.

As $[T_{\bf g},B_{\bf g}]$ is a normal subgroup of $n$-bounded order, it
follows that there is an $n$-bounded number $e$ such that $Z_{e}(G)$ contains
$[T_{\bf g},B_{\bf g}]$ for every ${\bf g}\in{\bf G}$. Pass to the quotient
$G/Z_{e}(G)$ and assume that $[T_{\bf g},B_{\bf g}]=1$ for any ${\bf g}\in{\bf
G}$.

We now prove the following claim:

  * (*)

Suppose that $G$ is $l$-Engel for some $l\geq 1$. Then $G$ is nilpotent of
$(k,l,n)$-bounded class.

Since $G/T_{\bf g}$ has bounded order, $G=\langle x_{1},\dots,x_{r},T_{\bf
g}\rangle$ for some elements $x_{1},\dots,x_{r}\in G$ and bounded $r$.
Moreover $G$ is metabelian and $l$-Engel, so $N\langle x_{i}\rangle$ has class
at most $l$ for each $i$, and therefore $N\langle x_{1},\dots,x_{r}\rangle$
has nilpotency class at most $lr$ by Fitting’s theorem. In particular $B_{\bf
g}\langle x_{1},\dots,x_{r}\rangle$ has bounded class. It follows from
$[T_{\bf g},B_{\bf g}]=1$ that $B_{\bf g}$ is contained in $Z_{j}(G)$ for some
bounded number $j$. Moreover $N_{\bf g}/B_{\bf g}$ has bounded order, so
$N_{\bf g}$ is contained in $Z_{j^{\prime}}(G)$ for some bounded number
$j^{\prime}$. Thus $[N,g_{1},\dots,g_{k-1}]$ is contained in
$Z_{j^{\prime}}(G)$ for all $g_{1},\dots,g_{k-1}\in G$. As $N=G^{\prime}$, we
conclude that $\gamma_{k+1}(G)\leq Z_{j^{\prime}}(G)$ and so $G$ is nilpotent
of class at most $k+j^{\prime}$. Hence, the claim follows.

Thus, it suffices to find a bounded-index subgroup of $G$ which is $l$-Engel
for some bounded $l$. We will use the notation $[x,{}_{i}y]$ to denote the
long commutator $[x,y,\dots,y]$, where $y$ is repeated $i$ times.

If $g\in G$, for short we will write

${\bf\tilde{g}}=(g,\dots,g)\in{\bf G},$

so $N_{\bf\tilde{g}}=[N,{}_{k-1}g]$. Since $N_{\bf\tilde{g}}/B_{\bf\tilde{g}}$
is a $p$-group of bounded order, there is a bounded number $f$ such that

$[N_{\bf\tilde{g}},\;_{f}g]=[N,{}_{k-1+f}g]\leq B_{\bf\tilde{g}}$

for all $g\in G$. For $g\in G$ and $i\geq k-1$ write $B_{i,g}$ for
$[B_{\bf\tilde{g}},{}_{i-k+1}\,g]$. As $B_{\bf\tilde{g}}\leq N$ is abelian and
$g^{y}\in Ng$ for each $y\in G$, it follows that $B_{i,g}$ is normal in $G$.
Put $C_{i,g}=C_{G}(B_{ig})$. Let $\beta_{i}$ be the maximal index of
$C_{i,g}$, where $g$ ranges over $G$. Since
$[T_{\bf\tilde{g}},B_{\bf\tilde{g}}]=1$, it follows that $\beta_{k-1}$ is
($|S|$,$n$)-bounded. It is clear that
$\beta_{k-1}\geq\beta_{k}\geq\beta_{k+1}\geq\dots$, thus there is a bounded
number $u$ such that $k-1\leq u$ and $\beta_{u}=\beta_{2u+f+k-1}$.

Choose $g\in G$ such that $C_{u,g}=C_{2u+f+k-1,g}$ is of index $\beta_{u}$ in
$G$. We have $C_{u,g}=C_{j,g}$ for $j=u,u+1,\dots,2u+f+k-1$. Let $h\in
C_{u,g}$. Note that

$\displaystyle B_{2u+f+k-1,g}$ $\displaystyle=$
$\displaystyle[B_{u,g},{}_{u+f+k-1}g]=[B_{u,g},{}_{u+f+k-1}\,(gh)]$
$\displaystyle\leq$
$\displaystyle[N,{}_{u+f+k-1}\,(gh)]\leq[B_{\bf\tilde{gh}},{}_{u}\,(gh)]$
$\displaystyle\leq$ $\displaystyle B_{u,gh}.$

So $C_{u,gh}\leq C_{2uf+k-1,g}=C_{u,g}$. Since $\beta_{u}$ is maximal and
since $h$ was chosen in $C_{u,g}$ arbitrarily, we are forced to conclude that

$C_{u,g}=C_{u,gh}\text{ whenever }h\in C_{u,g}.$

Set $C_{u,g}=C_{0}$ and note that $C_{0}$ centralizes $B_{u,gh}$ for any $h\in
C_{0}$.

Obviously, $C_{0}$ is a normal subgroup of index $\beta_{u}$ in $G$. Let
$\overline{G}=G/Z(C_{0})$. Since $Z(C_{0})$ contains $B_{u,gh}=[B_{\tilde{\bf
gh}},{}_{u-k+1}\,(gh)]$ and also $[N,{}_{k-1+f}\,(gh)]\leq B_{\bf\tilde{gh}}$
we conclude that $\overline{gh}$ is $(u+f+1)$-Engel in $\overline{G}$ for any
$h\in C_{0}$. We deduce from Lemma 4.3 that $C_{0}$ is $l$-Engel for
$l=2u+2f+3$. Now it follows from the above claim $(*)$ that $C_{0}$ is
nilpotent of bounded class and so in the case where $G$ is a metabelian
$p$-group the result follows.

Now we deal with the particular case where $G/N$ is an abelian $p$-group while
$N$ is a $p^{\prime}$-group.

Since $N_{\bf g}\cap T_{\bf g}\leq Z_{3}(T_{\bf g})$, we have that $[[N_{\bf
g},T_{\bf g}],T_{\bf g},T_{\bf g},T_{\bf g}]=1$. It follows from Lemma 4.4
that $T_{\bf g}$ centralizes $N_{\bf g}$.

So we simply assume that $B_{\bf g}=N_{\bf g}$ and $T_{\bf g}=C_{G}(N_{\bf
g})$. Note that if $g\in G$ and ${\bf g}=(g,\dots,g)\in{\bf G}$, then $N_{\bf
g}=[N,g]$ so in what follows we write $N_{g}$ in place of $N_{\bf g}$ and
$T_{g}$ in place of $T_{\bf g}$.

Choose $g\in G$ such that the index $[G:T_{g}]$ is maximal. Let $h\in T_{g}$.
Because of Lemma 4.4 we have

$N_{g}=[N_{g},g]=[N_{g},gh]\leq N_{gh}.$

It follows that $T_{gh}\leq T_{g}$ for any $h\in T_{g}$. Since $[G:T_{g}]$ is
maximal, deduce that $T_{gh}=T_{g}$ for any $h\in T_{g}$. So $T_{g}$
centralizes both $N_{g}$ and $N_{gh}$. It is easy to see that $N_{h}\leq
N_{g}N_{gh}$, so $T_{g}$ centralizes $N_{h}$. This holds for every $h\in
T_{g}$ and so $[N,T_{g},T_{g}]=1$. In view of Lemma 4.4 deduce that
$[N,T_{g}]=1$. Therefore $T_{g}$ is nilpotent of class at most $2$ and this
proves the theorem in the coprime metabelian case.

We now handle the case where $G$ is metabelian without making any assumptions
on the order of $N$ and $G/N$. Let $e=n!$. As $G$ acts on $B$ by conjugation
and each orbit has size at most $n$, it follows that $G^{e}$ centralizes $B$.
So $\langle B\rangle\cap G^{e}\leq Z(G^{e})$. The covering condition implies
that the number of $\gamma_{k}$-values in the group $G^{e}/Z(G^{e})$ is at
most $s$. Taking into account that $\gamma_{k}$ is a boundedly concise word it
follows that $\gamma_{k}\left(G^{e}/Z(G^{e})\right)$ has bounded order, and
its centralizer in $G^{e}$ is a bounded-index subgroup of nilpotency class at
most $k+1$. For each prime $p\leq n$ let $G_{p}$ be the preimage in $G$ of the
Sylow $p$-subgroup of $G/N$. For $p>n$, the Sylow $p$-subgroups of $G$ are
contained in $G^{e}$. Hence $G=G^{e}\prod_{p\leq n}G_{p}$. Therefore by
Fitting’s theorem it suffices to prove the result for $G=G_{p}$. Thus we may
assume that $G/N$ is a $p$-group. Let $P$ be a Sylow $p$-subgroup of $G$. Then
$G=NP$. By the result on $p$-groups, we can replace $P$ with a subgroup of
bounded index and bounded class, so without loss of generality we assume that
$P$ has $(k,n)$-bounded class $c$, say. It follows that $[P\cap N,{}_{c}G]=1$
and so $P\cap N\leq Z_{c}(G)$. Factoring out $Z_{c}(G)$, we may thus assume
that $N$ is a $p^{\prime}$-group. By the result in the coprime case we are
done.

We now drop the assumption that $G$ is metabelian and consider the general
case. As $\gamma_{3}(G)$ is abelian, $G^{\prime}$ is metabelian. Therefore, by
the above, $G^{\prime}$ has a nilpotent subgroup $A$ of bounded index and
bounded class. In view of [10] we can assume that $A$ is characteristic in
$G^{\prime}$ and in particular normal in $G$. Applying Theorem 4.1 we reduce
to the case where $A$ is abelian.

If $A=G^{\prime}$, then $G$ is metabelian and the result holds. We therefore
assume that $A<G^{\prime}$. Further, replacing $G$ by $C_{G}(G^{\prime}/A)$ we
assume that $G/A$ is nilpotent of class at most $2$. In particular,
$\gamma_{2}(G^{\prime})\leq A$. Fix a set $X$ of size at most $|G^{\prime}/A|$
such that $G^{\prime}$ is generated by $A$ and $X$. Set

${\bf X}=X\times\dots\times X\ \ (k-1\text{ factors}).$

If ${\bf x}=(x_{1},\dots,x_{k-1})\in{\bf X}$, the subgroup $A_{\bf
x}=[A,x_{1},\dots,x_{k-1}]$ consists of $\gamma_{k}$-values. Hence, arguing as
above, we deduce that $\mathrm{Pr}(A_{\bf x},G)\geq\frac{1}{sn^{2}}$. As $G$
centralizes $G^{\prime}/A$, we have that $x_{i}^{g}\in Ax_{i}$ for each $i$,
and because of the fact that $A$ is a normal abelian subgroup of $G$ it
follows that $A_{\bf x}$ is normal in $G$. By [2, Remark 2.7] there is a
normal subgroup $T_{\bf x}$ and a normal subgroup $B_{\bf x}\leq A_{\bf x}$
such that the indices $[G:T_{\bf x}]$ and $[A_{\bf x}:B_{\bf x}]$ are
$(s,n)$-bounded and $T_{\bf x}\cap A_{\bf x}\leq Z_{3}(T_{\bf x})$. Set

$H=\bigcap_{{\bf x}\in{\bf X}}T_{\bf x}.$

Since $|{\bf X}|\leq|G^{\prime}/A|^{k-1}$, observe that the index of $H$ in
$G$ is bounded. Therefore it is sufficient to show that $H$ has a nilpotent
subgroup of bounded index and bounded class. Note that $A_{\bf x}\cap H\leq
Z_{3}(H)$ for any ${\bf x}\in{\bf X}$. Since $\gamma_{2}(G^{\prime})\leq A$,
we have $\gamma_{2}(H^{\prime})\leq A\cap H^{\prime}\leq G^{\prime}=A\langle
X\rangle.$ Thus every $\gamma_{2+k-1}$-value of $H^{\prime}$ can be written as
a product of elements each of whom belongs to $A_{\bf x}\leq Z_{3}(H)$, for
some ${\bf x}\in{\bf X}$. Therefore $\gamma_{2+k-1+3}(H^{\prime})=1$ and
$H^{\prime}$ is nilpotent of class at most $k+3$. An application of Hall’s
Theorem 4.1 reduces the problem to the case where $H^{\prime}$ is abelian. We
already know that for metabelian groups the theorem holds. This completes the
proof of Theorem 1.2. ∎

## References

  * [1] M. Aschbacher, Finite group theory. Second edition. Cambridge Studies in Advanced Mathematics, 10. Cambridge University Press, Cambridge, 2000.
  * [2] E. Detomi, P. Shumyatsky, On the commuting probability for subgroups of a finite group, Proc. Roy. Soc. Edinburgh Sect. A 152 (2022), 1551–1564.
  * [3] E. Detomi, M. Morigi, P. Shumyatsky. On profinite groups with word values covered by nilpotent subgroups. Israel J. Math. 226 (2018), 993–1008.
  * [4] E. Detomi, M. Morigi, P. Shumyatsky, BFC theorems for higher commutator subgroups, Quarterly J. Math. 70 (2019), 849–858.
  * [5] G. Dierings, P. Shumyatsky, Groups with boundedly finite conjugacy classes of commutators, Quarterly J. Math. 69 (2018), 1047–1051.
  * [6] A. Fernández-Alcober and M. Morigi, Generalizing a theorem of P. Hall on finite-by-nilpotent groups, Proc. Amer. Math. Soc. 137 (2009), 425–429.
  * [7] G. A. Fernández-Alcober, M. Morigi, Outer commutator words are uniformly concise. J. London Math. Soc. 82 (2010), 581–595.
  * [8] S. Eberhard and P. Shumyatsky, Probabilistically nilpotent groups of class two, Math. Ann. 2023 https://link.springer.com/article/10.1007/s00208-023-02567-0
  * [9] P. Hall, Some sufficient conditions for a group to be nilpotent, Illinois J. Math., 2 (1958), 787– 801.
  * [10] E. I. Khukhro and N. Yu. Makarenko, Large characteristic subgroups satisfying multilinear commutator identities, J. Lond. Math. Soc. (2) 75 (2007), 635–646.
  * [11] B. H. Neumann, Groups covered by permutable subsets, J. London Math. Soc. (3) 29 (1954), 236–248.
  * [12] P. M. Neumann, Two combinatorial problems in group theory, Bull. Lond. Math. Soc., 21 (1989), 456–458.
  * [13] D. J. S. Robinson, A course in the theory of groups, Second, Graduate Texts in Mathematics, vol. 80, Springer-Verlag, New York, 1996.
  * [14] P. Shumyatsky, Finite-by-nilpotent groups and a variation of the BFC-theorem, Mediterr. J. Math. 19 (2022), Paper No. 202, 11 pp.