11institutetext: Keosys Medical Imaging, Nantes, France

# Kidney tumor segmentation using an ensembling multi-stage deep learning
approach. A contribution to the KiTS19 challenge

Gianmarco Santini 11 Noémie Moreau and Mathieu Rubeaux 1111

###### Abstract

Precise characterization of the kidney and kidney tumor characteristics is of
outmost importance in the context of kidney cancer treatment, especially for
nephron sparing surgery which requires a precise localization of the tissues
to be removed. The need for accurate and automatic delineation tools is at the
origin of the KiTS19 challenge. It aims at accelerating the research and
development in this field to aid prognosis and treatment planning by providing
a characterized dataset of 300 CT scans to be segmented. To address the
challenge, we proposed an automatic, multi-stage, 2.5D deep learning-based
segmentation approach based on Residual UNet framework. An ensambling
operation is added at the end to combine prediction results from previous
stages reducing the variance between single models. Our neural network
segmentation algorithm reaches a mean Dice score of 0.96 and 0.74 for kidney
and kidney tumors, respectively on 90 unseen test cases. The results obtained
are promising and could be improved by incorporating prior knowledge about the
benign cysts that regularly lower the tumor segmentation results.

###### Keywords:

CNN, Kidney tumor, Segmentation, Deep Learning

##  1 Introduction

Kidney cancer represents 2.4% of the cancers worldwide with more than 400 000
new cases in 2018 [1], with large variations in incidence rates based on
geography, ethnicity, gender and over the time [2]. In the last two decades,
the growing use of medical imaging had two effects: the renal tumor size
diagnosed has consistently decreased [3], and the number of localized renal
masses found incidentally, on the other hand, has raised [4].

While the standard for renal cancer treatment has been traditionally radical
nephrectomy (i.e. removal of both the tumor and damaged kidney), more
preservative nephronsparing surgery (i.e. open or laparoscopic partial
nephrectomy) has more than quadrupled in the last 20 years [3] and is now a
treatment of choice [5].

In this context, the information related to the kidney and tumor positions,
shapes and sizes, is of outmost importance for the surgery evaluation and
planning process. While imaging modalities such as CT allow precise detection
of the tumors, there is still a lack of automatic delineation tools, and the
evaluation continues to rely on the use of nephrometry scoring systems based
on manual and relatively simple imaging features [6, 7, 8].

A series of methods have been proposed in literature to perform automatic
kidney segmentation in various imaging modalities. When dealing with CT,
traditional methods based on imaging features, deformable models, active shape
models as well as atlases have been suggested [9]. More recently the global
interest towards deep learning algorithms has led to an incredible variety of
applications in the medical imaging field [10], and the area of kidney
segmentation is no exception [11, 12, 13, 14].

However, most of these achievements mainly focus on the automatic kidney
segmentation, generally on healthy patients, without considering the
specificity of cancerous tissue if present. Indeed, the renal tumor
characterization can be challenging because of its variety in terms of
position, extension and gray scale values. Moreover, its distinction with
benign renal cysts is not always trivial, as illustrated by the IIF (Follow-
up) categorization of the reference Bosniak classification [15], where the
cysts have a 5-10% risk of being a kidney cancer.

A first attempt to identify and segment kidney cancer was proposed by [16],
consisting in a computer-aided method able to distinguish healthy renal tissue
from cancerous one in CT scans, by using a region growing algorithm and a gray
level thresholding.

More recently, Zhou et al. [17] proposed another approach where a two-step
segmentation strategy was developed by employing a single atlas model to
isolate kidneys as first action, and then using supervoxels to identify
possible cancerous areas from the previous segmentation. However, both methods
require user interactions; in the first case to initialize some seeds for the
region growing, whilst in the second work the user is demanded to manually
select suspect regions from the super-voxel probability maps. Moreover in
[17], only cases where an evident contrastographic difference between kidneys
and tumor existed were considered and succeeded in the final segmentation.

In this context, the KiTS19 challenge [18] was proposed to stimulate the
research and development of trustworthy kidney and kidney tumors segmentation
methods, by providing a database of 300 annotated CT scans.

In this paper we propose an automatic segmentation method based on a multi-
stage 2.5D deep learning approach to address the KiTS19 MICCAI challenge on
tumor kidney segmentation. The rest of the paper is organized as follows.
Section 2 presents a detailed overview of the data and methods employed.
Section 3 describes the configuration used for training our model and gives
the results obtained. Section 4 opens a discussion on the results and presents
the conclusions.

##  2 Methods

###  2.1 Dataset

We used the KiTS19 Challenge database [18]. It consists of 300 contrast
enhanced CT scans, acquired in the pre-operative arterial phase and selected
from a cohort of subjects who underwent a partial or a radical nephrectomy
between 2010 and 2018 at the university of Minnesota. The included volumes are
characterized by different in plane resolutions ranging from 0.437 to 1.04 mm,
with a slice thickness varying among cases from a minimum of 0.5 mm up to 5.0
mm.

The dataset provides also for each included case the ground truth mask of both
tumor tissue and healthy kidney tissue. Ground truth labels have been manually
created by a pool of medical students under the supervision of an expert
clinician, by using only image axial projections. A detailed description of
the ground truth segmentation strategy is described in [18].

###  2.2 Preprocessing

Before training our multi-stage model, data were standardized to take into
account differences and reduce heterogeneities between the available scans. A
re-slicing operation was first performed bringing all volumes to the same
slice thickness of 3 mm. This value was chosen as a compromise between the
most common axial resolutions observed in the dataset.

Secondly, the training data values were bracketed in the Hounsfield Unit (HU)
range of -30 HU and 300 HU. Hence it was possible to easily remove from the
segmentation the fat regions surrounding the kidney, that are characterized by
values smaller than -30 HU [18]. On the other hand, the higher threshold was
chosen to highlight the high intensity structures and borders that generally
characterize the cancerous tissue. After that, data were standardized to a
zero mean and unit variance distribution of the pixel values.

Figure 1: Block diagram of the proposed segmentation model.

###  2.3 Multi-stage Deep learning approach

The proposed model is characterized by three different stages, as summarized
in Figure 1. The first stage aims to roughly segment the region of interest
(ROI) where to focalize the subsequent analysis, in this case the kidney
region. In the second stage, the segmentation of the kidneys and cancerous
tissue, is carried out by two different neural networks, which work on the
image sub-portions extracted thanks to the use of the approximate kidney
predictions from stage one. The results are finally combined in the last
stage, where the final segmentation is obtained by using an ensembling
operation. In the following, we provide details about the implementation of
each stage.

Figure 2: Res-UNet architecture. On the picture left side, the residual block
configurations. Marked in red the residual blocks where stride 2 convolutions
are used. Upsampling layers precede the decoding res-blocks.

####  2.3.1 Stage 1:

The initial kidney identification was performed using a Residual-UNet (or Res-
UNet) designed with a standard encoder-decoder structure of a UNet [19].
Particularly, we built it with four encoding levels, as many to recreate the
final prediction mask. For each level we used a pre-activated residual block
[20] characterized by two convolutional layers and a shortcut connection to
add the block input to its output, with the aim of speeding up the model
convergence. In all the shortcut connections a convolutional layer was also
added to fix some dimension differences occurring in the feature map
processing. In all the residual blocks of both encoding and decoding paths, a
1$\times$1 convolution was added to adjust the number of channels and make the
addition operation possible, without any dimension mismatches. Moreover, in
residual blocks two to four (highlighted in red in Figure 2) we used a
1$\times$1 convolution with stride two to cope with the size reduction passing
from a level to another. In all cases a linear activation function was
adopted, in order to make the connection purpose as close as possible to the
original one, proposed in [20].

To upsample the data and restore the original dimension a transposed
convolution with stride two was applied before every residual block in the
expanding path of the network. Skip connections were finally included to
concatenate low level, but with higher resolution, feature maps from the
encoder to the high level features in the decoding part. A detailed
description can be seen in Figure 2.

As our purpose in the first stage was to roughly segment kidneys with
cancerous tissue we tried to enhance the perception of global information
rather than local ones by feeding the Res-UNet with subsampled versions of the
original images, halved at 256$\times$256 pixels in the x-y plane. Moreover,
instead of a full 3D, we employed a 2.5D approach.

Using 2.5D inputs generally means to provide the network 2D slices coming from
coronal, axial and sagittal planes [21, 22, 23, 24]. In this case it consisted
in passing to the model a series of adjacent axial slices stacked together
along the channel dimension and make the network able to predict a mask
corresponding only to the central slice.

The final prediction was carried out with a softmax function to assess whether
pixels belonged to kidneys, tumor or background. Even if a three-labelled mask
was produced, we merged the last two (kidneys and tumor labels) in a single
meta-class and we used it in the subsequent stage to localize kidney region
and extract two ROIs, i.e. one for each kidney, if present.

Figure 3: Res-Net architecture. On top of each layer the number of kernels is
specified.

####  2.3.2 Stage 2:

As presented above, this stage was used to perform the actual segmentation of
both kidney and cancerous tissue. We employed two different convolutional
neural network (CNN) architectures to this aim: the first one is identical to
the Res-UNet of stage one, while the second network resumes a model proposed
in [25] and extended in another work [26] to solve a different segmentation
task. (i.e. the SegTHOR challenge).

Compared to [26] we adopted the same network architecture (see Figure 3) but
we simply modified it in order to work with 2.5D input images, that are bigger
in size, and directly predict three classes with a softmax classifier. Dropout
layer (p=0.5) and batch normalization with pre-activation were used throughout
all the residual blocks.

The aforementioned networks were both trained trying to minimize a weighted
categorical cross-entropy, with image sub-portions at the original full
resolution along x-y and an interpolated slice thickness at 3 mm.

The ROIs to segment were extracted creating a bounding box to circumscribe the
union of the kidney plus the tumor (or just the kidneys) by using the (x, y,
z) coordinates from the first stage segmentations. After that, the bounding
box was symmetrically expanded to reach the final size of 256$\times$256
pixels. We experimentally verified that such dimension allowed to fully
include the structures of interest inside the ROIs, even in case of extended
tissue anomalies. Moreover, such input configuration allowed to reuse the
first stage network configuration avoiding an interpolation process on the
extracted images.

####  2.3.3 Stage 3:

The proposed model ends with an ensembling stage, which combines prediction
masks coming from the two networks of stage two. A higher segmentation
accuracy is indeed demonstrated by combining more models with respect to the
use of the single ones [27, 28].

###  2.4 Post-Processing

A simple post-processing operation was carried out at the end of both the
first and second stages (red dashed lines in Figure 1). It mainly consisted in
a labelling operation performed on the predicted segmentations, to identify
and subsequently remove additional disconnected structures that could occur
around kidneys or in other image positions. Such false positive structures
were filtered away by counting the number of pixels belonging to every labeled
object detected, leaving only the ones composed by more than 5000 pixels. We
expected to find the kidneys with or without cancerous tissue as the largest
connected structures after the two segmentation stages.

###  2.5 Evaluation metric

As proposed by the challenge organizers, the output image segmentation quality
was assessed with the Dice score index (Eq. 1) computed on tumor and kidneys,
considered as a single entity and on tumor as a standalone object. Both
structures segmented with our method ($S_{DL}$) were compared with the ground
truth segmentations ($S_{GT}$) provided.

$Dice(S_{GT},S_{DL})=\frac{2\cdot(S_{GT}\cap S_{DL})}{|S_{GT}+S_{DL}|}$ (1)

##  3 Experimental settings and results

From 210 patients we received in the first round, we used 190 cases to train
our model and the remaining 20 to validate it. The final 90 cases were
provided by the organizers without the ground truth masks and were used to
rank the methods proposed in the context of the challenge according to the
average score obtained on the 90 test cases, combining the tumor and the
kidney dice indexes.

###  3.1 Training settings

We designed our networks using Tensorflow [29] and trained them from scratch
on single NVIDIA GTX 1080 of 11 GB, minimizing in all cases a weighted
categorical cross-entropy, and speeding up the process using an Adam
optimizer. An L2 regularization on kernel weights with 0.1 scale factor was
also used.

250 epochs were fixed as upper bound limit for training each single network,
but in all the conducted trials, the best model was always reached around 170
$\pm$ 30 epochs.

As anticipated above, all networks were trained with 2.5D input images
resulting from the concatenation of two slices above and two slices below the
current axial slice to segment.

In every training iteration, a balanced batch size of 32 samples was used.
During stage one, the batch cases were balanced according to three image group
types (roughly 33% each) characterized as follows. Group B: images in which
neither kidneys nor tumors appear, group K: images with healthy kidney
portions, group KT: images with kidney and tumor tissue. For stage two, 50% of
cases belonged to group K, while the remaining 50% was filled with cases from
group KT.

In order to prevent overfitting on the tumor segmentation, different data
augmentation strategies such as axial rotation (angle
$\in[-30^{\circ},30^{\circ}]$), horizontal flip, and central crop plus zoom
were also employed on-the-fly on KT cases only, as they appear in the training
dataset less frequently and with higher variability than the K cases.

The use of diverse combinations of initializations, training settings and
network architectures generally help subsequent ensembling operations [27,
28]. For this reason, we carried out different training configurations
(especially for stage 2) in order to reduce the generalization error of the
final prediction. Table 1 synthesizes different training configurations
adopted for the networks employed.

Table 1: Training configurations and data augmentation strategies employed for
each network. Data augmentation operations refer only to input images
presenting tumors (group KT). T.N. stands for Truncated Normal weights
initialization.

Data Augmentation Training Settings Rotation H-Flip Central crop Weights Loss
weights Learning $(p=1.0)$ $(p=0.5)$ $(p=0.66)$ Init [B, K, KT] rate Res-Unet1
$\bullet$ T.N. ($std\ 0.1$) [0.3 1.0 3.0] ${10}^{-4}$ Res-Unet2 $\bullet$
$\bullet$ T.N. ($std\ 0.1$) [0.3 1.0 3.0] ${10}^{-4}$ Res-Net $\bullet$
$\bullet$ $\bullet$ He uniform [0.2 0.25 0.55] ${10}^{-3}$

###  3.2 Evaluation results

With the above configurations we obtained the results expressed in Table 2, on
the 20 cases chosen randomly as validation set. We finally used the trained
ensemble model to perform the prediction on the 90 test cases. We obtained a
dice score of 0.96 and 0.74 for kidneys and tumor respectively. Segmentation
image examples follow.

Table 2: Segmentation results for each stage designed, considering kidneys
plus tumor (first column) and tumor alone (second column) on the validation
set.

Dice kidneys Dice tumor Res-Unet1 $0.96\pm 0.02$ $0.52\pm 0.32$ Res-Unet2
$0.97\pm 0.02$ $0.70\pm 0.28$ Res-Net $0.97\pm 0.02$ $0.72\pm 0.26$ Ensembling
$\bf{0.98\pm 0.01}$ $\bf{0.73\pm 0.25}$

Figure 4: Segmentation examples from four patients. On the top row the ground
truth labels are reported (red for kidneys, green for tumors). The proposed
model predictions are on the second row. Best viewed in color.

##  4 Discussion and conclusion

We presented an automatic method for semantic segmentation of kidneys and
kidney cancerous tissue from contrastographic CT acquisitions.

As shown in Table 2, the use of a multi-step segmentation algorithm allowed us
to obtain better results than using only a single segmenting step. Moreover,
the ensembling operation of different CNN predictions proved to give slightly
better results, compared to the individual network performances.

On the other hand, the use of an ensembling training strategy combined with a
multi-stage segmentation process led to an increase in training time, due to
the need to try more networks with different configuration settings. A single
network took indeed around three days of training with our hardware
configuration.

The use of a relatively large batch size allowed to better exploit batch
normalization properties compared to smaller batches previously tried (8 or
16). Among the evaluations, the results obtained with a batch size of 32
samples were better. However, this choice penalized the use of state-of-the-
art neural network architectures like Tiramisu [30] or DeepLab [31], which
couldn’t be trained with such input sizes, essentially due to memory
constraints.

Finally, using 2.5D input tensor helped to provide some kind of volumetric
information to the network resulting in a better segmentation level compared
to simple 2D inputs.

Considering the final performance obtained from the 90 test patients, we can
assert that the overall segmentation results, especially concerning the tumor
identification remain promising, but are quite low as a consequence of false
positive cases that were sometimes detected on kidney cysts, or false negative
cases where the tumor lesion was not east to identify on the CT. For future
improvements we plan to design a new training strategy, in which more of these
specific cases could be passed to the model, in order to differentiate them
from cases where cancerous tissues actually occur. Furthermore, the we would
like to work on even more focused input data, by maybe adding an additional
stage in order to focus on the analysis of even more detailed local features.

## Acknowledgments

This work is financed by FEDER funds through ”Programme opérationnel regional
FEDER-FSE Pays de la Loire 2014-2020” n∘ PL0015129 (EPICURE).

## References

  * [1] Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A.:  Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries.  CA: a cancer journal for clinicians 68(6) (2018) 394–424
  * [2] Scelo, G., Larose, T.L.:  Epidemiology and risk factors for kidney cancer.  Journal of Clinical Oncology 36(36) (2018) 3574–3581
  * [3] Nguyen, M.M., Gill, I.S., Ellison, L.M.:  The evolving presentation of renal carcinoma in the united states: trends from the surveillance, epidemiology, and end results program.  The Journal of urology 176(6) (2006) 2397–2400
  * [4] Sun, M., Abdollah, F., Bianchi, M., Trinh, Q.D., Jeldres, C., Thuret, R., Tian, Z., Shariat, S.F., Montorsi, F., Perrotte, P., et al.:  Treatment management of small renal masses in the 21st century: a paradigm shift.  Annals of surgical oncology 19(7) (2012) 2380–2387
  * [5] Dominguez-Escrig, J.L., Vasdev, N., O’Riordon, A., Soomro, N.:  Laparoscopic partial nephrectomy: Technical considerations and an update.  Journal of minimal access surgery 7(4) (2011) 205
  * [6] Ficarra, V., Novara, G., Secco, S., Macchi, V., Porzionato, A., De Caro, R., Artibani, W.:  Preoperative aspects and dimensions used for an anatomical (padua) classification of renal tumours in patients who are candidates for nephron-sparing surgery.  European urology 56(5) (2009) 786–793
  * [7] Kutikov, A., Uzzo, R.G.:  The renal nephrometry score: a comprehensive standardized system for quantitating renal tumor size, location and depth.  The Journal of urology 182(3) (2009) 844–853
  * [8] Simmons, M.N., Ching, C.B., Samplaski, M.K., Park, C.H., Gill, I.S.:  Kidney tumor location measurement using the c index method.  The Journal of urology 183(5) (2010) 1708–1713
  * [9] Torres, H.R., Queiros, S., Morais, P., Oliveira, B., Fonseca, J.C., Vilaca, J.L.:  Kidney segmentation in ultrasound, magnetic resonance and computed tomography images: A systematic review.  Computer methods and programs in biomedicine 157 (2018) 49–67
  * [10] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.:  A survey on deep learning in medical image analysis.  Medical image analysis 42 (2017) 60–88
  * [11] Thong, W., Kadoury, S., Piché, N., Pal, C.J.:  Convolutional networks for kidney segmentation in contrast-enhanced ct scans.  Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 6(3) (2018) 277–282
  * [12] Zheng, Y., Liu, D., Georgescu, B., Xu, D., Comaniciu, D.:  Deep learning based automatic segmentation of pathological kidney in ct: local versus global image context.  In: Deep Learning and Convolutional Neural Networks for Medical Image Computing.  Springer (2017) 241–255
  * [13] Jackson, P., Hardcastle, N., Dawe, N., Kron, T., Hofman, M., Hicks, R.J.:  Deep learning renal segmentation for fully automated radiation dose estimation in unsealed source therapy.  Frontiers in oncology 8 (2018) 215
  * [14] Sharma, K., Rupprecht, C., Caroli, A., Aparicio, M.C., Remuzzi, A., Baust, M., Navab, N.:  Automatic segmentation of kidneys using deep learning for total kidney volume quantification in autosomal dominant polycystic kidney disease.  Scientific reports 7(1) (2017) 2049
  * [15] Bosniak, M.A.:  The current radiological approach to renal cysts.  Radiology 158(1) (1986) 1–10
  * [16] Kim, D.Y., Park, J.W.:  Computer-aided detection of kidney tumor on abdominal computed tomography scans.  Acta radiologica 45(7) (2004) 791–795
  * [17] Zhou, B., Chen, L.:  Atlas-based semi-automatic kidney tumor detection and segmentation in ct images.  In: 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), IEEE (2016) 1397–1401
  * [18] Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., et al.:  The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes.  arXiv preprint arXiv:1904.00445 (2019)
  * [19] Ronneberger, O., Fischer, P., Brox, T.:  U-net: Convolutional networks for biomedical image segmentation.  In: International Conference on Medical image computing and computer-assisted intervention, Springer (2015) 234–241
  * [20] He, K., Zhang, X., Ren, S., Sun, J.:  Identity mappings in deep residual networks.  In: European conference on computer vision, Springer (2016) 630–645
  * [21] Roth, H.R., Lu, L., Seff, A., Cherry, K.M., Hoffman, J., Wang, S., Liu, J., Turkbey, E., Summers, R.M.:  A new 2.5 d representation for lymph node detection using random sets of deep convolutional neural network observations.  In: International conference on medical image computing and computer-assisted intervention, Springer (2014) 520–527
  * [22] Wolterink, J.M., Leiner, T., de Vos, B.D., van Hamersvelt, R.W., Viergever, M.A., Išgum, I.:  Automatic coronary artery calcium scoring in cardiac ct angiography using paired convolutional neural networks.  Medical image analysis 34 (2016) 123–136
  * [23] Setio, A.A.A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., Van Riel, S.J., Wille, M.M.W., Naqibullah, M., Sánchez, C.I., van Ginneken, B.:  Pulmonary nodule detection in ct images: false positive reduction using multi-view convolutional networks.  IEEE transactions on medical imaging 35(5) (2016) 1160–1169
  * [24] Roth, H.R., Wang, Y., Yao, J., Lu, L., Burns, J.E., Summers, R.M.:  Deep convolutional networks for automated detection of posterior-element fractures on spine ct.  In: Medical Imaging 2016: Computer-Aided Diagnosis. Volume 9785., International Society for Optics and Photonics (2016) 97850P
  * [25] Johnson, J., Alahi, A., Fei-Fei, L.:  Perceptual losses for real-time style transfer and super-resolution.  In: European conference on computer vision, Springer (2016) 694–711
  * [26] van Harten, L., Noothout, J.M., Verhoeff, J., Wolterink, J.M., Isgum, I.:  Automatic segmentation of organs at risk in thoracic ct scans by combining 2d and 3d convolutional neural networks.  In: SegTHOR@ ISBI. (2019)
  * [27] Lyksborg, M., Puonti, O., Agn, M., Larsen, R.:  An ensemble of 2d convolutional neural networks for tumor segmentation.  In: Scandinavian Conference on Image Analysis, Springer (2015) 201–211
  * [28] Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M., Pawlowski, N., Rajchl, M., Lee, M., Kainz, B., Rueckert, D., et al.:  Ensembles of multiple models and architectures for robust brain tumour segmentation.  In: International MICCAI Brainlesion Workshop, Springer (2017) 450–462
  * [29] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.:  Tensorflow: A system for large-scale machine learning.  In: 12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16). (2016) 265–283
  * [30] Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.:  The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. (2017) 11–19
  * [31] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.:  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40(4) (2017) 834–848


# On automorphisms of free metabelian Lie algebras

###### Abstract.

The well known Bachmuth-Mochizuki-Roman’kov Theorem [2, 31] states that every
automorphism of the free metabelian group of rank $\geq 4$ is tame. In 1992
Yu. Bahturin and S. Nabiyev [3] claimed that every nontrivial inner
automorphism of the free metabelian Lie algebra $M_{n}$ of any rank $n\geq 2$
over a field of characteristic zero is wild. More examples of wild
automorphisms of $M_{n}$ of rank $n\geq 4$ were given in 2008 by Z. Özcurt and
N. Ekici [27]. The main goal of this note is to show that both articles
contain uncorrectable errors and to draw the attention of specialists to the
fact that the question of tame and wild automorphisms for free metabelian Lie
algebras $M_{n}$ of rank $n\geq 4$ is still widely open.

Ualbai Umirbaev111Department of Mathematics, Wayne State University, Detroit,
MI 48202, USA and Institute of Mathematics and Mathematical Modeling, Almaty,
050010, Kazakhstan e-mail<EMAIL_ADDRESS>
Mathematics Subject Classification (2020): 17B40, 17B01, 17B30.

Key words: Automorphism, derivation, Lie algebra, free metabelian Lie algebra.

##  1\. Introduction

Let $K$ be an arbitrary field, and let ${\mathfrak{M}}$ be an arbitrary
homogeneous variety of linear algebras over $K$. Denote by
$A=K_{\mathfrak{M}}\langle x_{1},x_{2},\ldots,x_{n}\rangle$ the free algebra
of ${\mathfrak{M}}$ with a free set of generators
$X=\\{x_{1},x_{2},\ldots,x_{n}\\}$. If ${\mathfrak{M}}$ is a unitary variety
of algebras (see, for example [48]) then we assume that $A$ has an identity
element $1$. Let $\mathrm{Aut}(A)$ be the group of all automorphisms of $A$.
Denote by $\phi=(f_{1},f_{2},\ldots,f_{n})$ the automorphism $\phi$ of $A$
such that $\phi(x_{i})=f_{i},\,1\leq i\leq n$. An automorphism

$\displaystyle(x_{1},\ldots,x_{i-1},\alpha x_{i}+f,x_{i+1},\ldots,x_{n}),$

where $0\neq\alpha\in K,\ f\in K_{\mathfrak{M}}\langle
X\setminus\\{x_{i}\\}\rangle$, is called elementary. The subgroup
$\mathrm{TAut}(A)$ of $\mathrm{Aut}(A)$ generated by all elementary
automorphisms is called the tame automorphism group, and the elements of this
subgroup are called tame automorphisms of $A$. Nontame automorphisms of $A$
are called wild.

The question about tame and wild automorphisms of free algebras plays an
important role in combinatorial algebra. It is well known that all
automorphisms of the polynomial algebra $K[x,y]$ in two variables $x,y$ over a
field $K$ are tame [12, 15]. Similar results hold for free associative
algebras [9, 18] and for free Poisson algebras (in characteristic zero) [20]
(see also [19, 21]). Moreover, the automorphism groups of polynomial algebras,
free associative algebras, and free Poisson algebras in two variables are
isomorphic.

The automorphism groups of commutative and associative algebras generated by
three elements are much more complicated. The well-known Nagata automorphism
[25]

$\displaystyle\sigma=(x+2y(zx-y^{2})+z(zx-y^{2})^{2},y+z(zx-y^{2}),z)$

of the polynomial algebra $K[x,y,z]$ over a field $K$ of characteristic $0$ is
proven to be wild [36]. The so-called Anick automorphism (see [8, p. 343])

$\displaystyle\delta$ $\displaystyle=$ $\displaystyle(x+z(xz-zy),\,y+(xz-
zy)z,\,z)$

of the free associative algebra $K\langle x,y,z\rangle$ over a field $K$ of
characteristic $0$ is also proven to be wild [46]. The Nagata automorphism
gives an example of a wild automorphism of free Poisson algebras in three
variables. In 2006 I.P. Shestakov (unpublished) constructed an analogue of the
Anick automorphism

$\displaystyle\tau=(x+\\{xz-\\{y,z\\},z\\},y+z(xz-\\{y,z\\}),z)$

for free Poisson algebras in three variables.

It is well known [42] that the Nagata and the Anick automorphisms are stably
tame.

In 1981 Shafarevich [34] and in 1983 Anick [1] independently proved that every
automorphism of the polynomial algebra $K[x_{1},x_{2},\ldots,x_{n}]$ over a
field $K$ of characteristic zero can be approximated by tame automorphisms
with respect two different but similar topologies. This can be considered as a
weaker generalization of the Jung-van der Kulk Theorem for polynomial algebras
of rank $n\geq 3$. In particular, the Nagata automorphism is wild but can be
approximated by tame automorphisms.

In 1964 P. Cohn proved [8] that all automorphisms of finitely generated free
Lie algebras over a field are tame. Later this result was extended to free
algebras of Nielsen-Schreier varieties [17]. Recall that a variety of
universal algebras is called Nielsen-Schreier, if any subalgebra of a free
algebra of this variety is free, i.e., an analog of the classical Nielsen-
Schreier theorem is true. The varieties of all non-associative algebras [16],
commutative and anti-commutative algebras [38], Lie algebras and Lie
$p$-algebras [37, 47], and Lie superalgebras [22, 41] and Lie
$p$-superalgebras [23] over a field are Nielsen-Schreier. Some other examples
of Nielsen-Schreier varieties can be found in [35, 24, 44, 45, 7].

It was recently shown [10] that the varieties of pre-Lie (also known as right-
symmetric) algebras and Lie-admissible algebras over a field of characteristic
zero are Nielsen–Schreier. In particular, every automorphism of a free right-
symmetric and a free Lie-admissable algebra of finite rank is tame. The
tameness of automorphisms of free right-symmetric algebras in two variables
over any field was proven in [14]

One of the classical and well studied varieties of algebras is the variety of
metabelian Lie algebras. Let $L_{n}$ be a free Lie algebra of rank $n$ in the
variables $x_{1},\ldots,x_{n}$ over a field $K$. Then
$M_{n}=L_{n}/L^{\prime\prime}_{n}=L_{n}/[[L_{n},L_{n}],[L_{n},L_{n}]]$ is the
free metabelian Lie algebra of rank $n$ in the variables
$y_{i}=x_{i}+L^{\prime\prime}_{n}$, where $1\leq i\leq n$. Let $0\neq
z\in[M_{n},M_{n}]$. Then $\mathrm{ad}\,z:M_{n}\to M_{n}(m\mapsto[z,m])$ is a
locally nilpotent derivation and $(\mathrm{ad}\,z)^{2}=0$. The exponential
$\exp(\mathrm{ad}\,z)$ of $M_{n}$ can be written as

$\displaystyle\exp(\mathrm{ad}\,z)=\mathrm{id}+\mathrm{ad}\,z=(y_{1}+[z,y_{1}],\ldots,y_{n}+[z,y_{n}]).$

Automorphisms of this type are called inner.

Obviously, the group of tame automorphisms of $M_{2}$ coincides with the group
of linear automorphisms $\mathrm{GL}_{2}(K)$. It follows that any nontrivial
inner automorphism of $M_{2}$ is wild (see also [39, Proposition 4] and [29,
Theorem 3]). In 1992, V. Drensky [11] proved that the inner automorphism
$\mathrm{exp}(\mathrm{ad}[x_{1},x_{2}])$ of $M_{3}$ is wild. In 2008, V.
Roman’kov [32] proved that the group of automorphisms of $M_{3}$ cannot be
finitely generated modulo the subgroup generated by all inner and tame
automorphisms.

An analogue of the Anick-Shafarevich Theorem [1, 34] for free metabelian Lie
algebras was proven in 1993 by Bryant and Drensky [6]. They proved that the
group of tame automorphisms is dense in the group of all automorphisms of
$M_{n}$ if $n\geq 4$. A sharpened version of this result can be found in [13,
26].

In 1992 Yu. Bahturin and S. Nabiyev [3] announced that any nontrivial
exponential automorphism of the free metabelian Lie algebra $M_{n}$ of rank
$n\geq 2$ over a field of characteristic zero is wild. Unfortunately the proof
contains some uncorrectable mistakes. The paper [3] is not well written and it
seems that many specialists in the area accepted the main result without
checking the details of the proof. We explain the main steps of the proof
given in [3] using more modern terminology and explicitly show the first main
mistake in Section 2.

One more example of a wild automorphism of $M_{n}$ for $n\geq 4$ was given by
Z. Özcurt and N. Ekici [27]. The proof given in [27], in fact, contradicts
some results from [5, 6]. We discuss this in Section 3.

Thus the following problem is wide open.

######  Problem 1.

Is any automorphism of the free metabelian Lie algebra $M_{n}$ of rank $n\geq
4$ over any field tame?

The well known Bachmuth-Mochizuki-Roman’kov Theorem [2, 31] states that every
automorphism of the free metabelian group of rank $\geq 4$ is tame. More
information about automorphisms of free metabelian groups can be found in
[33].

In 1992 V. Drensky [11], in 1993 Papistas [28], and in 1995 Bahturin and
Shpilrain [4] proved that free algebras of rank $n\geq 2$ of any polynilpotent
variety of Lie algebras that is not abelian or metabelian have wild
automorphisms. This result also makes the case of free metabelian algebras
more important.

##  2\. On the paper by Bahturin and Nabiyev

###  2.1. The structure of $M_{n}$

Let $M_{n}$ be the free metabelian Lie algebra over a field $K$ freely
generated generated by $x_{1},\ldots,x_{n}$. We give a description of $M_{n}$
using Shmel’kin’s wreath products [39]. Let $Y_{n}$ be the abelian Lie algebra
with a linear basis $y_{1},y_{2},\ldots,y_{n}$. The universal enveloping
algebra $U(Y_{n})$ is the polynomial algebra $U(Y_{n})=K[y_{1},\ldots,y_{n}]$.
Let $T_{n}$ be the free (left) $U(Y_{n})$-module with a basis
$t_{1},\ldots,t_{n}$. Consider the direct sum

$\displaystyle M=Y_{n}\oplus T_{n}.$

Turn $M$ into a Lie algebra by

$\displaystyle[a+t,b+s]=as-bt$

where $a,b\in Y_{n}$ and $t,s\in T_{n}$. Then the subalgebra of $M$ generated
by

$\displaystyle x_{i}=y_{i}+t_{i},\ \ 1\leq i\leq n,$

is the free metabelian Lie algebra $M_{n}$ with the free variables
$x_{1},\ldots,x_{n}$ [39].

###  2.2. The Jacobian matrix of an endomorphism of $M_{n}$

Let $f\in M_{n}$ be an arbitrary element $M_{n}$. Then $f$ is uniquely
represented as

$\displaystyle f=y+t,y\in Y_{n},t\in T_{n}.$

Moreover, $t$ is uniquely represented as

$\displaystyle t=d_{1}t_{1}+\ldots+d_{n}t_{n},d_{1},\ldots,d_{n}\in
K[y_{1},\ldots,y_{n}]=U(Y_{n}).$

We call $d_{i}=\frac{\partial f}{\partial x_{i}}$ the Fox derivatives of $f$
(see [43]). Set

$\displaystyle\partial(f)=(d_{1},\ldots,d_{n})=\big{(}\frac{\partial
f}{\partial x_{1}},\ldots,\frac{\partial f}{\partial x_{n}}\big{)}.$

Also set $Y=(y_{1},\ldots,y_{n})^{t}$, where $t$ means transpose. There is a
well known statement ([43, Lemma 5]) that says $f\in[M_{n},M_{n}]$ if and only
if

(1) $\displaystyle\partial(f)Y=0.$

Let $\phi=(f_{1},\ldots,f_{n})$ be an arbitrary endomorphism of $M_{n}$ such
that $\phi(x_{i})=f_{i}$ for all $i$. In particilar, $X=(x_{1},\ldots,x_{n})$
is the identity automorphism. We define the Jacobian matrix of $\phi$ by

$\displaystyle J(\phi)=[\frac{\partial f_{i}}{\partial x_{j}}]_{1\leq i,j\leq
n}=\left[\begin{array}[]{ccc}\partial(f_{1})\\\ \ldots\\\ \partial(f_{n})\\\
\end{array}\right].$

If $\phi=(f_{1},\ldots,f_{n})$ and $\psi=(g_{1},\ldots,g_{n})$ then

$\displaystyle\phi\circ\psi(x_{i})=g_{i}(f_{1},\ldots,f_{n})$

for all $1\leq i\leq n$. The corresponding Chain Rule gives [43] that

(3) $\displaystyle J(\phi\circ\psi)=\overline{\phi}(J(\psi))J(\phi),$

where $\overline{\phi}$ denotes the endomorphism of $K[y_{1},\ldots,y_{n}]$,
first induced by $\phi$ to $Y_{n}$ and then extended to $U(Y_{n})$.

Denote by $\mathrm{IAut}\,M_{n}$ the group of all automorphisms of $M_{n}$
that induces the identity automorphism on $M_{n}/[M_{n},M_{n}]$.

######  Lemma 1.

The map

$\displaystyle J:\mathrm{IAut}(M_{n})\to\mathrm{GL}_{n}(U(Y_{n}))$

is an injective antimorphism of groups.

This is a direct corollary of (3) since $\overline{\phi}=\mathrm{id}$.

######  Remark 1.

The matrix $E+QA_{n}$ in [3, Theorem 2] is the Jacobian matrix of an
automorphism. It is mentioned (without proof) in the middle of [3, Theorem 2]
that $J$ is an isomorphism. Sometimes it depends on the definition of
composition of automorphisms.

###  2.3. Tame automorphisms

The group of tame automorphisms $\mathrm{TAut}\,M_{n}$ is generated by all
linear automorphisms and the automorphisms of the form

$\displaystyle\phi_{f}=(x_{1}+f(x_{2},\ldots,x_{n}),x_{2},\ldots,x_{n}),f\in[M_{n},M_{n}].$

Notice that

$\displaystyle J(\phi_{f})=E+\left[\begin{array}[]{cccc}\partial(f)\\\ 0\\\
\ldots\\\ 0\\\ \end{array}\right]=E+e_{1}\partial(f),$

where $E$ is the identity matrix and $e_{1}=(1,0,\ldots,0)^{t}$, since
$\partial(f)=(0,\frac{\partial f}{\partial x_{2}},\ldots,\frac{\partial
f}{\partial x_{n}})$. Notice that $\partial(f)e_{1}=0$.

######  Lemma 2.

Let $\alpha$ be an arbitrary linear automorphism of $M_{n}$. Then

$\displaystyle J(\alpha\phi_{f}\alpha^{-1})=E+\Phi\Psi,$

where $\Phi$ is a column vector and $\Psi$ is a row vector such that

$\displaystyle\Psi\Phi=0,\Psi Y=0.$

Proof. Let $J(\alpha)=A$. Using (3) we get

$\displaystyle
J(\alpha\phi_{f}\alpha^{-1})=\overline{\alpha}(J(\phi_{f}\alpha^{-1})J(\alpha)=\overline{\alpha}(\overline{\phi_{f}}(J(\alpha^{-1}))J(\phi_{f}))A$
$\displaystyle=\overline{\alpha}(A^{-1}J(\phi_{f}))A=A^{-1}\overline{\alpha}(J(\phi_{f}))A=A^{-1}\overline{\alpha}(E+e_{1}\partial(f))A$
$\displaystyle=A^{-1}(E+e_{1}\overline{\alpha}(\partial(f)))A=E+A^{-1}e_{1}\overline{\alpha}(\partial(f))A=E+\Phi\Psi,$

where $\Phi=A^{-1}e_{1}$ and $\Psi=\overline{\alpha}(\partial(f))A$. We have

$\displaystyle\Psi\Phi=\overline{\alpha}(\partial(f))e_{1}=\overline{\alpha}(\partial(f)e_{1})=0.$

Obviously,

$\displaystyle\alpha\phi_{f}\alpha^{-1}=X+\alpha(f,0,\ldots,0)^{t}\alpha^{-1}=X+(f_{1},\ldots,f_{n})^{t}$

for some $f_{1},\ldots,f_{n}\in[M_{n},M_{n}]$. Consequently,

$\displaystyle
J(\alpha\phi_{f}\alpha^{-1})=E+\left[\begin{array}[]{cccc}\partial(f_{1})\\\
\ldots\\\ \partial(f_{n})\\\ \end{array}\right]$

and

$\displaystyle\Phi\Psi=\left[\begin{array}[]{ccc}\partial(f_{1})\\\ \ldots\\\
\partial(f_{n})\\\ \end{array}\right].$

By (1) we get

$\displaystyle\Phi\Psi Y=\left[\begin{array}[]{ccc}\partial(f_{1})\\\
\ldots\\\ \partial(f_{n})\\\
\end{array}\right]Y=\left[\begin{array}[]{cccc}\partial(f_{1})Y\\\ \ldots\\\
\partial(f_{n})Y\\\ \end{array}\right]=0.$

Notice that $\Psi Y$, the product of a row and a column, belongs to
$U(Y_{n})$. Since $\Phi=A^{-1}e_{1}$ is a nonzero column it follows that $\Psi
Y=0$. $\Box$

###  2.4. Exponential automorphisms

Let $0\neq z\in[M_{n},M_{n}]$. Then $\mathrm{ad}\,z:M_{n}\to
M_{n}(m\mapsto[z,m])$ is a locally nilpotent derivation and
$(\mathrm{ad}\,z)^{2}=0$. We get

$\displaystyle\exp(\mathrm{ad}\,z)=\mathrm{id}+\mathrm{ad}\,z=(x_{1}+[z,x_{1}],\ldots,x_{n}+[z,x_{n}])^{t}.$

Notice that $\partial([z,x_{1}])=-y_{1}\partial(z)$ since
$z\in[M_{n},M_{n}]\subseteq T_{n}$ and $[z,x_{1}]=[z,y_{1}+t_{1}]=-y_{1}z$.
Consequently,

(11) $\displaystyle
J(\exp(\mathrm{ad}\,z))=E-\left[\begin{array}[]{ccc}y_{1}\partial(z)\\\
\ldots\\\ y_{n}\partial(z)\\\ \end{array}\right]=E-Y\partial(z).$

###  2.5. Wild automorphisms

######  Theorem 1.

[3, Theorem 4] Every nontrivial exponential automorphism
$\sigma=\exp(\mathrm{ad}\,z)$ is wild.

Proof. Suppose that $\sigma$ is tame. Then $\sigma$ is a product of linear
conjugates of the automorphisms of the type $\phi_{f}$ since $\sigma$ induces
the identity automorphism on $Y_{n}$. Taking the Jacobian matrices, by Lemmas
1 and 2, we get an equality of the form

$\displaystyle(E+\Phi_{1}\Psi_{1})\ldots(E+\Phi_{n}\Psi_{n})=E-Y\partial(z).$

From now on we deal only with the matrix multiplication. Recall that
$\Phi_{i}$ are columns and $\Psi$ are rows. Let’s consider the case when $n=3$
for clarity (the case $n=2$ is trivial). Then

(12)
$\displaystyle(E+\Phi_{1}\Psi_{1})(E+\Phi_{2}\Psi_{2})(E+\Phi_{3}\Psi_{3})=E-Y\partial(z).$

Set $\lambda_{ij}=\Psi_{i}\Phi_{j}\in K[y_{1},\ldots,y_{n}]=U(Y_{n})$. Then
(12) implies that

$\displaystyle\Phi_{1}\Psi_{1}+\Phi_{2}\Psi_{2}+\Phi_{3}\Psi_{3}+\Phi_{1}\lambda_{12}\Psi_{2}+\Phi_{1}\lambda_{13}\Psi_{3}+\Phi_{2}\lambda_{23}\Psi_{3}$
(13) $\displaystyle+\Phi_{1}\lambda_{12}\lambda_{23}\Psi_{3}=-Y\partial(z).$

Notice that $\lambda_{ii}=0$ by Lemma 2. Multiplying this equality by
$\Psi_{1}$ from the left hand side, we get

(14)
$\displaystyle\lambda_{12}\Psi_{2}+\lambda_{13}\Psi_{3}+\lambda_{12}\lambda_{23}\Psi_{3}=0.$

This exactly corresponds to [3, equation (17)].

######  Remark 2.

Notice that $\Psi_{k}=\overline{\Psi_{k}}A_{n}$ in [3] and the multiplication
on the left first by $A_{N}$ and then by $\overline{\Psi_{k}}$ is equivalent
to the multiplication by $\Psi_{k}$.

###  2.6. The mistake

Representation of (12) in the form

$\displaystyle(E+\Phi_{1}\Psi_{1})(E+\Phi_{2}\Psi_{2})(E-\Phi_{1}\Psi_{1})(E+\Phi_{1}\Psi_{1})(E+\Phi_{3}\Psi_{3})$
$\displaystyle=(E+(\Phi_{2}+\lambda_{12}\Phi_{1})(\Psi_{2}-\lambda_{21}\Psi_{1}))(E+\Phi_{1}\Psi_{1})(E+\Phi_{3}\Psi_{3})=E-Y\partial(z)$

is useless since the equation (2.5) corresponding to this equation is the
same.

But, interestingly, multiplication of (2.5) by $\Psi_{2}-\lambda_{21}\Psi_{1}$
leads to a ”reduced” result. Of course, we can get the same result
systematically, first multiplying (2.5) by $\Psi_{2}$ and then subtracting
$\lambda_{21}$ times (14). In fact, multiplying (2.5) by $\Psi_{2}$ gives

(15)
$\displaystyle\lambda_{21}\Psi_{1}+\lambda_{23}\Psi_{3}+\lambda_{21}\lambda_{12}\Psi_{2}+\lambda_{21}\lambda_{13}\Psi_{3}+\lambda_{21}\lambda_{12}\lambda_{23}\Psi_{3}=0.$

Combining (14) and (15) we get

(16) $\displaystyle\lambda_{21}\Psi_{1}+\lambda_{23}\Psi_{3}=0.$

We can also get this equality directly from (2.5) by multiplying it by
$\Psi_{2}-\lambda_{21}\Psi_{1}$ and this equality corresponds to the last
equality on [3, page 51]. Unfortunately this equation contains the additional
summand $\lambda_{21}\Psi_{1}$.

This is the main mistake. It does not allow us to make further ”triangular”
calculations.

##  3\. On the paper by Özkurt and Ekici

The authors of [27] claim, in particular, that the automorphism

$\displaystyle\psi=(x_{1}+[[x_{1},[x_{2},x_{3}]],x_{4}],x_{2},\ldots,x_{n})$

of $M_{n}$ is wild for all $n\geq 4$.

Let

$\displaystyle M_{n}=A_{1}\oplus\ldots\oplus A_{k}\oplus\ldots$

be the standard degree grading of $M_{n}$, i.e., $A_{k}$ is the linear span of
(Lie) monomials of degree $k$.

Let $\mathrm{IAut}_{i}(M_{n})$ be the subgroup of all automorphisms of $A$
that induces the identity automorphism on the factor-algebra

$\displaystyle M_{n}/(A_{i+1}+A_{i+2}+\ldots).$

We have

$\displaystyle\mathrm{Aut}(M_{n})=\mathrm{IAut}_{0}(M_{n})\supset\mathrm{IAut}_{1}(M_{n})\supset\ldots\supset\mathrm{IAut}_{k}(M_{n})\supset\ldots$

Obviously, $\mathrm{IAut}(M_{n})=\mathrm{IAut}_{1}(M_{n})$.

The automorphism $\psi$ belongs to $\mathrm{IAut}_{3}(M_{n})$. By the main
result of [6] there exists a tame automorphism $\overline{\phi}$ of $M_{n}$
such that $\overline{\phi}\in\mathrm{IAut}_{3}(M_{n})$ and
$\overline{\phi}^{-1}\psi\in\mathrm{IAut}_{4}(M_{n})$. This means that

$\displaystyle\overline{\phi}=(x_{1}+[[x_{1},[x_{2},x_{3}]],x_{4}]+u_{1},x_{2}+u_{2},\ldots,x_{n}+u_{n}),$

where $u_{1},\ldots,u_{n}\in A_{5}+A_{6}+\ldots$.

Let $L_{n}$ be the free Lie algebra of rank $n$ in the variables
$z_{1},\ldots,z_{n}$ such that
$M_{n}=L_{n}/L^{\prime\prime}_{n}=L_{n}/[[L_{n},L_{n}],[L_{n},L_{n}]]$ is the
free metabelian Lie algebra of rank $n$ in the variables
$x_{i}=z_{i}+L^{\prime\prime}_{n}$, where $1\leq i\leq n$.

Suppose that $\overline{\phi}$ is induced by the tame automorphism $\phi$ of
$L_{n}$. Then

$\displaystyle\phi=(z_{1}+[[z_{1},[z_{2},z_{3}]],z_{4}]+v_{1}+w_{1},z_{2}+v_{2}+w_{2},\ldots,z_{n}+v_{n}+w_{n}),$

where $v_{1},\ldots,v_{n}\in L^{\prime\prime}_{n}$ are homogeneous polynomials
of degree $4$ and $w_{1},\ldots,w_{n}$ does not contain Lie monomials of
degree less than $5$.

As in [27], denote by $\frac{\partial f}{\partial
z_{i}}=\frac{\partial}{\partial z_{i}}(f)\in U(L_{n})$ the (left) Fox
derivatives of $f$ [30, 40, 43], i.e.,

$\displaystyle f=\frac{\partial f}{\partial z_{1}}z_{1}+\ldots+\frac{\partial
f}{\partial z_{n}}z_{n}.$

Then

$\displaystyle\frac{\partial}{\partial
z_{1}}([[z_{1},[z_{2},z_{3}]],z_{4}]+v_{1})+\frac{\partial}{\partial
z_{2}}(v_{2})+\ldots+\frac{\partial}{\partial z_{n}}(v_{n})$
$\displaystyle=z_{4}[z_{2},z_{3}]+\frac{\partial}{\partial
z_{1}}(v_{1})+\frac{\partial}{\partial
z_{2}}(v_{2})+\ldots+\frac{\partial}{\partial
z_{n}}(v_{n})\in[U(L_{n}),U(L_{n})]$

by [5, Theorem 3.7].

This contradicts the final conclusion of the proof of [27, Theorem 4.1].

## References

  * [1] D.J. Anick, Limits of tame automorphisms of $k[x_{1},\ldots,x_{n}]$. J. Algebra 82(1983), no.2, 459–468.
  * [2] S. Bachmuth, H. Mochizuki, $\mathrm{Aut}F\to\mathrm{Aut}(F/F")$ is surjective for free group $F$ of rank $\geq 4$. Trans. Amer. Math. Soc. 292 (1985), no. 1, 81–101.
  * [3] Y. Bahturin, S. Nabiyev, Automorphisms and derivations of abelian extensions of some Lie algebras. Abh. Math. Sem. Univ. Hamburg 62 (1992), 43–57.
  * [4] Yu. Bahturin, V. Shpilrain, On Lie algebras with wild automorphisms. Results Math. 28 (1995), no. 3–4, 209–213.
  * [5] R.M. Bryant, V. Drensky, Obstructions to lifting automorphisms of free algebras. Comm. Algebra 21 (1993), no. 12, 4361–4389.
  * [6] R.M. Bryant, V. Drensky, Dense subgroups of the automorphism groups of free algebras. Canad. J. Math. 45 (1993), no. 6, 1135–1154.
  * [7] E. Chibrikov, On free Sabinin algebras. Comm. Algebra 39 (2011), no. 11, 4014–4035.
  * [8] P.M. Cohn, Subalgebras of free associative algebras. Proc. London Math. Soc. (3)14 (1964), 618–632.
  * [9] A.J. Czerniakiewicz, Automorphisms of a free associative algebra of rank 2. I, II. Trans. Amer. Math. Soc. 160 (1971), 393–401; 171 (1972), 309–315.
  * [10] V. Dotsenko, U. Umirbaev, An Effective Criterion for Nielsen–Schreier Varieties. International Mathematics Research Notices, rnad092, https://doi.org/10.1093/imrn/rnad092
  * [11] V. Drensky, Wild automorphisms of nilpotent-by-abelian Lie algebras. Manuscripta Math. 74 (1992), no. 2, 133–141.
  * [12] H.W.E. Jung, Uber ganze birationale Transformationen der Ebene. J. reine angew Math. 184 (1942), 161–174.
  * [13] C.E. Kofinas and A.I. Papistas, Automorphisms of free metabelian Lie algebras, Inter. J. Algebra and Compt. 26(4) (2016) 751–762.
  * [14] D. Kozybaev, L. Makar-Limanov, U. Umirbaev, The Freiheitssatz and the automorphisms of free right-symmetric algebras. Asian-Eur. J. Math. 1 (2008), no. 2, 243–254.
  * [15] W. van der Kulk, On polynomial rings in two variables. Nieuw Archief voor Wisk. (3)1 (1953), 33–41.
  * [16] A.G. Kurosh, Non-associative free algebras and free products of algebras. Rec.Math. [Mat. Sbornik] N.S. 20(62) (1947), 239–262.
  * [17] J. Lewin, On Schreier varities of linear algebras. Trans. Amer. Math. Soc. 132 (1968), 553–562.
  * [18] L.G. Makar-Limanov, The automorphisms of the free algebra of two generators. Funktsional. Anal. i Prilozhen. 4 (1970), no. 3, 107–108; English translation: in Functional Anal. Appl. 4 (1970), 262–263.
  * [19] Makar-Limanov, L., Shestakov, I., Polynomial and Poisson dependence in free Poisson algebras and free Poisson fields. Journal of Algebra 349 (2012), 372–379.
  * [20] L. Makar-Limanov, U. Turusbekova, U. Umirbaev, Automorphisms and derivations of free Poisson algebras in two variables. J. Algebra 322 (2009), 3318–3330.
  * [21] L. Makar-Limanov, U. Umirbaev, Free Poisson fields and their automorphisms. J. Algebra Appl. 15 (2016), no. 10, 1650196, 13 pp.
  * [22] A.A. Mikhalev, Subalgebras of free colored Lie superalgebras. Mat. Zametki 37 (1985), no. 5, 653–661.
  * [23] A.A. Mikhalev, Subalgebras of free Lie p-superalgebras. (Russian) Mat. Zametki 43 (1988), no. 2, 178–191, 300; translation in Math. Notes 43 (1988), no. 1–2, 99–106.
  * [24] A.A. Mikhalev, I.P. Shestakov, PBW-pairs of varieties of linear algebras. Comm. Algebra 42(2014), no.2, 667–687.
  * [25] M. Nagata, On the automorphism group of $k[x,y]$. Lect. in Math., Kyoto Univ., Kinokuniya, Tokio, 1972.
  * [26] R. Nauryzbaev, On generators of the tame automorphism group of free metabelian Lie algebras. Comm. Algebra 43 (2015), no. 5, 1791–1801.
  * [27] Z. Özkurt, N. Ekici, An application of the Dieudonné determinant: detecting non-tame automorphisms. J. Lie Theory 18 (2008), no. 1, 205–214.
  * [28] A.I. Papistas, Automorphisms of free polynilpotent Lie algebras. Comm. Algebra 21 (1993), no. 12, 4391–4395.
  * [29] A.I. Papistas, $\mathrm{IA}$-automorphisms of 2-generator metabelian Lie algebras, Algebra Colloq. 3 (1996), 193–198.
  * [30] C. Reutenauer, Applications of a noncommutative Jacobian matrix, J. Pure Appl. Algebra 77 (1992), 169–181.
  * [31] V.A. Roman’kov, The automorphism group of a free metabelian group. The Interconnection Problems of Abstract and Applied Algebra, Novosibirsk (1985), pp. 53–80 (in Russian).
  * [32] V. Roman’kov, On the automorphism group of a free metabelian Lie algebra Internat. J. Algebra Comput. 18 (2008), no. 2, 209–226.
  * [33] V.A. Roman’kov, Primitive elements and automorphisms of the free metabelian group of rank 3. Sib. Èlektron. Mat. Izv. 17 (2020), 61–76.
  * [34] I.R. Shafarevich, On some infinite-dimensional groups. II.(Russian)Izv. Akad. Nauk SSSR Ser. Mat.45(1981), no.1, 214–226.
  * [35] I.P. Shestakov and U.U. Umirbaev, Free Akivis algebras, primitive elements and their relations with loops. J. Algebra 250 (2002), 533–548.
  * [36] I.P. Shestakov, U.U. Umirbaev, Tame and wild automorphisms of rings of polynomials in three variables. J. Amer. Math. Soc. 17 (2004), 197–227.
  * [37] A.I. Shirshov, Subalgebras of free Lie algebras. (Russian) Mat. Sbornik 33(75) (1953), 441–452.
  * [38] A.I. Shirshov, Subalgebras of free commutative and free anticommutative algebras. (Russian) Mat. Sbornik 34(76) (1954), 81–88.
  * [39] A.L. Shmel’kin, Wreath products of Lie algebras and their application in the theory of groups, Trudy Moskov. Mat. Obshch. 29(1973), 247–260, Russian; Translation, Trans. Moscow Math. Soc. 29(1973), 239–252.
  * [40] V. Shpilrain, On generators of $L/R^{2}$ Lie algebras. Proc. Amer. Math. Soc. 119 (1993), no. 4, 1039–1043.
  * [41] A.S. Shtern, Free Lie superalgebras. Sibirsk. Mat. Zh. 27 (1986), no. 1, 170–174.
  * [42] M.K. Smith, Stably tame automorphisms, J. Pure and Appl. Algebra, 58(1989), 209–212.
  * [43] U.U. Umirbaev, Partial derivations and endomorphisms of some relatively free Lie algebras. (Russian) Sibirsk. Mat. Zh. 34 (1993), no. 6, 179–188.
  * [44] U.U. Umirbaev, On Schreier varieties of algebras. Algebra i logika 33 (1994), no. 3, 317–340; Ehglish translation: In Algebra and Logic 33 (1994), no. 3, 180–193.
  * [45] U.U. Umirbaev, Universal derivations and subalgebras of free algebras. Proc. 3rd Internat. Conf. Algebra, Krasnoyarsk, Russia, Walter de Gruyter, Berlin, 1996, 255–271.
  * [46] U. Umirbaev, The Anick automorphism of free associative algebras. J. Reine Angew. Math. 605 (2007), 165–178.
  * [47] E. Witt, Die Unterringe der freien Lieschen Ringe. Math. Z. 64 (1956), 195–216.
  * [48] K.A. Zhevlakov, A.M. Slinko, I.P. Shestakov, A.I. Shirshov, Rings that are nearly associative. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1982.