missing

# Light In The Black: An Evaluation of Data Augmentation Techniques for
COVID-19 CT’s Semantic Segmentation

Bruno A. Krinski, Daniel V. Ruiz, and Eduardo Todt

###### Abstract

With the COVID-19 global pandemic, computer-assisted diagnoses of medical
images have gained much attention, and robust methods of Semantic Segmentation
of Computed Tomography (CT) became highly desirable. Semantic Segmentation of
CT is one of many research fields of automatic detection of COVID-19 and has
been widely explored since the COVID-19 outbreak. In this work, we propose an
extensive analysis of how different data augmentation techniques improve the
training of encoder-decoder neural networks on this problem. Twenty different
data augmentation techniques were evaluated on five different datasets. Each
dataset was validated through a five-fold cross-validation strategy, thus
resulting in over 3,000 experiments. Our findings show that spatial level
transformations are the most promising to improve the learning of neural
networks on this problem.

  * Resumo.

Com a COVID-19, diagnósticos de imagens médicas assistidos por computador
ganharam muita atenção, e métodos robustos de Segmentação Semântica de
Tomografia Computadorizada (TC) tornaram-se altamente desejáveis. A
Segmentação Semântica de TC é um dos muitos campos de pesquisa de detecção
automática da COVID-19 e foi amplamente explorado desde o surto da COVID-19.
Neste trabalho, propomos uma análise extensiva sobre o quanto diferentes
técnicas de aumento de dados contribuem para melhorar o treinamento de redes
neurais codificador-decodificador sobre este problema. Vinte técnicas
diferentes de aumento de dados foram avaliadas em cinco conjuntos de dados
diferentes. Cada conjunto de dados foi validado através de uma estratégia de
validação cruzada de cinco subconjuntos, resultando assim em mais de 3.000
experimentos. Nossas descobertas mostram que as transformações de nível
espacial são as mais promissoras para melhorar o aprendizado das redes neurais
sobre este problema.

##  1 Introduction

Since 2019 the world has struggled with the new coronavirus (COVID-19)
pandemic, with millions of infections and deaths worldwide [Wang et al. 2020].
Until now, there are a total of 427,169,421 global cases and a total of
5,902,878 global deaths [of Medicine 2022] (updated February 22th, 2022). Due
to the virus’s quick dissemination, early diagnosis is highly desirable for
faster treatment and tracking infected people [Chen et al. 2020a]. Automatic
detection of COVID-19 infections in Computed Tomographys (CTs) shows to be a
great help for early diagnoses [Shi et al. 2021], with the Semantic
Segmentation [Cao and Bao 2020] of CTs being widely explored since the
COVID-19 outbreak [Shi et al. 2021]. Deep Learning based techniques and Deep
Neural Networks achieved impressive results in the segmentation of COVID-19
CTs [Shi et al. 2021, Krinski et al. 2021]. However, it has two limiting
factors. The first one is that labeling Semantic Segmentation is a labor-
intensive and timing-consuming process, and each pixel of the image must
receive the correct label. Otherwise, the network could converge to wrong
results [Shi et al. 2021, Cao and Bao 2020]. In addition to that, labeling CT
segmentation datasets must be made by highly specialized doctors to properly
label the lesion regions of the image [Shi et al. 2021].

With new approaches being proposed quickly, an urgency aggravated by the
global pandemic, the need for a proper evaluation becomes apparent. A broad
benchmark of architectures was presented by [Krinski et al. 2021], and one of
their conclusions was that the models’ generalization was impaired by the
small number of samples on the field’s datasets which also suffer from class
imbalance introducing some bias to the models. Data augmentation can mitigate
this issue; however, the influence of data augmentation during training was
left out.

In this work, we propose an extensive analysis of how different data
augmentation techniques improve the training of encoder-decoder neural
networks on this problem. Twenty different data augmentation techniques were
evaluated, see section 4, in three distinct experiments using five CTs
datasets: MedSeg [MedSeg 2021], Zenodo [Jun et al. 2020], CC-CCII [Zhang et
al. 2020], MosMed [Morozov et al. 2020], Ricord1a [Tsai et al. 2020]. Each
dataset was validated through a five-fold cross-validation strategy, thus
resulting in over 3,000 experiments. The code for running these same
experiments is publicly available111https://github.com/VRI-
UFPR/SparkInTheDarkLars2021.

##  2 Related Work

Data augmentation aims to generate a synthetic image by applying different
operations to a preexisting labeled image [Ruiz et al. 2020b]. The most common
operation are variations of an affine transformation such as flip, translate,
rotate, scale. The Random Erasing [Zhong et al. 2020] is one example of data
augmentation that adds information to the original image. In this technique, a
rectangle with random values is positioned in the image. This rectangle’s
height, width, and center points are random values. This data augmentation
helps the network learning process be more robust to object occlusions. Also,
it reduces the overfitting in the training step. The CutMix [Yun et al. 2019]
follows the same idea of the Random Erasing. However, instead of using
constant values or even random ones, the technique mixes two images by adding
an image A to some region of image B. This reduces information loss and
encourages generalization.

Following the same line of CutMix [Yun et al. 2019], the study presented by
[Summers and Dinneen 2019] evaluated several different non-linear mixing
algorithms. The authors showed that non-linear mixing algorithms are also
effective as linear mixing data augmentations. [Hendrycks et al. 2020]
proposed a mixing data augmentation called AugMix. In AugMix, sequences of
data augmentations are applied in parallel, generating a different image for
each data augmentation sequence. In the end, an element-wise convex
combination is applied to mix all generated images. In [Kisantal et al. 2019],
the authors proposed a data augmentation technique for small objects to
improve such objects’ detection and segmentation. The authors applied ”copy
and pasting” strategy to create several copies of the objects of interest.
They showed that this strategy increases the number of anchor boxes generated
by the Mask-RCNN [He et al. 2017], which helps the network to learn and detect
small objects. The ANDA [Ruiz et al. 2019] and IDA [Ruiz et al. 2020a]
techniques follow the idea of introducing new objects, however since those are
techniques focused on the generic problem of Salient Object Detection (SOD),
some additional operations are necessary such as Image Inpainting to erase the
original object and some additional computation to choose which combination of
background and object produce a significant salience and the affine
transformations to be applied to the new object that will replace the original
one.

In the Grid Mask [Chen et al. 2020b], a mask composed of black squares
uniformly distributed is generated and applied on top of the image. This
augmentation follows the same idea of Random Erasing, which forces the network
to learn from occluded regions in the image and reduces overfitting. The
advantage over others that randomly remove regions from the image is that
random algorithms can remove relevant regions from the image. The InstaBoost
[Fang et al. 2019] uses an inpainting technique [Bertalmio et al. 2001] to
remove the interest object from the image and place it in another region in
the image. An appearance consistency heatmap [Field et al. 1993] is used to
estimate the new region of the image where the object will be placed.

The authors of [Liu et al. 2019] proposed a data augmentation based on image-
to-image divided into two steps: training and deployment. The proposed method
uses several images of different classes in the training step, called source
images, and learns to translate the images between these classes. Then, in the
deployment step, a small set of images from the target class is used, with the
proposed method being able to translate from the source classes to the target
class. For each image to be mixed in the SuperMix [Dabouei et al. 2020], a
mixing binary mask with the salience information of the respective image is
generated. Then, a teacher model already trained in the problem is applied to
mix the images, optimize the position of the salient region in the mixed
image, and ensure that both salient regions are presented in the final mixed
image.

These Data Augmentations (DAs) are, in general, for generic segmentation
problems. However, there is no proper comparison of DA methods for the
COVID-19 segmentation problem. In this work, we focus on performing a
extensive comparison of generic DA methods in the approached problem.

##  3 Metrics and Datasets

The models were trained and evaluated across five different CTs datasets:
MedSeg [MedSeg 2021], Zenodo [Jun et al. 2020], CC-CCII [Zhang et al. 2020],
MosMed [Morozov et al. 2020], and Ricord1a [Tsai et al. 2020]. The MedSeg has
929 images and labels for four classes, with the following pixel proportion:
Background (0.98563), Ground Glass Opacity (GGO) (0.01072), Consolidation
(0.00351), and Pleural Effusion (0.0001). The Zenodo dataset has 3,520 images
and labels for four classes, with the following pixel proportion: Background
(0.89893), Left Lung (0.04331), Right Lung (0.04923), and Infections
(0.00852). The MosMed dataset is composed of 2,049 images, with labels for two
classes, with the following pixel proportion: Background (0.99810) and GGO-
Consolidation (0.00189). Ricord dataset is divided into three sets: 1a, 1b,
and 1c. The set 1a is the only one with segmentation masks and has 9,166
images with labels for two classes, with the following pixel proportion:
Background (0.95295) and Infections (0.04704). We also used a sub-set of CC-
CCII with segmentation masks composed of 750 images and has labels for four
classes, with the following pixel proportion: Background (0.87152), Lung Field
(0.11691), GGO (0.00802), and Consolidation (0.00353).

One of the problems pointed out in [Krinski et al. 2021] was the class
imbalance due to several images with just the background class; in fact,
recent work has shown that several problems suffer from class imbalance
[Laroca et al. 2021, Laroca et al. 2022]. Therefore, to mitigate this problem,
in the first step of this evaluation, we removed from the datasets images with
no lesion in the ground-truth mask. In the CC-CCII dataset, 4 images were
removed; in the MedSeg dataset, 457 images were removed; in the MosMed, 1,264
images were removed; in the Zenodo dataset, 547 were removed; and in the
Ricord1a, no image was removed. The datasets were split into 80% for training
and 20% for testing. Then, a five-fold cross-validation strategy further
divided the training set between training and validating sets. The metrics
used for evaluation were the F-score described by equation 1 and Intersection
over Union (IoU) described by equation 2.

$F-score=\frac{TruePositive}{TruePositive+\frac{FalsePositive+FalseNegative}{2}}$
(1)

$IoU=\frac{intersection}{union}$ (2)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

Figure 1: Illustration of twenty data augmentation techniques applied in a CT
image. In 1(a), the original image is presented. The data augmentations are
illustrated in the following sequence: Contrast Limited Adaptive Histogram
Equalization (CLAHE) 1(b), Coarse Dropout 1(c), Elastic Transform 1(d), Emboss
1(e), Flip 1(f), Gaussian Blur 1(g), Grid Distortion 1(h), Grid Dropout 1(i),
Image Compression 1(j), Median Blur 1(k), Optical Distortion 1(l), Piecewise
Affine Transformation 1(m), Posterize 1(n), Random Brightness Contrast (RBC)
1(o), Random Crop 1(p), Random Gamma 1(q), Random Snow 1(r), Rotate 1(s),
Sharpen 1(t), Shift Scale Rotate 1(u).

##  4 Experiments

In general, when dealing with the COVID-19 CT segmentation problem it is usual
to either completely neglect a dedicated evaluation of the impact of DA
techniques or merely report using a limited set of generic DA, not optimized
or specially designed for medical images, like Flip and Rotation operations,
such as in [Zhao et al. 2021, Qiblawey et al. 2021, Raj et al. 2021, Müller et
al. 2020, Chen et al. 2020c, Xu et al. 2020]. To properly measure the impact
of data augmentation on the COVID-19 CT segmentation problem, we evaluated
twenty data augmentation techniques.

In this work we evaluated the following twenty data augmentation techniques:
CLAHE, Coarse Dropout, Elastic Transform, Emboss, Flip, Gaussian Blur, Grid
Distortion, Grid Dropout, Image Compression, Median Blur, Optical Distortion,
Piecewise Affine Transformation, Posterize, RBC, Random Crop, Random Gamma,
Random Snow, Rotate, Sharpen, Shift Scale Rotate. Figure 1 illustrates the
twenty data augmentation techniques applied to a CT image 1(a). All data
augmentation techniques evaluated here are available in the Albumentation
library [Buslaev et al. 2020].

The encoder-decoder network chosen to evaluate the dataset augmentations was
the RegNetx-002 [Xu et al. 2021] encoder and U-net++ [Zhou et al. 2018]
decoder. Since the encoders achieved close results in the comparison performed
in [Krinski et al. 2021], the RegNetx-002 was chosen due to being the network
with a smaller number of parameters, making the RegNetx-002 faster for
training. The U-net++ was chosen because it achieved the highest F-score
compared with other decoders [Krinski et al. 2021]. The evaluation of how data
augmentation affects the results of different encoders and decoders was left
for future evaluation. Also, all experiments were evaluated through a five-
fold cross-validation strategy. In total, we performed three sets of data
augmentation evaluation, with each set varying network training parameters.
The varied parameters were the number of epochs trained, the learning rate,
and learning rate decay.

Table 1: Results of the first data augmentation evaluation in the test set.
The blue-colored values indicate the best F-scores, and the red-colored values
indicate the best IoUs values. Two probabilities of applying the data
augmentation were evaluated: 0.1 and 0.2. The green highlighted values show
the data augmentations in which the P-value achieved values lower than 0.05,
and the null hypothesis was rejected.

Probability | Augmentation | CC-CCII | MedSeg | MosMed | Ricord1a | Zenodo
---|---|---|---|---|---|---
|  |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU
|  No Augmentation |  0.6419 |  0.6011 |  0.4763 |  0.4289 |  0.7859 |  0.7211 |  0.8956 |  0.8372 |  0.8497 |  0.8152
0.1 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6363 0.6410 0.6454 0.6409 0.6458 0.6419 0.6443 0.6409 0.6427 0.6423 0.6439 0.6430 0.6446 0.6452 0.6425 0.6421 0.6415 0.6473 0.6426 0.6452 |  0.5962 0.6004 0.6048 0.6007 0.6044 0.6011 0.6036 0.5999 0.6019 0.6015 0.6032 0.6025 0.6039 0.6039 0.6020 0.6017 0.6010 0.6064 0.6024 0.6043 |  0.4754 0.4756 0.4781 0.4749 0.4787 0.4758 0.4764 0.4748 0.4769 0.4764 0.4768 0.4756 0.4763 0.4780 0.4757 0.4759 0.4737 0.4775 0.4760 0.4785 |  0.4278 0.4281 0.4305 0.4275 0.4309 0.4283 0.4291 0.4272 0.4295 0.4290 0.4292 0.4282 0.4289 0.4303 0.4285 0.4280 0.4263 0.4299 0.4285 0.4307 |  0.7927 0.7904 0.7977 0.7922 0.7957 0.7892 0.7967 0.7916 0.7936 0.7851 0.7932 0.8000 0.7954 0.7859 0.7863 0.7907 0.7887 0.7998 0.7885 0.8001 |  0.7268 0.7244 0.7311 0.7265 0.7296 0.7238 0.7308 0.7265 0.7276 0.7201 0.7269 0.7335 0.7285 0.7202 0.7218 0.7252 0.7231 0.7335 0.7225 0.7336 |  0.8933 0.8941 0.8915 0.8960 0.8858 0.8942 0.8911 0.8954 0.8950 0.8955 0.8922 0.8909 0.8931 0.8930 0.8952 0.8933 0.8936 0.8893 0.8941 0.8924 |  0.8340 0.8350 0.8322 0.8374 0.8256 0.8353 0.8314 0.8367 0.8361 0.8371 0.8330 0.8320 0.8338 0.8336 0.8367 0.8345 0.8346 0.8297 0.8350 0.8330 |  0.8490 0.8488 0.8496 0.8490 0.8468 0.8494 0.8502 0.8493 0.8499 0.8496 0.8500 0.8499 0.8492 0.8493 0.8497 0.8496 0.8492 0.8475 0.8494 0.8494 |  0.8142 0.8140 0.8149 0.8143 0.8110 0.8148 0.8156 0.8146 0.8155 0.8151 0.8155 0.8154 0.8147 0.8146 0.8151 0.8149 0.8143 0.8118 0.8149 0.8146
0.2 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6386 0.6379 0.6454 0.6420 0.6465 0.6421 0.6476 0.6398 0.6436 0.6431 0.6455 0.6480 0.6417 0.6419 0.6456 0.6415 0.6422 0.6458 0.6418 0.6482 |  0.5978 0.5976 0.6050 0.6015 0.6054 0.6015 0.6070 0.5990 0.6022 0.6025 0.6045 0.6072 0.6014 0.6009 0.6044 0.6006 0.6015 0.6050 0.6009 0.6070 |  0.4723 0.4744 0.4785 0.4755 0.4771 0.4755 0.4802 0.4740 0.4776 0.4762 0.4781 0.4771 0.4772 0.4761 0.4752 0.4761 0.4711 0.4781 0.4770 0.4806 |  0.4252 0.4270 0.4303 0.4280 0.4292 0.4288 0.4325 0.4264 0.4299 0.4289 0.4300 0.4297 0.4299 0.4280 0.4279 0.4279 0.4237 0.4302 0.4289 0.4326 |  0.7844 0.7931 0.8015 0.7927 0.7991 0.7876 0.8038 0.7868 0.7965 0.7888 0.7941 0.8049 0.7967 0.7875 0.7943 0.7935 0.7875 0.8010 0.7852 0.8004 |  0.7192 0.7273 0.7345 0.7270 0.7328 0.7230 0.7368 0.7209 0.7310 0.7231 0.7279 0.7385 0.7317 0.7212 0.7287 0.7274 0.7215 0.7340 0.7189 0.7340 |  0.8955 0.8920 0.8839 0.8942 0.8889 0.8928 0.8886 0.8855 0.8963 0.8951 0.8938 0.8899 0.8923 0.8867 0.8953 0.8916 0.8883 0.8813 0.8954 0.8856 |  0.8367 0.8329 0.8229 0.8355 0.8289 0.8338 0.8288 0.8250 0.8378 0.8364 0.8352 0.8304 0.8330 0.8266 0.8368 0.8323 0.8284 0.8203 0.8370 0.8253 |  0.8491 0.8489 0.8491 0.8487 0.8444 0.8494 0.8496 0.8487 0.8499 0.8497 0.8503 0.8495 0.8497 0.8493 0.8497 0.8493 0.8484 0.8473 0.8495 0.8491 |  0.8142 0.8141 0.8141 0.8137 0.8080 0.8146 0.8148 0.8140 0.8155 0.8150 0.8159 0.8147 0.8151 0.8145 0.8150 0.8146 0.8133 0.8114 0.8148 0.8141

In the first evaluation of data augmentations, the architecture was trained
for 50 epochs with a learning rate of 0.001. The learning rate was divided by
10 every 10 epochs. Two probabilities of applying the data augmentation were
evaluated: 0.1 and 0.2. As presented in Table 1, most of the data
augmentations did not improve the F-score and the IoU. The MosMed dataset was
the only dataset that applying data augmentations improved the results, with
improvements in the F-score of 1% and 2% in most of the data augmentations
applied. In the Zenodo dataset, most of the augmentations achieved similar
results with the baseline. However, the Grid Distortion with the probability
of 0.1 and the Optical Distortion with the probability of 0.2 improved the
F-score by 1%. In the MedSeg, only the Shift Scale Rotate augmentation with
the probability of 0.2 achieved better results with an F-score 1% higher than
the baseline. The CC-CCII and Ricord1a datasets did not achieve improvements
with data augmentations. To perform a statistical analysis of the data
augmentation evaluation, the one-sided Wilcoxon signed-rank test was applied
with the null hypothesis as the F-scores of the distribution without data
augmentation are greater than the distributions with data augmentation.

The datasets CC-CCII and MosMed were the datasets most sensitive to data
augmentation and achieved better F-scores in seven data augmentations when the
data augmentation was applied with probability 0.1 and eight data
augmentations when applied with probability 0.2. In the MosMed dataset, the
null hypothesis was rejected in most of the data augmentations applied in both
probabilities. In the MedSeg, the null hypotheses were rejected in only one
data augmentation when applied with probability 0.1 and two data augmentations
when applied with probability 0.2. The Ricord1a and Zenodo did not achieve
statistical significance to reject the null hypotheses in any data
augmentation.

Table 2: Results of the second data augmentation evaluation in the test set.
The blue-colored values indicate the best F-scores, and the red-colored values
indicate the best IoUs values. Two probabilities of applying the data
augmentation were evaluated: 0.1 and 0.2. The green highlighted values show
the data augmentations in which the P-value achieved values lower than 0.05,
and the null hypothesis was rejected.

Probability | Augmentation | CC-CCII | MedSeg | MosMed | Ricord1a | Zenodo
---|---|---|---|---|---|---
|  |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU
|  No Augmentation |  0.6447 |  0.6048 |  0.4780 |  0.4308 |  0.7938 |  0.7292 |  0.9031 |  0.8472 |  0.8511 |  0.8174
0.1 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6431 0.6429 0.6499 0.6458 0.6493 0.6448 0.6506 0.6412 0.6462 0.6462 0.6487 0.6495 0.6469 0.6439 0.6470 0.6449 0.6435 0.6522 0.6441 0.6504 |  0.6025 0.6016 0.6095 0.6054 0.6087 0.6045 0.6102 0.6011 0.6052 0.6060 0.6079 0.6092 0.6070 0.6037 0.6057 0.6043 0.6037 0.6115 0.6036 0.6096 |  0.4781 0.4769 0.4817 0.4777 0.4812 0.4799 0.4814 0.4802 0.4787 0.4765 0.4819 0.4819 0.4796 0.4805 0.4772 0.4783 0.4770 0.4810 0.4778 0.4823 |  0.4305 0.4304 0.4342 0.4308 0.4344 0.4329 0.4341 0.4324 0.4315 0.4299 0.4341 0.4349 0.4326 0.4326 0.4303 0.4305 0.4301 0.4341 0.4306 0.4350 |  0.7939 0.7985 0.8065 0.7953 0.8115 0.7917 0.8121 0.7927 0.7961 0.7959 0.8050 0.8018 0.7985 0.7940 0.7915 0.7949 0.7958 0.8073 0.7915 0.8060 |  0.7288 0.7335 0.7404 0.7296 0.7451 0.7265 0.7459 0.7279 0.7318 0.7314 0.7391 0.7370 0.7329 0.7276 0.7264 0.7300 0.7296 0.7416 0.7269 0.7397 |  0.9033 0.9027 0.8993 0.9039 0.8990 0.9035 0.9022 0.9019 0.9033 0.9023 0.9035 0.9022 0.9021 0.9025 0.9009 0.9020 0.9027 0.9007 0.9026 0.9026 |  0.8476 0.8464 0.8421 0.8482 0.8417 0.8473 0.8452 0.8453 0.8472 0.8458 0.8473 0.8454 0.8457 0.8460 0.8441 0.8455 0.8463 0.8439 0.8467 0.8460 |  0.8512 0.8510 0.8514 0.8509 0.8497 0.8511 0.8523 0.8511 0.8508 0.8516 0.8516 0.8522 0.8515 0.8512 0.8511 0.8510 0.8511 0.8501 0.8512 0.8518 |  0.8174 0.8174 0.8176 0.8171 0.8149 0.8173 0.8188 0.8174 0.8170 0.8179 0.8181 0.8186 0.8178 0.8174 0.8173 0.8171 0.8173 0.8158 0.8177 0.8181
0.2 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6408 0.6420 0.6526 0.6430 0.6501 0.6431 0.6543 0.6408 0.6460 0.6467 0.6492 0.6482 0.6472 0.6415 0.6449 0.6459 0.6416 0.6526 0.6466 0.6534 |  0.6005 0.6014 0.6121 0.6026 0.6091 0.6026 0.6139 0.6006 0.6057 0.6063 0.6085 0.6079 0.6063 0.6008 0.6049 0.6051 0.6018 0.6121 0.6063 0.6127 |  0.4764 0.4776 0.4840 0.4790 0.4813 0.4775 0.4835 0.4778 0.4782 0.4769 0.4821 0.4815 0.4793 0.4791 0.4771 0.4772 0.4752 0.4831 0.4778 0.4843 |  0.4293 0.4307 0.4366 0.4316 0.4341 0.4300 0.4365 0.4308 0.4308 0.4302 0.4346 0.4343 0.4318 0.4312 0.4299 0.4301 0.4280 0.4357 0.4310 0.4369 |  0.7919 0.7952 0.8137 0.7990 0.8066 0.7920 0.8102 0.7950 0.7962 0.7964 0.8057 0.8106 0.7930 0.7995 0.7963 0.7929 0.7961 0.8119 0.7867 0.8135 |  0.7273 0.7303 0.7469 0.7326 0.7408 0.7276 0.7441 0.7298 0.7310 0.7305 0.7396 0.7444 0.7280 0.7333 0.7309 0.7274 0.7291 0.7456 0.7209 0.7462 |  0.9025 0.9033 0.9012 0.9043 0.8977 0.9021 0.9007 0.9019 0.9048 0.9022 0.9030 0.9024 0.9027 0.9019 0.9034 0.9025 0.9028 0.8945 0.9035 0.8964 |  0.8461 0.8471 0.8439 0.8484 0.8397 0.8457 0.8432 0.8452 0.8494 0.8455 0.8466 0.8454 0.8463 0.8453 0.8474 0.8462 0.8467 0.8362 0.8474 0.8385 |  0.8512 0.8512 0.8517 0.8510 0.8486 0.8509 0.8524 0.8507 0.8512 0.8510 0.8518 0.8516 0.8512 0.8512 0.8513 0.8512 0.8507 0.8498 0.8515 0.8517 |  0.8173 0.8176 0.8178 0.8170 0.8135 0.8171 0.8189 0.8167 0.8175 0.8173 0.8183 0.8179 0.8174 0.8174 0.8175 0.8173 0.8166 0.8153 0.8180 0.8180

In the second evaluation of data augmentations, the architecture was trained
for 100 epochs with a learning rate of 0.001. The learning rate was divided by
10 every 20 epochs. Two probabilities of applying the data augmentation were
evaluated: 0.1 and 0.2. As presented in Table 2, the MosMed achieved the most
significant improvements, with the Grid Distortion with probability 0.1 and
the Elastic Transform with probability 0.2 increasing the F-score by 2%
compared with the baseline. However, unlike the first evaluation, the MosMeg
achieved better F-scores in only seven augmentations instead of fourteen
augmentations.

Table 3: Results of the third data augmentation evaluation in the test set.
The blue-colored values indicate the best F-scores, and the red-colored values
indicate the best IoUs values. Two probabilities of applying the data
augmentation were evaluated: 0.1 and 0.2. The green highlighted values show
the data augmentations in which the P-value achieved values lower than 0.05,
and the null hypothesis was rejected.

Probability | Augmentation | CC-CCII | MedSeg | MosMed | Ricord1a | Zenodo
---|---|---|---|---|---|---
|  |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU |  F-score |  IoU
|  No Augmentation |  0.6317 |  0.5893 |  0.4658 |  0.4183 |  0.7736 |  0.7089 |  0.9133 |  0.8610 |  0.8496 |  0.8153
0.1 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6288 0.6275 0.6330 0.6282 0.6348 0.6310 0.6328 0.6321 0.6296 0.6311 0.6328 0.6339 0.6308 0.6275 0.6295 0.6301 0.6278 0.6350 0.6322 0.6381 |  0.5873 0.5863 0.5912 0.5863 0.5922 0.5890 0.5911 0.5900 0.5876 0.5886 0.5911 0.5926 0.5889 0.5858 0.5874 0.5883 0.5863 0.5929 0.5895 0.5961 |  0.4618 0.4647 0.4679 0.4654 0.4686 0.4675 0.4704 0.4637 0.4653 0.4659 0.4696 0.4675 0.4645 0.4628 0.4626 0.4627 0.4610 0.4719 0.4650 0.4710 |  0.4149 0.4179 0.4208 0.4182 0.4214 0.4202 0.4229 0.4171 0.4182 0.4193 0.4219 0.4206 0.4179 0.4159 0.4156 0.4159 0.4146 0.4243 0.4177 0.4236 |  0.7698 0.7831 0.7907 0.7758 0.7933 0.7751 0.7907 0.7809 0.7725 0.7720 0.7862 0.7923 0.7762 0.7674 0.7764 0.7744 0.7675 0.7922 0.7740 0.7962 |  0.7068 0.7178 0.7249 0.7108 0.7278 0.7110 0.7241 0.7158 0.7086 0.7088 0.7205 0.7266 0.7113 0.7029 0.7112 0.7099 0.7032 0.7268 0.7099 0.7298 |  0.9140 0.9142 0.9136 0.9139 0.9130 0.9145 0.9144 0.9139 0.9142 0.9141 0.9141 0.9147 0.9141 0.9131 0.9128 0.9139 0.9141 0.9135 0.9146 0.9139 |  0.8617 0.8620 0.8610 0.8618 0.8603 0.8625 0.8621 0.8616 0.8620 0.8620 0.8620 0.8625 0.8621 0.8607 0.8605 0.8616 0.8620 0.8609 0.8626 0.8614 |  0.8498 0.8496 0.8505 0.8499 0.8493 0.8498 0.8509 0.8495 0.8477 0.8504 0.8496 0.8504 0.8502 0.8497 0.8498 0.8494 0.8496 0.8490 0.8499 0.8506 |  0.8155 0.8152 0.8163 0.8157 0.8147 0.8156 0.8169 0.8152 0.8144 0.8165 0.8160 0.8163 0.8159 0.8154 0.8156 0.8150 0.8151 0.8143 0.8157 0.8165
0.2 |  CLAHE Coarse Dropout Elastic Transform Emboss Flip Gaussian Blur Grid Distortion Grid Dropout Image Compression Median Blur Optical Distortion Piecewise Affine Posterize RBC Random Crop Random Gamma Random Snow Rotate Sharpen Shift Scale Rotate |  0.6272 0.6286 0.6373 0.6288 0.6365 0.6297 0.6370 0.6304 0.6312 0.6297 0.6335 0.6344 0.6303 0.6273 0.6315 0.6313 0.6258 0.6368 0.6323 0.6380 |  0.5854 0.5870 0.5956 0.5870 0.5944 0.5880 0.5951 0.5887 0.5888 0.5882 0.5914 0.5927 0.5886 0.5860 0.5894 0.5893 0.5841 0.5948 0.5901 0.5960 |  0.4649 0.4625 0.4714 0.4639 0.4696 0.4659 0.4725 0.4649 0.4661 0.4681 0.4691 0.4726 0.4647 0.4668 0.4625 0.4629 0.4599 0.4739 0.4668 0.4712 |  0.4177 0.4160 0.4235 0.4171 0.4226 0.4188 0.4249 0.4177 0.4186 0.4212 0.4217 0.4253 0.4177 0.4193 0.4157 0.4155 0.4131 0.4259 0.4196 0.4243 |  0.7663 0.7822 0.7925 0.7745 0.7935 0.7755 0.7992 0.7795 0.7754 0.7779 0.7855 0.7940 0.7733 0.7748 0.7694 0.7722 0.7709 0.8004 0.7750 0.7967 |  0.7027 0.7174 0.7268 0.7101 0.7283 0.7120 0.7327 0.7150 0.7118 0.7145 0.7205 0.7280 0.7098 0.7096 0.7049 0.7074 0.7067 0.7338 0.7119 0.7308 |  0.9140 0.9137 0.9122 0.9142 0.9127 0.9140 0.9140 0.9133 0.9144 0.9144 0.9147 0.9145 0.9144 0.9131 0.9137 0.9132 0.9145 0.9127 0.9147 0.9134 |  0.8618 0.8614 0.8589 0.8622 0.8597 0.8619 0.8612 0.8608 0.8625 0.8624 0.8628 0.8621 0.8625 0.8603 0.8615 0.8610 0.8622 0.8596 0.8628 0.8602 |  0.8497 0.8495 0.8508 0.8495 0.8487 0.8501 0.8512 0.8492 0.8498 0.8501 0.8508 0.8509 0.8497 0.8498 0.8497 0.8496 0.8484 0.8495 0.8495 0.8511 |  0.8154 0.8152 0.8166 0.8151 0.8138 0.8160 0.8172 0.8148 0.8155 0.8159 0.8168 0.8168 0.8156 0.8155 0.8154 0.8152 0.8142 0.8148 0.8154 0.8171

The CC-CCII achieved better F-scores with the Grid Distortion, Rotate, and
Shift Scale Rotate. Also, MedSeg had better F-scores in nine augmentations
with the probability of 0.1 and seven augmentations with the probability of
0.2. Data augmentations did not improve the F-score in the Zenodo and Ricord1a
datasets. The statistical analysis pointed out that the CC-CCII rejected the
null hypotheses in seven data augmentations in both probabilities. Training
for 100 epochs made the MedSeg achieve better F-scores in eleven data
augmentations with probability 0.1 and seven data augmentations with
probability 0.2. The results achieved in the MosMed dataset got worse when
compared with the results presented in Table 1, with only seven data
augmentations rejecting the null hypotheses. In the Ricord1a and Zenodo
datasets, besides the average F-score of the data augmentations being very
close to the average F-score without data augmentation, in the Ricord1a
dataset, the null hypothesis was rejected in two data augmentations with
probability 0.2, and in the Zenodo dataset, the null hypotheses were rejected
in three data augmentations with probability 0.1 and two data augmentations
with probability 0.2.

In the third evaluation, the architecture was trained for 100 epochs with a
learning rate of 0.0001. The learning rate was divided by 10 every 25 epochs.
Two probabilities of applying the data augmentation were evaluated: 0.1 and
0.2. As presented in Table 3, in the MosMed, the Shift Scale Rotate with the
probability of 0.1 and the Rotate with the probability of 0.2 increased the
F-score by 2% compared with the baseline. Also, the MosMed achieved the best
F-scores in nine augmentations with probability 0.1 and eight augmentations
with the probability of 0.2, pointing this training configuration with the
highest effectiveness for this dataset. Also, this training configuration
showed significant effectiveness in the Zenodo dataset, which achieved the
best F-scores in six augmentations with probability 0.1 and seven
augmentations with probability 0.2.

However, this training configuration did not significantly affect other
datasets. The MedSeg achieved higher F-scores in only three augmentations with
a probability of 0.1 and five augmentations with a probability of 0.2. Also,
the CC-CCII and Ricord1a datasets did not achieve improvements with any data
augmentation. Statistical analysis also was performed and, besides the average
F-score of the data augmentations being very close to the baseline without
data augmentation, the Zenodo and Ricord1a datasets achieved the highest
number of data augmentations with the null hypotheses rejected. The MosMed was
the dataset with the most promising results, with the null hypotheses rejected
in twelve data augmentations. The CC-CCII and MedSeg had fewer data
augmentations with the null hypotheses rejected compared with previews
experiments, showing that this training configuration is not proper for data
augmentations in these datasets.

##  5 Conclusion

These three experiments demonstrated that, although necessary, the generic
data augmentation techniques evaluated did not majorly improve the results in
the COVID-19 segmentation problem. The MosMed achieved the most significant
improvements with data augmentation, with the F-score being up to 2% higher in
comparison with baseline. This dataset was the most sensitive to data
augmentation techniques due to its imbalance problem. Data augmentations also
improved the CC-CCII and MedSeg dataset results, but it was necessary to train
the network for more epochs, and the data augmentations only achieved 1% of
improvements in the F-score. The Ricord1a and Zenodo datasets were the most
challenging and did not show improvements with data augmentations. Although
they achieved statistical significance to reject the null hypotheses in many
data augmentations, the average F-score slightly improved.

Another result of these data augmentation experiments is that spatial level
transformations such as Elastic Transform, Flip, Grid Distortion, Piecewise
Affine, Rotate, and Shift Scale Rotate were the operations that showed to be
the most favorable data augmentations to this problem. These data
augmentations improved the results in most of the experiments performed and
thus are the most promising techniques for future experiments with data
augmentation. The evaluation of more domain-specific data augmentation
techniques was left for future works.

## Acknowledgment

The authors would like to thank the Coordination for the Improvement of Higher
Education Personnel (CAPES) for the PhD scholarship. We gratefully acknowledge
the founders of the publicly available datasets, the support of NVIDIA
Corporation with the donation of the GPUs used for this research and the C3SL-
UFPR group for the computational cluster infrastructure.

## References

  * [Bertalmio et al. 2001] Bertalmio, M., Bertozzi, A. L., and Sapiro, G. (2001).  Navier-stokes, fluid dynamics, and image and video inpainting.  In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–I.
  * [Buslaev et al. 2020] Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A. A. (2020).  Albumentations: Fast and flexible image augmentations.  Information, 11(2).
  * [Cao and Bao 2020] Cao, F. and Bao, Q. (2020).  A survey on image semantic segmentation methods with convolutional neural network.  In 2020 International Conference on Communications, Information System and Computer Engineering (CISCE). IEEE.
  * [Chen et al. 2020a] Chen, M., Tu, C., Tan, C., Zheng, X., Wang, X., Wu, J., Huang, Y., Wang, Z., Yan, Y., Li, Z., Shan, H., Liu, J., and Huang, J. (2020a).  Key to successful treatment of covid-19: accurate identification of severe risks and early intervention of disease progression.  medRxiv, DOI:10.1101/2020.04.06.20054890.
  * [Chen et al. 2020b] Chen, P., Liu, S., Zhao, H., and Jia, J. (2020b).  Gridmask data augmentation.  arXiv preprint, arXiv:2001.04086.
  * [Chen et al. 2020c] Chen, X., Yao, L., and Zhang, Y. (2020c).  Residual attention u-net for automated multi-class segmentation of covid-19 chest ct images.  arXiv preprint, arXiv:2004.05645.
  * [Dabouei et al. 2020] Dabouei, A., Soleymani, S., Taherkhani, F., and Nasrabadi, N. M. (2020).  Supermix: Supervising the mixing data augmentation.  arXiv preprint, arXiv:2003.05034.
  * [Fang et al. 2019] Fang, H.-S., Sun, J., Wang, R., Gou, M., Li, Y.-L., and Lu, C. (2019).  Instaboost: Boosting instance segmentation via probability map guided copy-pasting.  arXiv preprint, arXiv:1908.07801.
  * [Field et al. 1993] Field, D. J., Hayes, A., and Hess, R. F. (1993).  Contour integration by the human visual system: Evidence for a local “association field”.  Vision Research, 33(2):173–193.
  * [He et al. 2017] He, K., Gkioxari, G., Dollar, P., and Girshick, R. (2017).  Mask r-CNN.  In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE.
  * [Hendrycks et al. 2020] Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2020).  AugMix: A simple data processing method to improve robustness and uncertainty.  Proceedings of the International Conference on Learning Representations (ICLR).
  * [Jun et al. 2020] Jun, M. et al. (2020).  Covid-19 ct lung and infection segmentation dataset.  Zenodo. Available at: https://zenodo.org/record/3757475. Accessed: 2022-02-22.
  * [Kisantal et al. 2019] Kisantal, M., Wojna, Z., Murawski, J., Naruniec, J., and Cho, K. (2019).  Augmentation for small object detection.  In 9th International Conference on Advances in Computing and Information Technology (ACITY 2019). Aircc Publishing Corporation.
  * [Krinski et al. 2021] Krinski, B. A., Ruiz, D. V., and Todt, E. (2021).  Spark in the dark: Evaluating encoder-decoder pairs for COVID-19 CT’s semantic segmentation.  In 2021 Latin American Robotics Symposium (LARS). IEEE.
  * [Laroca et al. 2021] Laroca, R., Araujo, A. B., Zanlorensi, L. A., De Almeida, E. C., and Menotti, D. (2021).  Towards image-based automatic meter reading in unconstrained scenarios: A robust and efficient approach.  IEEE Access, 9:67569–67584.
  * [Laroca et al. 2022] Laroca, R., Cardoso, E. V., Lucio, D. R., Estevam, V., and Menotti, D. (2022).  On the cross-dataset generalization in license plate recognition.  In International Conference on Computer Vision Theory and Applications (VISAPP), pages 166–178.
  * [Liu et al. 2019] Liu, M.-Y., Huang, X., Mallya, A., Karras, T., Aila, T., Lehtinen, J., and Kautz, J. (2019).  Few-shot unsupervised image-to-image translation.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE.
  * [MedSeg 2021] MedSeg (2021).  Covid-19 ct segmentation dataset.  Available at: http://medicalsegmentation.com/covid19/.  Accessed: 2021-05-03.
  * [Morozov et al. 2020] Morozov, S., Andreychenko, A., Pavlov, N., Vladzymyrskyy, A., Ledikhova, N., Gombolevskiy, V., Blokhin, I., Gelezhe, P., Gonchar, A., and Chernina, V. (2020).  Mosmeddata: Chest ct scans with covid-19 related findings dataset.  medRxiv, DOI:10.1101/2020.05.20.20100362.
  * [Müller et al. 2020] Müller, D., Rey, I. S., and Kramer, F. (2020).  Automated chest ct image segmentation of covid-19 lung infection based on 3d u-net.  arXiv preprint, arXiv:2007.04774.
  * [of Medicine 2022] of Medicine, J. H. U. (2022).  Coronavirus resource center.  Available at: https://coronavirus.jhu.edu/.  Accessed: 2022-02-22.
  * [Qiblawey et al. 2021] Qiblawey, Y., Tahir, A., Chowdhury, M. E. H., Khandakar, A., Kiranyaz, S., Rahman, T., Ibtehaz, N., Mahmud, S., Al-Madeed, S., and Musharavati, F. (2021).  Detection and severity classification of covid-19 in ct images using deep learning.  arXiv preprint, arXiv:2102.07726.
  * [Raj et al. 2021] Raj, A. N. J., Zhu, H., Khan, A., Zhuang, Z., Yang, Z., Mahesh, V. G. V., and Karthik, G. (2021).  ADID-UNET—a segmentation model for COVID-19 infection from lung CT scans.  PeerJ Computer Science, 7:e349.
  * [Ruiz et al. 2019] Ruiz, D. V., Krinski, B. A., and Todt, E. (2019).  ANDA: A novel data augmentation technique applied to salient object detection.  In 2019 19th International Conference on Advanced Robotics (ICAR), pages 487–492.
  * [Ruiz et al. 2020a] Ruiz, D. V., Krinski, B. A., and Todt, E. (2020a).  IDA: Improved data augmentation applied to salient object detection.  In 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 210–217.
  * [Ruiz et al. 2020b] Ruiz, D. V., Salomon, G., and Todt, E. (2020b).  Can giraffes become birds? an evaluation of image-to-image translation for data generation.  Anais do Computer on the Beach, 11(1):176–182.  DOI: 10.14210/cotb.v11n1.p176-182, also available as arXiv preprint, arXiv:2001.03637.
  * [Shi et al. 2021] Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., and Shen, D. (2021).  Review of artificial intelligence techniques in imaging data acquisition, segmentation, and diagnosis for COVID-19.  IEEE Reviews in Biomedical Engineering, 14:4–15.
  * [Summers and Dinneen 2019] Summers, C. and Dinneen, M. J. (2019).  Improved mixed-example data augmentation.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE.
  * [Tsai et al. 2020] Tsai, E. et al. (2020).  Medical imaging data resource center - rsna international covid radiology database release 1a - chest ct covid+ (midrc-ricord-1a).  DOI: 10.7937/VTW4-X588.
  * [Wang et al. 2020] Wang, C., Horby, P. W., Hayden, F. G., and Gao, G. F. (2020).  A novel coronavirus outbreak of global health concern.  The Lancet, 395(10223):470–473.
  * [Xu et al. 2021] Xu, J., Pan, Y., Pan, X., Hoi, S., Yi, Z., and Xu, Z. (2021).  Regnet: Self-regulated network for image classification.  arXiv preprint, arXiv:2101.00590.
  * [Xu et al. 2020] Xu, Z., Cao, Y., Jin, C., Shao, G., Liu, X., Zhou, J., Shi, H., and Feng, J. (2020).  Gasnet: Weakly-supervised framework for covid-19 lesion segmentation.  arXiv preprint, arXiv:2010.09456.
  * [Yun et al. 2019] Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., and Choe, J. (2019).  CutMix: Regularization strategy to train strong classifiers with localizable features.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE.
  * [Zhang et al. 2020] Zhang, K. et al. (2020).  Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography.  Cell, 181(6):1423–1433.e11.
  * [Zhao et al. 2021] Zhao, X., Zhang, P., Song, F., Fan, G., Sun, Y., Wang, Y., Tian, Z., Zhang, L., and Zhang, G. (2021).  D2a u-net: Automatic segmentation of covid-19 lesions from ct slices with dilated convolution and dual attention mechanism.  arXiv preprint, arXiv:2102.05210.
  * [Zhong et al. 2020] Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2020).  Random erasing data augmentation.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):13001–13008.
  * [Zhou et al. 2018] Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., and Liang, J. (2018).  UNet++: A nested u-net architecture for medical image segmentation.  In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 3–11. Springer International Publishing.


# Impact of Device Thermal Performance on 5G mmWave Communication Systems
††thanks:  This research on is supported in part by the National Science
Foundation under grant numbers CNS-1618692 and CNS-1618836.

Muhammad Iqbal Rochman1, Damian Fernandez2, Norlen Nunez2, Vanlin Sathya1,
Ahmed S. Ibrahim2, Monisha Ghosh1, and William Payne1  1Department of Computer
Science, University of Chicago, Chicago, Illinois, USA
2Department of Electrical and Computer Engineering, Florida International
University, Miami, Florida, USA
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

5G millimeter wave (mmWave) cellular networks have been reported to deliver
1-2 Gbps downlink throughput, via speed-tests. However, these speed-tests
capture only a few seconds of throughput and are not representative of
sustained throughput over several minutes. We report the first measurements of
sustained throughput in three cities, Miami, Chicago, and San Francisco, where
we observe throughput throttling due to rising skin temperature of the phone
when it is connected to a deployed 5G mmWave base-station (BS). Radio Resource
Control (RRC) messaging between the phone and the BS indicates the reduction
in the number of aggregated mmWave channels from 4 to 1 followed by a switch
to 4G. We corroborate these measurements with infra-red images as the phone
heats up. Thus, mmWave throughput will be limited not by network
characteristics but by device thermal management.

###### Index Terms:

5G, mmWave, throughput, thermal, throttling, skin temperature, CPU
temperature.

##  I Introduction

5G New Radio (5G NR) cellular networks are being rapidly deployed around the
world in low ($<1$ GHz), mid ($1-6$ GHz), and high ($>24$ GHz) bands. Of
these, the high, or mmWave bands, are being deployed predominantly in dense
urban areas in the US while the low and mid-bands, including the C-Band (3.7 -
3.98 GHz) are witnessing deployments that can provide wider coverage in
suburban and rural areas due to the favorable propagation characteristics.
Recent measurements on deployed 5G mmWave networks in major US cities
demonstrate that indeed 5G mmWave can deliver extremely high throughput in the
range of 1 - 2 Gbps [1, 2, 3]. These high throughputs are enabled by
aggregating up to eight 100 MHz mmWave channels, depending on network and
device capabilities.

However there is still a number of challenges associated with guaranteeing QoS
in 5G mmWave: beam-tracking, beam management, building blockage, and rain
attenuation, to name a few, and these are being addressed in many research
efforts [4, 5]. In this paper we address a question that has received less
attention by the research community: what is the sustained downlink
throughput, over several minutes, that can be delivered by a 5G mmWave
connection? Most reported speed-test measurements use commonly used speedtest
apps such as Ookla111https://www.speedtest.net/ or the FCC
Speedtest222https://play.google.com/store/apps/details?id=com.samknows.fcc,
where the test runs for only 5 - 10 seconds and is not indicative of the
average throughput when an application is running over several minutes at the
high throughput. We postulate that a high-throughput data transfer over
several minutes using multiple mmWave channels will cause device heating with
a resultant increase in the skin temperature that will then trigger throttling
of the throughput until the device cools to acceptable levels. We present
results from detailed experiments conducted with consumer 5G smartphones
operating over deployed 5G mmWave networks to demonstrate that indeed this
phenomenon occurs repeatedly when the ambient temperatures are high enough to
deter the cooling of the device. We demonstrate that as the skin temperature
measured by the device increases, the number of mmWave channels being
aggregated drops from 4 to 1 followed by handover to 4G LTE, with the
throughput dropping at each step. With external cooling, e.g., using an ice-
pack or low ambient temperature (e.g., on a Chicago winter day), high
throughput can be sustained over several minutes. Furthermore, we identify
explicit message exchanges in the Radio Resource Control (RRC) layer between
the user equipment (UE) and the base-station (BS) that confirm that the reason
for handing over to 4G LTE is thermal and not network congestion or other
considerations. Lastly, we used an infra-red (IR) camera to further
corroborate the effect of temperature rise at the mmWave antenna locations on
throughput.

TABLE I: Experiment Parameters Parameter | Value
---|---
Operator | Verizon (Band n261/28 GHz)
# of experiment locations | Chi.: 2, Mia.: 1, SF: 1
Device model | Google Pixel 5
# of devices used | Chi.: 2, Mia.: 1, SF: 1
Cumulative # of meas. runs | 32
Distance between BS and UE |  $\sim$2 meter
Average RSRP over all meas. | -92.63 dBm

(a) Chicago

(b) Miami

(c) San Francisco

Figure 1: Measurements locations in the three different cities (see Table II). TABLE II: Experiment Locations Location | # | Address | GPS coordinate | Traffic used | # of meas. runs | Ambient temp. at meas. time
---|---|---|---|---|---|---
Chicago | 1 | 61st St. & Woodlawn Ave. | 41.7874024,-87.5965241 | BG DL+FCC ST | 16 | Oct: $\sim$$24\degree$ C, Jan: $\sim$$-10\degree$ C
2 | 61st St. & Midway Plai. | 41.7844559,-87.5962098 | BG DL+FCC ST | 9
Miami | 3 | Bayfront Park | 25.7753436,-80.1853549 | BG DL | 6 | Sep: $\sim$$31\degree$ C
San Francisco | 4 | Pine & Stockton St. | 37.7914117,-122.407218 | BG DL+FCC ST | 1 | Sep: $\sim$$15\degree$ C

##  II Related work

Recent literature [1, 2] has demonstrated the feasibility of achieving very
high throughput with consumer smartphones over commercially deployed 5G
mmWave, in spite of the well-known limitations of mmWave propagation due to
beam tracking, beam management, mobility management and building blockage.
Advanced techniques, based on machine learning and artificial intelligence,
have been proposed for addressing these limitations, for example in [4, 5].
Most recently, [6] presents detailed measurements of 5G mmWave deployments by
two major commercial 5G operators in the US in diverse environments using
smartphone-based tools. The measurement-driven propagation analysis
demonstrated performance differences due to terrain, frequency of operation,
antenna pattern, etc. However, the relationship between device temperature and
sustained 5G mmWave throughput was not explored. The results presented in this
paper seek to address the gap in the literature on the effect of device
thermal management on end-user throughput on 5G mmWave networks.

In particular, we seek to demonstrate that the drop in throughput is indeed
due to thermal. According to the 3GPP standard [7], a UE can provide
information to the BS about its thermal state via the the RRC_CONNECTED
message field. Upon receiving such a message from the UE, the BS will respond
by temporarily reducing the number of aggregated data streams, in both
component carriers and MIMO layers, in both downlink and uplink transmissions
until the thermal warning messages are no longer received. This reduction in
component carriers (e.g. reduction from 4 to 1 mmWave channels) will lead to a
reduction in throughput until the skin temperature drops to below a pre-
specified threshold.

##  III Measurement Tools and Methodology

In order to demonstrate the effect of device skin temperature on sustained
throughput over 5G mmWave, the following requirements need to be met:

  * •

A sustained download of a high-bandwidth data stream over $\sim$15 minutes
while connected to a 5G mmWave BS.

  * •

A method of measuring temperature while the download is occurring, and

  * •

A method of extracting RRC messaging between the UE and the BS while the
download is occurring.

In this section, the tools and methodology used in this paper to satisfy the
above requirements are described.

Table I summarizes the parameters of the experiments conducted in two
locations in Chicago, one location in Miami and one location in San Francisco.
Data was collected in all 3 locations over September - October 2021, and in
Chicago in January 2022 for performance comparison under cooler ambient
temperatures. All experiments were conducted using the same UE model and
network: Google Pixel 5, running Android 11 on a Verizon network with an
unlimited data plan333Subscribed Verizon plan indicates a throttling after 50
GBytes for 4G and 5G low/mid-band data, and no throttling for 5G mmWave data..
Fig. 1 shows the specific measurement locations in Chicago, Miami and San
Francisco, while Table II shows detailed information of each location. The
Verizon 5G mmWave network at each location utilizes band n261 at $28$ GHz.

Downlink throughput saturation is achieved using a combination of two methods:

  * •

Background Download (BG DL) using HTTP download of a 10 GB dataset file [8].

  * •

FCC Speed Test (FCC ST) app: the 5 sec downlink throughput test is run
repeatedly to ensure that the link stays saturated continuously.

Thermal throttling was observed using either one of the above methods, but
combining both methods ensures that the link is fully saturated. The Miami
measurements used only the BG DL traffic, while BG DL + FCC ST was used in the
Chicago and San Francisco measurements. Due to this minor difference in
methodology, there are two separate throughput measurements: PHY level
throughput collected by Network Signal Guru (NSG), described below, and APP
level throughput collected by FCC ST. Using all the measurements reported in
this paper, we verified that, as expected, the APP throughput is always lower
than the PHY throughput. APP throughput values are easier to extract from FCC
ST than PHY throughput from NSG (requiring manual data input). Thus, these
different types of throughput measurements are carefully separated and only
the same type of throughput values are compared whenever needed in our
analysis.

The following Android apps were used to collect measurements systematically
for the experiments described in §IV:

  * •

SigCap [3], an Android app developed at the University of Chicago which
collects Global Positioning System (GPS), time and location information along
with signal and network parameters (e.g., 4G and 5G RSRP, RSRQ, RSSI, PCI, 4G
frequency, etc) through APIs that extract information directly from the modem
chip and hence is compliant to relevant standards. Instantaneous skin, CPU and
GPU temperature measurements from the APIs were added to SigCap [9] for the
work reported in this paper.

  * •

Network Signal Guru (NSG)444https://m.qtrun.com/en/product.html, a commercial
app that utilizes the phone’s root capability to provide more detailed
information about the transmission such as operating frequency, number of
carrier components, bandwidth, PHY throughput, and RRC messaging. However,
data export from NSG is difficult compared to SigCap and hence we use it only
for certain measurements.

##  IV Experimental Results

In this section, we demonstrate the impact of 5G mmWave transmissions on the
device temperature as measured by SigCap, and the resulting effect on downlink
throughput under various operating conditions that impact the UE temperature
(e.g., phone cover and ambient temperature). Furthermore, we confirm our
findings using an IR camera to image the phone as the mmWave transmission
progressed over time to demonstrate that the throughput drop correlated with
the rise in temperature at the mmWave antenna locations.

###  IV-A Impact of 5G mmWave on UE Temperature

(a) Throughput degradation correlated with device temperatures.

(b) Correlation of throughput with number of NR channels.

Figure 2: Representative results of throughput degradation.

5G mmWave throughput and UE temperature Vs. time In order to demonstrate the
effect of 5G mmWave on device temperature we conducted numerous measurements
using the tools and methodologies describe in the previous section. We
performed a total of 32 measurement runs over all locations, where each
measurement run starts with a cool phone. Fig. 2 shows a representative
measurement at Location 1 in Chicago (taken on Oct 9, 2021) using the combined
BG DL + FCC ST method. The PHY data rate and number of mmWave channels are
manually transcribed from NSG, while the temperature data is collected from
SigCap, both taken at $5$ sec intervals and synchronized using timestamps from
both apps. Fig. 2 only shows the PHY throughput when the FCC ST is running (at
$\sim$20 sec intervals) in order to display the conditions when the downlink
to the UE is fully loaded.

Fig. 2(a) shows that a PHY throughput of almost $2$ Gbps was achieved soon
after the experiment was started at the $200$ sec mark, which is the result of
aggregating 4 mmWave channels as shown in Fig. 2(b). The throughput increase
is accompanied by a rise on all three temperature measurements: skin, CPU and
GPU. At the $300$ sec mark, the number of aggregated mmWave channels reduces
to 1 and the resultant throughput is reduced significantly. At this point, the
CPU and GPU temperatures are reduced slightly, but the skin temperature does
not reduce sufficiently to restore the throughput to the levels seen at the
beginning of the experiment. The download was completed at $800$ sec.

Figure 3: CDF of CPU and skin temperature when number of NR channels changed,
and when NR channel is 4.

Analysis of skin temperature effect on throughput. We observe two events: (1)
when the number of 5G mmWave channels is reduced from 4 to 1 (i.e., $300$ sec
on Fig. 2(a)), and (2) when the device is handed over to the LTE network
($500$ sec). At both events, we recorded a ”Secondary Cell Group Failure”
signalling packet in the NSG log, which shows compliance to the 3GPP standard
[7]. Moreover, using the Android Temperature API [9], we obtained the static
temperature threshold values: $96\degree$ C for CPU and GPU, and $43\degree$ C
for Skin. Fig. 3 shows the skin and CPU temperature distribution of all our
data from all locations for the following cases: (i) the temperature when 4
mmWave channels are being aggregated, (ii) the temperature just after the
switch from 4 mmWave channels to 1 mmWave channel, and (ii) the temperature
just after the switch from 5G mmWave to LTE. We omit GPU temperature since we
observe that the CPU and GPU temperatures are similar. The figure clearly
shows that throttling to 1 mmWave channel happens mostly at skin temperature
of $\sim$$43\degree$ C, while throttling down to LTE happens mostly at skin
temperature of $\sim$$45\degree$ C. On the other hand, the CPU temperature
does not exhibit any correlation with the events since the its threshold is
never crossed. Hence, we infer that the skin temperature is the trigger that
causes the throughput degradation. While it appears that there is some
oscillation between states, on a larger time scale, we can still observe an
on-off pattern as shown by Fig. 2 between the $500$ to $770$ sec mark.

Figure 4: Correlation between APP throughput and Skin temperature.

Fig. 4 displays all 5G mmWave measurements collected at Chicago and San
Francisco, using both BG DL and FCC ST to saturate the downlink transfer.
Clearly, the higher skin temperature correlates to lower 5G mmWave throughput,
with lower throughput values recorded mostly in summer (Sep-Oct) and the
higher values recorded in Chicago in winter (Jan).

###  IV-B Thermal performance as a function of ambient conditions

(a) APP Throughput comparison between summer and winter in Chicago.

(b) PHY Throughput comparison between Chicago, Miami, and San Francisco in
summer.

Figure 5: Throughput over multiple experiments, time-normalized to the first
data

Effect of ambient temperature over seasons and location. Fig. 5 shows the
mmWave throughput versus time, where the time axis has been normalized i.e.,
$0$ sec is the timestamp of the first data point. Fig. 5(a) shows the
comparison of APP throughput between summer (Sep-Oct) and winter (Jan) in
Chicago. These measurements are from Location 1 and 2, using the combination
of FCC ST and BG DL. It is clear from the figure that in the warmer months,
when the ambient temperature was $\sim$$24\degree$ C there is a degradation of
throughput after $200$ sec, while no such degradation is observed in the
winter months when the ambient temperature was $\sim$$-10\degree$ C.

Fig. 5(b) shows the comparison of measurements in Chicago, Miami, and San
Francisco collected in summer. Since the data in Miami was captured using BG
DL traffic only, PHY throughput from NSG is used in this analysis. The
throughput in Miami data degrades faster (at $\sim$$60$ sec) than Chicago and
San Francisco, which can be explained by the climate difference between these
cities and the time of experiment. The Miami data was taken with ambient
temperature of $\sim$$31\degree$ C, while Chicago and San Francico data was
taken with ambient temperature of $\sim$$24\degree$ C and $\sim$$15\degree$ C,
respectively.

Figure 6: Prolonging high rate of 5G NR session via an ice bag.

Effect of external cooling and phone cover. To further confirm the correlation
between skin temperature and reduced 5G mmWave throughput, the following
experiment was conducted in Miami in summer. Measurements were taken with the
phone either held in the hand or placed on an ice-pack. Fig. 6 shows that a
mmWave throughput above $1$ Gbps is sustained when the phone is placed on top
of an ice-pack.

(a) Throughput degradation between covered and uncovered phones.

(b) Skin temperature rise between covered and uncovered phones.

Figure 7: Phone cover experiments in Miami.

Second, we investigate the impact of using a phone protective case on
extending the 5G mmWave throughput. Using a standard commercially available
case555https://www.spigen.com/products/pixel-5-case-tough-armor to cover the
phone, we compare throughput performance when the phone is with and without
cover. Each experiment type is repeated 3 times and both types were ran in
Miami. Fig. 7(a) shows the achievable PHY throughput for all six runs. Without
a phone cover, the phone can sustain up to $60$ sec of a higher $\sim$$1$ Gbps
5G mmWave throughput using $4$ channels. With the cover, the phone can only
sustain the higher rate up to $30$ sec.

The lower throughput performance of the covered phone can be explained by Fig.
7(b), which shows the corresponding skin temperature over all six runs. The
covered phone breached the $43\degree$ C threshold at $20$ sec, compared to
the uncovered phone at $40$ sec. Hence, the phone cover restricts heat
dissipation and causes a higher skin temperature. While further experiments
with more variables (e.g., phone model, case type, climate) are needed, this
experiment has demonstrated that faster heat dissipation allows for longer
utilization of the 5G mmWave network at full capacity.

###  IV-C Thermal performance investigation using an IR camera

In addition to extracting the skin temperature from the API, we also performed
IR camera measurements in May 2022. We set a FLIR One Pro LT IR camera up to
mount stably $\sim$17.5 cm above a case-less Samsung S21+ phone with the
Qualipoc measurement tool running666https://www.rohde-
schwarz.com/us/products/test-and-measurement/network-data-collection/qualipoc-
android_63493-55430.html. The phone initiates BG DL traffic similar to prior
experiments to capture PHY layer data (e.g., per channel SS-RSRP, SS-RSRQ, and
throughput) and HTTP/application layer throughput. We captured 4 $\times$ 10
minutes runs, with the phone connected to the 5G mmWave BS at Location 1 in
Fig. 1(a) with a good signal condition ($\sim$90 dBm RSRP). The ambient
temperature at the time of the experiment was 30$\degree$ C.

(a) IR thermal capture on O1 (BS is to the right of phone).

(b) IR thermal capture on O2 (BS is to the left of phone).

Figure 8: IR thermal captures in orientations O1 and O2.

(a) Throughput and temperatures vs. time in orientation O1.

(b) Throughput and temperatures vs. time in orientation O2.

Figure 9: Throughput and temperatures vs. time.

We identified three spots on the phone with a high likelihood of heating up
during the mmWave experiment: CPU and modem area (CPU), upper mmWave antenna
(UppAnt), and lower mmWave antenna (LowAnt). x-y coordinates are defined for
each spot, relative to the phone frame to ensure that the data is comparable
between different experiment runs. Note that none of these temperature spots
directly translate to the CPU and skin temperature value collected by the
Android API since the temperature API is not directly accessible.

Fig. 8(a) and Fig. 8(b) show representative thermal images777Full video
available at https://youtu.be/fm29QwdbVW8, with two orientations, O1 and O2,
w.r.t. the mmWave BS. In O1, the BS is located to the right of the phone, and
both Fig. 8(a) and Fig. 9(a) shows a higher temperature on the UppAnt spot
which indicates higher activity on this antenna. Conversely, Fig. 8(b) shows
the O2 orientation where the BS is located to the left of the phone, and Fig.
9(b) shows higher LowAnt temperature when the phone is connected to mmWave. In
both orientations, the phone started with 4 mmWave channels then throttled to
1 channel at the 95 sec and 85 sec mark for O1 and O2, respectively.
Subsequently, LTE handover occurred at the 154 sec and 174 sec mark for O1 and
O2, respectively. We observed a better overall throughput performance on O2
compared to O1 on all 4 runs. This is due to two reasons: location of each
antenna w.r.t. the CPU (O1 is closer), and which antenna is activated during
the experiment.

While we could not correlate the skin temperature to the spot temperatures, it
is highly likely that the skin temperature sensor is located in the upper half
of the phone, given that average temperature of the lower half was
$\sim$35$\degree$ C on all runs. Additionally, we performed UL traffic
experiments using a similar setup and observed that the phone was connected to
1 mmWave channel during the entire 10-minute run, even as the temperature of
the UppAnt spot increased to a maximum of 42$\degree$ C. This observation
further supports our hypothesis, i.e., fewer aggregated mmWave channels (and
consequently, lower throughput) do not cause a significant rise in skin
temperature, even with the additional power consumption due to transmitting:
transmitting over 1 mmWave channel caused less rise in skin temperature
compared to receiving over 4 mmWave channels.

##  V Conclusions and Future Work

We presented the first detailed measurements demonstrating the dramatic impact
of device skin temperature on 5G mmWave throughput. The experiments were
conducted in three different cities under various ambient conditions, with all
results indicating that 5G mmWave sustained throughput is limited due to the
rising skin temperature of the UE. We demonstrated a repeatable 3-step
throughput profile which starts with a high rate above $1$ Gbps due to
aggregating $4$ mmWave component carriers, which is downgraded to $1$ mmWave
carrier before finally falling back to the baseline 4G/LTE system as the
device skin temperature increases. Further, we have shown that the duration of
sustained 5G mmWave throughput can be significantly increased by not using a
phone covering or by using improved cooling mechanisms (e.g., an ice-pack for
proof of concept purposes). Finally, IR imaging confirms that the mmWave
antennas are a major contribution to the skin temperature rise. These results
indicate that device skin temperature should be considered in scheduling and
resource allocation algorithms so that the user does not experience a
fluctuating throughput and the device does not heat up beyond the skin
temperature limit of the phone. Our future work in this area will focus on
profiling the effects of CPU, modem, and mmWave antenna utilization on skin
temperature, thermal throttling, and throughput.888All measurements collected
for this paper will be made available on request.

## References

  * [1] A. Narayanan, E. Ramadan, R. Mehta, X. Hu, Q. Liu, R. A. Fezeu, U. K. Dayalan, S. Verma, P. Ji, T. Li _et al._ , “Lumos5G: Mapping and predicting commercial mmWave 5G throughput,” in _Proceedings of the ACM Internet Measurement Conference_ , 2020, pp. 176–193.
  * [2] A. Narayanan, E. Ramadan, J. Carpenter, Q. Liu, Y. Liu, F. Qian, and Z.-L. Zhang, “A first look at commercial 5G performance on smartphones,” in _Proceedings of The Web Conference 2020_ , 2020, pp. 894–905.
  * [3] M. I. Rochman, V. Sathya, N. Nunez, D. Fernandez, M. Ghosh, A. S. Ibrahim, and W. Payne, “A comparison study of cellular deployments in Chicago and Miami using apps on smartphones,” in _Proceedings of the 15th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & CHaracterization (WiNTECH’21)_, 2022, p. 61–68.
  * [4] M. Giordani, M. Polese, A. Roy, D. Castor, and M. Zorzi, “A tutorial on beam management for 3GPP NR at mmWave frequencies,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 1, pp. 173–196, 2018.
  * [5] Y.-N. R. Li, B. Gao, X. Zhang, and K. Huang, “Beam management in millimeter-wave communications for 5G and beyond,” _IEEE Access_ , vol. 8, pp. 13 282–13 293, 2020.
  * [6] A. Narayanan, M. I. Rochman, A. Hassan, B. S. Firmansyah, V. Sathya, M. Ghosh, F. Qian, and Z.-L. Zhang, “A comparative measurement study of commercial 5G mmWave deployments,” in _2022 IEEE International Conference on Computer Communications (INFOCOM)_. IEEE, 2022, to appear.
  * [7] 3rd Generation Partnership Project, “TS 38.331 5G Radio Resource Control (RRC) version 16.1.0 release 16,” 2020.
  * [8] (2021) Derf’s test media collection. [Online]. Available: https://media.xiph.org/video/derf
  * [9] Android, “Android developers: Documentation for HardwarePropertiesManager,” https://developer.android.com/reference/android/os/HardwarePropertiesManager, accessed October 2021.