11institutetext: Department of Computer Science and Software Engineering,
Concordia University, Montreal, Canada

# Weakly supervised segmentation of intracranial aneurysms using a 3D focal
modulation UNet

Amirhossein Rasoulian  Soorena Salari  Yiming Xiao

###### Abstract

Accurate identification and quantification of unruptured intracranial
aneurysms (UIAs) are essential for the risk assessment and treatment decisions
of this cerebrovascular disorder. Current assessment based on 2D manual
measures of aneurysms on 3D magnetic resonance angiography (MRA) is sub-
optimal and time-consuming. Automatic 3D measures can significantly benefit
the clinical workflow and treatment outcomes. However, one major issue in
medical image segmentation is the need for large well-annotated data, which
can be expensive to obtain. Techniques that mitigate the requirement, such as
weakly supervised learning with coarse labels are highly desirable. In this
paper, we leverage coarse labels of UIAs from time-of-flight MRAs to obtain
refined UIAs segmentation using a novel 3D focal modulation UNet, called
FocalSegNet and conditional random field (CRF) postprocessing, with a Dice
score of 0.68 and 95% Hausdorff distance of 0.95 mm. We evaluated the
performance of the proposed algorithms against the state-of-the-art 3D UNet
and Swin-UNETR, and demonstrated the superiority of the proposed FocalSegNet
and the benefit of focal modulation for the task.

###### Keywords:

Aneurysm Deep learning Weak segmentation Transformer Focal modulation
Conditional random field

##  1 Introduction

An intracranial aneurysm is an abnormal focal bulging or ballooning of the
vasculature in the brain due to weakened blood vessel walls. Although often
asymptomatic, large aneurysms can cause headaches, visual disturbance,
neurological deficits, and other issues [7]. Furthermore, a ruptured aneurysm
can lead to life-threatening subarachnoid hemorrhage, a severe stroke subtype.
Typical treatments for unruptured intracranial aneurysms (UIAs) include
endovascular coiling and surgical clipping. When screening for UIAs and
determining the treatment plans, risk assessment of UIA growth and rupture is
crucial. Thanks to modern medical imaging techniques, such as computed
tomography (CT) and magnetic resonance imaging (MRI), the ability to
accurately and reliably identify and measure aneurysms can ensure therapeutic
outcomes. Compared with CT angiography (CTA), time-of-flight magnetic
resonance angiography (TOF-MRA) does not expose the patients to radiation or
adverse reactions toward contrast agents. It thus is better suited for routine
follow-up imaging.

Manual identification and measurements of UIAs can be difficult and time-
consuming, especially for those that are of small sizes ($<$5mm), and it is
estimated that 10$\%$ of all UIAs are missed during screening [22]. Therefore,
automatic algorithms that allow detection and especially 3D segmentation of
the aneurysms from medical scans can greatly facilitate the clinical workflow
and enable more fine-grained risk analysis based on UIA shapes. To date, many
techniques [19, 6] have been proposed for UIA detection, but only a few
reported performance for 3D UIA segmentation from CTA/MRA. Notably, the ADAM
Challenge [19] hosted in conjunction with MICCAI 2020 attracted nine DL-based
UIA segmentation algorithms. Among these, various approaches of convolutional
neural networks (CNNs), especially variants of the UNet have been proposed,
with the winning algorithm achieving an average Dice score of 0.64 and a mean
95$\%$ Hausdorff distance (95-HD) of 2.62mm for correctly detected UIAs.
However, most existing techniques were developed based on private clinical
data with refined manual annotations, and many focused on CTA. Further
investigation is still needed for this challenging task, especially for MRA
with weaker contrast. The lack of large well-annotated databases, which are
costly to acquire and demand high domain expertise, poses significant
challenges in developing deep learning methods in medical imaging
applications, particularly for anatomical segmentation. To mitigate this,
weakly supervised segmentation [14, 23, 15, 16] leverages coarse annotations
that are easier to obtain (e.g., rough segmentation and categorical image
labels) to derive refined segmentation. In this paper, we employ weak labels
of UIAs from TOF-MRA to detect and segment the pathology by using a novel 3D
focal modulation UNet, called FocalSegNet, and filtering with a fully
connected conditional random field (CRF) model. In summary, our work has three
major novel contributions. First, inspired by the recent focal modulation
technique [24] and the UNet architecture [25], we proposed a novel 3D UNet
with focal modulation; Second, we thoroughly compared the performance of self-
attention [21] and focal modulation [24] in weak segmentation of UIAs; Lastly,
we revealed the key factors that contribute to UIA segmentation for the
proposed method through various ablation studies.

##  2 Methodology

###  2.1 Network architecture of FocalSegNet

While 3D UNet was a common choice for UIA detection and segmentation [19], the
newer Vision Transformer (ViT) and its variants (e.g., Swin Transformer) that
leverage self-attention mechanisms [21, 12] have become increasingly popular
in medical imaging applications, providing superior performance. More
recently, to better encode contextual information with a lighter model, Yang
et al. [24] proposed focal modulation for DL in vision tasks, and claimed it
performs better than the Swin Transformer. In focal modulation networks, self-
attention is replaced by a stack of depth-wise convolutional layers that
focally encode visual contexts and selectively gather them into a modulator
using a gated aggregation. Next, the modulator is injected into the query and
passed to the next block. The main difference between these two methods is
that in the self-attention, a query-key interaction is performed before the
result is aggregated with the value while in focal modulation, contexts around
each query are focally aggregated before the result is adaptively modulated
with the query.

Despite excellent performance in classification and segmentation of natural
images, focal modulation’s performance in 3D medical images is yet to be
verified. Recently, Naderi et al. [13] proposed a 2D UNet-like architecture
with focal modulation blocks as the encoder and decoder for abdominal CT
segmentation. It offered a higher mean Dice score (not confirmed by
statistical tests) than the Swin-Unet [2], another UNet-like model with Swin
Transformers as the encoder and decoder. Different from [13], we followed the
approach of the recent 3D Swin-UNETR [8], a popular Swin-CNN hybrid model to
build a new architecture by replacing the Swin Transformer with 3D focal
modulation for the encoder. At the encoder branch, each layer’s output,
consisting of image tokens and their embeddings, is reshaped to a volumetric
patch and passed through a residual block to work as skip connections.
Starting from the bottleneck, each feature map is expanded using a transposed
convolutional block and then concatenated with the corresponding skip
connection before passing through another residual block to the next layer.
The detailed architecture is depicted in Fig. 1.

Figure 1: Network architecture of the proposed FocalSegNet

###  2.2 Post-processing with fully connected CRF

As the deep learning (DL) model is trained on rough labels that overestimate
the true segmentation, we used a fully connected CRF [10] to further refine
our model’s initial prediction ($P(x_{i})$). If we define $X_{i}$ as a random
variable representing the assigned label to pixel $i$ (foreground/background),
and $I_{i}$ as a global observation characterizing the pixel information in
the input image, the CRF is modeled as the pair of $(I,X)$ which follows a
Gibbs distribution of the form $P(X=x|I)=\frac{1}{Z(I)}exp(-E(x|I))$ [11].
Here, $Z(I)$ is a normalization factor, and $E(x|I)$ is the energy function of
assigning labels $x$ to the image. Through minimizing this energy function,
the new labels will have a greater likelihood of being assigned to the image.
Inspired by [4], we define the energy function as:

$E(x|I)=\sum_{i}{\psi_{u}(x_{i})}+\sum_{i<j}{\psi_{p}(x_{i},x_{j})}$ (1)
$\psi_{u}(x_{i})=-logP(x_{i})$

$\psi_{p}(x_{i},x_{j})=\mu(x_{i},x_{j})\left[\omega_{1}exp(-\frac{{||p_{i}-p_{j}||}^{2}}{2\sigma_{\alpha}^{2}}-\frac{{||I_{i}-I_{j}||}^{2}}{2\sigma_{\beta}^{2}})\\\
+\omega_{2}exp(-\frac{{||p_{i}-p_{j}||}^{2}}{2\sigma_{\gamma}^{2}})\right]$

$\mu(x_{i},x_{j})=1\;\text{if}\;x_{i}\neq
x_{j}\;\text{and}\;0\;\text{otherwise}$

Here, $\psi_{u}(x_{i})$ is a unary term measuring the cost of assigning label
$x_{i}$ to pixel $i$, and $\psi_{p}(x_{i},x_{j})$ is a pairwise term that
shows the cost of assigning labels $x_{i}$ and $x_{j}$ to pixels $i$ and $j$
simultaneously, which uses a bilateral kernel and a Gaussian kernel. The first
kernel forces the pixels with similar positions ($p$) and intensities ($I$) to
have similar labels, and the second enforces the smoothness of prediction by
only considering the spatial proximity. $\omega_{1}$, $\omega_{2}$,
$\sigma_{\alpha}$, $\sigma_{\beta}$, and $\sigma_{\gamma}$ are hyper-
parameters that control the scale of these kernels and are chosen empirically.
We used connected-component analysis [17] to identify different clusters when
the algorithms predict multiple UIAs, and each connected component was treated
as an individual detected aneurysm. To further remove noise, any cluster that
had lower than 10 voxels was discarded.

##  3 Experiments and Evaluation

###  3.1 Dataset and preprocessing

Unfortunately, we couldn’t obtain the ADAM challenge dataset [19] for our
study. Instead, we relied on a publicly accessible TOF-MRA dataset of UIAs
described in [5] to develop and validate our algorithms. It contains 284
subjects (170 females,127 healthy controls/157 patients with 198 aneurysms).
For 246 subjects, coarse spheres that enclose a whole aneurysm were manually
labeled, while 38 subjects have voxel-wise segmentations. All scans were
preprocessed with skull-stripping and bias field correction [20], and finally
resampled to a median resolution of $0.39\times 0.39\times 0.55$ $mm^{3}$. To
allow efficient computation, we performed segmentation for image patches
instead of the entire brain volume. Furthermore, we followed the “anatomically
informed” approach in [5] to extract image patches of $64\times 64\times 64$
voxels guided by a probabilistic cerebrovascular atlas with anatomical
landmarks, resulting in $\sim$50 patches per subject. The obtained patches
will balance the positive samples with aneurysms and negative ones that
contain no aneurysms according to their occurrence, as well as variations of
the anatomy within. For more details and the script for image patch
extraction, we refer the readers to the original data paper [5]. Finally, for
DL model training and inferencing, all image patches were normalized with
z-transformation. Note that the cases with coarse labels were divided subject-
wise for training (95%) and validation (5%), and those with refined
segmentation masks were saved for model testing only.

###  3.2 Implementation details

In our proposed FocalSegNet, we utilized a 3D version of the original Focal
modulation network [24] as the encoder. For the baseline models, we
implemented a 3D UNet that had been used in the original data paper [5], and a
Swin-UNETR with default parameters [8] to compare two similar mechanisms,
self-attention and focal modulation. The only difference between Swin-UNETR
and FocalSegNet is their encoders, which makes our comparison valid. All
networks had four layers/hierarchies and were trained with a batch size of 12
and an AdamW optimizer (weight decay for L2 regularization=1e-6 and initial
learning rate=1e-3). We used a step learning scheduler that reduces the
learning rate by 2% every 100 iterations. Furthermore, we used sampling, data
augmentation, and a combination of different loss functions to tackle the
class imbalance problem. Early stopping is also used to avoid model
overfitting.

####  3.2.1 Sampling and data augmentation:

The ratio of patches without and with UIAs is 9:1, which can cause the model
to be biased toward the negative class. To tackle this, each patch in the
training set is assigned a probability distribution inversely proportional to
its label’s frequency of occurence. Then, each batch is randomly sampled from
this distribution with replacement, ensuring the number of positive and
negative patches in a batch is almost the same. Although some samples may not
be seen in one epoch, there is a high likelihood that the network will see all
of them after several epochs. We also use online image augmentation
techniques, including random flipping and rotation of up to 15 degrees along
x-, y-, and z-directions, and random Gaussian noise addition to mitigate
overfitting.

####  3.2.2 Loss function:

Since the small sizes of UIAs cause a large class imbalance, we composed our
total loss function with likelihood-based cross-entropy loss (CE), regional-
based generalized Dice loss (GD) [18], and distance-based boundary loss
(Boundary) [9] as: $Loss=\alpha CE+\beta GD+\gamma Boundary$ , where
$\alpha=\beta=2\gamma=0.40$ were determined empirically. These individual loss
functions help improve the detection and segmentation of UIAs for the proposed
and baseline models.

####  3.2.3 Ablation studies:

Besides comparing the performance of the full setup, where fully connected CRF
was used to filter initial results of the DL models, including FocalSegNet,
Swin-UNETR, and UNet, we also conducted ablation studies to test the impact of
CRF post-processing, as well as each component of the full loss function for
the proposed FocalSegNet. We evaluate the metrics for UIA detection and
segmentation for all the experiments to gain the required insights.

Table 1: UIA detection and segmentation performance (mean±std)

| UNet | Unet + CRF | Swin-UNETR | Swin-UNETR + CRF | FocalSegNet | FocalSegNet + CRF
---|---|---|---|---|---|---
FP rate | 0.322 ± 0.537 | 0.277 ± 0.500 | 0.476 ± 0.596 | 0.403 ± 0.560 | 0.231 ± 0.488 | 0.212 ± 0.464
Sensitivity | 0.793 ± 0.405 | 0.778 ± 0.415 | 0.866 ± 0.340 | 0.841 ± 0.366 | 0.827 ± 0.379 | 0.801 ± 0.399
Dice | 0.587 ± 0.144 | 0.668 ± 0.129 | 0.625 ± 0.137 | 0.668 ± 0.144 | 0.638 ± 0.130 | 0.677 ± 0.141
IoU | 0.430 ± 0.140 | 0.515 ± 0.137 | 0.468 ± 0.137 | 0.518 ± 0.149 | 0.481 ± 0.132 | 0.527 ± 0.144
95-HD | 2.870 ± 1.262 | 2.315 ± 1.125 | 2.495 ± 0.957 | 2.214 ± 1.032 | 2.504 ± 1.140 | 2.148 ± 1.082

###  3.3 Evaluation metrics

For the outcomes of the proposed algorithm and its counterparts, as well as
the associated ablation studies, we included five different metrics to
evaluate their performance in detecting and segmenting the aneurysm based on
the test dataset on a per-image-patch basis. Specifically, for UIA detection
quality, we measured the sensitivity and false positive (FP) rate. Note that
within the same image patch, it is feasible to contain multiple UIAs. Thus, we
separated distinct connected components in the prediction map and ground
truth, and evaluated them per aneurysm. Here, a correct aneurysm detection is
defined as when the centroid of the predicted segmentation component lies
within the boundary of an aneurysm and is considered a false positive
otherwise. Next, for all the correctly identified aneurysms, we further
evaluated the segmentation accuracy using the Dice coefficient, Intersection
over Union (IoU), and 95-$\%$ Hausdorff distance (95-HD) in voxels to verify
the quality of region and boundary overlaps. Because an aneurysm may be
detected by one model and missed by the other, we performed an unpaired two-
sided t-test to compare the performance between different experimental setups.
A $\text{p-value}<0.05$ was used to indicate a significant difference.

##  4 Results

###  4.1 UIA detection and segmentation

The UIA detection and segmentation performance for all three DL models with
CRF filtering is listed in Table 1, with segmentation results for two subjects
demonstrated in Fig. 2. In terms of aneurysm detection, for FP rate, our
FocalSegNet well outperformed the other two networks ($\text{p}<0.05$), while
for sensitivity, all perform similarly ($\text{p}>0.05$). This implies that
FocalSegNet is better at distinguishing true/false UIAs, which could be hard
even for human raters at times. Although FocalSegNet has the best mean
segmentation metrics, the difference with Swin-UNETR was not significant
($\text{p}>0.05$). However, both outperform the UNet ($\text{p}<0.05$).

Table 2: Influence of loss functions and CRF post-processing for the proposed
FocalSegNet in UIA detection and segmentation results (mean±std)

| FP rate | Sensitivity | Dice | IoU |  95-HD (voxels)
---|---|---|---|---|---
Cross-Entropy loss | 0.002 ± 0.048 | 0.006 ± 0.075 | 0.018 ± 0.007 | 0.009 ± 0.004 | 4.215 ± 0.121
\+ Generalized Dice loss | 0.527 ± 0.729 | 0.864 ± 0.343 | 0.604 ± 0.143 | 0.447 ± 0.144 | 2.854 ± 1.291
\+ Boundary loss | 0.231 ± 0.488 | 0.827 ± 0.379 | 0.638 ± 0.130 | 0.481 ± 0.132 | 2.504 ± 1.140
\+ CRF | 0.212 ± 0.464 | 0.801 ± 0.399 | 0.678 ± 0.141 | 0.527 ± 0.144 | 2.148 ± 1.082

###  4.2 Ablation studies

From Table 1, we observe that CRF filtering is crucial in boosting the accuacy
of the initial segmentation of all models ($\text{p}<0.05$) while FocalSegNet
offers the best performance without CRF ($\text{p}<0.05$). Table 2
demonstrates the impact of design choices on FocalSegNet. The network trained
solely with the Cross-Entropy loss function performs poorly due to the
aneurysm size being negligible compared to background voxels, leading to a
bias toward negative predictions. Incorporating Generalized Dice Loss enhances
performance, but the false positive (FP) rate remains high. To address this,
boundary loss is added, enforcing closer proximity between predictions and
ground truths. Ultimately, post-processing with a CRF further enhances
segmentation performance.

Figure 2: Comparison of segmentation results for two different patients (one
patient per row).Red label=ground truths and green label=automatic
segmentation.

##  5 Discussion

Cerebral aneurysm segmentation is still challenging due to its high similarity
in adjacent vessel structures. As previous reports [5, 19] primarily focus on
binary detection and/or localization of cerebral aneurysms, the relevant
reports on the assessment of segmentation algorithms are limited. Primarily,
many earlier works rely on contrast-enhanced CT angiographies, which have
sharper contrast for vasculatures than TOF-MRA, and in-house collected data
with refined annotations. Compared with the best segmentation results from the
ADAM Challenge (for detected true UIAs) with a 0.64 mean Dice score and 95-HD
of 2.62 mm [19], our proposed FocalSegNet, together with CRF post-processing
has achieved a Dice score of 0.677$\pm$0.141 and a 95-HD of 2.15 voxel
($\sim$0.95mm), only by leveraging rough labeling of the aneurysms. Regarding
UIA detection/localization, our proposed method obtained a sensitivity of
0.801 and an FR rate of 0.212, which are also excellent results. Note that
since our UIA detection performance was assessed on a per image-patch basis,
the resulting metrics are not directly comparable to those that were done
subject-wise. In almost all previous reports on the relevant topic, the UNet
and its variants have been widely adopted. Here, we employed the 3D UNet as a
baseline model to properly assess the segmentation quality against the
FocalSegNet and Swin-UNETR, whose general architectures were also inspired by
the UNet, but with the encoding branch modified with Transformer or focal
modulation blocks, respectively. This comparison also allows us to probe the
characteristics and performance of these two new DL techniques that model
long-range contextual information through different mechanisms. For our
application in a 3D weakly supervised segmentation, we confirm the claim of
[24] for the benefit of focal modulation, with Swin-UNETR ranking the second.
This is likely because aneurysms, especially the small ones, are often
attached to the main arteries near the branching points, and thus contextual
knowledge of anatomy will be beneficial. This is in addition to the anatomical
prior-based approach that we adopted from Di Noto et al. [5] in training data
sampling. In terms of GPU memory usage during training, FocalSegNet, Swin-
UNETR, and UNet took 11.2GB, 17.4GB, and 9.7GB respectively, based on the
batch size of 12. For all networks, the inference time per patch is around
70ms based on a desktop computer with Intel Corei9 CPU, Nvidia GeForce RTX
3090 GPU, and 24GB RAM. This offers a glance of the efficiency of the proposed
network. Compared to [24, 13], we further extended the 2D focal modulation to
3D in a new UNnet-like architecture for segmentation tasks for the first time.

For all the DL models, we used a combination of different loss functions,
including cross-entropy loss, generalized Dice loss, and boundary loss. While
generalized Dice loss and cross-entropy loss have been popular for many
medical image segmentation tasks, boundary loss [9] is helpful in segmenting
structures with high-class imbalance, such as aneurysms. With the intention to
reduce the reliance on carefully annotated manual segmentations, many have
attempted different strategies in weakly or semi-supervised learning [3],
among which CRFs have often been used either in preprocessing pseudo ground
truth labels or post-processing results from DL models trained on weak labels.
In our case, we designed the system using a fully connected CRF model as a
post-processing step, which further improved the accuracy of the initial
automatic segmentations based on spatial and intensity consistency of the
aneurysms. Some other works have also reported the application of a subsequent
refinement neural network [1], but this approach often requires a small number
of refined manual segmentations and will lead to a semi-supervised framework
instead.

##  6 Conclusion

In conclusion, we have proposed a novel 3D focal modulation UNet called
FocalSegNet in combination with CRF for weakly supervised segmentation and
detection of brain aneurysms from TOF-MRA. By leveraging coarse segmentation
ground truths, the proposed technique was able to achieve excellent
performance. By comparing its performance with the popular UNet and the most
recent Swin-UNETR, we demonstrated its superior performance and will extend
its application in other domain tasks in the near future.

## References

  * [1] Adiga V, S., Dolz, J., Lombaert, H.: Manifold-driven attention maps for weakly supervised segmentation. arXiv preprint arXiv:2004.03046 (2020)
  * [2] Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin-unet: Unet-like pure transformer for medical image segmentation. In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III. pp. 205–218. Springer (2023)
  * [3] Chan, L., Hosseini, M.S., Plataniotis, K.N.: A comprehensive analysis of weakly-supervised semantic segmentation in different image domains. International Journal of Computer Vision 129, 361–384 (2021)
  * [4] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4), 834–848 (2017)
  * [5] Di Noto, T., Marie, G., Tourbier, S., Alemán-Gómez, Y., Esteban, O., Saliou, G., Cuadra, M.B., Hagmann, P., Richiardi, J.: Towards automated brain aneurysm detection in tof-mra: Open data, weak labels, and anatomical knowledge. Neuroinformatics 21(1), 21–34 (2023)
  * [6] Din, M., Agarwal, S., Grzeda, M., Wood, D.A., Modat, M., Booth, T.C.: Detection of cerebral aneurysms using artificial intelligence: a systematic review and meta-analysis. J Neurointerv Surg 15(3), 262–271 (2023)
  * [7] Frösen, J., Tulamo, R., Paetau, A., Laaksamo, E., Korja, M., Laakso, A., Niemelä, M., Hernesniemi, J.: Saccular intracranial aneurysm: pathology and mechanisms. Acta Neuropathol 123(6), 773–86 (2012)
  * [8] Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 7th International Workshop, BrainLes 2021, Held in Conjunction with MICCAI 2021, Virtual Event, September 27, 2021, Revised Selected Papers, Part I. pp. 272–284. Springer (2022)
  * [9] Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., Ayed, I.B.: Boundary loss for highly unbalanced segmentation. Medical image analysis 67, 101851 (2018)
  * [10] Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems 24 (2011)
  * [11] Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data (2001)
  * [12] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 9992–10002 (2021)
  * [13] Naderi, M., Givkashi, M., Piri, F., Karimi, N., Samavi, S.: Focal-unet: Unet-like focal modulation for medical image segmentation. arXiv preprint arXiv:2212.09263 (2022)
  * [14] Rajchl, M., Lee, M.C.H., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M.A., Hajnal, J.V., Kainz, B., Rueckert, D.: Deepcut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE transactions on medical imaging 36(2), 674—683 (February 2017)
  * [15] Rasoulian, A., Salari, S., Xiao, Y.: Weakly supervised intracranial hemorrhage segmentation using hierarchical combination of attention maps from a swin transformer. In: Machine Learning in Clinical Neuroimaging: 5th International Workshop, MLCN 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings. pp. 63–72. Springer (2022)
  * [16] Rasoulian, A., Salari, S., Xiao, Y.: Weakly supervised intracranial hemorrhage segmentation using head-wise gradient-infused self-attention maps from a swin transformer in categorical learning. arXiv preprint arXiv:2304.04902 (2023)
  * [17] Silversmith, W.: cc3d: Connected components on multilabel 3D and 2D images. (sep 2021), https://zenodo.org/record/5535250
  * [18] Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3. pp. 240–248. Springer (2017)
  * [19] Timmins, K.M., van der Schaaf, I.C., Bennink, E., Ruigrok, Y.M., An, X., Baumgartner, M., Bourdon, P., De Feo, R., Noto, T.D., Dubost, F., Fava-Sanches, A., Feng, X., Giroud, C., Group, I., Hu, M., Jaeger, P.F., Kaiponen, J., Klimont, M., Li, Y., Li, H., Lin, Y., Loehr, T., Ma, J., Maier-Hein, K.H., Marie, G., Menze, B., Richiardi, J., Rjiba, S., Shah, D., Shit, S., Tohka, J., Urruty, T., Walińska, U., Yang, X., Yang, Y., Yin, Y., Velthuis, B.K., Kuijf, H.J.: Comparing methods of detecting and segmenting unruptured intracranial aneurysms on tof-mras: The adam challenge. Neuroimage 238, 118216 (2021)
  * [20] Tustison, N.J., Avants, B.B., Cook, P.A., Zheng, Y., Egan, A., Yushkevich, P.A., Gee, J.C.: N4itk: improved n3 bias correction. IEEE Trans Med Imaging 29(6), 1310–20 (2010)
  * [21] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [22] White, P.M., Wardlaw, J.M., Easton, V.: Can noninvasive imaging accurately depict intracranial aneurysms? a systematic review. Radiology 217(2), 361–70 (2000)
  * [23] Yang, G., Wang, C., Yang, J., Chen, Y., Tang, L., Shao, P., Dillenseger, J.L., Shu, H., Luo, L.: Weakly-supervised convolutional neural networks of renal tumor segmentation in abdominal cta images. BMC Medical Imaging 20(1), 37 (2020)
  * [24] Yang, J., Li, C., Gao, J.: Focal modulation networks. ArXiv abs/2203.11926 (2022)
  * [25] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: Learning dense volumetric segmentation from sparse annotation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2016)


# FedAR+: A Federated Learning Approach to Appliance Recognition with
Mislabeled Data in Residential Buildings

Ashish Gupta1, Hari Prabhat Gupta2, and Sajal K. Das1<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2Dept. of Computer Science and
Engineering, Indian Institute of Technology (BHU), Varanasi, India 1Dept. of
Computer Science, Missouri University of Science and Technology, Rolla, MO
65409, USA

###### Abstract.

With the enhancement of people’s living standards and rapid growth of
communication technologies, residential environments are becoming smart and
well-connected, increasing overall energy consumption substantially. As
household appliances are the primary energy consumers, their recognition
becomes crucial to avoid unattended usage, thereby conserving energy and
making smart environments more sustainable. An appliance recognition model is
traditionally trained at a central server (service provider) by collecting
electricity consumption data, recorded via smart plugs, from the clients
(consumers), causing a privacy breach. Besides that, the data are susceptible
to noisy labels that may appear when an appliance gets connected to a non-
designated smart plug. While addressing these issues jointly, we propose a
novel federated learning approach to appliance recognition, called FedAR+,
enabling decentralized model training across clients in a privacy preserving
way even with mislabeled training data. FedAR+ introduces an adaptive noise
handling method, essentially a joint loss function incorporating weights and
label distribution, to empower the appliance recognition model against noisy
labels. By deploying smart plugs in an apartment complex, we collect a labeled
dataset that, along with two existing datasets, are utilized to evaluate the
performance of FedAR+. Experimental results show that our approach can
effectively handle up to $30\%$ concentration of noisy labels while
outperforming the prior solutions by a large margin on accuracy.

Appliance recognition, federated learning, noisy labels, smart plug

††ccs: Computing methodologies Supervised learning by classification††ccs:
Hardware Energy metering

##  1\. Introduction

Energy consumption in residential buildings is increasing rapidly with the
growth of electrical household appliances. According to the United States
Energy Information Administration (US EIA) (United States Energy Information
Administration (US EIA), [n.d.]), $22\%$ of the total energy consumption in
$2020$ is accounted by residential buildings, requiring dedicated efforts to
reduce the usage of electricity. A practical solution is encouraging consumers
to use electric appliances efficiently, which involves recognizing the
appliances uniquely based on their consumption patterns recorded via
appliance-wise smart plugs (Veloso et al., 2019; Soe and Belleudy, 2019; Reddy
et al., 2017). By having information about currently running appliances, the
consumers can minimize the electricity usage by restricting high power
appliances (e.g., electric heater, air conditioner) during peak hours
(Farrokhifar et al., 2018). Moreover, the utility company (service provider)
may also incentivize the consumers by offering direct monetary benefit through
a dynamic pricing policy (Farrokhifar et al., 2018; Jang et al., 2021) and
indirect benefit through an appliance-wise breakage of consumption bill.
Literature indicates that appliance recognition has been a building block in
wide range of important applications such as load forecasting (Chen et al.,
2019; Wang and Wang, 2021), occupancy detection (Kleiminger et al., 2015), and
energy management in smart buildings (Ma et al., 2018; Schwermer et al.,
2022). However, the current appliance recognition approaches have disregarded
the following two practical issues.

(i) Privacy preservation of consumers’ data – As recognition model is
essentially a machine learning model, it requires a large amount of labeled
training data which, in general, collected from many consumers at a central
server (service provider). Sharing of data brings in a privacy concern to the
consumers as the data may be misused by adversaries via theft or burglary, and
by detecting home occupancy (Allik et al., 2020; Kleiminger et al., 2015).
Hence, the consumers may be reluctant to upload the data and as a consequence,
the existing approaches (Yan et al., 2019; Vadakattu and Suthaharan, 2018; Liu
et al., 2019; Soe and Belleudy, 2019; Veloso et al., 2019; Ganu et al., 2014)
would fail to train the recognition model, indicating a need of a model that
can be trained collaboratively at the consumer side without sharing any data.

(ii) Mislabeled training data – Some data samples may appear with wrong
(noisy) labels when an appliance is mistakenly connected to a non-designated
smart plug.111Assuming that each smart plug, during deployment, is designated
to a specific appliance to collect the labeled data automatically. Moreover, a
compromised consumer may also flip the labels in its local dataset with an
intent to poison the model. Such consumers might receive monetary benefits
from rival service provider. As prior studies (Vadakattu and Suthaharan, 2018;
Liu et al., 2019; Soe and Belleudy, 2019; Veloso et al., 2019) do not
incorporate any noisy-label handling mechanism, they can not withstand
mislabeled training data. Slightly on a different track, learning with noisy
labeled data has been a topic of great interest in computer vision; however,
the proposed solutions (Han et al., 2019; Tanaka et al., 2018; Li et al.,
2017; Wang et al., 2019; Wang et al., 2020; Gao et al., 2017) mainly relied
upon visual features, thus their applicability to time series data (generated
from smart plugs) is discouraged.

Although there exist some works (Schwermer et al., 2022; Qureshi et al., 2022)
on privacy-preserving appliance recognition using Federated Learning (FL),
they do not consider the presence of noisy labels in training data. In this
paper, we address this important challenge by building an appliance
recognition model in a collaborative manner using mislabeled training data
while preserving consumers’ privacy. To the best of our knowledge, this is the
first work to tackle the practical issues of privacy preservation and
mislabeled training data jointly for appliance recognition in residential
buildings.

Contributions: Major contributions of this paper are given below:

  * •

With a goal to train an appliance recognition model across distributed
consumers using their local private data, we propose a novel federated
learning approach, called FedAR+, in presence of a coordinating server
(service provider). The server initializes the training by broadcasting the
model (i.e., weights) to all the clients (consumers); each client re-trains
the model using its local data and dispatches the updated model back to the
server for aggregation. By repeating the above steps for some iterations,
FedAR+ produces a generalized model without exposing the consumers’ data.

  * •

FedAR+ incorporates an innovative aggregation function to deal with the
biasing problem caused due to non independent and identically distributed
(non-iid) data across clients.

  * •

We propose an adaptive noise handling method that strategically exploits a
joint loss function, incorporating the weight parameters and label
distributions, to enable the learning with mislabeled training data.

  * •

Finally, we collect real data by deploying smart plugs in three houses in an
apartment complex, to experimentally validate the performance of FedAR+.
Moreover, to demonstrate its efficacy, we also employ two widely used datasets
from the same domain. The overall results show that FedAR+ outperforms prior
solutions by a large margin while achieving an accuracy of more than $86\%$
even when the concentration of noisy labels in training data is as high as
$30\%$.

The paper is organized as follows. Section 2 reviews the related work while
Section 3 proposes our federated learning approach, FedAR+. Section 4
discusses the dataset creation steps and elaborates the causes for the
presence of noisy labels. Section 5 builds the underlying appliance
recognition model with a noise handling method. Section 6 evaluates the
performance of FedAR+ and compares with prior solutions. Finally, Section 7
concludes the paper.

##  2\. Related Work

This section discusses the notable and relevant existing works to position our
proposed approach.

###  2.1. Appliance recognition

Many works exist on appliance recognition as it has been a building block to
energy monitoring applications. For example, in (Ma et al., 2018), a
lightweight appliance recognition model is developed for energy management in
smart buildings. The authors in (Ganu et al., 2014) attempted to identify a
malfunctioning appliance and its operating states by leveraging electricity
consumption patterns. While a line of works (Yan et al., 2019; Veloso et al.,
2019; Liu et al., 2019) involved in distinguishing the appliances from one to
another, the work in (Soe and Belleudy, 2019) aimed to identify load profile
as intermittent, continuous, or phantom, for energy management in smart home
settings. Slightly different from above works, Codispoti et al. (Codispoti et
al., 2022) presented a $K$-active neighbors based appliance recognition
approach to learn from unlabeled data collected via Arduino operated smart
plugs.

Although the aforementioned prior approaches achieve good performance using
machine learning and deep learning algorithms, their performance heavily
relies on the assumption that the training data are correctly labeled and do
not contain any noisy labels. However, in practice, satisfying this assumption
requires additional care from the consumers during data collection (via smart
plugs (Ganu et al., 2014; Yan et al., 2019; Veloso et al., 2019; Codispoti et
al., 2022)), restricting their flexibility and thereby the consumers may be
reluctant to adopt such solutions. Besides that the recognition model should
not fully rely upon the consumers’ actions rather it should be robust enough
to leverage mislabeled training data.

###  2.2. Learning with noisy labels

Literature indicates that learning with mislabeled (noisy labeled) training
data has been a widely studied problem in computer vision and image processing
because the manual labeling is time consuming and costly (Song et al., 2022;
Liang et al., 2022). The work in (Han et al., 2019) presented an iterative
learning approach to re-label the noisy-labeled training samples while in
another work (Tanaka et al., 2018) the authors estimated correct labels
against noisy ones during training by jointly optimizing the model parameters
and intermediary corrected labels. In (Wang et al., 2019), a symmetric
learning approach is proposed to simultaneously address the presence of noisy
labels and overfitting problem of Deep Neural Networks (DNNs). A distillation
process leveraging knowledge graph is introduced in (Li et al., 2017) to learn
with noisy labels. Recently, a meta-learning approach is developed in (Wang et
al., 2020) to directly learn correct labels from the training data. However,
as the prior approaches mostly work around visual features, they can not offer
an accurate solution to mislabeled time series data.

###  2.3. Federated learning

In last few years, a new learning paradigm, Federated Learning (FL) (McMahan
et al., 2017) has received an unprecedented attention because it facilitates
collaborative model training without compromising clients’ privacy. Prior
works illustrate the effectiveness of FL in real-world applications such as
next word prediction (Hard et al., 2018), keyword spotting (Leroy et al.,
2019), and visual object detection (Liu et al., 2020). However, FL is yet to
be explored for the appliance recognition models that are otherwise trained at
the central server by collecting data from multiple clients (consumers) and
revealing the client’s privacy. FL offers an effective solution to this
problem by keeping the data locally with the clients while allowing
participate in collaborative training of the model. Recently, a few studies
(Qureshi et al., 2022; Zhang et al., 2022) have also attempted to apply FL in
smart energy management to enable load forecasting and load disaggregation at
consumer side. In another work (Schwermer et al., 2022), the authors presented
an FL approach to identify office plug load, however they do not consider the
presence of noisy labels in the training data, which we aim to address in this
work. Besides all, the application of FL to appliance recognition needs to be
investigated from robustness perspective in the presence of noisy labels.

##  3\. FedAR+ Approach

This section presents an overall setup of our FL approach, FedAR+, with
multiple clients222A client refers to a low-end computing device (e.g.,
personal computer) installed at consumer’s house to collect data from smart
plugs. The device is capable enough to train the underlying appliance
recognition model. and a common remote server, as depicted in Figure 1. In
appliance recognition scenario, the consumer acts as a client and the service
provider works as a remote server. A client may have many appliance-specific
smart plugs, each connected to a designated appliance to measure the
appliance’s electricity consumption and transfer that data to a local in-house
computing device. To initialize training, the server dispatches an appliance
recognition model to all the clients. Each client retrains the model using its
local data and sends the weight updates to the server for aggregation. Next,
the aggregated (or global) model is sent back to the clients. By repeating the
above steps for a certain number of global rounds, the model eventually
converges to an optimal solution. With this, FL helps achieve a more
generalized and accurate model without sharing the client’s local data.

In FedAR+, we build a deep learning model for appliance recognition which
requires a large amount of data for training; however, at the beginning of the
deployment, the clients may not have sufficient data. Therefore, we consider
the availability of an auxiliary dataset with the clients before the
initialization of FL training. To avoid flow disruption, we discuss the
auxiliary dataset (collected beforehand from some anonymous houses) in Section
4 and appliance recognition model with noise handing method in Section 5,
separately. Besides that, we formulate an aggregate function to alleviate the
bias that might be introduced by the clients having substantially larger
dataset than the others.

Figure 1. Overview of FedAR+. At each client, a noise handling method is
incorporated to enable the model learn with mislabeled data.

Let $K$ denote the number of clients collaborating in the learning to build
the recognition model. These clients need not be the same from which the
auxiliary data ware collected. Let
$\mathcal{D}^{j}=\\{\mathbfcal{X},\mathbfcal{Y}\\}$ be the dataset with
$j^{th}$ client, which includes both auxiliary and local data collected over a
fixed period of time, where $1\leq j\leq K$. At each update round, the
objective of the remote server is to learn optimal weight parameters
$\boldsymbol{\theta}$ by minimizing an empirical loss function as

(1)
$\displaystyle\underset{\boldsymbol{\theta}}{\text{argmin}}\quad\left\\{\mathcal{F}(\boldsymbol{\theta})=\Psi\left(\\{f^{j}(\boldsymbol{\theta})\\}_{1\leq
j\leq K}\right)\right\\},$

where $\Psi(\cdot)$ is an aggregate function and $f^{j}(\cdot)$ is the local
objective function used by the $j^{th}$ client. We also propose a noise
handling method, in Section 5.2, to facilitate learning with mislabeled
training data at the client.

###  3.1. Local model update at client

FedAR+ uses second-order method to perform local updates at the client.
Particularly, we adopt canonical Newton’s method of the form
$-\nabla^{2}(f^{j})^{-1}\nabla f^{j}$ (Nocedal and Wright, 2006) as it
improves the convergence rate and reduces the accumulation of errors. Given
the weight parameters $\boldsymbol{\theta}_{[t]}$ of the global model at
update round $t$, the client $j$ first computes the local gradient as

(2) $\displaystyle g^{j}_{[t]}=\nabla
f^{j}\big{(}\boldsymbol{\theta}_{[t]}\big{)}.$

The client next computes the second-order gradient (Hessian matrix) at
$\boldsymbol{\theta}_{[t]}$ as follows

(3) $\displaystyle
h^{j}_{[t]}=\nabla^{2}f^{j}\big{(}\boldsymbol{\theta}_{[t]}\big{)}.$

Now, the local model at the client $j$ is updated as

(4)
$\displaystyle\boldsymbol{\theta}_{[t+1]}^{j}=\boldsymbol{\theta}^{j}_{[t]}-\eta(h^{j}_{[t]})^{-1}g^{j}_{[t]}$

and $\eta$ is the learning rate. Finally, the local updates are sent back to
the server for aggregation.

###  3.2. Global model update at server

The problem of aggregation at the server becomes quite simple if we assume
that all the clients have independent identically distributed (iid) data, and
it can be easily solved by using FedAvg (McMahan et al., 2017) as

(5) $\displaystyle\Psi\left(\\{f^{j}(\boldsymbol{\theta})\\}_{1\leq j\leq
K}\right)\overset{\underset{\mathrm{def}}{}}{=}\sum_{j=1}^{K}\frac{N^{j}}{N^{total}}\boldsymbol{\theta}_{[t+1]}^{j},$

where $N^{j}$ is the size of $\mathcal{D}^{j}$ and
$N^{total}=N^{1}+N^{2}+\cdots+N^{h}$. However, this assumption is unrealistic
for applying FL to appliance recognition as the clients may have different
number of appliances (essentially non-iid data). With FedAvg, the model may be
biased towards the clients who have substantially larger dataset than others.
To deal with this situation, we introduce an aggregation function

(6) $\displaystyle\Psi\left(\\{f^{j}(\boldsymbol{\theta})\\}_{1\leq j\leq
K}\right)\overset{\underset{\mathrm{def}}{}}{=}\sum_{j=1}^{K}\frac{1}{K}\cdot\boldsymbol{\theta}_{[t+1]}^{j}.$

With the new aggregation function, each client would receive an unbiased model
regardless of number of appliances the client possesses. Algorithm 1
summarizes the major steps of FedAR+ with $K$ clients for $T$ number of global
rounds.

Initialization:

1 The server builds and broadcasts a recognition model along with an auxiliary
dataset to all $K$ clients.

2

3for _$t\leftarrow 1$ to $T$_ do

Local model update at round $t$:

4 for _each client $j\in\\{1,2,\cdots,K\\}$_ do

/* $\boldsymbol{\theta}^{j}_{[t]}$ is the weight parameters of local model */

5 Obtain $\boldsymbol{\theta}^{j}_{[t+1]}$ using Eq. 4. // the underlying loss
function is formulated in Section 5.2.

6 Dispatch $\boldsymbol{\theta}^{j}_{[t+1]}$ to the server.

Global model update using aggregation at round $t$:

7
$\boldsymbol{\theta}_{[t+1]}=\sum_{j=1}^{K}\frac{1}{K}\cdot\boldsymbol{\theta}_{[t+1]}^{j}$,
using Eq. 6.

8 Broadcast $\boldsymbol{\theta}_{[t+1]}$ to the clients

Algorithm 1 FedAR+

$\bullet$ Model convergence: The global recognition model (at the server)
advances as the training progresses and it is said to be converged when stops
advancing or reaches to the optimal solution. We study the model convergence,
under the standard assumptions on the function $\mathcal{F}(\cdot)$ (McMahan
et al., 2017), in terms of the optimality gap
$\delta=\mathcal{F}(\boldsymbol{\theta}_{[T]})-\mathcal{F}(\boldsymbol{\theta}^{*})$,
where $T$ denotes the maximum number of rounds and $\boldsymbol{\theta}^{*}$
denotes the weights of the optimal model. Ideally, $\delta\approx 0$ for a
sufficiently large $T$. Assuming $K$ clients, FedAR+ can achieve
$O(\frac{1}{\sqrt{KT}})$ convergence for our DNN model (i.e., non-convex
optimization problem). Our experimental results, reported in Section 6.4.1,
show that the global model converges in $T=30$ rounds (with $50$ local
iterations on each clients at each round) with $10$ clients even when there
exist $30\%$ noisy labels. The theoretical proof of the convergence rate and
the theoretical bounds over the local gradients with heterogeneous clients are
to be investigated in our future work.

##  4\. Auxiliary Dataset

In FedAR+, the server is assumed to provide an auxiliary dataset to the
clients, before initializing the training, to enable them for an effective
local update of the model. We therefore first discuss data collection and
preprocessing for creating an auxiliary dataset. We utilize the power
consumption data for recognizing appliances such as refrigerator, electric
kettle, television, etc. The data are collected by connecting the appliance to
power socket through a designated smart plug that provides a sequence of time
stamped readings at a preset sampling rate.

###### Definition 0 (Time Series of Consumption).

It is a temporal sequence of data points collected over a period of time. Let
$X=\\{x_{1},x_{2},\cdots,x_{n}\\}$ denote the Time Series of power Consumption
(TSC) readings from a designated smart plug, where $n$ is the total number of
data points collected during the entire experiment; and $x_{i}$ denotes a
reading taken at time $t_{i}$, where $1\leq i\leq n$ and $t_{i-1}<t_{i}$.

###  4.1. Dataset creation

We construct a dataset using TSCs of different appliances. As the appliance
can change its state from ON to OFF or vise-versa several times, each TSC
(denoted by $X$) includes readings corresponding to both the states. We first
separate out only the subsequences (of $X$) that correspond to ON states
occurred at distinct time steps along $X$. Then for each separated
subsequence, an appliance footprint is computed and stored as an instance of
the respective appliance.

####  4.1.1. Data collection

To create an auxiliary dataset, we collected power consumption data from three
different houses (within an apartment complex) where each possesses six common
household appliances: refrigerator, microwave oven, television, washing
machine, air conditioner, and mixer grinder. Each appliance is connected to a
designated smart plug that transmits the readings to a in-house data collector
(e.g., personal computer) at 1Hz. As we collected the data for a period of one
month from each house, we got total $18$ TSCs (i.e., six time series from each
of the three houses).

####  4.1.2. Preprocessing

Let us first discuss the terminology for better illustration of data
preprocessing.
$\bullet$ Switch point: For a TSC $X=\\{x_{1},x_{2},\cdots,x_{n}\\}$, a time
instance $t$ is said to be a switch point if the following conditions hold:
(i) The difference $\delta(t)=|X(t)-X(t-1)|>\phi_{1}$, a predefined threshold,
where $X(t)$ and $X(t-1)$ denote the power consumption readings at time $t$
and $t-1$, respectively; and (ii) The rate of change in power readings
$\delta_{r}(t)=\delta(t)/X(t)>\phi_{2}$, another threshold. For setting an
appropriate value for $\phi_{1}$ and $\phi_{2}$, we visualized several time
series for different appliances including both low power (e.g., television)
and high power (e.g., air conditioner). We observed that with $\phi_{1}=30$
watts and $\phi_{2}=0.2$ (i.e., $20\%$) jointly, the switch points can be
detected correctly for most commonly available appliances such as
refrigerator, microwave oven, air conditioner, etc. Further, as the thresholds
are set empirically, their values are subject to change according to the
appliance’ operating environment (such as brand and power rating standards of
different countries). With small thresholds, we may get frequent false
positive; on the contrary, some ON states may get lost with large thresholds.

$\bullet$ Steady point: A time point $t$ along the time series $X$ is said to
be steady if $\delta_{r}(t)<\phi_{2}$.

$\bullet$ Steady period: Given a time series $X$, a steady period is a
subsequence $X_{t:m}=\\{x_{t+1},x_{t+2},\cdots,x_{t+m}\\}$ if all of its time
points are steady. Here, $t$ and $m$ respectively denote a switch point and
the length of the steady period, where $t<(n-m)$.

###### Definition 0 (Appliance Footprint).

For a given steady period $X_{t:m}$, corresponding to the ON state of the
appliance, we define the appliance footprint as:

(7) $\displaystyle X_{af}=\\{X_{t:m}(i)-X_{t:m}(i-1)\ |\ 1\leq i\leq m\\},$

where $X_{t:m}(i)$ denotes the $i^{th}$ data point of steady period.

We compute single-order differences between the consecutive data points to
capture subtle fluctuations, revealing better identifiable patterns than those
with higher-order statistics. Moreover, the single order difference
automatically scales down the values to a smaller range, eliminating the need
of normalization. The appliance footprint essentially represents the power
consumption pattern of the appliance when it is active. For a given TSC
$X=\\{x_{1},x_{2},\cdots,x_{n}\\}$, the extraction of footprints includes
following three steps:

  1. (1)

Identify switch points in $X$ under thresholds $\phi_{1}$ and $\phi_{2}$.

  2. (2)

For each identified switch point $t$, follow two sub-steps: (a) search for a
steady period of length $m$ after $t$. Let
$X_{t:m}=\\{x_{t+1},x_{t+2},\cdots,x_{t+m}\\}$ be a steady period obtained
after the switch point $t$. (b) if $X(t)-X(t+m)<0$, then the steady period
$X_{t:m}$ corresponds to ON state of the appliance; otherwise OFF state.

  3. (3)

The steady periods corresponding to ON states, are used to obtain appliance
footprints; each of which along with its label (name of the appliance) is
stored as an instance.

Figure 2. Illustrating switch points, steady points, and steady period, in a
TSC of a refrigerator for a window of $15$ minutes.

Figure 2 illustrates a TSC of a refrigerator with identified switch points,
steady points, and steady period. The obtained steady periods are used to
compute the appliance footprints. Upon obtaining the footprints by processing
TSCs of all the appliances, we perform padding on shorter instances to make
all the instances of equal length and store them in the dataset. which later
provided to the clients as an auxiliary dataset.

###  4.2. Presence of noisy labels

During local data collection at the client, some TSCs may get associated with
noisy (wrong) labels due to following reasons:

  * •

From deployment perspective: An appliance (say $A_{1}$) mistakenly got
connected to a non-designated smart plug that was marked to connect with some
other appliance (say $A_{2}$). Consequently, the generated TSC receives a
noisy label $A_{2}$, creating several mislabeled instances (wrong appliance
footprints) in the training dataset. It is true that such noisy labels may be
avoided at the cost of additional care from the consumers, however it is not
preferable rather the model should be robust against mislabeled training data.

  * •

From security perspective: A malicious consumer (or compromised client) may
attempt to inject a wrong label intentionally, to gain some incentive from a
rival service provider. As a consequence, the client’s local model would
generate corrupted local updates, which eventually would diminish the
performance of global model.

###### Definition 0 (Noisy label).

Let $\mathcal{D}=\\{(\mathbfcal{X},\mathbfcal{Y})\\}$ be a dataset where
$\mathcal{X}^{i}\in\mathbfcal{X}$ is the $i^{th}$ instance associated with a
class label $\mathcal{Y}^{i}\in\mathbf{Y}=\\{y_{1},y_{2},\cdots,y_{C}\\}$, the
set of all $C$ classes (appliances). The label $\mathcal{Y}^{i}$ is said to be
noisy if either of the following holds: (i) $\mathcal{Y}^{i}$ is mislabeled as
other class label, i.e., $\mathcal{Y}^{i}\in\\{\mathbf{Y}-y_{c}\\}$, where
$y_{c}$ denotes the correct class label of $\mathcal{X}^{i}$, or (ii)
$\mathcal{Y}^{i}$ is an arbitrary class label, i.e.,
$\mathcal{Y}^{i}\notin\mathbf{Y}$.

##  5\. Appliance Recognition Model

This section presents a deep neural network (DNN) for appliance recognition
that trains collaboratively on the locally collected data and the auxiliary
dataset provided by the server. The choice of DNN is inspired by its success
at recognition tasks with a rich set of learnable features. The network (or
model) learns from a training dataset and predicts the class label of a new
instance. On the top of that, we propose an adaptive noise handling method to
enable the model learning with mislabeled training data at the clients.

###  5.1. Base model overview

We build a DNN with three convolutional layers (connected sequentially)
followed by a flatten and a Fully Connected (FC) layer, as shown in Figure 3.
Let $\mathcal{D}=\\{\mathbfcal{X},\mathbfcal{Y}\\}$ be a training dataset
(including auxiliary dataset) with the client. The model takes a training
dataset $\mathcal{D}$ and yields a set of class probabilities using a softmax
function. The convolutional layers are all one-dimensional, each consisting of
$128$ filters of size $1\times 1$ with input shape $(1,m)$, where $m$ denotes
the number of data points in each instance (i.e., appliance footprint).
Considering there exist total $C$ labels in $\mathcal{D}$, we use $C$ neurons
at the FC layer. Finally, a softmax function is applied on the output of FC
layer to get the class probabilities.

Figure 3. An overview of the appliance recognition model in FedAR+.

Now, we present mathematical formulation of our base model (i.e., excluding
the noise handling method). Given the dataset $\mathcal{D}$, the recognition
model mainly attempts to learn a mapping
$\mathcal{H}:\mathbfcal{X}\rightarrow\mathbfcal{Y}$, which usually expressed
as

(8)
$\displaystyle\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})=\sigma_{\mathbf{Y}}(\boldsymbol{\theta}\mathbfcal{X}),$

where $\sigma(\cdot)$ is a softmax function and
$\mathbf{Y}=\\{y_{1},y_{2},\cdots,y_{C}\\}$ is a set of different class labels
in $\mathcal{D}$. The function $\sigma(\cdot)$ can transform a vector into
probability distribution over its elements. For a vector $z\in\mathbb{R}^{C}$,
the softmax function is:

$\displaystyle\sigma_{c}(z)$
$\displaystyle=\frac{\text{e}^{z_{c}}}{\sum_{c=1}^{C}\text{e}^{z_{c}}}=p(c|z)\quad\forall
c\in\\{1,2,\cdots,C\\}.$

Rewriting Eq. 8,

(9)
$\displaystyle\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})=p(\mathbf{Y}|\mathbfcal{X},\boldsymbol{\theta}).$

To this end, the appliance recognition problem with base model, at the client,
can be observed as a local optimization problem

(10) $\displaystyle
f(\boldsymbol{\theta})=\underset{\boldsymbol{\theta}}{\text{argmin
}}\big{\\{}\mathcal{L}\big{(}\mathbfcal{Y},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}\big{\\}},$

where $\mathcal{L}(\cdot)$ is an underlying empirical loss function. The base
model employs cross-entropy loss function as widely used in DNNs for solving
recognition problems (Goodfellow et al., 2016). The cross-entropy loss
function for $\mathcal{D}$ with $N$ instances, is given as

$\displaystyle\mathcal{L}\big{(}\mathbfcal{Y},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}\mathbb{I}(\mathcal{Y}^{i},y_{c})\text{log}\
\mathcal{H}\big{(}\mathcal{X}^{i},\boldsymbol{\theta}\big{)},$ (11)
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}\mathbb{I}(\mathcal{Y}^{i},y_{c})\text{log}\
p\big{(}\mathcal{Y}^{i}=y_{c}\big{|}\mathcal{X}^{i},\boldsymbol{\theta}\big{)},$

where $\mathbb{I}(\mathcal{Y}^{i},y_{c})=1$ if $\mathcal{Y}^{i}$ is $y_{c}$,
and $0$ otherwise.

###  5.2. Noise handling method

To enable the base model learning from mislabeled training data, this section
proposes a noise handling method that learns correct labels by iteratively
updating the label distributions probabilistically, which is significantly
different from the existing approaches (Xu et al., 2022; Li et al., 2022; Gao
et al., 2017) where constant distributions were used in all the iterations. In
our method, the model optimizes label distributions along with weight
parameters during training, and therefore the local objective function (i.e.,
Eq. 10) can be written as

(12) $\displaystyle
f(\boldsymbol{\theta})=\underset{\boldsymbol{\theta},\mathbfcal{Y}_{d}}{\text{argmin
}}\big{\\{}\mathcal{L}\big{(}\mathbfcal{Y}_{d},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}\big{\\}},$

where $\mathbfcal{Y}_{d}$ denotes the label distributions among $C$ classes
for all $N$ instances of the dataset $\mathcal{D}$. To solve Eq. 12, we
introduce a noise handling method consisting of three steps explained below.

####  5.2.1. Learn weight parameters $\boldsymbol{\theta}$

First, we train the base model with cross-entropy loss function (Eq. 5.1) on
the training dataset $\mathcal{D}$. By optimizing the loss function, the model
learns the weight parameters $\boldsymbol{\theta}$. Due to noisy labels, the
learned weights may be far from optimality; nevertheless, they can certainly
be used for the initial estimation of the label distributions over the
training data.

####  5.2.2. Estimate label distributions $\mathbfcal{Y}_{d}$:

Given the trained model, the label distributions $\mathbfcal{Y}_{d}$ can be
estimated for all instances of $\mathcal{D}$ through validation as
$\mathbfcal{Y}_{d}=\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})$. For each
instance, the model provides probability distribution of labels using learned
$\boldsymbol{\theta}$. The label with highest probability is assigned to the
instance. In general, if the assigned label is the same as true, then its
probability should differ substantially from that of other labels. However,
this statement holds only if the training dataset does not contain any noisy
label. Hence, our method utilizes the distribution instead of only the highest
probable label while computing the loss.

####  5.2.3. Optimize $\boldsymbol{\theta}$ and $\mathbfcal{Y}_{d}$:

This step aims to optimize $\mathbfcal{Y}_{d}$ using Kullback-Leibler (KL)
divergence (MacKay and Mac Kay, 2003) and subsequently fine-tuning the
parameters $\boldsymbol{\theta}$ with optimized version of
$\mathbfcal{Y}_{d}$. The KL-divergence measures the difference between two
probability distributions. Thus, the loss function of the base model, defined
Eq. 5.1, is replaced by

(13)
$\displaystyle\mathcal{L}\big{(}\mathbfcal{Y}_{d},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}=\frac{1}{N}\sum_{i=1}^{N}$
$\displaystyle
KL(\mathcal{Y}_{d}^{i}\parallel\mathcal{H}(\mathcal{X}^{i},\boldsymbol{\theta})),$
$\displaystyle\text{where}\quad
KL(\mathcal{Y}_{d}^{i}\parallel\mathcal{H}(\mathcal{X}^{i},\boldsymbol{\theta}))=$
$\displaystyle\sum_{c=1}^{C}\mathcal{Y}_{d}^{i,c}\
\text{log}\left(\frac{\mathcal{Y}_{d}^{i,c}}{\mathcal{H}_{c}(\mathcal{X}^{i},\boldsymbol{\theta})}\right).$

Let us first compute the gradient of $\mathcal{L}(\cdot)$ for all $i$ and $c$
as

(14)
$\displaystyle\frac{d\mathcal{L}\big{(}\mathbfcal{Y}_{d},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}}{d\mathcal{Y}_{d}^{i,c}}=1+\sum_{c=1}^{C}\
\text{log}\left(\frac{\mathcal{Y}_{d}^{i,c}}{\mathcal{H}_{c}(\mathcal{X}^{i},\boldsymbol{\theta})}\right),$

and then update $\mathbfcal{Y}_{d}$ by

(15)
$\displaystyle\mathbfcal{Y}_{d}=\mathbfcal{Y}_{d}-\eta\frac{d\mathcal{L}\big{(}\mathbfcal{Y}_{d},\mathcal{H}(\mathbfcal{X},\boldsymbol{\theta})\big{)}}{d\mathbfcal{Y}_{d}},$

where $\eta$ is the learning rate. Once $\mathbfcal{Y}_{d}$ stabilizes, the
model stops learning. With learned $\mathbfcal{Y}_{d}$, the weight parameters
$\boldsymbol{\theta}$ are then fine tuned for a fixed number of local
iterations usually preset by the server before initializing the training.
Finally, the updated weights $\boldsymbol{\theta}$ are collected by the server
from all the clients for aggregation. Note that the proposed noise handling
method is adaptive as it automatically adapts to mislabeled training data
without requiring any additional mechanism to correct the labels beforehand.

###  5.3. Prediction

Given the trained model and a testing TSC (generated from the smart-plug), we
first need to extract appliance footprints using the preprocessing steps
(discussed in Section 4.1) to prepare input to the model. Let
$\boldsymbol{\theta}^{*}$ be the optimized model, obtained after $T$ global
rounds, dispatched from the server to all the clients to predict the class
label (or recognitze an appliance) using the footprint. Figure 4 illustrates
the recognition process using DNN-based model.

Figure 4. Appliance recognition using the optimized recognition model
$\boldsymbol{\theta}^{*}$.

Let $\mathcal{X}{{}^{\prime}}\notin\mathcal{D}$ be a testing instance (an
appliance footprint) for which the class label is to be predicted. The client
utilizes the model with $\boldsymbol{\theta}^{*}$ to first compute the
posterior class probabilities and then assigns the highest probable class
label to $\mathcal{X}{{}^{\prime}}$ as expressed below

(16)
$\displaystyle\vspace{-0.1in}\mathcal{Y}{{}^{\prime}}=\underset{y_{c}}{\text{argmax}}\
\\{\sigma_{y_{c}}(\boldsymbol{\theta}^{*}\mathcal{X}{{}^{\prime}})\\}.\vspace{-0.1in}$

Time complexity of our appliance recognition model mainly depends on the
operations at the convolutional layers. As the model comprises three layers
with identical configurations, the time complexity is
$O\big{(}m(\sum_{i=1}^{3}p_{i-1}\cdot s^{2}_{i}\cdot f_{i}\cdot
o_{i}^{2})\big{)}$ (He and Sun, 2015), where $m$ is the length of the instance
(i.e., appliance footprint), $p_{i-1}$ is the number of input channels,
$s_{i}$ and $f_{i}$ are respectively the number and size of filters, and
$o_{i}$ is the spatial size of output feature vector at $i^{th}$ layer. As the
model is trained across different clients, which can be done in parallel,
FedAR+ imposes only communication overheads due to aggregation after each FL
round, however, the communication analysis is out-of-scope of this work.

##  6\. Performance Evaluation

We evaluate the performance of FedAR+ on a dataset collected from three
different houses, and also on two existing datasets, namely UK-DALE (Kelly and
Knottenbelt, 2015) and Tracebase (Reinhardt et al., 2012), using accuracy,
precision, recall, and $F_{1}$-score. To demonstrate the superiority of our
proposed approach, we compare it with three existing ones and report the
results using the considered metrics and execution time.

###  6.1. Datasets

####  6.1.1. Collected dataset

Data collection is done by deploying appliance-specific smart plugs in three
different houses (not included in the auxiliary dataset) for a period of one
month. The power consumption data are collected at a sampling rate $1$Hz.
After preprocessing (discussed in Section 4.1), we get a dataset of $840$
instances for six different appliances including refrigerator, microwave oven,
television, washing machine, air conditioner, and mixer grinder. Each instance
corresponds to a footprint of a particular appliance. We call this dataset as
appliance footprints in residential buildings (Res-AF).

####  6.1.2. UK-DALE dataset (Kelly and Knottenbelt, 2015)

It contains power consumption data of various household appliances from five
houses for a period of one year. The data are recorded at a sampling rate
$1/6$ Hz (i.e., one reading per six seconds). For the experiments, we selected
five appliances including refrigerator, washing machine, kettle, dishwasher,
and boiler.

####  6.1.3. Tracebase dataset (Reinhardt et al., 2012)

It contains more than $1000$ power consumption traces collected from $15$
different houses. One trace corresponds to a time series of readings taken for
one particular appliance over a window of $24$ hours. The readings are
reported at an average sampling rate of $1$ Hz. In (Reinhardt et al., 2012), a
Measurement and Actuation Unit (MAU) is developed to collect the power
consumption traces of the appliances. MAU is installed between wall mounted
power outlet and power plug of the appliance. For experimental evaluation, we
selected five appliances having sufficient number of traces (instances) in the
dataset. The five selected appliances are refrigerator, microwave oven,
kettle, television, and dishwasher.

$\bullet$ Preprocessing: In the existing datasets, the readings are collected
for a continuous period and thus the resulting time series include both ON and
OFF states of the appliances. We therefore preprocess these datasets to
prepare them for training and testing the proposed recognition model. By
following the preprocessing steps described in Section 4.1, we obtained
$1,860$ and $930$ instances (i.e., appliance footprints) in UK-DALE and
Tracebase datasets, respectively.

###  6.2. Experimental setup

Prior to conducting experiments, we split each dataset into two parts:
training with $80\%$ and testing with $20\%$ instances. The training data are
further split into $10$ non-iid chunks using Dirichlet distribution with
parameter $\alpha=0.9$ and $20\%$ overlapping. These chunks are then provided
to $K=10$ clients. It is to note that each client may have different number of
instances per class due to non-iid data. While working with Res-AF dataset, a
copy of auxiliary dataset (collected in Section 4) is given to each client.
Similarly, for UK-DALE and Tracebase datasets, $20\%$ of the training data are
used as auxiliary dataset which is same for all clients and required for the
model initialization. Further, to produce noisy labels in the training
datasets, we flip the labels of a fixed percentage (indicated in the
respective results) of instances across all class labels.

We simulated the FedAR+ algorithm with a server and $10$ clients in Python
programming language through Tensorflow libraries. In the implementation, we
chose the following parameters: optimizer = ‘sgd’, activation = ‘relu’ with
each convolutional layer, and learning rate $\eta=0.1$. The number of FL
global rounds are set based on the results obtained after rigorous experiments
(see Section 6.4.1). Since our recognition model consists of only three
convolutional layers, each with $128$ filters, it does not overfit with small
training datasets.

###  6.3. Performance metrics

The following metrics evaluate the performance of FedAR+.

  * •

Precision (P): It is the ratio of the number of correctly classified instances
of an appliance $x$ to the total number of instances classified as $x$.
Precision indicates a quality aspect of the appliance recognition model.

  * •

Recall (R): It is the ratio of the number of correctly classified instances of
an appliance $x$ to the total number of instances actually belonging to $x$.
Recall measures the completeness and relevance of the recognition model.

  * •

F1 score: It is a harmonic mean of precision and recall, and is computed as
$\frac{2\times\text{P}\times\text{R}}{\text{P}+\text{R}}$.

  * •

Accuracy: It is the proportion of correctly classified instances of the
testing dataset. It is expressed as

$\displaystyle\text{Accuracy}=\frac{1}{N_{test}}\sum_{i=1}^{N_{test}}\mathbb{I}(\mathcal{Y}^{i},\mathcal{Y}{{}^{\prime}}),$

where $\mathbb{I}(\mathcal{Y}^{i},\mathcal{Y}{{}^{\prime}})=1$ if
$\mathcal{Y}^{i}==\mathcal{Y}{{}^{\prime}}$ and $0$ otherwise; and $N_{test}$
is the number of instances in testing dataset.

###  6.4. Experimental results

Through experiments, we seek answers to the following six questions: (1) How
does the accuracy of the model improve over rounds in FedAR+? (2) How does the
concentration of noisy labels influence the testing performance? (3) What is
the appliance-wise performance of the model? (4) How does FedAR+ scale to the
number of clients? (5) How efficiently does FedAR+ outperform the prior
approaches? The results are presented below.

####  6.4.1. Training accuracy over FL rounds

At first, we analyze the training accuracy of the recognition model over $35$
rounds with different concentrations (from $5\%$ to $30\%$) of noisy labels in
the training dataset. We set the local epochs to $50$ at the clients in all
the experiments. Figure 5 demonstrates the results for $5\%$ and $30\%$ cases.
The results clearly indicate that the model is able to achieve more than
$92\%$ of training accuracy at $30^{th}$ round, even when $30\%$ training
instances are mislabeled. As initial model is far from the optimality in first
few rounds, it shows low accuracy for all datasets. The accuracy increases
rapidly up to $15^{th}$ rounds and starts stabilizing afterwards. As no change
is observed in the accuracy between $30^{th}$ and $35^{th}$ rounds, we report
all the subsequent results with $30$ FL rounds and $50$ local epochs at the
clients.

(a) With $5\%$ noisy labels

(b) With $30\%$ noisy labels

Figure 5. Training accuracy of the proposed appliance recognition model with
the noise handling method in FedAR+.

####  6.4.2. Testing performance with varying concentration of noisy labels

In Figure 6, we report the impact of noisy labels with varying concentrations
on the performance of the model. As the concentration of noisy labels
increases, the accuracy and F1 score decrease, which seems bit obvious but,
such a drop is substantial for the base model compared to the one with noise
handling method. For instance, in case of Tracebase dataset, with just $5\%$
noisy labels, the base model immediately loses more than $4\%$ accuracy. On
the flip side, for Res-AF dataset with $30\%$ noisy labels, the model gains
$14.2\%$ on accuracy by utilizing the proposed noise handling method,
indicating the effectiveness of the proposed approach. For all the datasets,
FedAR+ achieves an accuracy of more than $84\%$ and F1 score of above $81\%$
up to $30\%$ of noisy labels; however, the performance drops sharply
afterwards, signaling the noise handling upper limit of our approach. With
higher concentrations, the reason for such a drop is the increase in confusion
while differentiating between correct and noisy labels. Similar observations
can be made from F1 score, shown in parts (b), (d) and (f) of Figure 6.

(a) Accuracy for Res-AF dataset

(b) F1 score for Res-AF dataset

(c) Accuracy for UK-DALE dataset

(d) F1 score for UK-DALE dataset

(e) Accuracy for Tracebase dataset

(f) F1 score for Tracebase dataset

Figure 6. Performance results of FedAR+ using the recognition model with and
without noise handling method.

####  6.4.3. Appliance-wise performance of the recognition model

Next, we analyze the appliance-wise performance results of the model with
noise handling method for Res-AF and Tracebase datasets in Tables 1 and 2,
respectively. The results are reported using precision, recall, and F1 score
evaluation metrics for $5\%$ and $30\%$ concentration of noisy labels. Results
indicate that the model performs best on “television” and “refrigerator”
classes; they were identified with more than $90\%$ of recall even when the
concentration of noisy labels is $30\%$, witnessing the robustness of our
approach FedAR+ against noisy labels. We also observed that the precision
values are marginally ($1\sim 4$ approximately) differ from the recall ones,
indicating the ability of FedAR+ to manage the good balance between the
relevance and completeness of the appliance recognition model.

Table 1. Appliance-wise performance of the model with noisy handling method
for Res-AF dataset using precision (P), recall (R), and F1 score (F).

| 5% Noisy labels | 30% Noisy labels
---|---|---
P (%) | R (%) | F (%) | P (%) | R (%) | F (%)
Refrigerator | 90.4 | 94.0 | 90.8 | 90.2 | 89.8 | 90.0
Microwave oven | 92.8 | 92.7 | 93.5 | 85.3 | 85.5 | 85.4
Television | 93.1 | 94.4 | 94.7 | 90.5 | 90.8 | 90.6
Washing machine | 89.0 | 91.4 | 89.1 | 87.2 | 80.4 | 87.8
Air conditioner | 89.3 | 92.7 | 90.7 | 80.1 | 82.4 | 81.2
Mixer grinder | 91.3 | 90.0 | 90.5 | 83.8 | 85.2 | 84.5
Average | 90.8 | 92.5 | 91.5 | 85.8 | 87.3 | 86.5

Table 2. Appliance-wise performance of the model with noise handling method
for Tracebase dataset using precision (P), recall (R), and F1 score (F).

| 5% Noisy labels | 30% Noisy labels
---|---|---
P (%) | R (%) | F (%) | P (%) | R (%) | F (%)
Refrigerator | 91.1 | 94.2 | 92.6 | 86.3 | 91.5 | 88.6
Microwave oven | 88.4 | 91.2 | 89.7 | 81.2 | 89.1 | 84.9
Kettle | 86.7 | 88.2 | 87.4 | 80.3 | 85.2 | 82.6
Television | 88.1 | 93.2 | 90.5 | 85.3 | 92.1 | 88.5
Dishwasher | 87.2 | 90.1 | 88.6 | 81.1 | 87.9 | 84.3
Average | 88.3 | 91.4 | 89.8 | 82.8 | 89.2 | 85.7

####  6.4.4. Scalability analysis

The scalability of our FedAR+ algorithm can be measured in terms of the number
of clients it can support without affecting the performance of the model. We
investigate the scalability by increasing the number of clients. To perform an
experiment with many clients (say $50\sim 500$), we first need to provide
sufficient data instances to every client. For this, we generated synthetic
training and testing data by adding Gaussian noise with $\mu=0$ and
$\sigma\in[0.1-0.5]$, and distributed the data uniformly among the clients.
Figure 7 shows the training loss of the global model over FL rounds for
different number of clients. One quick observation from the results is that
the model converges (i.e., loss stabilizes) after $20$ rounds when only $50$
clients exist, however it needs $10$ more rounds with $100$ to $500$ clients.
It indicates that by increasing the number of FL rounds, our approach can be
easily scaled to large number of clients without affecting the convergence.
Furthermore, we conducted some experiments with $0\%$ and $30\%$ noisy labels
and reported the accuracy (obtained by the model trained for $30$ rounds) in
Table 3 for both Res-AF and Tracebase datasets. The results demonstrate the
scalability of FedAR+ to $500$ clients with a marginal drop in accuracy that
cab be easily recovered by training the model for more FL rounds.

(a) For Res-AF dataset

(b) For Tracebase dataset

Figure 7. Training loss of the model in FedAR+ with $0\%$ noisy labels. Table
3. Accuracy results for the model trained over $30$ rounds.

No. of Clients | Res-AF dataset | Tracebase dataset
---|---|---
| 0% Noisy
---
labels
| 30% Noisy
---
labels
| 0% Noisy
---
labels
| 30% Noisy
---
labels
10 | 93.9 | 88.1 | 93.5 | 87.3
50 | 94.1 | 87.9 | 92.1 | 86.2
100 | 91.8 | 85.2 | 91.2 | 84.8
500 | 92.5 | 84.7 | 91.1 | 83.3

Table 4. Performance comparison of the proposed approach with the existing
ones using accuracy (in %). [P: Precision, R: Recall, F: $F_{1}$ score, A:
Accuracy]

Datasets |  Approaches | Concentration of noisy labels in training data
---|---|---
$0\%$ | $10\%$ | $20\%$ | $30\%$
P | R | F | A | P | R | F | A | P | R | F | A | P | R | F | A
Res-AF |  HARB (Yan et al., 2019) | $84.2$ | $87.2$ | $85.7$ | $89.5$ | $81.2$ | $83.1$ | $82.1$ | $84.9$ | $72.4$ | $67.2$ | $69.7$ | $70.2$ | $56.1$ | $60.1$ | $58.0$ | $58.6$
LSTM-AR (Schwermer et al., 2022) | $60.2$ | $59.2$ | $59.7$ | $60.5$ | $59.2$ | $55.2$ | $57.1$ | $60.9$ | $52.2$ | $50.2$ | $50.7$ | $51.1$ | $43.2$ | $45.7$ | $44.4$ | $46.6$
CNN-AR (Schwermer et al., 2022) | $55.2$ | $53.4$ | $54.2$ | $57.3$ | $52.4$ | $52.3$ | $52.3$ | $50.8$ | $45.1$ | $44.1$ | $43.6$ | $47.3$ | $48.2$ | $41.1$ | $42.1$ | $41.7$
FedAR (Proposed) | $92.4$ | $93.8$ | $93.1$ | $94.4$ | $90.1$ | $87.5$ | $88.8$ | $89.1$ | $82.1$ | $79.5$ | $80.8$ | $81.5$ | $73.6$ | $72.4$ | $73.0$ | $73.4$
FedAR+ (Proposed) | $\mathbf{93.1}$ | $\mathbf{95.1}$ | $\mathbf{94.1}$ | $\mathbf{94.8}$ | $\mathbf{91.2}$ | $\mathbf{89.5}$ | $\mathbf{90.3}$ | $\mathbf{92.3}$ | $\mathbf{88.5}$ | $\mathbf{87.1}$ | $\mathbf{87.8}$ | $\mathbf{89.5}$ | $\mathbf{87.2}$ | $\mathbf{85.7}$ | $\mathbf{86.4}$ | $\mathbf{88.4}$
UK-DALE |  HARB (Yan et al., 2019) | $87.6$ | $89.5$ | $88.5$ | $90.5$ | $85.7$ | $81.2$ | $83.4$ | $83.5$ | $70.2$ | $68.6$ | $69.4$ | $72.5$ | $60.2$ | $56.1$ | $58.1$ | $59.2$
LSTM-AR (Schwermer et al., 2022) | $59.1$ | $58.9$ | $59.0$ | $60.5$ | $56.3$ | $59.4$ | $57.8$ | $58.9$ | $45.6$ | $47.3$ | $45.9$ | $43.2$ | $43.2$ | $41.1$ | $42.1$ | $43.6$
CNN-AR (Schwermer et al., 2022) | $56.1$ | $50.1$ | $53.0$ | $54.8$ | $50.6$ | $59.2$ | $59.9$ | $51.2$ | $43.5$ | $43.2$ | $41.3$ | $42.2$ | $36.4$ | $38.1$ | $37.7$ | $41.9$
FedAR (Proposed) | $92.4$ | $91.2$ | $91.8$ | $94.9$ | $87.2$ | $89.2$ | $88.2$ | $90.3$ | $75.2$ | $76.1$ | $75.6$ | $81.1$ | $76.2$ | $70.3$ | $73.1$ | $74.2$
FedAR+ (Proposed) | $\mathbf{94.1}$ | $\mathbf{90.5}$ | $\mathbf{92.3}$ | $\mathbf{95.1}$ | $\mathbf{94.8}$ | $\mathbf{90.2}$ | $\mathbf{92.4}$ | $\mathbf{93.5}$ | $\mathbf{86.4}$ | $\mathbf{87.7}$ | $\mathbf{87.0}$ | $\mathbf{90.8}$ | $\mathbf{84.2}$ | $\mathbf{80.9}$ | $\mathbf{82.5}$ | $\mathbf{87.2}$
Tracebase |  HARB (Yan et al., 2019) | $87.4$ | $88.1$ | $87.7$ | $90.8$ | $78.6$ | $82.1$ | $80.3$ | $80.3$ | $71.5$ | $68.2$ | $69.8$ | $70.3$ | $58.2$ | $52.1$ | $55.0$ | $57.4$
LSTM-AR (Schwermer et al., 2022) | $56.2$ | $60.3$ | $58.2$ | $59.5$ | $58.2$ | $60.2$ | $59.2$ | $59.9$ | $50.5$ | $51.2$ | $49.3$ | $51.2$ | $45.2$ | $45.4$ | $45.0$ | $42.6$
CNN-AR (Schwermer et al., 2022) | $49.2$ | $50.8$ | $50.0$ | $51.2$ | $48.7$ | $52.4$ | $50.5$ | $50.5$ | $43.6$ | $38.7$ | $41.1$ | $43.3$ | $37.4$ | $36.9$ | $40.1$ | $39.2$
FedAR (Proposed) | $94.2$ | $92.8$ | $\mathbf{93.5}$ | $93.5$ | $91.7$ | $90.7$ | $91.2$ | $90.2$ | $80.5$ | $78.7$ | $79.6$ | $80.8$ | $70.1$ | $71.9$ | $71.0$ | $72.7$
FedAR+ (Proposed) | $\mathbf{94.6}$ | $\mathbf{91.4}$ | $\mathbf{93.0}$ | $\mathbf{93.8}$ | $\mathbf{93.7}$ | $\mathbf{90.5}$ | $\mathbf{92.1}$ | $\mathbf{92.9}$ | $\mathbf{89.2}$ | $\mathbf{86.3}$ | $\mathbf{87.7}$ | $\mathbf{88.9}$ | $\mathbf{87.5}$ | $\mathbf{84.3}$ | $\mathbf{85.9}$ | $\mathbf{86.1}$

###  6.5. Comparison with existing approaches

We compare FedAR+ approach with three state-of-the-art solutions including two
best performing plug-load identification models from (Schwermer et al., 2022)
and Household Appliance Recognition through Bayes classification (HARB) (Yan
et al., 2019). The work (Schwermer et al., 2022) leverages FL to train four
different deep learning models; we pick two best performers, long-short term
memory (LSTM) and convolutional neural network (CNN), named as LSTM-AR and
CNN-AR for the convenience. Similar to the proposed approach, LSTM-AR and CNN-
AR models are also trained across $10$ clients (possessing non-iid data) over
$30$ FL rounds with aggregation at every $50$ local epochs. On the other hand,
HARB (Yan et al., 2019) follows a central learning paradigm. In HARB, the time
series data are transformed into a set of statistical features (e.g., working
time, power range, frequency of the use, etc.) on which Bayesian learning is
applied to obtain posterior class probabilities that are used for appliance
prediction. As the existing solutions do not incorporate any noise handling
method, we consider two variants of the proposed approach: 1) FedAR: without
noise handling and 2) FedAR+: with noise handling, in the interest of fair
comparison. Table 4 shows the comparison results using precision, recall,
$F_{1}$ score, and accuracy. We make following important observations:

  * •

Both FedAR and FedAR+ gain over the existing solutions on all the evaluation
metrics by a large margin (approximately $15\%\sim 27\%$) even in the presence
of $30\%$ noisy labels.

  * •

Prior FL models, LSTM-AR and CNN-AR, perform much worse than the centralized
algorithm HARB in all the cases, indicating their inability to learn with non-
iid data across the clients. On the contrary, our FL approach FedAR
outperforms HARB with a substantial margin of more than $5\%$ in case of no
noisy label and more than $12\%$ in case of $30\%$ noisy labels, showing the
effectiveness of learning with non-iid data with the aggregation function
(defined in Eq. 6).

  * •

Even with $30\%$ mislabeled data, FedAR+ secured the accuracy and F1 of more
than $85\%$ on all the datasets, validating the success of our noise handling
method. It is worth to notice that the performance gain of FedAR+ over other
methods increases significantly with the surge in noisy labels.

####  6.5.1. IID versus non-IID data

Considering $10$ clients in FL setup, we now report the test accuracy results
in Figure 8 with iid (or uniformly) and non-iid data (simulated using
Dirichlet distribution as mentioned in experimental setup). For the sake of
fair comparison, this experiment does not include noisy labels. For both
datasets, FedAR+ shows its capability to learn with non-iid data by securing
almost equal accuracy as with iid data, however it is not true for prior
approaches; for instance, LSTM-AR loses $16.4\%$ accuracy when clients possess
non-iid training data. Although the performance of prior approaches seem to
improve significantly with iid data, they could never reach beyond $81\%$.

(a) For Res-AF dataset

(b) For Tracebase dataset

Figure 8. Comparing accuracy results obtained with iid and non-iid data (with
$0\%$ noisy labels) across clients in FL based approaches.

####  6.5.2. Execution time

Finally, we compare the execution time of FedAR+ with the existing approaches.
Here, the execution time indicates the total time taken by an approach to
classify the entire testing dataset. To better understand the comparison, we
compute the percentage difference ($T_{diff}$) in the execution times of any
existing approach from FedAR+, as follows:

$\displaystyle T_{diff}=\frac{T_{x}-T_{FedAR+}}{T_{x}}\times 100,$

where $T_{x}$ denotes the execution time of an existing approach $x$. Figure 9
shows the comparison results in terms of $T_{diff}$, which is an average over
50 executions. It is apparent that $T_{diff}$ is positive in all the cases,
indicating that FedAR+ is faster ($10.5\%$ to $16.8\%$) than the existing
approaches.

(a) For Res-AF dataset

(b) For Tracebase dataset

Figure 9. Comparison results using percentage difference in the execution time
of the existing approaches from FedAR+.

##  7\. Conclusion and Discussions

This paper proposed an FL approach, FedAR+, for identifying household
appliances using their electricity consumption patterns. The approach dealt
with two important issues related to the appliance recognition model: 1)
presence of noisy labels in the training dataset, and 2) model building at the
client (consumer) side without sharing local data to the server. By employing
deep learning and incorporating a noise handling method, we developed an
accurate appliance recognition model that can learn from mislabeled training
data. To preserve client’s privacy, the model is trained and updated locally
without sharing the local data with the server. By deploying smart plugs in an
apartment complex, we collected a real world dataset to validate the
effectiveness of FedAR+. Through rigorous experimental analysis, we
demonstrated the superiority of our approach over existing ones and showed
that FedAR+ can effectively accommodate up to $30\%$ noisy labels while
compromising the accuracy only slightly. Considering the availability of
sufficient training data, FedAR+ can be scaled to large number of clients,
enabling its adoption to real-world energy monitoring applications.

In future, we plan to work on theoretical guarantees of FedAR+. We will also
explore the robustness aspects of the appliance recognition model under the
presence of alien and malicious clients.

Alien appliance: An alien (unseen) appliance is one for which there exists no
instance in the training dataset to build the recognition model. Identifying
alien appliances is an interesting problem as it gives flexibility to the
consumer to introduce new household appliances without providing any
additional information to the service provider. In other words, the model
should rely upon only the semantic information of native (seen) appliances to
identify the alien ones. We plan to develop effective strategies for
extracting semantic information, thereby enhancing the capability of our
recognition model in FedAR+.

Malicious client: A client with wrong intention may try to attack the model by
altering its weight parameters when transmitted from the client to server.
Such malicious client can affect the performance of the global model in
FedAR+. We plan to detect the malicious clients who send incorrect parameters
by either adding random-noise or backdoor patterns in the dataset. Our idea is
to exploit the history of each client’ gradients with an appropriate
similarity measures (e.g., cosine distance) to distinguish malicious clients
from the normal ones and exclude their parameters from the aggregation.

## References

  * (1)
  * Allik et al. (2020) A. Allik, S. Muiste, and H. Pihlap. 2020.  Smart meter data analytics for occupancy detection of buildings with renewable energy generation. In _9th International Conference on Renewable Energy Research and Application_. 248–251.
  * Chen et al. (2019) K. Chen, K. Chen, Q. Wang, Z. He, J. Hu, and J. He.  Short-term load forecasting with deep residual networks.  _IEEE Transactions on Smart Grid_ 10, 4 (2019), 3943–3952.
  * Codispoti et al. (2022) J. Codispoti, A. R. Khamesi, N. Penn, S. Silvestri, and E. Shin.  Learning from Non-experts: An Interactive and Adaptive Learning Approach for Appliance Recognition in Smart Homes.  _ACM Transactions on Cyber-Physical Systems_ 6, 2 (2022), 1–22.
  * Farrokhifar et al. (2018) M. Farrokhifar, F. Momayyezi, N. Sadoogi, and A. Safari.  Real-time based approach for intelligent building energy management using dynamic price policies.  _Sustainable cities and society_ 37 (2018), 85–92.
  * Ganu et al. (2014) T. Ganu, D. Rahayu, D. P. Seetharam, R. Kunnath, A. P. Kumar, V. Arya, S. A. Husain, and S. Kalyanaraman. 2014\.  SocketWatch: an autonomous appliance monitoring system. In _IEEE International Conference on Pervasive Computing and Communications (PerCom)_. 38–43.
  * Gao et al. (2017) B.-B. Gao, C. Xing, C.-W. Xie, J. Wu, and X. Geng.  Deep label distribution learning with label ambiguity.  _IEEE Transactions on Image Processing_ 26, 6 (2017), 2825–2838.
  * Goodfellow et al. (2016) I. Goodfellow, Y. Bengio, and A. Courville. 2016.  _Deep learning_.  MIT press.
  * Han et al. (2019) J. Han, P. Luo, and X. Wang. 2019.  Deep self-learning from noisy labels. In _IEEE/CVF international conference on computer vision (ICCV)_. 5138–5147.
  * Hard et al. (2018) A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage.  Federated learning for mobile keyboard prediction.  _arXiv preprint arXiv:1811.03604_ (2018).
  * He and Sun (2015) K. He and J. Sun. 2015\.  Convolutional neural networks at constrained time cost. In _IEEE conference on computer vision and pattern recognition_. 5353–5360.
  * Jang et al. (2021) D. Jang, L. Spangher, T. Srivistava, M. Khattar, U. Agwan, S. Nadarajah, and C. Spanos. 2021.  Offline-online reinforcement learning for energy pricing in office demand response: lowering energy and data costs. In _8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys)_. 131–139.
  * Kelly and Knottenbelt (2015) J. Kelly and W. Knottenbelt.  The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes.  _Scientific Data_ 2, 150007 (2015).
  * Kleiminger et al. (2015) W. Kleiminger, C. Beckel, and S. Santini. 2015.  Household occupancy monitoring using electricity meters. In _ACM international joint conference on pervasive and ubiquitous computing (UbiComp)_. 975–986.
  * Leroy et al. (2019) D. Leroy, A. Coucke, T. Lavril, T. Gisselbrecht, and J. Dureau. 2019.  Federated learning for keyword spotting. In _IEEE International Conference on Acoustics, Speech and Signal Processing_. 6341–6345.
  * Li et al. (2022) Q. Li, J. Wang, Z. Yao, Y. Li, P. Yang, J. Yan, C. Wang, and S. Pu. 2022.  Unimodal-Concentrated Loss: Fully Adaptive Label Distribution Learning for Ordinal Regression. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 20513–20522.
  * Li et al. (2017) Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li. 2017.  Learning from noisy labels with distillation. In _IEEE/CVF international conference on computer vision (ICCV)_. 1910–1918.
  * Liang et al. (2022) K. J. Liang, S. B. Rangrej, V. Petrovic, and T. Hassner. 2022\.  Few-shot learning with noisy labels. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 9089–9098.
  * Liu et al. (2020) Y. Liu, A. Huang, Y. Luo, H. Huang, Y. Liu, Y. Chen, L. Feng, T. Chen, H. Yu, and Q. Yang. 2020.  Fedvision: An online visual object detection platform powered by federated learning. In _AAAI Conference on Artificial Intelligence_ , Vol. 34. 13172–13179.
  * Liu et al. (2019) Y. Liu, X. Wang, and W. You.  Non-Intrusive Load Monitoring by Voltage-Current Trajectory Enabled Transfer Learning.  _IEEE Transactions on Smart Grid_ 10, 5 (2019), 5609–5619.
  * Ma et al. (2018) M. Ma, W. Lin, J. Zhang, P. Wang, Y. Zhou, and X. Liang.  Toward Energy-Awareness Smart Building: Discover the Fingerprint of Your Electrical Appliances.  _IEEE Transactions on Industrial Informatics_ 14, 4 (2018), 1458–1468.
  * MacKay and Mac Kay (2003) D. J. MacKay and D. J. Mac Kay. 2003.  _Information theory, inference and learning algorithms_.  Cambridge university press.
  * McMahan et al. (2017) B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. 2017.  Communication-efficient learning of deep networks from decentralized data. In _Artificial intelligence and statistics (AISTATS)_. 1273–1282.
  * Nocedal and Wright (2006) J. Nocedal and S. Wright. 2006.  _Numerical optimization_.  Springer Science & Business Media.
  * Qureshi et al. (2022) N. B. S. Qureshi, D.-H. Kim, J. Lee, and E.-K. Lee. 2022\.  Poisoning Attacks against Federated Learning in Load Forecasting of Smart Energy. In _IEEE/IFIP Network Operations and Management Symposium_. 1–7.
  * Reddy et al. (2017) R. Reddy, N. Keesara, V. Garg, and V. Pudi. 2017\.  Plug load identification using regression based nearest neighbor classifier. In _8th International Conference on Future Energy Systems_. 101–110.
  * Reinhardt et al. (2012) A. Reinhardt, P. Baumann, D. Burgstahler, M. Hollick, H. Chonov, M. Werner, and R. Steinmetz. 2012.  On the accuracy of appliance identification based on distributed load metering data. In _Sustainable Internet and ICT for Sustainability (SustainIT)_. 1–9.
  * Schwermer et al. (2022) R. Schwermer, J. Buchberger, R. Mayer, and H.-A. Jacobsen. 2022.  Federated office plug-load identification for building management systems. In _13th ACM International Conference on Future Energy Systems_. 114–126.
  * Soe and Belleudy (2019) W. T. Soe and C. Belleudy. 2019.  Load Recognition from Smart Plug Sensor for Energy Management in a Smart Home. In _IEEE Sensors Applications Symposium (SAS)_. 1–6.
  * Song et al. (2022) H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Tanaka et al. (2018) D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa. 2018\.  Joint optimization framework for learning with noisy labels. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5552–5560.
  * United States Energy Information Administration (US EIA) ([n.d.]) United States Energy Information Administration (US EIA). [n.d.].  How much energy is consumed in U.S. buildings?  https://www.eia.gov/tools/faqs/faq.php?id=86&t=1 Accessed: 10 Apr 2021.
  * Vadakattu and Suthaharan (2018) V. Vadakattu and S. Suthaharan. 2018.  Feature extraction using apparent power and real power for smart home data classification. In _17th IEEE International Conference on Machine Learning and Applications (ICMLA)_. 1290–1295.
  * Veloso et al. (2019) A. F. d. S. Veloso, R. G. de Oliveira, A. A. Rodrigues, R. A. Rabelo, and J. J. Rodrigues. 2019\.  Cognitive Smart Plugs for Signature Identification of Residential Home Appliance Load using Machine Learning: From Theory to Practice. In _IEEE International Conference on Communications (ICC) Workshops_. 1–6.
  * Wang et al. (2019) Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey. 2019.  Symmetric cross entropy for robust learning with noisy labels. In _IEEE/CVF international conference on computer vision (ICCV)_. 322–330.
  * Wang et al. (2020) Z. Wang, G. Hu, and Q. Hu. 2020.  Training Noise-Robust Deep Neural Networks via Meta-Learning. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4524–4533.
  * Wang and Wang (2021) Z. Wang and H. Wang. 2021\.  Improving load forecast in energy markets during COVID-19. In _8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys)_. 168–171.
  * Xu et al. (2022) N. Xu, J.-Y. Li, Y.-P. Liu, and X. Geng.  Trusted-Data-Guided Label Enhancement on Noisy Labels.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Yan et al. (2019) D. Yan, Y. Jin, H. Sun, B. Dong, Z. Ye, Z. Li, and Y. Yuan.  Household appliance recognition through a Bayes classification model.  _Sustainable Cities and Society_ 46 (2019), 101393.
  * Zhang et al. (2022) Y. Zhang, G. Tang, Q. Huang, Y. Wang, K. Wu, K. Yu, and X. Shao.  Fednilm: Applying federated learning to nilm applications at the edge.  _IEEE Transactions on Green Communications and Networking_ (2022).