§ INTRODUCTION

§ RELATED WORK

§ METHOD

§.§ Training and Inference of Binary Neural Networks

§.§ Hardware Architecture

§.§ BNN Interpretability with Grad-CAM

The output of the convolutional layers in a CNN contains localized information of the input image, without any prior bias on the location of objects and features during training. This information can be captured using Class Activation Mapping (CAM) <cit.> and Gradient-weighted Class Activation Mapping (Grad-CAM) <cit.> techniques. To apply CAM, the model must end with a global average pooling layer followed by a fully-connected layer, providing the logits of a particular input. The BNN models investigated in this work operate on a small input resolution of 32$\times$32, and achieve a high reduction of spatial information without incorporating a global average pooling layer. For this reason, the Grad-CAM approach is better-suited to obtain visual interpretations of 's attention and determine the important regions for its predictions of different inputs and classes.

To obtain the class-discriminative localization map, we consider the activations and gradients for the output of the  layer (see Tab. <ref>), which has spatial dimensions of 5$\times$5. We use average pooling for the corresponding gradients and reduce the channels by performing Einstein summation as specified in <cit.>. With this approach the base networks do not need any modifications or retraining. Due to the synthetically generated dataset used for training, we expect  models to generalize well against domain shifts.

§ RESULTS AND DESIGN SPACE EXPLORATION

§.§ Experimental Setup

§.§ Design Space Exploration

§.§ Grad-CAM and Confusion Matrix Analysis

The confusion matrix in Fig. <ref> shows the generalization of -CNV on all classes after balancing the dataset. As expected, it is extremely rare to mistake nose+mouth exposed with a correctly worn mask. Less critically, nose and nose+mouth have a slight misclassification overlap, still at only 2% of the total samples given for each class. Finally, the chin exposed and the correct class have some sample misclassifications ($\leq$1%), which could be attributed to the chin area being small in some images and hard to detect at low-resolution.

[Correctly masked Grad-CAM]

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

[Nose exposed Grad-CAM]

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

[Mouth + nose exposed Grad-CAM]
2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

[Chin exposed Grad-CAM]

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

Grad-CAM output of two  variants and a float-32 (FP32) CNN. Results are collected for all four wearing positions on a diverse set of individuals. Binarized models show distinct regions of interest which are focused on the exposed part of the face rather than the mask. The FP32 model is difficult to interpret in some cases.

We further analyze the output heat maps generated by Grad-CAM to interpret the predictions of our BNNs with respect to the diverse attributes of the MaskedFace-Net dataset. In Fig. <ref> and Fig. <ref> - Fig. <ref>, column 1 and 2 indicate the label and input image respectively. Columns 3, 4 and 5 highlight the heat maps obtained from the Grad-CAM output of -CNV, -$n$-CNV and a full-precision version of CNV with float-32 parameters (FP32). The heat maps are overlaid on the raw input images for better visualization. All raw images chosen have been classified correctly by all the networks, for fair interpretation of feature-to-prediction correlation.

In Fig. <ref>(a), we analyze the Region of Interest (RoI) for the correctly masked class. 's learning capacity allows it to focus on key facial lineaments of the human wearing the mask, rather than the mask itself. This potentially helps in generalizing on other mask types. For the child example shown in the first row, the focus of  lies on the nose area, asserting that it is fully covered to result in a correctly masked prediction. Similarly, for the adult in row 2, -CNV focuses on the upper edge of the mask, to predict its coverage of the face. This also holds for our small version of , with significantly reduced learning capacity. The RoI curves finely above the mask, tracing the exposed region of the face. In the third row example, -CNV falls back to focusing on the mask, whereas -$n$-CNV continues to focus on the exposed features. Both models achieve the same prediction by focusing on different parts of the raw image. In contrast to the  variants, the full-precision FP32 model seems to focus on a combination of several different features on all three examples. This can be attributed to its larger learning capacity and possible overfitting.

In Fig. <ref>(b), we analyze the Grad-CAM output of the uncovered nose class. -CNV and -$n$-CNV focus specifically on two regions, namely the nose and the straight upper edge of the mask. These clear characteristics cannot be observed with the oversized FP32 CNN. In Fig <ref>(c), the results show the RoI for predicting the exposed mouth and nose class. All models seem to distribute their attention onto several exposed features of the face. Fig. <ref>(d) shows Grad-CAM results for chin exposed predictions. Although the top region of the mask points upwards, similar to the correctly worn mask, the BNNs pay less attention to this region and instead focus on the neck and chin. With the full-precision FP32 model, it is difficult to interpret the reason for the correct classification, as little to no focus is given to the chin region, again hinting at possible overfitting.

Beyond studying the BNNs' behavior on different class predictions, we can use the attention heat maps to understand the generalization behavior of the classifier. In Fig. <ref> - Fig. <ref>, we test 's generalization over ages, hair colors and head gear, as well as complete face manipulation with double-masks, face paint and sunglasses. In Fig <ref>, we see that the smaller eyes of infants and elderly do not hinder 's ability to focus on the top region of the correctly worn masks. In Fig. <ref>, -CNV shows resilience to differently colored hair and head-gear, even when having a similar light-blue color as the face-masks (row 2 and 3). In contrast, the FP32 model's attention seems to shift towards the hair and head-gear for these cases. Finally, in Fig. <ref>, both  variants focus on relevant features of the corresponding label, irrespective of the obscured or manipulated faces. This empirically shows that the complex training of BNNs, along with their lower information capacity, constrains them to focus on a smaller set of relevant features, thereby generalizing well for unprecedented cases.

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

Grad-CAM results for age generalization.

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

Grad-CAM results for hair/headgear generalization.

2*Label     2*Raw     BCoP     BCoP     2*FP32

CNV     $n$-CNV

Grad-CAM results for face manipulation with double-masks, face paint and sunglasses.

§.§ Discussion and Comparison with Other Works

As mentioned in Sec <ref>, detection of masks has piqued the interest of many researchers in the computer vision domain due to its relevance in the context of the ongoing COVID-19 pandemic. NVIDIA proposed mask recognition using object detection models <cit.>. These models require INT8 or Float-16 numerical precision, with ResNet-18 as a backbone for input images of 960$\times$544. The complexity is orders of magnitude higher than the models we propose in this paper. A head-to-head comparison is difficult to make due to differences in the training approach, the CNN model, the datasets used and the application requirements. The networks are trained to predict only two classes (mask, no mask), which is a simpler problem compared to the exact positioning supported by . However, the localization and higher resolution makes it a more complex task overall. With the NVIDIA Jetson Nano hardware, which typically requires $\sim$10W of power on intensive workloads, a frame-rate of 21 FPS is achieved. The more powerful 25W Jetson AGX Xavier can achieve up to 508 FPS. Compared to the NVIDIA approach <cit.>,  is targeted at low-power, embedded applications with peak inference power of $\sim$2W and high classification rates of up to $\sim$6400 FPS on smaller resolution input images. It is worth noting that  can also classify high resolution images containing multiple individuals, by slicing the input into many 32$\times$32 frames and batch processing them. This application makes use of the high-throughput results presented in Tab. <ref>. Another approach proposed by Agarwal et al. <cit.> achieves the task of detecting a range of personal protective equipment (PPE). Processing takes place on cloud servers, which could raise privacy and data safety concerns in public settings. Wang et al. <cit.> propose an in-browser server-less edge computing method, with object detection models. The browser-enabled device must support the WebAssembly instruction format. The authors benchmarked their approach on an iPad Pro (A9X), an iPhone 11 (A13) and a MacBook pro (Intel i7-9750H), achieving 5, 10 and 20 FPS respectively. Needless to say, these devices (or similar) are expensive and cannot be placed in abundance in public areas. Similarly, <cit.> offers an Android application solution, which is suitable for users self-checking their masks. In this case, low-power, edge-hardware and continuous surveillance are not emphasized.

Our approach offers a unique, low-power, high-throughput solution, which is applicable to cheap, embedded FPGAs. Moreover, the  solution is not constrained to FPGA platforms. Software-based inference of  is also possible on other low-power microcontrollers, with binary instructions. Training on synthetic data allows us to generate more samples with different mask colors, shapes, and sizes <cit.>, further improving the generalizability of the BNNs, while keeping real-world data available for fine-tuning stages.

§ CONCLUSION