# ALET∗ (Automated Labeling of Equipment and Tools): A Dataset for Tool
Detection and Human Worker Safety Detection

Fatih Can Kurnaz1, Burak Hocaog̃lu1, Mert Kaan Yılmaz1, İdil Sülo1, and Sinan
Kalkan1
* The world ‘alet’ means ‘tool’ in Turkish. 1 KOVAN Research Lab, Dept. of Computer Engineering, Middle East Technical University, Ankara, Turkey {fatih.kurnaz, burak.hocaoglu, kaan.yilmaz, idil.sulo<EMAIL_ADDRESS>
###### Abstract

Robots collaborating with humans in realistic environments need to be able to
detect the tools that can be used and manipulated. However, there is no
available dataset or study that addresses this challenge in real settings. In
this paper, we fill this gap with a dataset for detecting farming, gardening,
office, stonemasonry, vehicle, woodworking, and workshop tools. The scenes in
our dataset are snapshots of sophisticated environments with or without humans
using the tools. The scenes we consider introduce several challenges for
object detection, including the small scale of the tools, their articulated
nature, occlusion, inter-class invariance, etc. Moreover, we train and compare
several state of the art deep object detectors (including Faster R-CNN,
Cascade R-CNN, RepPoint, and RetinaNet) on our dataset. We observe that the
detectors have difficulty in detecting especially small-scale tools or tools
that are visually similar to parts of other tools. In addition, we provide a
novel, practical safety use case with a deep network which checks whether the
human worker is wearing the safety helmet, mask, glass, and glove tools. With
the dataset, the code and the trained models, our work provides a basis for
further research into tools and their use in robotics applications. The
dataset, the trained networks, and all associated codes will be made available
at: https://github.com/metu-kovan/METU-ALET.

##  I INTRODUCTION

The near future will see a cohabitation of robots and humans, where they will
work together for performing tasks that are especially challenging, tiring or
unergonomic for humans. This requires robots to have abilities for perceiving
the humans, the task at hand, and the environment. An essential perceptual
component for these abilities is the detection of objects, especially the
tools.

Figure 1: Samples from the METU-ALET dataset, illustrating the wide range of
challenging scenes and tools that a robot is expected to recognize in a
clutter, possible with human co-workers using the tools. Since annotations are
too dense, only a small subset is displayed. [Best viewed in color]

The robotics community has paid marginal importance to tools that are used by
humans. For example, there are studies focusing on affordances of tools, or on
the detection and transfer of these affordances [1, 2, 3]. However, these
studies considered tools mostly in isolated and limited, toy environments.
Moreover, they have considered only a limited set of tools (see Table I). What
is more, the literature has not studied detection of tools, nor is there a
dataset available for it.

In this paper, we focus on the detection of tools in realistic, cluttered
environments (e.g. like those in Fig. 1) where collaboration between humans
and robots is expected. To be more specific, we study detection of tools in
real work environments that are composed of many objects (tools) that look
alike and that occlude each other. For this end, we first collect an extensive
tool detection dataset composed of 49 tool categories. Then, we compare the
widely used state-of-the-art object detectors on our dataset, as a baseline.
The results suggest that detecting tools is very challenging owing to tools
being too small and articulated, and bearing too much inter-class similarity.
Finally, we introduce a safety usecase from our dataset and train a novel CNN
network for this task.

The necessity for a dataset for tool detection: Tool detection requires a
dataset of its own since it bares novel challenges of its own: (i) Many tools
are small objects which elicit a problem to standard object detectors that are
tuned for detecting moderately larger objects. (ii) Many tools are
articulated, and in addition to viewpoint, scale and illumination changes,
object detectors need to address invariance to articulation. (iii) Tools are
generally used in highly cluttered environments posing challenges on clutter,
occlusion, appearance, and illumination – see Figure 1 for some samples. (iv)
Many tools exhibit low inter-class differences (e.g., between screwdriver,
chisel and file or between putty knives and scraper).

TABLE I: A comparison of the datasets that include tools. The Epic-Kitchens dataset provides videos, which makes analysis of scenes difficult. Moreover, the figures for the Epic-Kitchens dataset are estimated based on the provided data. *SO denotes the number of scenes that only include a single object.  Dataset |  |  Tool
---
Categories
|  Tool
---
Classes
|  # of
---
Images
|  # Instances
---
per Tool
Modality |  |  Dense
---
Bounding Boxes?
|  RGB-D Part Aff. [1]
---
| Kitchen, Workshop,
---
Garden
17 |  | 3
---
(SO*: 102)
6.17 | RGB-D | No
|  ToolWeb [4]
---
| Kitchen, Office,
---
Workshop
23 |  | 0
---
(SO*: 116)
5.03 | 3D model | No
|  Visual Aff. of Tools [5]
---
Toy | 3 |  | 0
---
(SO*: 5280)
377 |  | RGB stereo
---
(inc. semantic map)
No
|  Epic-Kitchens [6]
---
Kitchen |  60+ |  | NA
---
(Videos)
200+ |  | RGB stereo
---
(inc. semantic map)
Yes
METU-ALET (Ours) |  | Farm, Garden, Office,
---
Stonemasonry, Vehicles,
Woodwork, Workshop
49 |  | 2699
---
(SO*: 0)
200+ | RGB | Yes

###  I-A Related Work

Object Detection: Object detection is one of the most studied problems in
Computer Vision with many practical applications in many robotics scenarios.
Object detection generally follows a two-stage approach: (i) Region selection,
which pertains to the selection of image regions that are likely to contain an
object. (ii) Object classification, which deals with the classification of a
selected region into one of the object categories. With advances in deep
learning both stages have seen tremendous boost in performance in many
challenging settings, e.g., [7, 8, 9]. Faster R-CNN [9] is a well-known
representative of such two-stage detectors.

It has been shown that the two stages can be combined and objects can be
detected in one stage. Models such as [10, 11, 12] assume a fixed set of
localized image regions (anchors) for each object category, and estimate
objectness for each category and for each anchor. Among these, RetinaNet [10]
processes features and make classification at multiple scales (called a
feature pyramid network) and combines the results, yielding the state of the
art results among one-stage detectors.

These one-stage and two-stage detectors have a top-down approach. In contrast,
bottom-up object detection forms a detection pipeline from points or keypoints
that are likely to be on objects and identifies objects by combining such
points (examples include CenterNet [13], RepPoints [14]).

The current trend in deep object detectors is to take a pre-trained object
classification network as the feature extractor (called the backbone network),
then perform two-stage or one-stage object detection from those features.
Alternatives for the backbone network include deep classification networks
such as VGG [15], ResNet [16] and ResNext [17].

Tools in Robotics: The robotics community has extensively studied how tools
can be grasped, manipulated, and how such affordances can be transferred
across tools. For example, Kemp and Edsinger [18] focused on detection and the
3D localization of tool tips from optical flow. For the same goal, Mart et al.
[3] proposed using a CNN network (AlexNet) to first classify a blob into one
of the three tool labels they considered, and then used 3D geometric features
to identify tool tips.

Another study [1] addressed the problem of estimating the grasping positions,
scoops and supports of tools from RGB-D data. In a similar setting, Mar et al.
[19, 2] studied prediction of affordances for different categories of tools
separately. For this purpose, they first clustered tools using their 3D
geometric descriptors, and then estimated the affordances of each cluster
separately.

Related datasets: Comparing our dataset to the ones used in the robotics
literature (e.g., [1, 4, 5] – see also Table I), we see that they are limited
in the number of categories and the instances that they consider. Moreover,
since they mainly focus on detection of tool affordances, they are not
suitable for training a deep object detector.

For related problems, there are numerous datasets of objects in the robotics
literature, e.g. for 3D pose estimation and robot manipulation (e.g., LINEMOD
[20], YCB Objects [21], Table-top Objects [22], Object Recognition Challenge
[23]). These datasets generally include table-top objects with 3D models and
do not include tool categories.

For object detection, there are several datasets such as PASCAL [24], MS-COCO
[25] and ImageNet [26], which do include some tool categories (e.g., hammer,
scissors); however, these datasets are designed to be for general purpose
objects and scenes, unlike the ones we expect to see in tool-used environments
(such as the ones in Fig. 1). Therefore, they do not provide sufficient amount
of training instances for training a general purpose tool detector with a
reasonable performance.

Safety Detection: There are studies on detecting whether or not a human worker
is wearing safety helmet [27, 28] or a vest [29]. However, there are no
studies or datasets that include safety glass, mask, glove and headphone
together with helmet.

###  I-B Contributions of Our Work

The main contributions of our work can be summarized as follows:

  * •

A Tool Detection Dataset: To the best of our knowledge, ours is the first to
provide a dataset on detection of tools in the wild.

  * •

A Baseline for Tool Detection: On our dataset, we train and analyze many
state-of-the-art deep object detectors. Together with the dataset, the code
and the trained models, our work can form as a basis for robotics applications
that require detection of tools in challenging realistic work environments
with humans.

  * •

A Novel Usecase for Checking the Safety of Human Coworkers: Our dataset
includes humans performing tasks with and without wearing a safety helmet,
mask, glass, headphone and gloves. We form a ALET Safety subset of positive
and negative instances of these and train a deep CNN network that checks
whether a human coworker is wearing these safety tools.

##  II METU-ALET: A TOOL DETECTION DATASET

In this section, we present and describe the details of METU-ALET and how the
dataset was collected.

###  II-A Tools and Their Categories

In ALET, we consider 49 different tools that are used for six broad contexts
or purposes: Farming, gardening, office, stonemasonry, vehicle, woodworking,
workshop tools111It is better to call some of these objects as equipment.
However, since they provide similar functionalities (being used by a human or
a robot while performing a task), we will just use the term tool to refer to
all such objects, for the sake of simplicity.. The 20 most frequent tools from
our dataset are: Chisel, Clamp, Drill, File, Gloves, Hammer, Mallet, Meter,
Pen, Pencil, Plane, Pliers, Safety glass, Safety helmet, Saw, Screwdriver,
Spade, Tape, Trowel, Wrench.

We excluded tools used in kitchen since there is already an exclusive dataset
for this purpose [6]. Moreover, we limited ourselves to tools that can be
ultimately grasped, pushed, or manipulated in an easy manner by a robot.
Therefore, we did not consider tools such as ladders, forklifts, and power
tools that are bigger than a hand-sized drill.

###  II-B Dataset Collection

Our dataset is composed of three groups of images:

  * •

Images collected from the web: Using keywords and usage descriptions that
describe the tools listed in Section II-A, we crawled and collected royalty-
free images from the following websites: Creativecommons, Wikicommons, Flickr,
Pexels, Unsplash, Shopify, Pixabay, Everystock, Imfree.

  * •

Images photographed by ourselves: We captured photos of office and workshop
environments from our campus.

  * •

Synthetic images: In order to make sure that there are at east 200 instances
for each tool, we developed a simulation environment and collected synthetic
images (see Fig. 2 for examples). For this, we used the Unity3D platform with
3D models of tools acquired from UnityStore. For each scene to be generated,
the following steps are followed:

    * –

Scene Background: We created a room like environment with 4 walls, 10
different random objects (chair, sofa, corner-piece, television) in static
positions. At the center of the room, we spawned one of six different tables
selected randomly from $Uniform(1,6)$. To introduce more randomness, we also
dropped unrelated objects like mugs, bottles etc. randomly.

    * –

Camera: Each dimension of the camera position (x, y, z) was sampled randomly
from $Uniform(-3,3)$. Camera’s viewing direction was set towards the center of
the top of the table.

    * –

Tools: In each scene, we spawned $N\sim Uniform(5,20)$ tools which are
selected randomly from $Uniform(1,49)$. The spawn tools are dropped onto the
table from $[x,y,z]$ selected randomly from $Uniform(0,1)$ above the table.
Initial orientation (each dimension) is sampled from $Uniform(0,360)$.

The special cases when the sampled camera not seeing the table-top etc. are
handled using hand-designed rules. Some examples from this process can be seen
in Figure 2.

For annotating the tools in the downloaded and the photographed images, we
used the VGG Image Annotation (VIA) tool [30]. Annotation was performed by the
authors of the paper.

Figure 2: Some examples from the Synthetic Images. [Best viewed in color]

###  II-C Dataset Statistics

In this section, we provide some descriptive statistics about the METU-ALET
dataset.

Cardinality and Sizes of BBs: The METU-ALET dataset includes 22,835 bounding
boxes (BBs). As displayed in Fig. 3, for each tool category, there are more
than 200 BBs, which is on an order similar to the widely used object detection
datasets such as PASCAL [24]. As shown in Table II, METU-ALET includes tools
that appear small (area $<32^{2}$), medium ($32^{2}<$ area $<96^{2}$) and
large ($96^{2}<$ area) – following the naming convention from MS-COCO [25].
Although this is expected, as we will see in Section IV, deep networks have
difficulty especially detecting small tools.

Figure 3: The distribution of bounding boxes across classes. With the photographed and synthesized photos, each tool category has more than 200 bounding boxes. [Best viewed in color] TABLE II: The sizes of the bounding boxes (BB) of the annotated tools in METU-ALET. For calculating these statistics, we considered the scaled versions of the images that were fed to the networks, namely, $1333x800$. |  Subset
---
Category
Small BBs | Medium BBs | Large BBs | Total
Downloaded | 809 | 4650 | 5661 | 11114
Photographed | 13 | 309 | 443 | 765
Synthesized | 813 | 6934 | 3209 | 10956
Total | 1629 | 11893 | 9313 | 22835

Cardinality and Sizes of the Images: Our dataset is composed of 2699 images in
total, and on average, has size $1138\times 903$. See Table III for more
details. Although the number of images may appear low, the number of bounding
boxes (22835) is sufficient since there are more than 200 BBs per tool, and
the avg. number of BBs per image is rather large ($6.6$) compared to PASCAL
2012 ($2.3$).

TABLE III: The cardinality and the resolution of the images in METU-ALET.  Subset | Cardinality | Avg. Resolution
---|---|---
Downloaded | 1870 | $924\times 786$
Photographed | 89 | $3663\times 3310$
Synthesized | 740 | $1374\times 917$
Total/Avg | 2699 | $1138\times 903$

###  II-D ALET Safety Subset

In this section, we illustrate how the ALET dataset can be used for addressing
a critical issue in human-robot collaborative environments; that of checking
whether the human worker is conforming to the security guidelines and wearing
the required safety tools. ALET dataset contains a good number of safety tool
(“helmet, glass, mask, headphone and glove”) instances ($2159$ in total just
in real images, and $3104$ when combined with synthetic examples), which
suffice for training a deep classifier.

For forming the safety subset, first we used OpenPose [31] to both detect the
humans in an image and estimate their 2D poses. Then, we compared the
positions of safety tools with the positions of the corresponding joints (e.g.
for glove, the hand joints are used whereas head joints are used for the
helmet etc.). If, for a tool, the Euclidean distance between the positions
(normalized wrt. size of the BB of the human) were less than 0.2, the human
was considered wearing that safety tool.

##  III METHODOLOGY

In this section, we briefly describe the deep object detectors that we
evaluated as a baseline, and the safety as a straightforward usecase of our
dataset.

###  III-A Deep Object Detectors

As stated in Section I-A, top-down deep object detectors can be broadly
analyzed in two categories: (i) single-stage detectors, and (ii) multi-stage
detectors. To form a baseline, we evaluated strong representatives of both
single-stage (RetinaNet) and multi-stage (Faster R-CNN, Cascade R-CNN)
detectors. Moreover, we included RepPoints, a recent bottom-up object
detector.

Faster R-CNN. Faster R-CNN [9] is one of the first networks to use end-to-end
learning for object detection. It feeds features extracted from a backbone
network to a region proposal network, which estimates an objectness score and
the (relative) coordinates of a set of $k$ anchor boxes for each position on a
regular grid. For each such box with an objectness score above a threshold,
the object classification network (Fast R-CNN) is executed to classify each
box into one of the object categories.

For training the network, classification loss and box-regression loss (to
penalize the spatial mismatch between the detected box and the ground truth
box) are combined. The box-regression loss is weighted with a constant
($\lambda$), which we selected as 1.0 as suggested by the paper.

Cascade R-CNN. Cascade R-CNN [32] utilizes a region-proposal network with
multiple detection stages with an increasing amount of IoU threshold for each
of them. By doing so it eliminates negative samples better in each detection
stage while increasing its IoU threshold in each further detection stage.

RetinaNet. RetinaNet [10] is a one-stage detector which forms a multi-scale
pyramid from the features obtained from the backbone network and performs
classification and bounding box estimation in parallel for each layer (scale)
of the pyramid. In order to address the data imbalance problem that affects
single-stage detectors owing to background, RetinaNet proposes using focal
loss that decreases the contribution of the “easy” examples to the overall
loss. Compared to other single-stage detectors, RetinaNet considers a denser
set of bounding boxes to classify.

RepPoint Detection. Contrary to other approaches, RepPoint [14] is an anchor
free, bottom-up detector. It is based on identifying representative points on
objects and then combining these points into bounding boxes.

###  III-B A Safety Usecase for ALET

We created a CNN architecture consisting of three 2D convolutional layers and
two fully connected layers. After each convolutional layer we added a batch
normalization layer, and each layer is also followed by ReLu activation. The
final layer has five outputs with sigmoid activation. The network performs
five-class (one for each safety tool) multi-label classification with binary
cross-entropy. The network is trained on ALET Safety Dataset.

An alternative approach could be to combine the results of the tool detector
and the pose detector. However, considering that the tool detection networks
are having acceptable performance, we adopted an independent network for
safety detection. Moreover, a tool detector would be detecting 49 tools in a
scene 43 of which are irrelevant for our safety usecase.

##  IV EXPERIMENTS

###  IV-A Training and Implementation Details

We split the ALET dataset (unless otherwise stated, this means real and
synthetic images combined together) into $2112$ ($\%80$) training, $268$
($\%10$) validation and $264$ ($\%10$) testing samples. For training each
network, the following libraries and settings are used (in all networks, the
output layer is replaced with the tool categories, and the whole network
except for the feature extracting backbone is updated during training): For
each detector, the pre-trained network from mmdetection [33] is used with
backbone ResNet-50-FPN.

###  IV-B Quantitative Results for Tool Detection

On the testing subset of ALET, we compare the performance of the detectors
trained on the training subset of ALET. We use the MS COCO style average
precision (AP).

Table IV lists the AP and mAP values of the baseline networks on our dataset.
We notice that the baseline detectors perform well on tools that are very
distinctive and different from others; e.g. tape-dispenser, safety-helmet,
hole-punch, pencil-sharpener. On such tools, we observe AP of up to 84.057
(Cascade R-CNN for pencil-sharpener).

However, we see that the detectors have trouble in detecting especially tools
that are too narrow, like pen-pencil, knife, file. There are several reasons
for these results: (i) Object detectors have been designed for general object
detection and need to extended to consider a wider range of anchors with a
wider range of aspect ratios, which would increase the complexities of the
networks. (ii) A second reason is that annotated boxes of small tools such as
pen-pencil include more pixels of other objects than the annotated small
object itself. (iii) Moreover, tools such as screwdriver, chisel, file look
very similar to each other. Moreover, these tools appear very similar to parts
of other tools from a side view or from far (e.g., the front part of a drill
is likely to be classified as a screwdriver, and in many cases, one half of a
plier is detected as a chisel).

These suggest that tool detection is indeed a very challenging problem
especially owing to small tools and tools having very similar appearances to
other tools.

Table V evaluates the performance improvement provided by the synthetic images
from the simulation environment. We compared our dataset in with and without
synthetic images by using RetinaNet and it shows that synthetic images elevate
our dataset. The results for other networks are being produced and will be
included in the final version.

TABLE IV: MS COCO style AP of the baseline networks. Training and testing are both performed with the synthetic + real images.  |  Anvil |  Aviation-snip |  Axe |  Bench-vise |  Brush |  Caliper |  Caulk-Gun |  Chisel |  Clamp |  Crowbar |  Drill |  File |  Flashlight |  Gloves |  Grinder |  Hammer |  Hex-keys |  Hole-punch |  Knife |  Level |  Mallet |  Marker |  Meter |  Pen |  Pencil |  Pencil-sharpener |  Plane |  Pliers |  Putty-knives |  Rake |  Ratchet |  Riveter |  Ruler |  Safety-glass |  Safety-headphone |  Safety-helmet |  Safety-mask |  Saw |  Scissors |  Screwdriver |  Soldering-Iron |  Spade |  Square |  Staple-gun |  Stapler |  Tape |  Tape-dispenser |  Trowel |  Wrench |  mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
F. R-CNN |  72.459 |  10.055 |  32.406 |  52.399 |  10.387 |  39.164 |  29.794 |  9.846 |  27.552 |  33.349 |  33.041 |  8.015 |  44.904 |  27.469 |  47.988 |  19.544 |  21.196 |  53.863 |  16.563 |  19.478 |  24.066 |  20.332 |  21.664 |  10.174 |  3.615 |  79.609 |  40.590 |  26.324 |  35.579 |  34.509 |  21.822 |  19.221 |  6.183 |  19.673 |  55.061 |  54.212 |  43.853 |  18.990 |  15.883 |  16.548 |  32.060 |  26.620 |  16.801 |  32.707 |  47.723 |  31.348 |  44.904 |  12.473 |  23.657 |  29.504
Cascade |  76.828 |  12.192 |  43.472 |  55.545 |  13.906 |  44.049 |  29.496 |  11.990 |  34.169 |  42.046 |  35.007 |  11.296 |  51.400 |  28.866 |  47.635 |  23.255 |  18.630 |  60.845 |  19.880 |  26.841 |  33.148 |  21.388 |  27.889 |  10.981 |  8.870 |  84.057 |  45.020 |  26.691 |  35.582 |  43.111 |  19.815 |  26.048 |  3.957 |  25.455 |  58.154 |  56.597 |  46.912 |  20.759 |  19.303 |  21.622 |  38.022 |  29.994 |  18.053 |  43.114 |  51.709 |  30.281 |  52.868 |  19.746 |  29.843 |  33.395
RepPoint |  74.244 |  8.378 |  37.624 |  58.743 |  13.565 |  40.194 |  19.457 |  10.765 |  27.696 |  32.818 |  35.832 |  5.226 |  40.098 |  26.825 |  43.463 |  17.260 |  18.407 |  57.106 |  18.337 |  22.725 |  23.849 |  18.882 |  20.667 |  5.018 |  7.078 |  77.734 |  35.258 |  26.483 |  24.412 |  34.455 |  15.143 |  14.223 |  3.389 |  21.542 |  52.697 |  53.013 |  48.066 |  20.072 |  11.207 |  15.918 |  28.515 |  22.724 |  18.675 |  25.806 |  42.978 |  26.009 |  45.661 |  10.115 |  22.828 |  28.187
RetinaNet |  74.221 |  3.688 |  25.142 |  52.970 |  2.825 |  31.939 |  10.034 |  7.690 |  24.473 |  23.042 |  30.572 |  4.764 |  32.114 |  25.463 |  44.273 |  14.123 |  17.078 |  55.004 |  10.947 |  19.545 |  19.147 |  13.013 |  15.904 |  2.025 |  4.086 |  76.220 |  29.780 |  20.118 |  20.658 |  31.445 |  11.851 |  11.881 |  1.088 |  16.012 |  53.298 |  51.680 |  48.647 |  8.440 |  11.380 |  8.178 |  22.776 |  11.488 |  13.854 |  30.117 |  46.282 |  27.571 |  43.356 |  6.855 |  11.294 |  24.048
TABLE V: The gain obtained for RetinaNet using the synthetic examples generated from our simulation environment.  | Training w | Training w
---|---|---
| Real Data | Real+Synthetic Data
mAp on Real+Syn test images | $11.980$ | $20.118$

###  IV-C Sample Tool Detection Results

In Fig. 4, we display a few detection results on a few of the challenging
scenes in Fig. 1. We see that although many tools are detected, many are
missed.

Figure 4: Sample detections from RetinaNet (w 0.4 as confidence threshold) on
a few of the challenging scenes from Fig. 1. In the top example, almost all
detections are correct. However, detector detected screwdriver and pliers in
place of pens, which can be attributed to similarity in size and color. In the
second image detector wrongfully annotates a hat as safety-helmet. In
addition, it misses some of the crowded wrench examples, which is caused by
the complexity of the problem.

###  IV-D Safety Usecase Results

In this usecase, we analyze the method proposed in Section III-B. Our CNN
network obtained $0.50$ F1 score on the test set, and as illustrated with some
examples in Fig. 5 and accuracy results from Table VI, we see that a network
and a human detector & pose estimator can be used to easily identify whether a
human is wearing a safety equipment or not. However, the network is having
difficulty with safety glasses and gloves, because, as illustrated in Fig. 5,
the region around eyes may not be visible for a working man and hands tend to
be outside the BB in some cases.

TABLE VI: Class accuracies for the safety network.  | Helmet | Gloves | Mask | Headphone | Glass
---|---|---|---|---|---
Accuracy | $75\%$ | $69\%$ | $85\%$ | $85\%$ | $54\%$

Figure 5: Sample results for the safety usecase. Symbols: H: Helmet, M: Mask,
G: Gloves, L: Glass, D: Headphone. [Best viewed in color]

##  V CONCLUSION

In this paper, we have introduced an extensive dataset for tool detection in
the wild. Moreover, we formed a baseline by training and testing four widely-
used state-of-the-art deep object detectors in the literature, namely, Faster
R-CNN [9], Cascade R-CNN [32], and RetinaNet [10] and RepPoint Detection [14].
We demonstrated that such detectors especially have trouble in finding tools
whose appearance is highly affected by viewpoint changes and tools that
resemble parts of other tools.

Moreover, we have provided a very practical yet critical usecase for human-
robot collaborative scenarios. Combining the detected “helmet, glass, mask,
headphone and gloves” categories with the detection results of a human
detector & pose estimator, we have demonstrated how our dataset can be used
for practical applications other than merely detecting tools in an
environment.

## ACKNOWLEDGMENT

This work was supported by the Scientific and Technological Research Council
of Turkey (TÜBİTAK) through project called “CIRAK: Compliant robot manipulator
support for montage workers in factories” (project no 117E002). The numerical
calculations reported in this paper were partially performed at TÜBİTAK
ULAKBIM, High Performance and Grid Computing Center (TRUBA resources). We
would like to thank Erfan Khalaji for his contributions on an earlier version
of the work.

## References

  * [1] A. Myers, C. L. Teo, C. Fermüller, and Y. Aloimonos, “Affordance detection of tool parts from geometric features,” in _ICRA_ , 2015.
  * [2] T. Mar, V. Tikhanoff, and L. Natale, “What can i do with this tool? self-supervised learning of tool affordances from their 3-d geometry,” _TCDS_ , vol. 10, no. 3, pp. 595–610, 2018.
  * [3] T. Mar, L. Natale, and V. Tikhanoff, “A framework for fast, autonomous and reliable tool incorporation on icub,” _Frontiers in Robotics and AI_ , vol. 5, p. 98, 2018.
  * [4] P. Abelha and F. Guerin, “Learning how a tool affords by simulating 3d models from the web,” in _IROS_ , 2017.
  * [5] A. Dehban, L. Jamone, A. R. Kampff, and J. Santos-Victor, “A moderately large size dataset to learn visual affordances of objects and tools using icub humanoid robot,” in _ECCV Workshop on Action and Anticipation for Visual Learning_ , 2016.
  * [6] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling egocentric vision: The epic-kitchens dataset,” in _ECCV_ , 2018.
  * [7] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _CVPR_ , 2014\.
  * [8] R. Girshick, “Fast r-cnn,” in _ICCV_ , 2015.
  * [9] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _NIPS_ , 2015.
  * [10] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _ICCV_ , 2017.
  * [11] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _ECCV_ , 2016.
  * [12] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _CVPR_ , 2016.
  * [13] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _ICCV_ , 2019.
  * [14] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin, “Reppoints: Point set representation for object detection,” in _ICCV_ , 2019.
  * [15] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv:1409.1556_ , 2014.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [17] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _CVPR_ , 2017.
  * [18] C. C. Kemp and A. Edsinger, “Robot manipulation of human tools: Autonomous detection and control of task relevant features,” in _ICDL_ , 2006.
  * [19] T. Mar, V. Tikhanoff, G. Metta, and L. Natale, “Multi-model approach based on 3d functional features for tool affordance learning in robotics,” in _Humanoids_ , 2015.
  * [20] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in _ACCV_ , 2012.
  * [21] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols,” _arXiv:1502.03143_ , 2015.
  * [22] M. Sun, G. Bradski, B.-X. Xu, and S. Savarese, “Depth-encoded hough voting for joint object detection and shape recovery,” in _ECCV_ , 2010.
  * [23] N. Vaskevicius, K. Pathak, A. Ichim, and A. Birk, “The jacobs robotics approach to object recognition and localization in the context of the icra’11 solutions in perception challenge,” in _ICRA_ , 2012.
  * [24] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _IJCV_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [25] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _ECCV_ , 2014.
  * [26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR_ , 2009.
  * [27] K. Li, X. Zhao, J. Bian, and M. Tan, “Automatic safety helmet wearing detection,” _arXiv:1802.00264_ , 2018.
  * [28] J. Wu, N. Cai, W. Chen, H. Wang, and G. Wang, “Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset,” _Automation in Construction_ , vol. 106, p. 102894, 2019.
  * [29] N. D. Nath, A. H. Behzadan, and S. G. Paal, “Deep learning for site safety: Real-time detection of personal protective equipment,” _Automation in Construction_ , vol. 112, p. 103085, 2020.
  * [30] A. Dutta, A. Gupta, and A. Zissermann, “VGG image annotator (VIA),” http://www.robots.ox.ac.uk/ vgg/software/via/, 2016, version: 2.0.5, Accessed: 27 Feb 2019.
  * [31] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh, “OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields,” in _arXiv:1812.08008_ , 2018.
  * [32] Z. Cai and N. Vasconcelos, “Cascade R-CNN: delving into high quality object detection,” _arxiv 1712.00726_ , 2017.
  * [33] K. Chen, J. Wang, and J. e. a. Pang, “MMDetection: Open mmlab detection toolbox and benchmark,” _arXiv:1906.07155_ , 2019.