# Embedded Planogram Compliance Control System

M. Erkin Yücel
R&D
Migros Ticaret A.Ş.
Istanbul, Turkey
<EMAIL_ADDRESS>
& Serkan Topaloğlu
Faculty of Engineering, Department of Electrical and Electronics Engineering
Yeditepe University
Istanbul, Turkey
<EMAIL_ADDRESS>
& Cem Ünsalan
Faculty of Engineering, Department of Electrical and Electronics Engineering
Yeditepe University
Istanbul, Turkey
<EMAIL_ADDRESS>

###### Abstract

Smart retail stores are becoming an integral part of our lives. Various
computer vision and sensor-based systems collaborate to enable these complex
and automated operations. Furthermore, the retail sector presents several open
and challenging problems that could benefit from advanced pattern recognition
and computer vision techniques. One such critical challenge is planogram
compliance control. In this study, we propose a complete embedded system to
tackle this issue. Our system consists of four key components as image
acquisition and transfer via stand-alone embedded camera module, object
detection via computer vision and deep learning methods working on single
board computers, planogram compliance control method again working on single
board computers, and energy harvesting and power management block to accompany
the embedded camera modules. The image acquisition and transfer block is
implemented on the ESP-EYE camera module. The object detection block is based
on YOLOv5 as the deep learning method and local feature extraction. We
implement these methods on Raspberry Pi 4, NVIDIA Jetson Orin Nano, and NVIDIA
Jetson AGX Orin as single board computers. The planogram compliance control
block utilizes sequence alignment through a modified Needleman-Wunsch
algorithm. This block is also working along with the object detection block on
the same single board computers. The energy harvesting and power management
block consists of solar and RF energy harvesting modules with suitable battery
pack for operation. We tested the proposed embedded planogram compliance
control system on two different datasets to provide valuable insights on its
strengths and weaknesses. The results show that our method achieves F1 scores
of 0.997 and 1.0 in object detection and planogram compliance control blocks,
respectively. Furthermore, we calculated that the complete embedded system can
work in stand-alone form up to two years based on battery. This duration can
be further extended with the integration of the proposed solar and RF energy
harvesting options.

_Keywords_ embedded camera modules, computer vision at the edge, single board
computers, object detection, planogram compliance control, solar energy
harvesting, RF energy harvesting

##  1 Introduction

Technological advancements have been transforming the retail sector. To be
more specific, integration of sensor networks, internet of things, computer
vision, machine learning, and data analysis is making smart retail stores
increasingly common and efficient. These technologies are practically applied
to automatize operations and enhance customer experience. A significant
challenge in retail is planogram compliance control which can be addressed
through these technologies. Planogram is the diagram showing placement of
products in shelves of a retail store. A shelf is considered planogram
compliant when all the products on it are in the right place and quantity.
Planogram compliance, crucial for maintaining proper product placement and
stock levels, is often undermined by human error or stock issues, leading to
sales loss and penalties. Typically, around 70% compliance is observed in
stores and resetting planograms can boost sales by up to 7.8% in two weeks
[1].

Traditionally, planogram compliance is manually checked by employees, a
process prone to labor intensity and errors. Alternatives include using mobile
devices for photo-based checks [2], installing fixed IP cameras [3], or
employing robots [4, 5, 6]. However, these methods can be costly or laborious.
Recently, embedded systems have been used to capture shelf images for
efficient planogram control [7, 8, 9]. In literature, various computer vision
and pattern recognition methods address planogram compliance control and
object detection in shelf images, as summarized in our previous study [10].
There, we introduced a novel approach for planogram compliance control in
smart retail, utilizing object detection using local feature extraction and
implicit shape model (ISM) and iterative search processes, with sequence
alignment via a modified Needleman-Wunsch algorithm. Melek  _et al._ [11]
developed a multi-stage system for grocery shelf product recognition,
combining detection, classification, and refinement.

In this study, we extend our previous work on planogram compliance control in
two different ways. First, we propose using YOLO for bounding box detection
and feature filtering, replacing ISM for object detection. Second, we form a
complete embedded planogram compliance control system with all its components.
While forming the system, we followed design constraints as follows. The
system should operate stand-alone in a retail environment over extended
periods, without needing for shelf cabling due to safety considerations. Given
multiple shelves within a retail store, the system should also be cost-
effective. High-resolution cameras, and high-end computer systems or cloud
computing resources are impractical for this application due to their high
power consumption and high cost, thus necessitating the use of embedded
systems. The planogram compliance control system can work twice a day. For
example, the first control can be made in the morning before the store opens.
Hence, the employee can reorganize the shelves if compliance is below the
desired level. To handle all these constraints, we propose an embedded system
summarized as in Fig. 1.

Figure 1: Functional block diagram of the proposed system.

As can be seen in Fig. 1, the first stage of the system is the _image
acquisition and transfer_ block formed by an embedded camera module with Wi-Fi
connectivity. The camera to be used for this purpose should be low cost, low-
power, and easy to acquire. The best fit for these constraints is the ESP-EYE
module from Espressif Systems [12]. In the image acquisition and transfer
block, the system captures shelf images at predetermined time intervals and
transmits them to the second stage if it detects a significant change compared
to the previous captured image. We will explain the details of this block in
Section 2. The image acquisition and transfer block is positioned in front of
the shelf to work in stand alone form. Therefore, there should be an
accompanying _energy harvesting and power management_ block to supply power to
it. Moreover, the energy harvesting and power management block should have RF
and solar energy harvesting modules to enhance the battery life. We will
explain the working principles of this block in detail in Section 5.

The second stage of the embedded system is formed by a single-board computer
(SBC) to implement the _object detection_ and _planogram compliance control_
blocks. The SBC offers a cost-effective alternative to conventional PCs and
can be distributed throughout the store for increased efficiency. In this
study, we test Raspberry Pi 4, NVIDIA Jetson Orin Nano, and NVIDIA Jetson AGX
Orin SBCs. The object detection block works such that it preprocess the shelf
image and employs YOLO based bounding box detection and local feature
extraction frameworks upon receipt an image from the first stage. In the
planogram compliance control block, detected objects are converted into a
planogram-compatible format. Then, the modified Needleman-Wunsch algorithm is
applied for planogram compliance control of the shelf image. Furthermore, the
focused and iterative search technique is used for improved results [10]. We
will explain the usage scenarios for object detection and planogram compliance
control blocks in Sections 3 and 4, respectively.

We will explain working principles of the proposed embedded planogram
compliance control system in the following sections in detail. We test the
proposed methods on two different datasets for the object detection and
planogram compliance control steps. We will also analyze the overall system
power consumption via tests. Furthermore, we test our methods on PC for
benchmark.

##  2 The Image Acquisition and Transfer Block

The image acquisition and transfer block is in the first stage of the proposed
embedded planogram compliance control system. We will start with explaining
its hardware properties in this section. Then, we will provide details on
change detection and controlled image transfer operations performed on this
hardware.

###  2.1 Hardware used in Operation

The camera to be used for the image acquisition and transfer block should be
low cost and power. The best fit for these constraints is the ESP-EYE module
from Espressif Systems [12]. This module incorporates an embedded camera and
Wi-Fi chip. The camera can capture up to 2-Megapixel resolution images and
store them in the external memory of the module. Therefore, the ESP-EYE module
serves as a good candidate to implement and realize the image acquisition and
transfer block in our embedded system. We use the C language to program the
ESP-EYE module, benefiting from the Arduino IDE and ESP32 library.

A convenient retail store with 1000 $m^{2}$ sales area can have shelf length
up to 1200 m. An ESP-EYE can cover an average of 4 m wide shelf blocks with
three racks. Hence, it can cover a total of 12 m shelf area. As a result, we
will need approximately 100 of ESP-EYE modules to cover all the shelves in a
typical retail store.

###  2.2 Change Detection and Controlled Image Transfer

Planogram compliance control is necessary only when there is a major change in
the shelf. Therefore, we pick the image rationing with preprocessing method
from our previous work [13]. In this method, we first acquire the shelf image,
$I_{l}$, in grayscale format with QVGA resolution and then apply filtering to
obtain $\tilde{I_{l}}=G\ast I_{l}$, where $G$ is the simple Gaussian blurring
filter kernel. The previously captured and filtered image, $\tilde{I_{r}}$, is
stored in flash memory of the ESP-EYE module for reference during low power
hibernation mode. Subsequently, we calculate ratio of the images, where each
pixel ratio approaches zero with no change and nears $\pi/4$ with maximum
change. After thresholding each pixel ratio with a predetermined value
$\tau_{p}$ and counting the changed pixels, we determine if the shelf image
has changed based on a predefined threshold, $\tau_{c}$, of all pixels. If any
change is detected in a shelf image, a new JPEG image with UXGA resolution $I$
is acquired and transmitted to a remote location via Wi-Fi. Finally, the ESP-
EYE module enters low power hibernation mode, awaiting the next timer trigger
to wake up. We provide the pseudocode for the overall operation in Algorithm
1.

Algorithm 1 Pseudocode of image capture and rationing for shelf control.

$I_{l}\leftarrow\textsc{CaptureImage}($)

$\tilde{I_{l}}\leftarrow\textsc{FilterImage}(I_{l})$

$c\leftarrow\textsc{DetectChange}(\tilde{I_{r}},\tilde{I_{l}},\tau_{p})$

if $c\geq\tau_{c}$ then

$I\leftarrow\textsc{CaptureImage}($)

$\textsc{EnableWiFi}($)

$\textsc{TransferImage}(I)$

$\textsc{DisableWiFi}($)

end if

$\textsc{EnterHibernationMode}($)

##  3 The Object Detection Block

When any change in the shelf image is detected, we should find objects in the
shelf image to compare them with the actual planogram. We will explain the
operations needed for the object detection step in this section. We will start
with the hardware used in operation next.

###  3.1 Hardware used in Operation

The object detection block in our embedded planogram compliance system
requires high processing power and memory. Therefore, we deploy an embedded
SBC to implement it. To be more specific, we use SBCs with high processing
power as Raspberry Pi 4, NVIDIA Jetson Orin Nano, and NVIDIA Jetson AGX Orin
to find objects in the image. We utilize advanced software and libraries on
these embedded devices, leveraging their capability to run the Linux OS. We
use OpenCV library for local feature extraction and object detection
operations. We picked NodeJS as the development framework at SBCs.
Furthermore, we compile the OpenCV from its source to deploy the DNN and image
processing functions in the NodeJS environment. We enabled the CUDA support
for OpenCV while compiling it in NVIDIA Jetson devices which gives us to use
their GPU parallel processing capability for DNN operations.

###  3.2 Image Retrieval and Preprocessing

As the image acquisition and transfer block sends the shelf image to the
object detection block, an end-point should receive it. We implement this
operation using Express which is NodeJS web application framework [14]. We
create a web server application which listens for the incoming image data and
upon receive, it saves the image data to a file.

The received image contains full shelf view containing several shelf racks.
However, the planogram compliance control block works with a single shelf
rack. Thus, we split the image horizontally into the number of shelf racks
using OpenCV library functions. Depending on the distance of the camera to the
shelf, there are three to four shelf racks in the shelf view. Hence, a shelf
rack image approximately has dimensions as $1600\times 400$ pixels after
splitting. We also add padding to the top and bottom of the image. Hence, it
becomes suitable to be processed in the bounding box step via YOLO. Finally,
we employ a blurring filter to the image using OpenCV library functions to
remove noise.

###  3.3 Local Feature Extraction and Matching

Objects on shelves often display complex and closely resembling appearances in
the retail environment. This study employs local feature extraction,
specifically the SIFT method for object detection, a technique chosen based on
its superior performance in our previous study [10]. Let $I(x,y)$ represent a
shelf image with $J$ different objects in it. Assume that the $j^{th}$ object
on the shelf has a model (or planogram) image $I_{j}(x,y)$ for $j=1,\ldots,J$.
Let $w_{j}$ and $h_{j}$ be the width and height of $I_{j}(x,y)$, respectively.
We extract local feature vectors, $\overrightarrow{f_{jl}}$ for
$l=1,\ldots,L_{j}$ from $I_{j}(x,y)$ to represent the $j^{th}$ object with
$L_{j}$ keypoints, using SIFT functions in OpenCV library. Similarly, we
extract the local feature vectors, $\overrightarrow{f_{m}}$ for $m=1,\ldots,M$
from the shelf image, $I(x,y)$, to detect objects in it. Here, $M$ is the
total number of extracted keypoints from the shelf image. We provide the
pseudocode for the feature extraction operation in Algorithm 2.

Algorithm 2 Pseudocode of local feature extraction function.

function ExtractFeat($image$)

$kp\leftarrow\textsc{SiftKeypointDetect}(image)$

$f\leftarrow\textsc{SiftFeatureDescriptor}(image,kp)$

return $f$

end function

One way of finding the object of interest in the shelf image is matching
extracted local feature vectors from the $j^{th}$ object,
$\overrightarrow{f_{jl}}$, and shelf image, $\overrightarrow{f_{m}}$. Here, we
use the brute force matching functions in OpenCV library to find the matching
features. We provide the pseudocode for this operation in Algorithm 3.

Algorithm 3 Pseudocode of feature matching function.

function FeatMatch($f_{jl},f_{m},\tau_{\alpha}$)

$matches\leftarrow\textsc{BFMatcher}(f_{jl},f_{m})$

$f_{jn}\leftarrow\\{\\}$

for all $m\in matches$ do

if $m[0].d<\tau_{\alpha}\times m[1].d$ then

$f_{jn}\leftarrow f_{jn}\bigcup m$

end if

end for

return $f_{jn}$

end function

Furthermore, we also use the ratio test proposed by Lowe [15] to keep only
strong matches. Lowe suggests setting $\tau=0.8$ gives the best result for
SIFT. Instead of such a constant value, we set the matching threshold as
$\tau_{\alpha}=0.95-0.2\alpha$ where $\alpha$ is the iteration parameter
(initially set to one) to be introduced in 4.2. Assume that we have $N_{j}$
local feature vectors from $I(x,y)$ satisfying the threshold constraint for
the $j^{th}$ object. Hence, we will have
$\left\\{\overrightarrow{f_{jn}}\right\\}\subset\left\\{\overrightarrow{f_{m}}\right\\}$
for $n=1,\ldots,N_{j}$ and $N_{j}\leq M$.

###  3.4 Bounding Box Extraction and Filtering

YOLO (You Only Look Once) introduced by Redmon  _et al._ [16] is an advanced,
real-time object detection system, known for its efficiency and speed. It
integrates bounding box prediction and object classification into a single
process. Unlike traditional methods, YOLO divides the input image into a grid,
with each cell predicting bounding boxes and class probabilities, enabling
much faster processing speeds. YOLO has different versions where each version
builds upon the foundational principles of its predecessors, offering enhanced
performance in terms of training speed and prediction accuracy.

One of the most popular YOLO version is YOLOv5 [17] which represents a
significant leap in terms of performance and usability. This version
incorporates several state-of-the-art techniques, such as the use of Cross
Stage Partial networks (CSPNet) for enhancing the learning capability of the
model, and the application of Spatial Pyramid Pooling (SPP) layers to increase
the receptive field. Additionally, YOLOv5 is designed to be user-friendly,
with simpler model training, easier customization, and better support for
deployment across various platforms, including embedded devices. YOLOv5 has
ten available multiscale models having speed/accuracy trade-offs. It supports
11 different formats (both export and run time) and can be easily implemented
in embedded devices. Hence, in this study we choose YOLOv5 as our object
detection model, specifically YOLOv5s and YOLOv5x models. We did not use pre-
trained models. We trained them on the SKU110K dataset [18], which has single
class, using Nvidia Jetson AGX. We set the image size to $640\times 640$ and
epoch length to 100 in the training. We also set the batch size to 48 and 144
for YOLOv5x and YOLOv5s, respectively. The training lasts 16.6 hours and the
mAP50 reaches 0.59 for YOLOv5x, and it lasts 11.4 hours and the mAP50 reaches
to 0.56 for YOLOv5s, respectively. We give the graphs for training and
validation in Fig. 2.

(a) YOLOv5s

(b) YOLOv5s

Figure 2: YOLOv5 training results.

We employ the OpenCV DNN library to perform inference on shelf images using
the YOLOv5 model. OpenCV DNN library cannot use the YOLOv5 model file
directly. However, it can use ONNX format for model file. Hence, we first
export our custom trained YOLOv5 model to the ONNX format. The model produces
all the candidate bounding boxes as a result of the inference operation. Each
bounding box, $B_{b}$, is characterized by a tuple containing the confidence
score and spatial dimensions as $(cs_{b},x_{b},y_{b},w_{b},h_{b})$ for
$b=1,\ldots,B$. To identify the best candidates for the $j^{th}$ object, we
apply filtering and threshold techniques. Specifically, we use the width
$w_{j}$, height $h_{j}$, and aspect ratio $\frac{w_{j}}{h_{j}}$ of the
$j^{th}$ object as reference. We discard any bounding box with dimensions and
aspect ratio falling below half or exceeding twice these reference values.
Subsequently, we apply a confidence score threshold, which we set it at 5% of
the maximum confidence score, to further refine our candidates. Hence, we
discard any bounding box with very low confidence score. Following these
steps, we obtain $C_{j}$ candidate bounding boxes for the $j^{th}$ object. We
provide the pseudocode explaining these steps in Algorithm 4.

Algorithm 4 Pseudocode of bounding box extraction and filtering.

function ExtractBB($I$)

$B_{b}\leftarrow\textsc{Inference}(I)$

for all $Object_{j}\in planogram$ do

$B_{tmp}\leftarrow\\{\\}$

for all $box_{b}\in B_{b}$ do

if $(w_{j}\times 0.5\leq w_{b}\leq w_{j}\times 2)$ and

$(h_{j}\times 0.5\leq h_{b}\leq h_{j}\times 2)$ and

$(\frac{w_{j}}{h_{j}}\times
0.5\leq\frac{w_{b}}{h_{b}}\leq\frac{w_{j}}{h_{j}}\times 2)$ then

$B_{tmp}\leftarrow B_{tmp}\cup\\{box_{b}\\}$

end if

end for

$\tau_{b}\leftarrow 0.05\times\max(\forall cs_{tmp})$

$B_{jc}\leftarrow\\{\\}$

for all $box_{tmp}\in B_{tmp}$ do

if $(cs_{tmp}\geq\tau_{b})$ then

$tl\leftarrow(x_{tmp}-\frac{w_{tmp}}{2},y_{tmp}-\frac{h_{tmp}}{2})$

$br\leftarrow(x_{tmp}+\frac{w_{tmp}}{2},y_{tmp}+\frac{h_{tmp}}{2})$

$cnt\leftarrow(x_{tmp},y_{tmp})$

$B_{jc}\leftarrow B_{jc}\cup\\{tl,br,cnt,cs_{tmp}\\}$

end if

end for

end for

return $B_{jc}$

end function

As we detect possible object bounding boxes for the $j^{th}$ object in the
shelf image, the next step is finding matching local features inside bounding
boxes. Therefore, we find the number of features inside each bounding box,
$s_{c}$. We further eliminate weak objects which have lower feature number
than 5% of the number of features of the $j^{th}$ object, $L_{j}$. Then, we
normalize it with the total number of feature of the $j^{th}$ object, $L_{j}$.
We multiply it with the confidence score of the bounding box to use it as the
detection score for the detected object. Some bounding boxes may overlap,
meaning that there may be more than one candidate object in the given
location. Therefore, we eliminate weak candidates using the greedy algorithm
for non-maxima suppression (NMS) of bounding boxes. At the end of this
operation, we detect a total of $D$ objects, $D\leq\sum_{j=1}^{J}C_{j}$, from
the shelf image $I(x,y)$. We provide the pseudocode to filter out matching
features not lying in a bounding box in Algorithm 5.

Algorithm 5 Pseudocode of matching feature filtering.

function FeatFilter($B_{jd},f_{jn}$)

for all $Object_{j}\in planogram$ do

$B_{tmp}\leftarrow\\{\\}$

for all $box_{c}\in B_{jc}$ do

$s_{c}\leftarrow 0$

for all $feature_{n}\in f_{jn}$ do

if $(tlx_{c}\leq x_{n}\leq brx_{c})$ and

$(tly_{c}\leq y_{n}\leq bry_{c})$ then

$s_{c}\leftarrow s_{c}+1$

end if

end for

if $s_{c}>0.05\times L_{j}$ then

$weight_{c}\leftarrow\frac{s_{c}}{L_{j}}\times cs_{c}$

$B_{tmp}\leftarrow B_{tmp}\cup\\{box_{c},weight_{c}\\}$

end if

end for

$D_{jd}\leftarrow\textsc{NMS}(B_{tmp})$

end for

return $D_{jd}$

end function

##  4 The Planogram Compliance Control Block

Object detection results in Section 3 lead to planogram compliance control.
After representing detection results in an abstract planogram format, we use
the modified Needleman-Wunsch algorithm to align and compare the detected and
reference planograms as suggested in our previous study [10]. Then, we use the
iterative search method to focus on undetected or empty object locations in
the shelf image. We explain these steps in detail next. Here, we use the same
hardware used in Section 3 as well as the NodeJS framework.

###  4.1 Sequence Alignment

To compare the detected objects in previous section with the given planogram
data, we first need to convert detected object list into planogram format.
Here, we sort the detected objects from left to right according to their
center coordinates. Then, we group the items of same object hence we can
compare their location and quantity, later. We also keep the start and end
coordinates hence we can iteratively search for undetected objects only as
explained in our previous study [10]. Based on this, we represent the detected
objects in the shelf image (in sorted and grouped form) as
$L_{s}=\left[o_{d},q_{d},B_{d}\right]$ and reference planogram as
$L_{r}=\left[o_{t},q_{t},B_{t}\right]$. To do so, we use the algorithm given
in Algorithm 6.

Algorithm 6 Pseudocode of object to planogram method.

function Obj2Pla($D_{m}$)

$L\leftarrow\\{\\}$

$D_{m}\leftarrow\textsc{Sort}(D_{m})$

for all $o_{m}\in D_{m}$ do

if $L\owns o_{m}$ then

$q_{m}\leftarrow q_{m}+1$

$br\leftarrow br_{m}$

else

$q_{m}\leftarrow 1$

$tl\leftarrow tl_{m}$, $br\leftarrow br_{m}$

end if

$B_{m}\leftarrow\\{tl,br\\}$

$L\leftarrow L\cup\\{o_{d},q_{d},B_{d}\\}$

end for

return $L$

end function

We can control planogram compliance by comparing the reference, $L_{r}$, and
detected, $L_{s}$, planograms for a given shelf. To do so, we need to align
them using the modified Needleman-Wunsch (NW) algorithm [19]. The NW algorithm
has three steps: initialization, matrix fill and trace back. In the
initialization step we set the score matrix $F$ with size $(E+1)\times(T+1)$
and set the first row and column of $F$. In the matrix filling step we fill
the $F(d,t)$ using the gap penalties, $g_{ins}$ and $g_{del}$, for insert and
delete, and the substitution score, $s(o_{d},o_{t})$, for match and mismatch.
In this study, we set the gap penalties and substitution score dynamically
based on their importance. We set the gap penalty for the delete operation as
the required number of objects in the reference planogram since we could not
detect them. We set the gap penalty for the insert operation as the number of
extra items in the detected planogram. The aim here is that the more inserted
such false objects are, the more penalty there should be. The third step in
the modified NW algorithm is tracing back the filled matrix entries. Here, we
trace back the decisions we have made while calculating each matrix entry. We
use the pseudocode in Algorithm 7 for planogram alignment using the modified
NW algorithm.

Algorithm 7 Pseudocode of the Needleman-Wunsch algorithm.

function NW($L_{r},L_{s}$)

$\triangleright$ Initialize

$F\leftarrow[E+1][T+1]$

$F_{trace}\leftarrow[E+1][T+1]$

$t\leftarrow 1,d\leftarrow 1$

$F(0,0)\leftarrow 0$

while $t\leq T$ do

$F(0,t)\leftarrow F(0,t-1)-1$

$t\leftarrow t+1$

end while

while $d\leq E$ do

$F(d,0)\leftarrow F(d-1,0)-1$

$d\leftarrow d+1$

end while

$\triangleright$ Matrix Fill

for all $cell(d,t)\in F$ do

$up\leftarrow F(d,t-1)-g_{del}$

$left\leftarrow F(d-1,t)-g_{ins}$

$dia\leftarrow F(d-1,t-1)+s(o_{d},o_{t})$

$F(d,t)\leftarrow\max(up,left,dia)$

$F_{trace}(d,t)\leftarrow maxIndex(up,left,dia)$

end for

$\triangleright$ Trace Back

$d\leftarrow E+1,t\leftarrow T+1$

while $(d,t)\neq(0,0)$ do

if $F_{trace}(d,t)=dia$ then

$\hat{L_{r}}\leftarrow\hat{L_{r}}\bigcup\left[o_{t},q_{t},B_{t}\right]$

$\hat{L_{s}}\leftarrow\hat{L_{s}}\bigcup\left[o_{d},q_{d},B_{d}\right]$

$(d,t)\leftarrow(d-1,d-1)$

else if $F_{trace}(d,t)=up$ then

$\hat{L_{r}}\leftarrow\hat{L_{r}}\bigcup\left[A,0,-\right]$

$\hat{L_{s}}\leftarrow\hat{L_{s}}\bigcup\left[o_{d},q_{d},B_{d}\right]$

$(d,t)\leftarrow(d,t-1)$

else if $F_{trace}(d,t)=left$ then

$\hat{L_{r}}\leftarrow\hat{L_{r}}\bigcup\left[o_{t},q_{t},B_{t}\right]$

$\hat{L_{s}}\leftarrow\hat{L_{s}}\bigcup\left[D,0,-\right]$

$(d,t)\leftarrow(d-1,t)$

end if

end while

return $\hat{L_{r}},\hat{L_{s}}$

end function

Following the steps in Algorithm 7, we can obtain aligned object groups
$\hat{o}_{d}$ and $\hat{o}_{t}$ and their quantities $\hat{q}_{d}$ and
$\hat{q}_{t}$. We then calculate the planogram match ratio, $\mu_{s}$, ranging
from 0 (no compliance) to 1 (full compliance), using the Algorithm in
Algorithm 8. A $\mu_{s}$ of 0 means all items are misplaced, while 1 signifies
complete planogram compliance. A match ratio between 0 and 1 indicates partial
compliance, with some items missing, extra, or incorrectly placed.

Algorithm 8 Pseudocode of planogram control operations.

function PLAControl($\hat{L_{r}},\hat{L_{s}}$)

$result\leftarrow\\{\\}$

$sum\leftarrow 0,csum\leftarrow 0$

for all $\hat{d}\in\hat{L_{s}}$ and $\hat{t}\in\hat{L_{r}}$ do

if $o_{d}==o_{t}$ then

if $q_{d}==q_{t}$ then

$result\leftarrow result\bigcup MT$

$sum\leftarrow sum+q_{d}$

else if $q_{d}>q_{t}$ then

$result\leftarrow result\bigcup ME$

$sum\leftarrow sum+\min(q_{d},q_{t})$

else if $q_{d}<q_{t}$ then

$result\leftarrow result\bigcup MI$

$sum\leftarrow sum+\min(q_{d},q_{t})$

end if

else if $o_{d}\neq o_{t}$ then

$result\leftarrow result\bigcup NM$

end if

$csum\leftarrow csum+q_{t}$

end for

$\mu_{s}\leftarrow\frac{sum}{csum}$

return $result,\mu_{s}$

end function

###  4.2 Focused and Iterative Search

The object detection method and modified NW algorithm introduced in the
previous sections may not produce exact planogram alignment result in one
iteration. Therefore, we use the focused and iterative search method to
improve the results [10]. Here, we repeat the object detection operation by
relaxing the detection constraints and focusing on the region of interest
(ROI) in the shelf image $I(x,y)$. As we form the ROI, we iteratively decrease
the value of $\alpha$ as $\alpha_{new}=0.75\alpha_{old}$. We detect new
objects based on the new $\alpha$ value at each iteration and apply the NW
algorithm to the updated $L_{s}$ and $L_{r}$ until $\mu=1$ or $\mu$ does not
change for six successive iterations. We provide the pseudocode for the
focused and iterative search operation in Algorithm 9. As the iterations end,
we have the planogram compliance control result at hand.

Algorithm 9 Pseudocode of focused and iterative search operations.

function FocusedItrSearch($I,L_{r}$)

$\alpha\leftarrow 1,retryNum\leftarrow 0$

$\mu_{s}\leftarrow 0,\mu_{sp}\leftarrow 0$

$\overrightarrow{f_{m}}\leftarrow\textsc{ExtractFeat}(I)$

$B_{jd}\leftarrow\textsc{ExtractBB}(I)$

while $IterNum<6$ and $\mu_{s}\neq 1$ do

$\mu_{sp}\leftarrow\mu_{s}$

$\tau_{\alpha}\leftarrow 1-0.15\alpha$

for all $object_{j}\neq MT\in result$ do

$f_{jn}\leftarrow\textsc{FeatMatch}(f_{jl},f_{m},\tau_{\alpha})$

$D_{jd}\leftarrow\textsc{FeatFilter}(f_{jl},f_{jn})$

end for

$L_{s}\leftarrow\textsc{Obj2Pla}(D_{jd})$

$\hat{L_{r}},\hat{L_{s}}\leftarrow\textsc{NW}(L_{r},L_{s})$

$result,\mu_{s}\leftarrow\textsc{PLAControl}(\hat{L_{r}},\hat{L_{s}})$

if $\mu_{sp}==\mu_{s}$ then

$retryNum\leftarrow retryNum+1$

end if

end while

return $result,\mu_{s}$

end function

##  5 The Energy Harvesting and Power Management Block

We need a battery-based, stand-alone system for the image acquisition and
transfer block. Therefore, one important part of the proposed embedded
planogram compliance control system is the energy harvesting and power
management block. Various energy harvesting methods, including RF and WPT, are
increasingly utilized to extend the battery life of embedded systems [20, 21,
22, 23]. Kadir  _et al._ [24] introduced a Wi-Fi energy harvester suitable for
indoor, low-power applications, featuring multiple antennas and providing up
to 2 V output. Tepper  _et al._ [25] explored the use of WPT systems in
aircraft for powering sensors, employing commercial Powercast 2110 evalkit
with a transmitter operating at 915 MHz.

In this section, we explain suitable modules for solar and RF energy
harvesting. We use them in our embedded planogram compliance control system to
extend battery life, either separately or combined using complementary methods
or power ORing [26]. Therefore, we start with explaining the battery pack used
in operation.

###  5.1 Battery Pack

The battery generally has the largest size in an embedded system with its size
directly proportional to its capacity and price. Hence, determining the
battery life and capacity plays a vital role while determining the system
physical dimensions and overall cost. We maximize the battery life with the
help of ambient energy harvesting options in the proposed system. Therefore,
we should use a rechargeable battery to store harvested energy. To be more
specific, we picked the LiFePO4 battery, which is a type of Lithium-Ion (Li-
Ion) battery, to supply the system since its voltage range is suitable
directly use it with ESP-EYE module. Moreover, we minimize the current drawn
via utilizing hibernation mode for the ESP-EYE and periodic wake-ups via timer
interrupts. This operation significantly saves battery power and maximizes the
battery life.

###  5.2 Solar Energy Harvesting

The simplest energy harvesting method is to use ambient light with the help of
photovoltaic (PV) cells, as there is usually constant artificial lighting in a
retail store. We use a power management integrated circuit (PMIC) to charge
the battery from the harvested energy, track the maximum power point of PV
cells, and supply the regulated voltage to the system. We use the SPV1050 from
STMicroelectronics as the PMIC that can be used for indoor light energy
harvesting [27]. This module harvests energy from solar cells and stores the
harvested energy in a capacitor. Then, it supplies the regulated output
voltage to system components. It can harvest energy from PV cells even if the
output voltage of the PV cell is as low as 150 mV. It also supports secondary
input where other energy harvesting methods can be connected in complementary
to the PV cell. Hence, it can harvest energy from this input if there is not
enough lighting on PV cell. De Rossi  _et al._ [28] show that an amorphous
silicon (a-Si) PV cell can produce about 13 $\mu W/cm^{2}$ under 500 lux LED
light illumination, which is the average illumination level in a retail store.
Hence, we pick the a-Si PV cell in the proposed system.

###  5.3 RF Energy Harvesting

We can also use the RF energy harvesting chip P2210B from Powercast to back up
the battery power of ESP-EYE module. Basically, the system converts RF energy
to DC voltage and stores it in a capacitor. Then, the chip regulates and
supplies the voltage to output. The P2210B chip can harvest power level below
to -12 dBm from 902 MHz to 928 MHz RF signals. It can provide an output
current of 0.2 mA at an input power level of 0 dBm. The 0 dBm average RF power
at indoor is quite acceptable when it is recognized that maximum power level
is 36 dBm and reduced down to 4 dBm at 915 MHz for GSM communication [29].

##  6 Experiments

We test the proposed system in this section. To do so, we benefit from two
datasets. We first introduce them. Then, we provide the object detection
performance of the proposed method on these datasets. To note here, we take
the performance of the SIFT method in our previous work as the baseline method
[10]. Furthermore, we re-assess object detection performance the baseline
method using intersection over union (IoU) value of 50%. Afterward, we provide
experimental results on the planogram compliance control step. We also provide
timing and power consumption performance of the proposed embedded system.

###  6.1 Datasets used in Experiments

In the absence of a dedicated dataset for evaluating embedded system object
detection in retail, we utilized available datasets relevant to similar
problems, such as object detection in retail stores and bounding box
extraction. Specifically, we adapted a subset of the grocery product dataset
annotated by Tonioni and Di Stefano [30, 31]. We split the images for single
shelf rack focus and resized them to 400 pixels in height, matching the
dimensions used by the image retrieval block in our system. We also employed
the Migros dataset, selecting its first subset, comprising 28 shelf images
with 312 objects, already split for single rack views and resized to align
with our system’s requirements [10]. This approach allows an effective
assessment of our method and comparison with previous works.

###  6.2 Object Detection Performance

We focus on the object detection performance of the proposed method in this
section. Here, we use IoU value of 50% while computing the precision, recall,
and F1 score as our performance metrics. We tested the proposed object
detection method on the Migros and grocery product dataset. We provide a
representative image from each dataset for the detected objects in Fig. 3. As
can be seen in this figure, all objects are detected correctly with the
proposed method.

(a) Migros dataset.

(b) Grocery Product dataset.

Figure 3: Bounding box of the detected objects from the representative shelf
images of test datasets.

We provide object detection performance of the proposed method on the Migros
dataset in Table 1. As can be seen in this table, the proposed method in this
study achieves higher object detection performance compared to the results of
baseline object detection method. To be more specific, the overall performance
increases using YOLO instead of ISM while detecting bounding boxes. To note
here, bounding box detection and classification steps depend on the SIFT
keypoint extraction and feature matching performance in the baseline method.
We benefit from YOLO to detect bounding boxes in the proposed system.
Therefore, we can reach F1 scores around 1 for both YOLOv5 models. This is a
remarkable result for the given problem.

Table 1: Object detection performance of the proposed method on Migros dataset. Model | Precision | Recall | F1 Score
---|---|---|---
Baseline | 0.974 | 0.965 | 0.969
YOLOv5s | 0.997 | 0.997 | 0.997
YOLOv5x | 0.994 | 0.990 | 0.992

We provide object detection results on the grocery product dataset in Table 2.
As can be seen in this table, the proposed method with YOLOv5x model achieves
higher object detection performance on this dataset compared to the baseline
object detection method. However, overall performance of the YOLO models
decrease compared to the Migros dataset. Lower scores in this dataset are
because of two main reasons. First, images in this dataset are blurry and have
low resolution. Second, the proposed method is sensitive to the shelf height
and full visibility of the shelf. There are some images in the grocery product
dataset which violate this constraint. Therefore, they led to low performance.
Even with these negative effects, the proposed method was able to produce F1
scores around 0.927 with the YOLOv5x model.

Table 2: Object detection performance of the proposed method on the grocery product dataset. Model | Precision | Recall | F1 Score
---|---|---|---
Baseline | 0.910 | 0.935 | 0.922
YOLOv5s | 0.879 | 0.895 | 0.887
YOLOv5x | 0.924 | 0.929 | 0.927

###  6.3 Planogram Compliance Control Performance

There is no consensus on measuring the planogram compliance control
performance in literature. The closest study for this purpose is by Liu  _et
al._ [32]. Unfortunately, this study focuses on a graph theory based
representation. Therefore, we provide our own measures based on precision,
recall, and F1 score values. We define true positives as alignment results
matching the ground truth, false positives as misalignments not in the ground
truth, and false negatives as ground truth object groups missing in the
alignment results.

We provide planogram compliance control performance results for the Migros
dataset in Table 3. As can be seen in this table, the proposed method achieves
higher performance compared to the baseline method. This is because the
planogram compliance control performance is directly related to the object
detection performance given in Table 1. This can be expected since correctly
detecting objects in the shelf image directly affects the corresponding
planogram formation from the shelf.

Table 3: Planogram compliance control performance of the proposed method on the Migros dataset. Model | Precision | Recall | F1 Score
---|---|---|---
Baseline | 0.954 | 0.980 | 0.966
YOLOv5s | 1.000 | 1.000 | 1.000
YOLOv5x | 0.987 | 0.993 | 0.990

###  6.4 Timing and Power Consumption Performance of the Image Acquisition and
Transfer Block

Timing performance of the image acquisition and transfer block is important
since we would like to have a stand-alone system to achieve this operation.
Therefore, we first analyzed timing of the methods running on the ESP-EYE
module. We measured the results as 3.5 and 8.5 seconds for image acquisition
and transfer operations, respectively. Based on this timing performance
results and working scenario provided in Section 2, we can report the average
current drawn from the 3.3 V power supply (including wireless transfer) as
243.2 mA. Assuming that the 18650 sized LiFePO4 battery has 1500 mAh capacity,
we can claim that the ESP-EYE based first section of the proposed embedded
system can last approximately 24 months. This duration can be enhanced by
increasing battery capacity or adding an energy harvesting module to the
system.

According to our tests, the SPV1050 solar energy harvester and $52\times 27$
mm sized a-Si PV cell can extend battery life approximately 12 to 24 months
depending on lighting levels 250 lux and 500 lux, respectively. Moreover, the
P2110B RF energy harvester can feed 3.3 V and 0.018 mA continuously to the
system under -8 dBm average input power. Hence, operation durations can be
expanded further by approximately 12 more months. The harvesting options can
be used as complementary to further increase the battery life. Moreover, if
the lighting increase above the 1000 lux, a double sized a-Si PV cell is used
or the average input RF power increases above to -3 dBm, then the system can
operate infinitely without recharging or replacing the battery. Hence, the
battery size became insignificant and a LiFePO4 battery with the smallest
capacity can be used.

###  6.5 Timing Performance of the Object Detection Block

We analyzed timing performance of the object detection block on selected
hardware platforms at maximum GPU and CPU power levels in this section. While
calculating the timing performance for the object detection block, we pick the
average time required to process one shelf rack image as a measure. This
analysis was conducted using images from the Migros dataset. We provided the
results in Table 4. It is important to note that if the object detection block
fails to identify all objects in a shelf image, iterations in the focused and
iterative search method increase, significantly impacting the timing
performance.

Table 4: Timing performance (in seconds) of the proposed method on different SBCs. SBCs | Baseline | YOLOv5s | YOLOv5x
---|---|---|---
RPi4 | 21.61 | 28.90 | 36.13
Orin Nano | 2.70 | 5.70 | 7.43
AGX Orin | 1.86 | 3.65 | 3.76
PC | 5.96 | 6.17 | 9.65

As can be seen in Table 4, YOLOv5s emerges as the fastest model across all
hardware platforms despite having the lowest object detection score.
Nevertheless, it cannot outperform the baseline method in terms of speed. On
the other hand, YOLOv5x has the slowest timing performance, attributable to
its larger model size and increased number of parameters. Therefore, the
reader should take these values into account while picking the appropriate
method.

NVIDIA Jetson AGX Orin stands out as the fastest SBC for all models when the
hardware platforms of interest are compared. To be more specific, with GPU
support, it is almost 1.7 times times faster than a PC with an Intel i7–7600U
CPU while using YOLOv5s. As expected, Raspberry Pi4 is the slowest SBC due to
its less powerful CPU. These durations imply that complete planogram
compliance control in a store equipped with 100 ESP-EYE modules can take
approximately 2.4, 0.5, and 0.3 hours with YOLOv5s on a single Raspberry Pi4,
NVIDIA Jetson Orin Nano, and AGX, respectively. The same operations can take
around 3, 0.6, and 0.3 hours with YOLOv5x on the Raspberry Pi4, NVIDIA Jetson
Orin Nano, and AGX, respectively.

We should also consider the cost of mentioned SBCs in comparison. The price
for the NVIDIA Jetson AGX Orin and Nano boards are USD 1999 and USD 499,
respectively [33]. The price for the Raspberry Pi4 board is USD 75 [34]. Based
on the price of the SBCs and timing values, the same tasks using YOLOv5x could
be completed at cost of USD 1999, USD 998 or USD 750 using a single NVIDIA
Jetson AGX Orin SBC, two NVIDIA Jetson Orin Nano SBCs or ten Raspberry Pi 4
SBCs, respectively.

##  7 Final Comments

In this study, we proposed an embedded planogram compliance control system
with all its components. The methodology used in the system is the improved
version of our previous work. To be more precise, the improvement has been
done by adding YOLO or object detection. This way, our system precisely
detects and aligns objects on shelves which is crucial for planogram
compliance control accuracy. Our proposed embedded system architecture is
designed for effective operation within the constraints of power consumption,
cost, and physical space in retail environments. Therefore, we benefit from
low-cost ESP-EYE modules for image capture and more powerful single-board
computers like Raspberry Pi 4 or NVIDIA Jetson for implementation. We also
applied solar and RF energy harvesting techniques to extend the battery life
of our system while maintaining its compactness. This aspect is crucial for
seamless integration into retail environments. Our scalability analysis,
involving various hardware configurations, demonstrates the flexibility of the
system in adapting to diverse performance requirements and budget constraints.
Our experimental validation conducted using public datasets showcases the
strengths of our approach in comparison to existing methods. The results
affirm the viability of our system in smart retail applications, highlighting
its scalability, efficiency, and cost-effectiveness. The next step for us is
extending our method to employ machine learning and AI algorithms for better
planogram alignment and matching. Also we can train lightweight models to run
on low power embedded systems. Hence, it can be used in stand alone form in
retail sector.

## References

  * Shapiro [2009] M. Shapiro.  Executing the best planogram.  Technical report, Professional Candy Buyer, Norwalk, CT, USA, 2009.
  * [2] Planorama: Image recognition for Retail Execution and Merchandising.  Planorama: Image recognition for retail execution and merchandising.  URL https://planorama.com/.
  * [3] Shelfie.  Shelfie.  URL https://shelfieretail.com/.
  * [4] Simbe Robotics.  Simbe robotics.  URL https://www.simberobotics.com/.
  * [5] FellowAI: Autonomous Navigation and Mapping.  Fellowai: Autonomous navigation and mapping.  URL https://www.fellowai.com/.
  * [6] Badger Technologies: Retail Automation.  Badger technologies: Retail automation.  URL https://www.badger-technologies.com/.
  * [7] Shelf Cameras — Focal Systems.  Shelf cameras — focal systems.  URL https://focal.systems/topstock-backroom-cameras-1.
  * [8] Pricer ShelfVision.  Pricer shelfvision.  URL https://www.pricer.com/products/platform/shelf-vision.
  * [9] Improve On-Shelf Availability with Captana.  Improve on-shelf availability with captana.  URL https://www.ses-imagotag.com/products/captana/.
  * Yücel and Ünsalan [2023] M. Erkin Yücel and Cem Ünsalan.  Planogram compliance control via object detection, sequence alignment, and focused iterative search.  _Multimedia Tools and Applications_ , 2023.
  * Melek et al. [2023] Ceren Gulra Melek, Elena Battini Sonmez, Hakan Ayral, and Songul Varli.  Development of a hybrid method for multi-stage end-to-end recognition of grocery products in shelf images.  _Electronics_ , 12(17), 2023.  ISSN 2079-9292.  doi:10.3390/electronics12173640.  URL https://www.mdpi.com/2079-9292/12/17/3640.
  * [12] ESP-EYE.  Esp-eye.  URL https://www.espressif.com/en/products/devkits/esp-eye/overview.
  * Yücel and Ünsalan [2022] M. Erkin Yücel and Cem Ünsalan.  Shelf control in retail stores via ultra-low and low power microcontrollers.  _Journal of Real-Time Image Processing_ , 19, 08 2022.
  * [14] Express.  Express.  URL https://expressjs.com/.
  * Lowe [2004] David Lowe.  Distinctive image features from scale-invariant keypoints.  _International Journal of Computer Vision_ , 60(2):91–110, 2004.
  * Redmon et al. [2015] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  _arXiv preprint arXiv:1506.02640_ , 2015.
  * Jocher [2022] Glenn Jocher.  ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, November 2022.
  * Goldman et al. [2019] Eran Goldman, Roei Herzig, Aviv Eisenschtat, Jacob Goldberger, and Tal Hassner.  Precise detection in densely packed scenes.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5222–5231, 06 2019.
  * Needleman and Wunsch [1970] Saul B. Needleman and Christian D. Wunsch.  A general method applicable to the search for similarities in the amino acid sequence of two proteins.  _Journal of Molecular Biology_ , 48(3):443–453, 1970.
  * Riva and Zaim [2022] Cengiz Riva and Abdul Zaim.  A comparative study on energy harvesting battery-free lorawan sensor networks.  _Electrica_ , 23:40–47, 2022.
  * Choi [2018] Hyun-Sik Choi.  Application for outdoor dust monitoring using rf wireless power transmission.  pages 196–199, 01 2018.
  * Tripathi and Nasipuri [2019] Asheesh Tripathi and Asis Nasipuri.  Information based smart rf energy harvesting in wireless sensor networks.  pages 197–198, 10 2019.
  * Warnakulasuriya et al. [2022] Diluna Adeesha Warnakulasuriya, Konstantin Mikhaylov, and Onel Alcaraz López.  Wireless power transfer for bluetooth low energy based iot device: an empirical study of energy performance.  pages 1–7, 10 2022.
  * abd Kadir et al. [2014] Ermeey abd Kadir, Aiguo Hu, Morteza Biglari-Abhari, and Kean Aw.  Indoor wifi energy harvester with multiple antenna for low-power wireless applications.  06 2014.
  * Tepper et al. [2020] Jan Tepper, Aygün Baltaci, Bernd Schleicher, Andreas Drexler, Svetoslav Duhovnikov, Mustafa Ozger, Morteza Tavana, Cicek Cavdar, and Dominic Schupke.  Evaluation of rf wireless power transfer for low-power aircraft sensors.  In _2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC)_ , pages 1–6, 2020.
  * Estrada-López et al. [2018] Johan J. Estrada-López, Amr Abuellil, Zizhen Zeng, and Edgar Sánchez-Sinencio.  Multiple input energy harvesting systems for autonomous iot end-nodes.  _Journal of Low Power Electronics and Applications_ , 8(1), 2018.
  * SPV [1050] SPV1050.  Spv1050.  URL https://www.st.com/resource/en/datasheet/spv1050.pdf.
  * De Rossi et al. [2015] Francesca De Rossi, Tadeo Pontecorvo, and Thomas Brown.  Characterization of photovoltaic devices for indoor light harvesting and customization of flexible dye solar cells to deliver superior efficiency under artificial lighting.  _Applied Energy_ , 156:413–422, 2015.
  * 3GPP TS 05.05 V8.20. [0] 3GPP TS 05.05 V8.20.0.  3gpp ts 05.05 v8.20.0 - 3rd generation partnership project, technical specification group GSM/EDGE, radio access network, radio transmission and reception.  Standard, ETSI, 2005-11.
  * George and Floerkemeier [2014] M. George and C. Floerkemeier.  Recognizing products: A per-exemplar multi-label image classification approach.  In _European Conference on Computer Vision_ , pages 440–455, 09 2014.
  * Tonioni and Di Stefano [2017] A. Tonioni and L. Di Stefano.  Product recognition in store shelves as a sub-graph isomorphism problem.  In _International Conference on Image Analysis and Processing_ , pages 682–693, 10 2017.
  * Liu et al. [2016] S. Liu, W. Li, S. Davis, C. Ritz, and H. Tian.  Planogram compliance checking based on detection of recurring patterns.  _IEEE MultiMedia_ , 23(2):54–63, 2016.
  * [33] Jetson Orin Modules and Developer Kits.  Jetson orin modules and developer kits.  URL https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/.
  * [34] Raspberry Pi: Products.  Raspberry pi: Products.  URL https://www.raspberrypi.com/products/.