# Taking a PEEK into YOLOv5 for Satellite Component Recognition via Entropy-
based Visual Explanations111Article accepted at AIAA SCITECH Forum 2024
Copyright © 2024 by the American Institute of Aeronautics and Astronautics,
Inc. All rights reserved.

Mackenzie J. Meni222PhD Student, Department of Mathematics and Systems
Engineering. NEural TransmissionS Lab, Department of Mathematics and Systems
Engineering
Florida Institute of Technology, Melbourne, FL 32901, USA  Trupti
Mahendrakar333PhD Candidate, Department of Aerospace, Physics and Space
Sciences, AIAA Student Member. Autonomy Lab, Department of Aerospace, Physics
and Space Sciences
Florida Institute of Technology, Melbourne, FL 32901, USA  Olivia D. M.
Raney444MFA Student, Communication Media Arts, Scripps College of
Communication. Communication Media Arts, Scripps College of Communication,
Ohio University, Athens, OH 45701, USA  Ryan T. White555Assistant Professor,
Department of Mathematics and Systems Engineering. NEural TransmissionS Lab,
Department of Mathematics and Systems Engineering
Florida Institute of Technology, Melbourne, FL 32901, USA.  Michael L.
Mayo666Research Physicist, Environmental Laboratory. and Kevin R.
Pilkiewicz777Research Physicist, Environmental Laboratory. Environmental
Laboratory
U.S. Army Engineer Research and Development Center (ERDC), Vicksburg, MS
39180, USA.

###### Abstract

The escalating risk of collisions and the accumulation of space debris in Low
Earth Orbit (LEO) has reached critical concern due to the ever increasing
number of spacecraft. Addressing this crisis, especially in dealing with non-
cooperative and unidentified space debris, is of paramount importance. This
paper contributes to efforts in enabling autonomous swarms of small chaser
satellites for target geometry determination and safe flight trajectory
planning for proximity operations in LEO. Our research explores on-orbit use
of the You Only Look Once v5 (YOLOv5) object detection model trained to detect
satellite components. While this model has shown promise, its inherent lack of
interpretability hinders human understanding, a critical aspect of validating
algorithms for use in safety-critical missions. To analyze the decision
processes, we introduce Probabilistic Explanations for Entropic Knowledge
extraction (PEEK), a method that utilizes information theoretic analysis of
the latent representations within the hidden layers of the model. Through both
synthetic in hardware-in-the-loop experiments, PEEK illuminates the decision-
making processes of the model, helping identify its strengths, limitations and
biases.

##  1 Nomenclature

AP = | average precision
---|---
APF = | artificial potential field
CAM = | class activation map
CNN = | convolutional neural network
FPN = | feature pyramid network
GPU = | graphics processing unit
Grad-CAM = | gradient-weighted class activation map
$H(X)$ = | entropy of a random variable $X$
IoU = | intersection over union
LEO = | Low Earth Orbit
mAP = | mean average precision
<EMAIL_ADDRESS>= | mewan average precision at 0.5 IoU threshold
mAP@0.5:0.95 = | mean average precision averaged over 0.5, 0.55, …, 0.95 IoU thresholds
OOS = | on-orbit servicing
P = | precision
PANet = | path aggregation network
pdf = | probability density function
PEEK = | Probabilistic Explanations for Entropic Knowledge extraction
PR curve = | precision-recall curve
R = | recall
R-CNN = | region-based convolutional neural network
ReLU = | rectified linear unit activation function
SWaP = | size, weight, and power
RPN = | region proposal network
TP = | true positive
YOLO = | You Only Look Once

##  2 Introduction

The increasing number of spacecraft in Low Earth Orbit (LEO) from governments
and private companies has delivered immense benefits, including high-precision
remote sensing of the Earth, microgravity and space environment research, and
deep space imaging. However, this orbital crowding has precipitated growing
risks of collisions. In-space missile tests carried out by multiple nations
[1, 2, 3, 4] have scattered space debris that remains in LEO, and many
decommissioned or dysfunctional spacecraft are also still in orbit. This has
become such an acute problem that the UN General Assembly passed a resolution
in late 2022 calling for a halt to destructive direct-ascent anti-satellite
missile testing [5].

One approach to reduce the future proliferation of space debris is to extend
the lifetimes of satellites with on-orbit servicing (OOS). Previous missions
demonstrating on-orbit operations include DART [6], XSS-10 [7], XSS-11 [8],
and ANGELS [9]. More recently, Northrop Grumman performed OOS operations
around IS-901 [10] and IS-10-02 [11]. However, these missions are operated
around a known RSO with a single chaser. In reality, the greatest collision
risks are posed by large, non-cooperative RSOs whose structure, geometry, and
capabilities are not well known. At the same time, close proximity operations
are challenging: manned missions to such unknown environments are dangerous;
lag times in data transfer make real-time, ground-based analysis of the target
during approach infeasible; and on-board computers have limited capabilities.
The stakes are high–uncontrolled collisions between the approaching spacecraft
and target would exacerbate the debris problem and could cause significant
damage and injury. A promising solution is to use an autonomous swarm of small
chaser satellites to perform operations around the target. This would
substantially lower the risks required to de-tumble and perform other OOS
operations around non-cooperative spacecraft. This requires accurate and
trustworthy machine vision and sensing systems that can run in low-compute
environments onboard the swarm.

Computer vision has become exceptionally powerful in recent years. Even
challenging missions like autonomous rendezvous and docking with non-
cooperative satellites could be feasible if modern vision algorithms can be
harnessed for use in space. There are two primary barriers: spaceflight
hardware is much less powerful than the large conventional computers typically
used for state-of-the-art computer vision and the opaque decision processes of
these models make validation difficult. Algorithmic developments have
decreased computational costs while low size, weight, and power (SWaP)
computers have improved. Prior work has sufficiently narrowed the gap in
accuracy and runtime to permit satellite component recognition to run on
spaceflight-equivalent hardware. However, the problems of interpreting model
decisions and, consequently, building trust in these techniques for high-
stakes operations is largely unsolved.

This article aims to address this problem by developing a method to provide
explanations of a lightweight satellite component detection algorithm’s
decision process. We leverage information theoretic analysis of the input data
and the model’s internal latent data representations to efficiently generate
explanations of the decision process in visual and probabilistic terms. This
provides insight into failure modes, enables more robust validation, and will
guide future model developments.

The major contributions of this work are:

  1. 1.

Probabilistic Explanations for Entropic Knowledge extraction (PEEK), an
efficient technique for explaining CNN decisions by cross-referencing
empirical entropy estimates of internal data representations with input
images.

  2. 2.

Hardware-in-the-loop experiments on video feeds of a real-world satellite
mock-up under realistic lighting and motion conditions using a YOLOv5-based
satellite component detector.

  3. 3.

An in-depth analysis of failure modes of effective satellite component
detector algorithms and attribution of failures as true risks or artifacts of
the experimental environment.

##  3 Related Literature

This research draws significantly from existing literature, including work in
computer vision such as vision systems deployed onboard spacecraft, and the
mathematical field of information theory.

###  3.1 Computer Vision

Over the past 15 years, deep CNNs have revolutionized computer vision,
overtaking traditional algorithms in nearly all vision tasks. Several factors
converged resulting in these gains. Large, annotated image datasets intended
for training were made public [12, 13]). GPU acceleration [14] enabled extreme
parallelization of neural computations, which then enabled much larger, more
expressive neural architectures to be trained [15, 16, 17]. Rapid gains in
image classification prompted the community to focus on more challenging
computer vision tasks, such as image segmentation, object detection, and
object tracking.

Object detection simultaneously identifies multiple objects in an image,
classifies them, and localizes them with tight rectangular bounding boxes with
horizontal bases (illustrated in the figure below). In this case, the process
has detected two solar panels, the body, and antenna; labeled them properly;
and localized them with tight bounding boxes. For each object, the algorithm
predicts $(x,y)$ coordinates for the center of the bounding box, the
dimensions of the box, a class prediction (solar, body, antenna, or thruster
in this case), and a score measuring how confident the model is that it has
located an object (see the numbers with the predicted class labels in the
diagram).

Figure 1: Object detection performed on satellite components

Early algorithms [18] simply performed brute-force searches for objects–they
extracted all bounding boxes along sliding windows of different sizes, fed the
extracted sub-windows to high-quality CNN image classifiers, selected those
with high classification confidence, and then pruned away redundant
detections. This approach soon gave way to algorithms designed specifically
for object detection. The multi-stage character of leading object detection
algorithms was eroded over time to reduce computational costs and improve
accuracy. Region-based CNNs (R-CNNs) [19] deployed selective search to
generate intelligent region proposals where objects may exist, leading to far
fewer, high-quality bounding boxes to be fed into CNN classifiers. OverFeat
[20] and SPPNet [21] sped up the feature extraction with new convolutional
techniques and pool features in arbitrary regions. Faster R-CNN [22] replaced
selective search with a region proposal network (RPN), a CNN that proposes
regions. The RPN was paired with the CNN classifier, and the two were trained
in alternating stages, resulting in better-quality object detections with
lower inference times. Single-stage detectors go a step further and perform
the entire task in one network that maps raw input image pixels to bounding
boxes, class probabilities, and confidence (objectness) scores simultaneously.
These algorithms run an order of magnitude faster than multi-stage networks
but were initially plagued by poor accuracy. The You Only Look Once (YOLO)
[23, 24, 25, 26, 27, 28, 29, 30, 31] family of single-stage detectors have
eliminated the gap in performance in recent years, now competing with the best
multi-stage object detectors in a single-stage architecture.

We experimented with lightweight versions of several variations of YOLO to
address the gap in the literature regarding performance comparisons. YOLOv5s,
YOLOv6n, YOLOv7t, YOLOv8n, and YOLOv8s were tested in our experiments. YOLOv5s
and YOLOv8s yielded the best accuracy on a test split of synthetic image data.
However, YOLOv5s generalized significantly better to satellites unseen during
training than YOLOv8s in hardware-in-the-loop experiments.

###  3.2 On-board Computer Vision for In-Space Applications

Prior work by the authors have explored computer vision and sensor fusion for
onboard spacecraft component detection, 3D reconstruction of spacecraft [32]
from inspection orbits, and guidance and navigation for close proximity
operations. This research demonstrated YOLOv5 [28] can effectively detect
satellite bodies, solar arrays, antennas, and thrusters [33, 34] on a low
compute budget (Raspberry Pi with Intel Neural Compute Stick 2), both in
testing images of arbitrary spacecraft and in a lab setting with a satellite
mock-up not seen during training. Multi-stage detector Faster R-CNN displayed
high object detection accuracy but could not run at frame rates sufficient for
GNC applications. [35]. Video feeds were later fused with depth data estimated
by a stereographic camera to build 3D point-clouds marking bodies, solar
arrays, and antennas. Artificial potential field (APF) guidance algorithms
marked bodies as attractive and fragile components as repulsive. It
successfully guided software-simulated [36] chaser spacecraft around fragile
solar panels and antennas to dock safely with bodies of non-cooperative
spacecraft. Later flight tests with drones [37] demonstrate this approach to
autonomous rendezvous with non-cooperative spacecraft is feasible, though
ongoing research aims to further improve model accuracy and run times.

However, CNN decision-making remains complex and difficult for humans to grasp
fully. With such high human and financial costs involved with failure, a model
providing no explanations for its decisions is not likely to be adopted for
these safety-critical on-orbit missions. In addition, it is difficult to
develop remedies and introduce safeguards into the algorithm without a better
understanding of why failures occur. These concerns call for further study
into how, when, and why models fail.

###  3.3 Visualization of Neural Network Predictions

One of the most commonly used methods for CNN classifier visualization is
gradient-weighted class activation mapping (Grad-CAM) [38], which measures and
visualizes the importance of each pixel of the input image to the CNN’s
classification decision. For a given target class and an image, it performs a
forward pass with the CNN and extracts the class activation maps (CAMs) of the
final convolutional layer. This layer keys in on high-level features used in
the CNN’s decision process. Grad-CAM then finds the gradient of the target
class score with respect to CAMs from earlier layers using a backward pass.
These gradients are pooled spatially across the CAMs. After applying the ReLU
activation and upscaling using guided backpropagation [39], the result is
overlaid as a heatmap on the input image, where the higher intensities
correspond are pixels that were most influential to the class prediction.

Grad-CAM and several other visualization methods (e.g., Grad-CAM++ [40], CNN
fixations [41], DeepLIFT [42]) all fall under the class discriminative
methods. The performance of these methods heavily rely on the classification
being accurate. If objects are classified incorrectly, the visualizations will
be distorted or incorrect, defeating the purpose of visualizing the CNNs
decision process. To overcome the challenges of the class discriminative
methods, Eigen-CAM [43] works based entirely on the forward pass. It assumes
that during the training process of the CNN, all non-relevant features learned
from the training data will be regularized or smoothed out and only relevant
features are preserved in the model, hence eliminating the dependency on
classifier performance and providing class-agnostic analysis of CNN decisions.

Eigen-CAM uses singular value decomposition (SVD) to compute the principal
components of the feature maps and projects the input image onto the last
convolutional layer. The class activation map is then generated by projecting
this output onto the first eigenvector. While Eigen-CAM does not need to
compute a expensive backward pass, the SVD can still be computationally
intensive.

###  3.4 Entropy and Information Theory

Information theory is the study of information within data signals [44]. Of
primary interest to this work are input signals, i.e. images or video feeds,
and signals passing through the hidden layers of neural networks. Entropy is
an information theoretic measure of uncertainty in the distribution of a
random variable $X$. The (differentiable) entropy of a continuous random
variable $X$ valued in $\mathbb{R}^{k}$ with probability density function
(pdf) $f$ is defined as

$\displaystyle H(X)=\mathbb{E}\left[-\log
f(X)\right]=-\int_{\mathbb{R}^{k}}f(x)\log f(x)\,dx.$ (1)

Recent work [45] established the importance of entropy propagation through
CNNs to their performance. Hence, we hypothesize the entropy of the internal
feature maps of a CNN computed during a forward pass can help explain the
model’s decisions.

##  4 Motivation and Methods

###  4.1 Patterns in CNN Latent Representations

Inside CNN layers are convolutional filters that, when applied to an input
image, compute a feature map. More specifically, the weights inside the
filters are learned during model training to extract useful features such as
edges and color to create these feature maps, which are useful for the
downstream task of object detection. These features are stored as hidden
representations of the input data, called feature maps. An example of this
process on can be seen in the figure below, where we extracted two $6\times
6\times 3$ convolutional filters (the two rows of in Figure 2: one $6\times 6$
slice applied to each color channel) and the feature maps they produce from
the first convolutional module of YOLOv5. Figure 2 shows the impact of the
filters to achieve specific feature maps.

Figure 2: Features extracted by convolutional filters of the first layer of
YOLOv5

YOLOv5 was trained to perform object detection by minimizing classification
and localization errors. During training, these two filters learned to become
edge detectors despite no explicit guidance by the optimization process. In
particular, the first filter in Figure 2 has negative weights (darker) on the
bottom of the filter and positive weights (lighter) on the top, both with more
intensity near the center. This combination of blurring (more muted values in
the corners) and horizontal dividing line between positives and negatives is
equivalent to the classical top Sobel filter [46] for detecting horizontal
lines. Note the horizontal lines on the solar panels are easily visible but
the vertical lines are not. The resulting Feature Map 1 clearly enhances
horizontal lines. Similarly, the second filter is similar to the classical
right Sobel filter for detecting vertical lines, and we see the vertical lines
on the solar panel on Feature Map 2 in Figure 2.

The collection of feature maps in a convolutional layer contain the latent
representations of the data as it flows through the network (as seen above).
This is where we will be performing information theoretic analysis because
they contain signals with information content that helps us to perform the
downstream task.

###  4.2 PEEK

A pixel-level entropy measure applied to the feature maps will be used to
analyze the latent data representations within the YOLOv5 architecture and its
internal feature maps. This analysis adds transparency to the model’s
decision-making process and provides an avenue to detect failure modes and
offer remedies for them.

Our method, Probabilistic Explanations for Entropic Knowledge extraction
(PEEK), computes a depth-wise entropy-related measure at each “pixel” of the
latent feature maps of hidden convolutional layers of CNNs. These latent
representations $L\in\mathbb{R}^{l\times w\times d}$ are stacks of feature
maps of shape $l\times w$ extracted by the convolutional filters in the layer.
We first shift the “pixel” intensities of each feature map to be positive:

$\displaystyle L^{\text{pos}}_{ijk}=L_{ijk}+\left|\min\limits_{\\{i\leq
l,j\leq w\\}}L_{ijk}\right|$ (2)

Since all feature maps tend to have both positive and negative values, formula
(2) shifts the values to the positive scale for each feature map separately.
This both preserves the relative differences between pixels within a feature
map and retains the strength of intense feature maps. PEEK then computes the
entropy ($g(x)=-x\ln x$ as defined in the convex optimization context [47]) of
each depth-wise column of pixels in spatial location $(i,j)$, resulting in the
PEEK intensity at that location:

$\displaystyle P_{ij}$
$\displaystyle=-\sum\limits_{k=1}^{d}g\left(L^{\text{pos}}_{ijk}\right)$ (3)

We apply this PEEK calculation to each spatial location in the hidden
representation $L$ to compute a full PEEK map for the selected convolutional
layers of the neural network. This PEEK map can be resized and overlaid on the
input image, resulting in highlighting of high-entropy locations in the
feature maps corresponding to that location.

In this paper, we will employ PEEK and Eigen-CAM to visualize the latent data
representations within a YOLOv5s model trained for satellite component
detection. This will enable a direct comparison of visualization techniques
and their effectiveness.

##  5 Performance Evaluation and Experimental Conditions

This section specifies the object detection performance metrics, the datasets
used for training and testing the algorithms, the lab setting for hardware-in-
the-loop experiments.

###  5.1 Object Detection Metrics

The model’s performance is measured in terms of its ability to classify and
localize features. This paper evaluates and compares the mean average
precision (mAP) at the intersection over union (IoU) thresholds 0.5 and
0.5:0.95. IoU measures the overlap between two bounding boxes by dividing the
area of their intersection by the area of their union. It ranges from 0 (no
overlap) to 1 (full overlap).

$\displaystyle IoU$ $\displaystyle=\frac{\text{Area of intersection of
bounding boxes}}{\text{Area of union of bounding boxes}}$ (4)

A predicted class and bounding box is called a true positive (TP) in object
detection if the bounding box has IoU with a ground-truth bounding box above a
specified IoU threshold and it is classified correctly. Precision (P) is the
ratio of TP detections to all positive detections made by the model, which
descries how reliable model predictions are. Recall (R) is the ratio of TP
detections to all positive instances in the dataset, which describes how well
the model detects objects in images.

$\displaystyle P$ $\displaystyle=\frac{TP}{\text{Positive object
predictions}}$ (5) $\displaystyle R$ $\displaystyle=\frac{TP}{\text{Ground-
truth object instances}}$ (6)

A precision-recall (PR) curve is generated by plotting precision versus recall
for a range of confidence thresholds required to make detections for each
class. Average precision (AP) for a specified class is the area under the PR
curve for that class. Mean average precision (mAP) is the mean of class-
specific APs. The resulting mAP is a one-number summary of object detection
quality for a specified IoU threshold.

###  5.2 Training Dataset

The training data consists of images of spacecraft from Google, mostly
generated by STK and Kerbal Space Program as well as Blender-generated images
of NASAs stereo lithography models. A total of 1231 images are collected for
the training dataset. Figure 3 shows the total number of annotations present
in the dataset for each component.

Figure 3: Training Dataset

This dataset was randomly split into a train, validation, and test subsets
(60%/20%/20%). YOLOv5s was trained on the training split using image
augmentation to include rotation, black cut-out boxes, Gaussian noise, and
greyscale based on camera issues. These are shown in Figure 4.

Figure 4: Training Dataset Augmentation

The validation subset was used for tuning the model. The tuned model achieves
0.587<EMAIL_ADDRESS>on the test subset not seen during training. More rigorous
testing was then performed in the lab as discussed next.

###  5.3 Testing Dataset and Experimental Setting

This paper uses an experimental testing dataset [48] that includes hardware-
in-the-loop image frames extracted at 30 frames per second from two videos of
a mock-up satellite shot under extreme lighting conditions. The mock-up
satellite yaws at a constant angular rate of 1.5 deg/second for 3 minutes on
the ORION test bed [49] at the Autonomy Lab in Florida Tech. The ORION testbed
depicted in Figure 5 enables us to create in-space visual conditions suitable
for testing this algorithm. The test bed is equipped with a planar maneuver
kinematics simulator that accommodates pan-tilt mechanisms custom-built to
simulate on-orbit proximity operations used to simulate chaser satellite
motion, as seen in Figure 5A. The simulator uses a separate stationary pan-
tilt head to simulate the attitude motion of a target satellite model. The
model features geometry and surfaces typically found on satellites, such as
solar panels, parabolic antennas, body panels, and thruster nozzles. All the
walls, floors, ceiling and windows panels of the room are covered with high
absorptive black paint and black curtains. A Hilio D12 LED LitePanel capable
of generating light with a color of 5,600K (daylight balanced temperature) is
used to simulate space-like lighting conditions, as seen in Figure 5B.

A Target satellite mock-up and chaser simulator

B Testbed simulation of extreme lighting conditions

Figure 5: The ORION Testbed at the Autonomy Lab at Florida Tech

Figure 6 depicts the satellite mock-up with its features. It contains 1800
frames annotated with its two solar panels, thruster, body, and antenna. The
object detectors used below attempt to detect each visible feature in each
frame of the video.

Figure 6: Satellite mock-up and its features

The trained YOLOv5s model achieves 0.377<EMAIL_ADDRESS>The performance is degraded
in this real-world setting, but this remains relatively good performance when
compared to the object detection literature.

##  6 Experimental Results

###  6.1 Architecture Analysis via PEEK

The architecture of YOLOv5 as shown in Figure 7 is made up of 24 modules
separated into the backbone, neck, and head. The backbone (modules 0-9)
consists of strided convolutional layers and C3 modules, adapted from [50] and
optimized). As the convolutional filter scans over the input, it extracts
patterns from image patches spanning the entire input but represents them in a
lower dimension in the next layer. This shrinks the feature map shapes by half
while still representing the entire image, so each pixel of the output
represents a larger image patch. The next layer scans over those output
feature maps so they extract patterns from regions that represent more space
in the original image. Therefore, this portion of the backbone extracts
features at successively larger spatial scales. Last in the backbone is
spatial pyramidal pooling “fast” (SPPF) [21], which pooling at different
scales, helping the model capture features useful for detecting objects of
various sizes.

The neck (modules 10-23) synthesizes features extracted at different scales
and with different levels of abstraction. Modules 10-17 work with the backbone
to form a feature pyramid network (FPN) [51]. The FPN treats the successively
smaller feature maps learned by the backbone as the levels of the pyramid. The
bottom of the pyramid corresponds to larger feature maps that have high enough
resolution to encode small, fine-grained features. As we move up the pyramid,
the model starts to learn larger features. While the FPN focuses on features
of the objects themselves, modules 18-23 form a path aggregation network
(PANet) [52] that helps estimate object positioning. It also shortens the path
from detection heads that focus on detecting small, medium, and large-scale
objects to appropriately-sized features of those objects learned by the FPN.

The head (module 24) predicts bounding boxes and class probabilities for each
object identified in the input image. After performing non-max suppression to
remove redundant detections, the entire architecture is trained to minimize
errors for objects detected by the heads in terms of both object
classification and localization by bounding boxes.

Figure 7: YOLOv5 (v6.0/6.1) Architecture

PEEK reveals numerous intuitive results regarding the architecture of YOLOv5.
In Figure 8A, the high-entropy features in Module 5’s feature maps include the
satellite and its features as intended, but also the metal tracks along the
ground and the door frame on the wall to the left. In addition, the
satellite’s solar panels, body, and antenna fade into one another, which would
make disentangling the objects difficult. The C3 Module and its bottlenecks
reduce the relative entropy of the extraneous features and provide clear
separation between the satellite features. Upsampling Module 11 from the neck
simply performs nearest neighbor upscaling to increase the resolution of the
inputs. As expected, it has no impact on the PEEK features aside from making
them higher resolution like the feature maps: as we see in Figure 8B, they
just become more pixelated. Lastly, PEEK demonstrates that Concatenation
Module 16 successfully fuses the well-defined features in Module 4 and more
abstract features from Module 10 into intense but disentangled satellite
features and well-segmented extraneous features on the back wall of the lab,
as seen in Figure 8C.

A C3 Module

B Upsampling

C Concatenation

Figure 8: The effects of architectural structures on latent representations as
revealed by PEEK

###  6.2 PEEK vs Eigen-CAM

PEEK and Eigen-CAM both extract hidden representations computed in a forward
pass through the network and provide visual cues into the model’s detection
process. In Figure 9A is a frame from a synthetic satellite video with
YOLOv5’s detections annotated. In order to analyze how the algorithm made its
decisions, we compare Eigen-CAM and PEEK maps of 5 modules from the model’s
architecture in Figure 9B. Both methods were prone to distraction by the Earth
in the image, however, PEEK was far less impacted.

In Module 1, both Eigen-CAM and PEEK do a good job of finding edges,
specifically in the satellite itself. Yet in Module 7, both are being fooled
by the earth. However, PEEK’s strongest values in red are seen to be only on
the satellite while the opposite is true for Eigen-CAM. The further we move
through the algorithm, PEEK starts to clearly show the algorithm’s focus that
led to a detection. In Modules 16 and 19, PEEK shows some distraction in space
and on the Earth, but is most strongly focused on the satellite, while Eigen-
CAM is lost in the boundaries of the image and has minimal focus on the
satellite. Similarly, in Module 22, PEEK is focused specifically on the
location of the detections, while Eigen-CAM still has focus on the areas
surrounding.

These results show PEEK provides visual explanations with greater explanatory
value in the task of object detection.

A YOLOv5 Detections in Synthetic Video Frame

B PEEK vs. Eigen-CAM

Figure 9: PEEK Analysis of YOLOv5 Satellite Component Detector on a Synthetic
Image

###  6.3 PEEK Practical Uses

PEEK has allowed us to fully visualize and understand the structure of YOLOv5,
explain detections, and provide us a better visualization of the internal
representations of the network. In addition to the human interpretable
ability, PEEK has many practical uses.

In Figure 10C, we compare PEEK to Eigen-CAM applied to the same detection
frame from our lab testing dataset. Eigen-CAM only provides minimal
explanation of the detection and errors. For example, in Module 16, Eigen-CAM
focuses on the satellite, along with the edges of the frame. This is where we
have an incorrect detection of a solar panel on the far right of Figure 10A.
However, in every other Module, it provides us little to no explanation or
interpretation.

PEEK provides us explanations to possible reasons for errors in the detection
along with possible issues with the data itself. In Modules 7, 16, and 19, we
have high entropy visuals seen on the satellite, around the satellite in clear
circles or squares, and a highlighted portion on the far right of the image.
The highlighted portion on the satellite was where the algorithm correctly
detected and classified a solar panel and antenna. Though the algorithm misses
the body of the satellite, in PEEK’s modules 19 and 22, we see that the
entropy values are weak on areas around the antenna on the body, showing that
it is in fact only focusing on the antenna. In Modules 16 and 19, PEEK clearly
highlights a shape on the far right that is then detected and classified as a
solar panel incorrectly. However, due to the highlighted shapes surrounding
the the satellite, it was clear that the algorithm was picking up artifacts in
the frames that were not necessarily visible to the human eye without further
inspection.

Due to these artifacts seen in PEEK, we decided to take the same frame and
brighten the image. Doing this revealed some aspects of our mock set up that
could be confusing the algorithm. While the room is prepped to minimize the
amount of light as much as possible, there are slight exposures that were only
noticeable due to PEEK. In Figure 10B, we identified the artifacts seen in
PEEK in Figure 10C as air vents, a camera, a doorframe, and then a table in
the far right corner which later was incorrectly detected and classified as a
solar panel. This finding shows the value of PEEK and the ability to find
artifacts in data, explain errors, and interpret the detections as a whole.
This allows us to reevaluate our data and make changes that could improve the
quality of our training, minimize biases, and prevent confusion of the
algorithm.

A YOLOv5 Detections in the Lab

B Brightened Image with Visible Extraneous Lab Objects

C PEEK vs. Eigen-CAM

Figure 10: PEEK for Analysis of YOLOv5 Satellite Component Detector on a Real-
World Satellite Mock-Up

Lastly, the efficient PEEK calculations run far faster than Eigen-CAM. To
compare them, we ran both methods 10 times, computing heatmaps for all 23
layers of YOLOv5 for the same input frames, and averaged the runtimes for each
layer in Figure 11. The compute costs rise and fall with the dimensions of the
latent representations of each layer: higher at the top of the backbone in
layers and top of the FPN layer 16, and lower elsewhere. Overall, PEEK
runtimes are 100 to 1000 times faster than Eigen-CAM while exhibiting the
benefits demonstrated above.

Figure 11: PEEK vs. Eigen-CAM Runtimes

##  7 Conclusion

In conclusion, the growing congestion of LEO coupled with the impending risks
of space debris present a pressing challenge for space agencies and private
entities alike. This challenge has been exacerbated by the lack of complete
awareness of all orbiting spacecraft, especially non-cooperative ones, making
close-proximity operations a high-stakes endeavor. Addressing these issues
requires innovative solutions, and one promising approach is the use of
autonomous swarms of small chaser satellites.

However, the success of this approach hinges on the deployment of accurate and
trustworthy machine vision and sensing systems that can operate in low-compute
environments onboard the swarm. While computer vision has made significant
strides, the unique constraints of spaceflight hardware and the opaqueness of
decision-making in modern vision algorithms have presented substantial
hurdles.

This article tackled these challenges by introducing the Probabilistic
Explanations for Entropic Knowledge extraction (PEEK) method, a novel
technique designed to shed light on the decision process of lightweight
satellite component detection algorithms. By leveraging information theoretic
analysis and internal data representations, PEEK provides valuable insights
into the workings of these algorithms, offering explanations in both visual
and probabilistic terms. This not only aids in understanding failure modes but
also facilitates robust validation and guides the future development of these
critical models.

The ability to reevaluate our data in this manner offers a transformative
opportunity to enhance the quality of our training datasets and experimental
choices. By pinpointing sources of error and bias, we can take proactive steps
to rectify these issues, ultimately leading to more accurate and reliable
satellite component detection algorithms. Moreover, PEEK empowers us to
prevent potential confusion in the algorithm’s decision-making process,
thereby ensuring the safe and efficient operation of autonomous satellite
swarms in LEO.

PEEK represents a valuable asset in the ongoing quest to improve the
robustness and effectiveness of machine vision systems in space and beyond.

## Acknowledgments

R. T. White thanks the NVIDIA Applied Research Accelerator Program for
hardware support. The authors also thank Energy Management Aerospace, LLC, and
the Oak Ridge Institute for Science and Education (ORISE) for financial
support of M.J. Meni (MIPRs W81EWF30204272 and W81EWF31876813). Opinions,
interpretations, conclusions, and recommendations are those of the author(s)
and are not necessarily endorsed by the U.S. Army.

## References

  * Johnson [2021] Johnson, N. L., “Operation Burnt Frost: A View From Inside,” _Space Policy_ , Vol. 56, 2021, p. 101411.  10.1016/j.spacepol.2021.101411.
  * Kestenbaum [2007] Kestenbaum, D., “Chinese Missile Destroys Satellite in 500-Mile Orbit,” , Jan. 2007.  URL https://www.npr.org/2007/01/19/6923805/chinese-missile-destroys-satellite-in-500-mile-orbit.
  * Henry [2019] Henry, C., “India ASAT debris spotted above 2,200 kilometers, will remain a year or more in orbit,” , Apr. 2019.  URL https://spacenews.com/india-asat-debris-spotted-above-2200-kilometers-will-last-a-year-or-more/.
  * U.S. Space Command Public Affairs Office [2021] U.S. Space Command Public Affairs Office, “Russian direct-ascent anti-satellite missile test creates significant, long-lasting space debris,” , Nov. 2021.  URL https://www.spacecom.mil/Newsroom/News/Article-Display/Article/2842957/russian-direct-ascent-anti-satellite-missile-test-creates-significant-long-last/.
  * United Nations General Assembly [2022] United Nations General Assembly, “Resolution: Destructive direct-ascent anti-satellite missile testing,” , Dec. 2022.  URL http://digitallibrary.un.org/record/3996915.
  * Cheng et al. [2018] Cheng, A. F., Rivkin, A. S., Michel, P., Atchison, J., Barnouin, O., Benner, L., Chabot, N. L., Ernst, C., Fahnestock, E. G., Kueppers, M., Pravec, P., Rainey, E., Richardson, D. C., Stickle, A. M., and Thomas, C., “AIDA DART asteroid deflection test: Planetary defense and science objectives,” _Planetary and Space Science_ , Vol. 157, 2018, pp. 104–115.  10.1016/j.pss.2018.02.015.
  * Davis et al. [2003] Davis, T., Baker, M. T., Belchak, T., and Larsen, W., “XSS-10 micro-satellite flight demonstration program,” 2003.
  * Air Force Research Laboratory [2011] Air Force Research Laboratory, “XSS-11 Micro Satellite,” Tech. rep., 2011\.  URL https://www.kirtland.af.mil/Portals/52/documents/AFD-111103-035.pdf?ver=2016-06-28-110256-797.
  * Air Force Research Laboratory [2014] Air Force Research Laboratory, “Automated Navigation and Guidance Experiment for Local Space (ANGELS),” Tech. rep., 2014.
  * Sheetz [2020] Sheetz, M., “For the first time ever, a robotic spacecraft caught an old satellite and extended its life,” , Apr. 2020.  URL https://www.cnbc.com/2020/04/17/northrop-grumman-mev-1-spacecraft-services-intelsat-901-satellite.html.
  * Northrop Grumman [2021] Northrop Grumman, “Northrop Grumman and Intelsat Make History with Docking of Second Mission Extension Vehicle to Extend Life of Satellite,” , Apr. 2021.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., “ImageNet: A large-scale hierarchical image database,” _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.  10.1109/CVPR.2009.5206848.
  * Lin et al. [2014] Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L., “Microsoft COCO: Common objects in context,” _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , Springer, 2014, pp. 740–755.
  * Krizhevsky et al. [2017] Krizhevsky, A., Sutskever, I., and Hinton, G. E., “ImageNet classification with deep convolutional neural networks,” _Communications of the ACM_ , Vol. 60, No. 6, 2017, pp. 84–90.  10.1145/3065386.
  * Simonyan and Zisserman [2014] Simonyan, K., and Zisserman, A., “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * Szegedy et al. [2015] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A., “Going deeper with convolutions,” _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1–9.  10.1109/CVPR.2015.7298594.
  * He et al. [2016] He, K., Zhang, X., Ren, S., and Sun, J., “Deep residual learning for image recognition,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * Szegedy et al. [2013] Szegedy, C., Toshev, A., and Erhan, D., “Deep Neural Networks for Object Detection,” _Advances in Neural Information Processing Systems_ , Vol. 26, edited by C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Curran Associates, Inc., 2013.  URL https://proceedings.neurips.cc/paper_files/paper/2013/file/f7cade80b7cc92b991cf4d2806d6bd78-Paper.pdf.
  * Girshick et al. [2013] Girshick, R. B., Donahue, J., Darrell, T., and Malik, J., “Rich feature hierarchies for accurate object detection and semantic segmentation,” _CoRR_ , Vol. abs/1311.2524, 2013.  URL http://arxiv.org/abs/1311.2524.
  * Sermanet et al. [2013] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y., “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks,” , 2013.  10.48550/ARXIV.1312.6229.
  * He et al. [2015] He, K., Zhang, X., Ren, S., and Sun, J., “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , Vol. 37, No. 9, 2015, pp. 1904–1916.
  * Ren et al. [2015] Ren, S., He, K., Girshick, R. B., and Sun, J., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” Vol. abs/1506.01497, 2015.  URL http://arxiv.org/abs/1506.01497.
  * Redmon et al. [2015] Redmon, J., Divvala, S. K., Girshick, R. B., and Farhadi, A., “You Only Look Once: Unified, Real-Time Object Detection,” Vol. abs/1506.02640, 2015.  URL http://arxiv.org/abs/1506.02640.
  * Redmon and Farhadi [2016] Redmon, J., and Farhadi, A., “YOLO9000: Better, Faster, Stronger,” _arXiv preprint arXiv:1612.08242_ , 2016.
  * Redmon and Farhadi [2018] Redmon, J., and Farhadi, A., “YOLOv3: An Incremental Improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * Bochkovskiy et al. [2020] Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y. M., “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * Ge et al. [2021] Ge, Z., Liu, S., Wang, F., Li, Z., and Sun, J., “Yolox: Exceeding yolo series in 2021,” _arXiv preprint arXiv:2107.08430_ , 2021.
  * Jocher et al. [2022] Jocher, G., Ayush Chaurasia, Stoken, A., Borovec, J., NanoCode012, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang, Imyhxy, , Lorna, Zeng Yifu, Wong, C., Abhiram V, Montes, D., Zhiqiang Wang, Fati, C., Jebastin Nadar, Laughing, UnglvKitDe, Sonck, V., Tkianai, YxNONG, Skalski, P., Hogan, A., Dhruv Nair, Strobel, M., and Jain, M., “ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation,” , 2022.  10.5281/ZENODO.7347926.
  * Li et al. [2022] Li, C., Li, L., Jiang, H., Weng, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., Nie, W., et al., “YOLOv6: A single-stage object detection framework for industrial applications,” _arXiv preprint arXiv:2209.02976_ , 2022.
  * Wang et al. [2022] Wang, C.-Y., Bochkovskiy, A., and Liao, H.-Y. M., “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” _arXiv preprint arXiv:2207.02696_ , 2022.
  * Jocher et al. [2023] Jocher, G., Chaurasia, A., and Qiu, J., “YOLO by Ultralytics,” , Jan. 2023\.  URL https://github.com/ultralytics/ultralytics.
  * Caruso et al. [2023] Caruso, B., Mahendrakar, T., Nguyen, V. M., White, R. T., and Steffen, T., “3D Reconstruction of Non-cooperative Resident Space Objects using Instant NGP-accelerated NeRF and D-NeRF,” _arXiv preprint arXiv:2301.09060_ , 2023.
  * Mahendrakar et al. [2021a] Mahendrakar, T., White, R. T., Wilde, M., Kish, B., and Silver, I., “Real-time Satellite Component Recognition with YOLO-V5,” _35th Annual Small Satellite Conference_ , 2021a.
  * Mahendrakar et al. [2023a] Mahendrakar, T., Attzs, M. N., Tisaranni, A. L., Duarte, J. M., White, R. T., and Wilde, M., “Impact of Intra-class Variance on YOLOv5 Model Performance for Autonomous Navigation around Non-Cooperative Targets,” _AIAA SCITECH 2023 Forum_ , 2023a, p. 2374.
  * Mahendrakar et al. [2022] Mahendrakar, T., Ekblad, A., Fischer, N., White, R., Wilde, M., Kish, B., and Silver, I., “Performance Study of YOLOv5 and Faster R-CNN for Autonomous Navigation around Non-Cooperative Targets,” _2022 IEEE Aerospace Conference (AERO)_ , IEEE, 2022, pp. 1–12.
  * Mahendrakar et al. [2021b] Mahendrakar, T., Wilde, M., and White, R., “Use of artificial intelligence for feature recognition and flightpath planning around non-cooperative resident space objects,” _ASCEND 2021_ , 2021b, p. 4123.
  * Mahendrakar et al. [2023b] Mahendrakar, T., Holmberg, S., Ekblad, A., Conti, E., White, R. T., Wilde, M., and Silver, I., “Autonomous Rendezvous with Non-Cooperative Target Objects with Swarm Chasers and Observers,” _arXiv preprint arXiv:2301.09059_ , 2023b.
  * Selvaraju et al. [2017] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D., “Grad-cam: Visual explanations from deep networks via gradient-based localization,” _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * Springenberg et al. [2015] Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A., “Striving for Simplicity: The All Convolutional Net,” _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings_ , edited by Y. Bengio and Y. LeCun, 2015.  URL http://arxiv.org/abs/1412.6806.
  * Chattopadhay et al. [2018] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N., “Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks,” _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2018, pp. 839–847.  10.1109/WACV.2018.00097.
  * Mopuri et al. [2018] Mopuri, K. R., Garg, U., and Babu, R. V., “Cnn fixations: an unraveling approach to visualize the discriminative image regions,” _IEEE Transactions on Image Processing_ , Vol. 28, No. 5, 2018, pp. 2116–2125.
  * Shrikumar et al. [2017] Shrikumar, A., Greenside, P., and Kundaje, A., “Learning important features through propagating activation differences,” _International conference on machine learning_ , PMLR, 2017, pp. 3145–3153.
  * Muhammad and Yeasin [2020] Muhammad, M. B., and Yeasin, M., “Eigen-cam: Class activation map using principal components,” _2020 international joint conference on neural networks (IJCNN)_ , IEEE, 2020, pp. 1–7.
  * Cover and Thomas [2006] Cover, T. M., and Thomas, J. A., _Elements of information theory_ , Wiley John + Sons, 2006.
  * Meni et al. [2023] Meni, M. J., White, R. T., Mayo, M., and Pilkiewicz, K., “Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performance,” _arXiv preprint arXiv:2308.14938_ , 2023.
  * Marr and Hildreth [1980] Marr, D., and Hildreth, E., “Theory of edge detection,” _Proceedings of the Royal Society of London. Series B. Biological Sciences_ , Vol. 207, No. 1167, 1980, pp. 187–217.
  * Boyd and Vandenberghe [2004] Boyd, S., and Vandenberghe, L., _Convex Optimization_ , Cambridge University Press, 2004.  10.1017/CBO9780511804441.
  * Attzs et al. [2023] Attzs, M. N. J., Mahendrakar, T., Meni, M. J., White, R. T., and Silver, I., “Comparison of Tracking-By-Detection Algorithms for Real-Time Satellite Component Tracking,” _37th Annual Small Satellite Conference_ , 2023.
  * Wilde et al. [2016] Wilde, M., Kaplinger, B., Go, T., Gutierrez, H., and Kirk, D., “ORION: A simulation environment for spacecraft formation flight, capture, and orbital robotics,” _2016 IEEE Aerospace Conference_ , IEEE, 2016, pp. 1–14.
  * Wang et al. [2020] Wang, C.-Y., Liao, H.-Y. M., Wu, Y.-H., Chen, P.-Y., Hsieh, J.-W., and Yeh, I.-H., “CSPNet: A new backbone that can enhance learning capability of CNN,” _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 390–391.
  * Lin et al. [2017] Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S., “Feature pyramid networks for object detection,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * Liu et al. [2018] Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J., “Path aggregation network for instance segmentation,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.