# Georeferencing of Photovoltaic Modules from Aerial Infrared Videos using
Structure-from-Motion

Lukas Bommes  Forschungszentrum Jülich GmbH, Helmholtz-Institute Erlangen-
Nuremberg for Renewable Energies (HI ERN)  Claudia Buerhop-Lutz
Forschungszentrum Jülich GmbH, Helmholtz-Institute Erlangen-Nuremberg for
Renewable Energies (HI ERN)  Tobias Pickel  Forschungszentrum Jülich GmbH,
Helmholtz-Institute Erlangen-Nuremberg for Renewable Energies (HI ERN)  Jens
Hauch  Forschungszentrum Jülich GmbH, Helmholtz-Institute Erlangen-Nuremberg
for Renewable Energies (HI ERN)  Christoph Brabec  Forschungszentrum Jülich
GmbH, Helmholtz-Institute Erlangen-Nuremberg for Renewable Energies (HI ERN)
Institute Materials for Electronics and Energy Technology, Universität
Erlangen-Nürnberg (FAU)Correspondence to<EMAIL_ADDRESS>Ian Marius
Peters  Forschungszentrum Jülich GmbH, Helmholtz-Institute Erlangen-Nuremberg
for Renewable Energies (HI ERN)

###### Abstract

To identify abnormal photovoltaic (PV) modules in large-scale PV plants
economically, drone-mounted infrared (IR) cameras and automated video
processing algorithms are frequently used. While most related works focus on
the detection of abnormal modules, little has been done to automatically
localize those modules within the plant. In this work, we use incremental
structure-from-motion to automatically obtain geocoordinates of all PV modules
in a plant based on visual cues and the measured GPS trajectory of the drone.
In addition, we extract multiple IR images of each PV module. Using our
method, we successfully map $99.3\text{\,}\mathrm{\char 37\relax}$ of the
$35084$ modules in four large-scale and one rooftop plant and extract over
$2.2$ million module images. As compared to our previous work, extraction
misses $18$ times less modules (one in $140$ modules as compared to one in
eight). Furthermore, two or three plant rows can be processed simultaneously,
increasing module throughput and reducing flight duration by a factor of $2.1$
and $3.7$, respectively. Comparison with an accurate orthophoto of one of the
large-scale plants yields a root mean square error of the estimated module
geocoordinates of $5.87\text{\,}\mathrm{m}$ and a relative error within each
plant row of $0.22\text{\,}\mathrm{m}$ to $0.82\text{\,}\mathrm{m}$. Finally,
we use the module geocoordinates and extracted IR images to visualize
distributions of module temperatures and anomaly predictions of a deep
learning classifier on a map. While the temperature distribution helps to
identify disconnected strings, we also find that its detection accuracy for
module anomalies reaches, or even exceeds, that of a deep learning classifier
for seven out of ten common anomaly types. The software is published at
https://github.com/LukasBommes/PV-Hawk.

## I. Introduction

The large amount of global installed solar photovoltaics (PV) and expected
future growth require automatic image analysis for adequate quality control.
As PV modules may develop defects due to environmental influences, aging or
incorrect handling, PV plants need to be inspected regularly to ensure safe
operation and maximum yield. Due to the large size of most PV plants,
inspection is only economic if highly automated [1]. Thus, recent years have
seen a surge in automated PV plant inspection systems, such as the ones by
Zefri et al. [2], Pierddicca et al. [3], Henry et al. [4], and Carletti et al.
[5]. These systems rely on drones equipped with a thermal infrared (IR)
camera, that enables detection of abnormal PV modules based on their thermal
signature [6]. The large amounts of acquired IR images are automatically
processed by computer vision algorithms, which typically detect PV modules in
the images, predict module anomalies, and localize each module in the PV
plant.

In this work, we focus on the localization of PV modules in large-scale
plants. Localization is a crucial task as it enables targeted repairs of
abnormal modules. However, it is also notoriously difficult to identify the
correct module among millions of identically looking and densely packed
modules from a highly repetitive video with only a limited viewport. Previous
works attempted to solve this problem by stitching adjacent video frames of a
PV plant row into a panorama image [7, 8, 9]. This approach was successful,
yet only works well for short video sequences. And, as also shown in our
previous work [10], panorama stitching requires manual selection of the video
frames for each row and provides the module location only relative to other
modules. Niccolai et al. [11] also use panorama stitching and additionally
match each row panorama to a CAD plan. While this yields absolute module
locations, it requires a CAD plan, which is not always available and, even if
it is available, is by no means standardized across different PV plants.

Other works explore direct georeferencing of PV modules in each image based on
the measured GPS position and altitude of the drone [12, 13]. Georeferencing
requires a centimeter-accurate Realtime Kinematics GPS (RTK-GPS) and is prone
to GPS measurement errors as no additional visual cues are considered.
Further, georeferencing is limited to nadiral images, which may contain sun
reflections and exhibit sub-optimal contrast compared to images taken under
the optimal viewing angle. Being limited to nadiral images also makes drone
operation more difficult.

Another method for module localization is the creation of an orthophoto from a
few high-altitude images [14, 15, 16]. Orthophotos allow visualizing the
temperature distribution of the entire PV plant. One issue with this approach
is that it is not always possible to take images from high altitudes, e.g. if
there are nearby streets. Furthermore, a low spatial resolution and possible
visual artefacts impede accurate detection of abnormal modules based on the
orthophoto alone.

This work presents a new method for PV plant inspection based on aerial IR
videos. As opposed to the related works, our method is fully automated,
provides the absolute geocoordinates of each PV module instead of a relative
location, works on long video sequences of large-scale plants, requires no CAD
plan, works with both standard GPS and RTK-GPS and is not limited to nadiral
videos. Furthermore, videos can be acquired from low flight altitudes and
multiple high resolution images of each module are obtained, which are
important for downstream analysis.

Our method builds on our previous work [10], but features a more general
approach for PV module localization based on structure-from-motion (SfM) [17,
18] to obtain absolute geocoordinates of the PV modules in a plant. In
addition, PV module images are extracted from each video frame and tracked
over subsequent frames. Based on the extracted images, module anomalies can be
detected with a deep learning classifier [19] and visualized on a map. This
enables quick assessment of the health state of the entire PV plant and helps
performing targeted repairs. Similarly, module temperatures can be mapped
across the PV plant. Temperature mapping allows detecting abnormal modules by
comparison with neighbouring modules. This approach can replace more complex
deep learning classifiers for detecting abnormal modules, as we will show. As
opposed to the related works, our method relies on both visual cues and
measured GPS trajectory for georeferencing. This improves robustness to GPS
measurement errors and allows to use standard GPS instead of RTK-GPS. We
further use videos instead of individual images. Video analysis speeds up data
acquisition and works not only with automatic waypoint flights, but also with
manual flights performed ad-hoc for small and irregular plants. Videos also
yield larger amounts of data as each PV module is captured in multiple video
frames, which is beneficial for training machine learning algorithms on the
extracted data. Compared to our previous work, requirements on the flight
trajectory are less stringent, and, in principle, plants with non-row layouts,
such as rooftop plants, can be processed. Having a single tool for different
types of PV plants is more cost-effective and requires less maintenance than
multiple plant-specific solutions. We also show that for regular plants,
multiple rows can be scanned simultaneously, which significantly increases
throughput.

## II. Method

This section introduces our method for fully automatic extraction and
georeferencing of PV modules from aerial IR videos. For an overview see fig.
1. After acquisition with the drone, IR videos of a PV plant are split into
individual frames and the GPS trajectory of the drone is extracted and
interpolated. Following Bommes et al. [10], PV modules are segmented by Mask
R-CNN [20], tracked over subsequent frames, extracted and stored to disk. To
georeference PV modules, a subset of keyframes is selected based on travelled
GPS distance and visual overlap. Subsequently, a georeferenced $3$D
reconstruction of the PV plant is obtained by incremental SfM alongside the
$6$-DOF camera pose of each keyframe. This requires calibrated camera
parameters, which are obtained beforehand. The known keyframe poses are then
used to triangulate observed PV modules into the $3$D reconstruction, yielding
the desired module geocoordinates.

\begin{overpic}[scale={1.0},abs,unit=1mm,tics=5]{figures/overview_notext.pdf}
\put(10.583,60.239){\parbox[t][22.76219pt][c]{51.79823pt}{\footnotesize\centering
Thermal IR\\\ video file\@add@centering}}
\put(10.583,46.601){\parbox[t][22.76219pt][c]{51.79823pt}{\footnotesize\centering
Split video\\\ in frames\@add@centering}}
\put(10.583,21.314){\parbox[t][22.76219pt][c]{51.79823pt}{\footnotesize\centering
Extract GPS\\\ coordinates\@add@centering}}
\put(42.563,21.314){\parbox[t][22.76219pt][c]{51.79823pt}{\footnotesize\centering
Select\\\ keyframes\@add@centering}}
\put(65.482,21.314){\parbox[t][22.76219pt][c]{51.79823pt}{\footnotesize\centering
Calibrate\\\ camera\@add@centering}}
\put(45.872,80.751){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Segment PV modules\@add@centering}}
\put(91.997,80.751){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Track PV modules\@add@centering}}
\put(45.872,53.262){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Reconstruct camera\\\ poses (SfM)\@add@centering}}
\put(91.708,53.262){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Triangulate modules\@add@centering}}
\put(96.074,21.314){\parbox[t][22.76219pt][c]{72.92725pt}{\footnotesize\centering
Module\\\ geocoordinates\@add@centering}}
\put(136.087,80.751){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Extract rectified image\\\ patches of modules\@add@centering}}
\put(136.087,53.262){\parbox[t][22.76219pt][c]{97.77222pt}{\footnotesize\centering
Analyze modules\\\ (anomalies, temperatures)\@add@centering}}
\put(136.087,39.697){\parbox[t][14.79541pt][c]{97.77222pt}{\footnotesize\centering
Visualize on map\@add@centering}} \end{overpic} Figure 1: Overview of our
method for automatic extraction and georeferencing of PV modules from aerial
IR videos.

### A. Camera Model and Calibration

Several steps of our pipeline use a calibrated pinhole camera model to project
$3$D scene points into image coordinates and to triangulate image points into
a $3$D reconstruction of the scene. Lens distortion is modelled by a Brown-
Conrady radial distortion model [21] with five distortion coefficients.

Calibration is performed once for each camera using OpenCV’s [22] calibration
method with around $150$ images of a chessboard calibration target (see fig.
2). The target consists of foil patches applied to a polymer panel, providing
sufficient contrast in the IR image due to different emissivities. We obtain
best results when capturing calibration images outside on a cloudy day.

Figure 2: Examplary IR image of the camera calibration target.

### B. Drone Flight and Video Acquisition

Our method is intended to be used with IR videos acquired by a drone, which
scans one or multiple rows of a PV plant at an altitude of
$10\text{\,}\mathrm{m}$ to $30\text{\,}\mathrm{m}$ and at a velocity that
ensures blur-free images. Acquisition should take place under clearsky
conditions and solar irradiance above
$700\text{\,}\mathrm{W}\text{\,}{\mathrm{m}}^{-2}$. Similar to our previous
method [10], both nadiral and non-nadiral videos can be processed and the
camera angle and flight velocity may be varied during the flight. For accurate
georeferencing of the SfM reconstruction the drone needs to travel a
sufficient distance in at least two orthogonal directions. Furthermore, the
flight altitude should be kept approximately constant in case standard GPS is
used and no accurate altitude measurement is available. These requirements are
much less restrictive than those of our previous work, resulting in higher
flexibility and robustness. It is, for example, no problem, if the drone moves
non-monotonically along a plant row, or if the same row is scanned multiple
times. Furthermore, situations, in which the scanned row is cropped at the top
or bottom of the frame, or in which other rows become visible in the camera
viewport, can be handled.

For compatibility with the remaining processing steps, we split the acquired
IR videos into individual $16$-bit grayscale images, convert each image to
Celsius scale, normalize to the interval $\left[0,255\right]$ using its
minimum and maximum temperature value, convert to $8$-bit, and finally,
perform histogram equalization.

### C. Segmentation, Tracking and Extraction of PV modules

These steps correspond to our previous work [10] and are therefore described
only briefly. A Mask R-CNN instance segmentation model, which is trained on a
photovoltaic-specific dataset, obtains a binary segmentation mask for each PV
module in each video frame. After fitting a quadrilateral to each mask, the
underlying image region is extracted, warped to a rectangular region by a
homography and stored as a $16$-bit radiometric image file. A tracking
algorithm associates masks of the same PV module over subsequent frames and
assigns a unique tracking ID to each module. The tracking ID is then used to
group the extracted image patches of each module.

### D. Preprocessing of the GPS Trajectory

The drone records its latitude and longitude in WGS-$84$ coordinates at a rate
of $1\text{\,}\mathrm{Hz}$. As we do not use RTK-GPS, the measured altitude is
unreliable and we assume it as unknown in the subsequent steps. To match the
rates of GPS measurements to the higher frame rate of the camera, we perform
piecewise linear interpolation of the GPS trajectory and sample a GPS position
for each frame. Prior to this, we transform the trajectory from WGS-$84$
coordinates to local tangent plane (LTP) coordinates [23]. LTP coordinates are
Cartesian with their origin at or near the inspected site. This enables
accurate interpolation and enhances numerical stability in the subsequent SfM
procedure.

### E. Selection of Frames for Reconstruction

In this step we select a subset of partially overlapping video frames for the
subsequent SfM procedure, which we call keyframes. Subsampling the frames
keeps the computational cost of the SfM procedure, which is quadratic in the
number of frames, within an acceptable range. It also decouples SfM from the
video frame rate, simplifying the use of different cameras. SfM further
benefits from the larger parallax between any two keyframes, ensuring more
accurate triangulation of scene points.

We select a frame as a keyframe if i) its distance to the previous keyframe
along the GPS trajectory exceeds $0.75\text{\,}\mathrm{m}$, or if ii) its
intersection over union (IoU) with the previous keyframe is smaller than
$85\text{\,}\mathrm{\char 37\relax}$. To obtain the IoU, ORB features [24] of
the frame and the previous keyframe are extracted and matched. A homography is
estimated from the matches, which projects the bounding rectangle of the frame
onto that of the previous keyframe. The IoU is then the intersection area of
both rectangles divided by their total area.

One advantage of capturing videos over individual images, is the ability to
adjust the overlap between images after the data is already captured.

### F. Reconstruction of Camera Poses with SfM

In this step the $6$-DOF camera pose of each keyframe is reconstructed using
OpenSfM, an incremental SfM library [25]. Inputs are the calibrated camera
parameters and the selected keyframes with their GPS positions in LTP
coordinates. Due to unavailability of reliable measurements we set the GPS
altitude to zero and fix the dilution of precision (DOP) to
$0.1\text{\,}\mathrm{m}$. Outputs are the rotation and translation of each
keyframe in a LTP coordinate system and a $3$D point cloud of reconstructed
scene points, which is not further needed. An example is shown in fig. 4(a).
In the following, we explain briefly how the SfM library works.

#### 1) Feature detection and matching

The SfM library first finds HAHOG features [26], i.e. characteristic points,
in each keyframe. Overlapping frames are then found by matching these features
between pairs of frames. To limit the search space matches are computed only
for frame pairs which are at most $15\text{\,}\mathrm{m}$ apart.

#### 2) Initialization of the reconstruction

One frame pair with sufficient parallax is selected for initialization of the
reconstruction. The pose of the first frame is set as world coordinate origin.
The pose of the second frame relative to the first frame is estimated with the
five-point algorithm [27] or, in case of a planar scene, by decomposing a
homography [28]. An initial set of $3$D scene points is triangulated from the
matched feature points in both frames.

#### 3) Iterative reconstruction

Starting from the initial frame pair the other keyframes are added
incrementally to the reconstruction. In each iteration the frame with most
matches to any of the reconstructed frames is selected. Its pose is estimated
from observed $3$D scene points in the reconstruction and their corresponding
$2$D projections in the frame by solving the perspective-n-point problem [29].
Subsequently, new scene points are triangulated from feature points shared
between the newly added frame and other frames in the reconstruction.
Afterwards, the entire reconstruction is rigidly transformed, so that camera
positions best align with their measured GPS positions. In regular intervals
bundle adjustment optimizes all reconstructed camera poses and scene points
simultaneously by minimizing the reprojection error of the scene points in all
frames. Here, camera positions are kept close to their measured GPS positions.
Additionally, camera parameters are refined. We use these refined parameters
in all subsequent steps.

#### 4) Post-processing of the reconstruction

Under some circumstances the reconstruction of a long video sequence can fail
partially. This results in multiple partial reconstructions each with a
different LTP coordinate origin. To register all partial reconstructions in a
common LTP coordinate frame, we transform each partial reconstruction to
WGS-84 coordinates using the reconstruction-specific LTP origin. We then
transform the reconstruction back to LTP coordinates, this time using the
common LTP origin. This common origin is arbitrarily set to the origin of the
first partial reconstruction.

### G. Obtaining Geocoordinates of PV Modules

Once the keyframe poses are reconstructed, we triangulate the corner/center
points of segmented PV modules into the reconstruction, yielding corresponding
LTP geocoordinates. Examples of this are shown in fig. 4(b) and 4(c). Due to
inaccuracies in the module segmentation a robust triangulation procedure and
subsequent refinement of the obtained LTP coordinates are required.

#### 1) Triangulation of PV modules

For each tracked module, we obtain pixel-coordinates of the four corner points
and the center point in all keyframes, in which the module is visible. Modules
observed in less than two keyframes are skipped as they are likely spurious
detections and triangulation is impossible. The five module points are then
undistorted with the calibrated Brown-Conrady model, and triangulated from all
possible pairs of keyframes, in which they are observed. The so triangulated
points are only retained if the following two conditions are met: i) The angle
between the two viewing rays is larger than
$1\text{\,}\mathrm{\SIUnitSymbolDegree}$ for all five points, and ii) none of
the reprojection errors of the five points exceeds a threshold of $5$ pixels.
As there are typically several pairs of keyframes observing the same module,
we get a noisy set of triangulated modules (see fig. 3(a)). We fuse them
robustly by computing the median of corresponding points (see fig. 3(b)).

#### 2) Merging of duplicate detections

A PV module may be lost during tracking and reappear a few frames later with a
different tracking ID, resulting in multiple overlapping triangulations of the
module in the reconstruction (see fig. 3(b)). This step identifies and fuses
such duplicates. To this end, for each keyframe, all triangulated modules are
projected back into the frame. Two or more modules are identified as
overlapping if the mean Euclidean distance between the corresponding four
corner points and the center point is smaller than $20$ pixels. To merge
overlapping modules, module points are re-triangulated according to the
procedure above, this time using all keyframes of the overlapping modules. The
result is shown in fig. 3(c).

#### 3) Refinement of triangulated modules

The triangulation is further refined by moving nearby PV module corners closer
together, yielding a smoother result (see fig. 3(d)). To this end, we build a
graph containing all triangulated module points as vertices $\mathcal{P}$ and
edges between those points that are close to another. Points are considered
close if their Euclidean distance is at most $1\text{\,}\mathrm{m}$ in the
reconstruction and $20$ pixels in projected image coordinates. Given this
graph we use the g2o graph optimization framework [30] to obtain refined
module points $\mathcal{P}^{*}$ by optimizing the following objective

$\mathcal{P}^{*}=\operatorname*{arg\,min}_{\mathcal{P}}\sum_{\left<i,j\right>\in\mathcal{C}}\rho_{h}\left(\mathbf{e}_{ij}^{\intercal}\mathbf{\Omega}_{ij}\mathbf{e}_{ij}\right)\textrm{.}$
(1)

Here, $\mathcal{C}$ is the set of pairs of indices for which an edge exist,
$\mathbf{e}_{ij}=\mathbf{P}_{i}-\mathbf{P}_{j}$ is the difference between two
points, and $\mathbf{\Omega}_{ij}$ is the information matrix, which we set to
the identity matrix. The robust Huber cost function $\rho_{h}$ reduces the
impact of outliers.

(a)

(b)

(c)

(d)

Figure 3: Steps of PV module triangulation: a) initial triangulation from all
keyframe pairs, b) after computing median points, c) after merging duplicates,
and d) after iterative refinement.

We do not apply any further refinements, such as aligning surface normals of
modules, or enforcing a rectangular shape, to retain maximum flexibility of
our method with respect to the layout of PV modules.

(a)

(b)

(c)

Figure 4: Reconstruction results of the SfM procedure: a) reconstructed
feature points (grey) and camera poses (blue line and black camera frustrums),
b) with triangulated PV modules, c) top-down view on the triangulated modules
of an entire PV plant.

### H. Final Dataset Structure

After triangulation, LTP coordinates of PV module corners and center points
are transformed back to WGS-$84$ coordinates and stored in a GeoJSON file
together together with the module tracking ID. Similarly, the extracted IR
patches of each module are stored as image files in a directory named after
the tracking ID. This dataset structure allows for analysis of the extracted
image patches and visualization of results on a map.

## III. Experiments & Results

In this section, we apply our method to five different PV plants. We quantify
the module extraction success rate together with the georeferencing error, and
validate the tools’ ability to process multiple plant rows in parallel. We
further map predicted module anomalies and module temperatures and
investigate, to what extent the temperature distribution can replace a deep
learning-based classifier for the detection of abnormal modules.

### A. Video Dataset

To validate our method, we acquire IR videos of five PV plants in Germany with
a combined $35084$ PV modules using a drone of type DJI Matrice $210$. Tab. 1
contains details of the PV plants, drone flights and weather conditions during
data acquisition. Plants A to D are large-scale open-space plants with regular
row-based layouts. Plant E is a less regular arrangement of PV arrays mounted
on several rooftops. All plants consist of $60$-cell crystalline silicon
modules. Videos of plants A, B and E are recorded by a DJI Zenmuse XT$2$
thermal camera with $640\times 512$ pixels resolution, $8\text{\,}\mathrm{Hz}$
frame rate and $13\text{\,}\mathrm{mm}$ focal length. For plants C and D we
use another variant of the DJI Zenmuse XT$2$ with $30\text{\,}\mathrm{Hz}$
frame rate and $19\text{\,}\mathrm{mm}$ focal length. Fig. 5 shows exemplary
video frames from our dataset. Note, that we could not use the dataset from
our previous work due to an exchange of our camera and unavailability of
calibrated camera parameters.

Table 1: Details of PV plants, drone flights and weather conditions in our study. Start and end time are in UTC+2:00. Peak velocity is the $99.9\text{\,}\mathrm{\char 37\relax}$ quantile of all velocities estimated from position and time delta of subsequent video frames. Weather data is from Deutscher Wetterdienst [31]. We report mean and standard deviation of measurements taken at the nearest weather station every $10$ minutes during the flight. Plant Details | Flight Details | Weather Conditions
---|---|---
ID | # Modules | Type | Start time | End time | # Frames | Distance | Peak velocity | Air temp. | Global radiation | Wind speed | Wind dir.
A | $13\,640$ | open-space | 10:28:48 | 12:40:14 | $42\,272$ | $7612$ | $\mathrm{m}$ | $4.1$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | $25.9\pm 0.5$ | $\mathrm{\SIUnitSymbolCelsius}$ | $39.7\pm 1.8$ | $\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ | $2.8\pm 0.4$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | WSW
B | $5280$ | open-space | 13:37:59 | 14:14:30 | $13\,715$ | $2929$ | $\mathrm{m}$ | $4.1$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | $26.8\pm 0.4$ | $\mathrm{\SIUnitSymbolCelsius}$ | $30.3\pm 6.1$ | $\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ | $3.6\pm 0.4$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | SW
C | $6210$ | open-space | 12:16:21 | 12:39:05 | $34\,593$ | $2468$ | $\mathrm{m}$ | $6.6$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | $22.3\pm 0.3$ | $\mathrm{\SIUnitSymbolCelsius}$ | $46.2\pm 10.6$ | $\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ | $5.7\pm 0.6$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | WNW
D | $8460$ | open-space | 11:01:00 | 11:33:34 | $50\,348$ | $3479$ | $\mathrm{m}$ | $7.2$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | $23.4\pm 0.2$ | $\mathrm{\SIUnitSymbolCelsius}$ | $57.4\pm 1.5$ | $\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ | $2.0\pm 0.8$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | W
E | $1494$ | rooftops | 11:30:24 | 11:54:02 | $4527$ | $485$ | $\mathrm{m}$ | $4.2$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | $19.0\pm 0.3$ | $\mathrm{\SIUnitSymbolCelsius}$ | $42.8\pm 3.0$ | $\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ | $2.6\pm 0.3$ | $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ | SE

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_005700.jpg}
\put(5.0,5.0){Plant A/B} \end{overpic}

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_012326.jpg}
\put(5.0,5.0){A/B} \end{overpic}

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_000535.jpg}
\put(5.0,5.0){C/D} \end{overpic}

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_000574.jpg}
\put(5.0,5.0){E} \end{overpic}

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_000864.jpg}
\put(5.0,5.0){E} \end{overpic}

\begin{overpic}[width=433.62pt]{figures/dataset_images/frame_003752.jpg}
\put(5.0,5.0){E} \end{overpic}

Figure 5: Exemplary IR video frames of the PV plants in our study.

### B. Module Extraction Success Rate

We apply our method on the five PV plants in our video dataset and extract
over $2.2$ million IR images of the $35084$ PV modules of all plants (on
average $64.1$ images per module). Additionally, geocoordinates are obtained
for each module as exemplary shown for plant B in fig. 6 (for the other plants
see appendix A-A). As detailed in tab. 2, $99.3\text{\,}\mathrm{\char
37\relax}$ of all modules are successfully extracted and georeferenced. As
compared to the $87.8\text{\,}\mathrm{\char 37\relax}$ success rate of our
previous work, we now miss only one in $140$ modules instead of one in eight.
This $18$-fold improvement of the extraction success rate is mostly due to the
higher robustness of our new method to errors in the data acquisition process,
such as cropping of the scanned row, double acquisition of the same row, or
small loops in the drone trajectory. Such an error caused our previous method
to loose all modules in an entire plant row. Opposed to that, our new method
can handle many of those acquisition errors, and fails at most locally for a
few modules. The almost perfect success rate of our new method is important in
practice, as every missed module is a missed opportunity to increase yield and
profitability of the plant. Furthermore, safety critical anomalies (e.g. fire
hazards) could be overlooked.

Tab. 2 also contains a detailed breakdown of the failure modes of the $234$
modules missed by our method. In total, $13$ modules exhibit substantial
distortions, and $40$ modules are missing in the reconstructions, because they
are not covered by sufficiently many video frames to be accurately
triangulated. Another $181$ modules appear multiple times in the
reconstruction because the merging procedure (sec. II-G.2) failed. This
happens for modules appearing in video frames, which are temporally far apart.
As these frames have a large relative pose error (due to the use of standard
GPS) the triangulated modules do not align well and can not be merged
correctly. This can most likely be mitigated by using RTK-GPS. Finally, there
are $24$ false positive modules corresponding to other objects, which are
mistaken as PV modules by the Mask R-CNN segmentation model.

Figure 6: Map with estimated geocoordinates of PV modules in plant B. Table 2: Numbers of PV modules and module image patches extracted from the plants in our dataset. Failures are missing (MM), duplicate (DP), distorted (DS) and false positive (FP) modules. Plant | # Modules | # Patches | # Failures
---|---|---|---
| Total | Extracted | Extracted |  $\varnothing$/Module | MM | DP | DS | FP
A | $13\,640$ | $13\,463$ | $398\,221$ | $29.2$ | $18$ | $152$ | $7$ | $13$
B | $5280$ | $5246$ | $140\,120$ | $26.6$ | $6$ | $28$ | $0$ | $3$
C | $6210$ | $6200$ | $635\,437$ | $102.3$ | $4$ | $1$ | $5$ | $2$
D | $8460$ | $8453$ | $936\,867$ | $110.8$ | $7$ | $0$ | $0$ | $4$
E | $1494$ | $1488$ | $138\,008$ | $90.8$ | $5$ | $0$ | $1$ | $2$
Total | $35\,084$ | $34\,850$ | $2\,248\,653$ | $64.1$ | $40$ | $181$ | $13$ | $24$

### C. Georeferencing Accuracy

In this section we quantify the accuracy of the georeferenced PV module
locations in terms of the root mean square error (RMSE) between estimated LTP
geocoordinates $(\hat{e},\hat{n})$ and ground truth geocoordinates $(e,n)$ of
$N$ selected PV modules

$\textrm{RMSE}=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(\hat{e}_{i}-e_{i}\right)^{2}+\frac{1}{N}\sum_{i=1}^{N}\left(\hat{n}_{i}-n_{i}\right)^{2}}\textrm{.}$
(2)

Here, $e$ and $n$ are the east and north positions in the LTP coordinate
system. The altitude coordinate is omitted as for mapping only the horizontal
error is of interest. Because point correspondences have to be found manually,
we select only every $11$th module in every second row of the plant and
consider only the top-left module corners. We further limit the accuracy
analysis to plant A, as it is the largest plant in our dataset and the only
one, for which a ground truth is available. Ground truth positions are
obtained from an orthophoto of the plant. This is possible, as this orthophoto
exhibits a small RMSE of less than $2\text{\,}\mathrm{cm}$, facilitated by the
use of RTK-GPS, ground control points, high-resolution visual imagery and a
higher flight altitude.

The RMSE for the entire plant is $5.87\text{\,}\mathrm{m}$. This is close to
the expected $4.9\text{\,}\mathrm{m}$ accuracy of GPS under open sky
conditions [32]. However, the RMSE is not constant for the entire plant, but
instead smoothly increases from $0.42\text{\,}\mathrm{m}$ in the east to
$9.39\text{\,}\mathrm{m}$ in the west. This also becomes evident in fig. 7,
which shows the spatial interpolation of the RMSE over the entire plant. This
error drift in the SfM reconstruction is most likely caused by the low
accuracy and unknown DOP of the measured GPS trajectory of the drone. As the
SfM reconstruction consist of seven partial reconstructions and we do not use
ground control points, another possible cause is misalignment of the partial
reconstructions.

To analyze the distortion of each individual row, we remove the trend in the
RMSE distribution. To this end, we align each row with the respective ground
truth positions prior to computing the RMSE for the row. The resulting per-row
RSME values range from $0.22\text{\,}\mathrm{m}$ to $0.82\text{\,}\mathrm{m}$,
indicating low distortion of individual rows. Due to this, accurate
localization of PV modules within the plant is possible, despite the large
absolute RMSE of $5.87\text{\,}\mathrm{m}$.

Figure 7: RMSE between ground truth and estimated horizontal geocoordinates of
PV modules in plant A. The RMSE is computed for selected points (black dots)
and linearly interpolated on a $4000\times 800$ grid (heatmap).

### D. Simultaneous Processing of Multiple Rows

One important advantage of our new method is the ability to process multiple
PV plant rows simultaneously. We validate this experimentally by acquiring IR
videos of the first $12$ rows of plant A. We perform three flights, scanning
one, two and three rows at a time. Fig. 8 shows exemplary video frames of each
flight as well as the reconstructions of modules and flight trajectories
produced by our method.

As reported in tab. 3, scanning two and three rows simultaneously speeds up
the flight duration by a factor of $2.1$ and $3.7$, respectively. Module
throughput increases accordingly from $3.36\text{\,}{\mathrm{s}}^{-1}$ to
$7.03\text{\,}{\mathrm{s}}^{-1}$ and $12.57\text{\,}{\mathrm{s}}^{-1}$. This
means, scanning all $2376$ modules of the $12$ selected rows takes only $338$
or $189$ seconds when scanning two or three rows at a time. Additionally,
flight distance decreases by a factor of $1.9$ and $2.8$. This has the benefit
of increasing the range of the drone before a battery change is needed. The
cost for the improvement in throughput is a two- or threefold reduction in the
resolution of extracted module images. Furthermore, we found the manual flight
is slightly more complicated when scanning three rows at a time instead of one
or two, because it is easier to miscount the rows when shifting over to the
next row triplet. However, this is not a limitation when flying autonomously.

This experiment confirms the ability of our method to significantly increase
throughput simply by scanning more than one plant row at a time. This is
highly relevant in practice, as it significantly reduces duration and cost of
the inspection. It is also an improvement over our previous method [10], which
could process only one row at a time.

Figure 8: Top row: Exemplary video frames for scanning one, two and three PV plant rows simultaneously. Bottom row: Resulting reconstructions of modules and flight trajectory. Table 3: Results for simultaneous scanning of one, two and three rows. | One Row | Two Rows | Three Rows
---|---|---|---
Flight distance | $1307\text{\,}\mathrm{m}$ | $681\text{\,}\mathrm{m}$ | $461\text{\,}\mathrm{m}$
Flight duration | $707\text{\,}\mathrm{s}$ | $338\text{\,}\mathrm{s}$ | $189\text{\,}\mathrm{s}$
Average module resolution |  $141\text{\,}\mathrm{px}$ × $99\text{\,}\mathrm{px}$ |  $73\text{\,}\mathrm{px}$ × $50\text{\,}\mathrm{px}$ |  $46\text{\,}\mathrm{px}$ × $33\text{\,}\mathrm{px}$
Module throughput | $3.36\text{\,}{\mathrm{s}}^{-1}$ | $7.03\text{\,}{\mathrm{s}}^{-1}$ | $12.57\text{\,}{\mathrm{s}}^{-1}$

### E. Mapping Module Anomalies

In this section we apply a deep learning-based binary classifier to the
extracted IR image patches of each PV module in plant A, which predicts
whether the module is abnormal or not. 111We use the ResNet-$34$ convolutional
neural network (CNN) classifier from Bommes et al. [19], which is trained with
a supervised cross-entropy loss on labelled IR module patches of plant B in
the dataset of the original work. We then use the estimated module
geocoordinates to visualize the distribution of abnormal modules on a map (see
fig. 9). Since there are multiple images for each module, we can plot the
fraction of images, in which a module is predicted as abnormal. We call this
the _anomaly ratio_. As opposed to a simple binary prediction, the anomaly
ratio is an approximate indicator for the severity of a module anomaly. This
is, because for severe, i.e. clearly visually expressed, anomalies the
classifier is more confident, reaching a larger consensus of its predictions
over all images of a module.

The so obtained anomaly map enables not only targeted repairs of severely
abnormal modules, but also facilitates the identification of fundamental
problems of the plant. For the analyzed plant, we find for example, that
anomalies occur much more frequently in the bottom row, where modules are
closer to the ground, rather than in the top row. A possible explanation for
this is the intrusion of moisture into the PV modules near the ground. Being
aware of such an issue allows the operator to monitor affected modules more
thoroughly and to take action to prevent further damage to the plant.

Figure 9: Map of predicted module anomalies in a section of plant A. On the
left, IR images of the modules highlighted on the map are shown. Their
temperature range is $30\text{\,}\mathrm{\SIUnitSymbolCelsius}$ (black) to
$50\text{\,}\mathrm{\SIUnitSymbolCelsius}$ (white).

### F. Mapping Module Temperatures

Apart from module anomalies, we visualize the spatial distribution of module
temperatures in plant A (see fig. 10). Temperatures are obtained from the
extracted IR image patches of each module and plotted on a map using the
module geocoordinates. For each module the maximum, minimum, mean or median
temperature over the module area can be computed. Prior to this, we cut away a
few pixels ($5\text{\,}\mathrm{\char 37\relax}$ of the image width) from the
image borders to ignore module frames and mounting brackets. To obtain a final
temperature value for each module, we take the mean over the values estimated
for each of the image patches of the module. As opposed to using a single
representative image patch or the maximum over all patches, the mean is more
robust to artifacts, which may be present in some of the module images. Of
both mean and maximum temperature distributions, we find the maximum
temperatures (see fig. 10(b)) more informative as they are sensitive to the
local hot spots typically occurring in abnormal modules. However, both mean
and maximum module temperatures reveal the global temperature distribution of
the plant, which is not constant, but exhibits a low-frequency pattern with
temperature differences of up to $15\text{\,}\mathrm{K}$. As module images are
acquired over a duration of $132$ minutes, possible explanations for this
pattern are slow changes in the solar irradiance [33], cloud cover [34, 35],
air temperature, wind speed, and camera temperature [36]. The temperature
distribution is also affected by local differences in the radiative and
convective heat transfer, and by the number of neighbouring modules, leading
to cooler modules at the edges of each plant row [37]. Direct use of this
temperature distribution for anomaly detection is not possible, as there is no
common threshold value, which separates normal from abnormal modules. To
account for this, we compute local temperature differences between
neighbouring modules. Specifically, we subtract the median of the maximum
module temperatures of all neighbouring modules within a radius of
$7\text{\,}\mathrm{m}$ from each module. Fig. 10(c) shows the resulting
relative maximum module temperatures. These relative temperatures are
independent of the changes in environmental conditions during the flight, and
consequently facilitate detection of abnormal modules by selecting a suitable
temperature threshold.

Apart from locally overheated modules, the temperature distribution allows to
identify string anomalies. For example in plant A, there is an inactive string
(in the middle of the $19$th row counted from the bottom), which is clearly
visible in the temperature map (see fig. 10). Being able to identify such
anomalies is important, as an entire inactive string causes large yield and
power losses.

(a)

(b)

(c)

Figure 10: Map of plant A showing the distribution of mean (a) and maximum (b)
module temperatures. Each value is the average over all images showing a
module. For higher contrast temperatures are clipped below
$30\text{\,}\mathrm{\SIUnitSymbolCelsius}$ and above
$50\text{\,}\mathrm{\SIUnitSymbolCelsius}$. In (c) local differences of the
maximum module temperature are emphasized by subtracting the median
temperatures of neighbouring modules within a radius of
$7\text{\,}\mathrm{m}$.

### G. Anomaly Detection with Module Temperatures

In this section we analyze, whether the relative maximum module temperatures
(see fig. 10(c)) alone are sufficient to accurately identify abnormal modules,
and whether they can replace the more complex deep learning-based anomaly
classifier from sec. III-E. To this end, we manually label all $13463$ modules
of plant A as healthy or as abnormal with one out of the ten anomaly classes
shown in fig. 11. For each module a binary anomaly prediction is obtained by
comparing its relative maximum module temperature to a specified threshold
value. Similarly, for the deep learning classifier we compare the anomaly
ratio (see fig. 9) of the module to a threshold value.

As common in the anomaly detection literature [38, 39, 40], we quantify the
anomaly detection performance as the area under the receiver operating
characteristic (AUROC). This metric is independent of a specific threshold
value, and therefore, enables a fair comparison of both classifiers, which
depend differently on their threshold values. AUROC is defined as the area
under the true positive rate
$\textrm{TPR}=\textrm{TP}/(\textrm{TP}+\textrm{FN})$ plotted against the false
positive rate $\textrm{FPR}=\textrm{FP}/(\textrm{FP}+\textrm{TN})$ at
different threshold values. Here, TP and TN are the numbers of correctly
classified abnormal and healthy modules, and FP is the number of healthy
modules falsely classified as abnormal and FN the number of abnormal modules
falsely classified as healthy.

Tab. 4 reports the resulting AUROC scores for each anomaly class and an
overall AUROC score, which considers all anomaly classes. The results indicate
that both module temperature distribution and deep learning classifier perform
equally and nearly perfect on severe anomalies (Sh, Cs+, Cm+), while they
complement each other on the less severe anomaly classes. The deep learning
classifier performs better for Pid, Cs and Cm anomalies, which are
characterized by low temperature gradients, and are therefore not as
accurately identifiable by the temperature distribution. On the contrary, the
temperature distribution performs better for D, So, Chs anomalies, which have
large temperature gradients and a small spatial extent. The small spatial
extent makes detection of these anomalies difficult for a convolutional neural
network. Both classifiers perform poorly on homogeneously overheated modules
(Mh) because their predictions are based on temperature differences within the
image (deep learning classifier) or within the local neighbourhood of modules
(temperature distribution). However, using absolute instead of the relative
maximum module temperatures allows to accurately identify Mh anomalies (see
sec. III-F).

Summing up, the module temperature distribution can supersede a complex deep
learning-based anomaly classifier for the detection of seven out of ten common
module anomalies in a PV plant. This is beneficial for practical applications
because of the simplicity, higher speed and better interpretability of the
temperature distribution. Furthermore, no training is required, which saves
the effort of creating a labelled training dataset and circumvents the issue
of having to generalize from the training to the test dataset.

(a) Healthy

(b) Mh: Module open-circuit

(c) Sh: Substring open-circuit

(d) Pid: Potential- induced degrad.

(e) Cm+: Multiple hot cells

(f) Cs+: Single hot cell

(g) Cs/Cm: Warm cell(s)

(h) D: Diode overheated

(i) Chs: Hot spot

(j) So: Soiling

Figure 11: Exemplary IR images of the module anomaly classes in our analysis. Temperature ranges from $30\text{\,}\mathrm{\SIUnitSymbolCelsius}$ (black) to $60\text{\,}\mathrm{\SIUnitSymbolCelsius}$ (white). The figure is adopted from our previous work [10]. Table 4: AUROC scores for the detection of module anomalies in plant A by the module temperature distribution versus a deep learning classifier. Scores of the better classifier and scores above $99\text{\,}\mathrm{\char 37\relax}$ are in bold. Anomaly | # Modules | AUROC / %
---|---|---
|  | Temp. Distribution | Deep Learning Clf.
Mh | $22$ | $59.04$ | $52.73$
Sh | $32$ | $99.78$ | $99.95$
Pid | $149$ | $76.01$ | $95.92$
Cm+ | $11$ | $100.00$ | $99.61$
Cs+ | $30$ | $99.81$ | $99.90$
Cm | $420$ | $60.71$ | $86.16$
Cs | $294$ | $61.65$ | $78.84$
D | $294$ | $99.31$ | $62.06$
Chs | $23$ | $89.64$ | $81.53$
So | $136$ | $79.71$ | $61.29$
Overall | $1411$ | $74.31$ | $78.04$

## IV. Conclusion

In this work, we developed a method for the automatic extraction and
georeferencing of PV modules from aerial IR videos, which can be used for
fully automatic PV plant inspection. One possible future improvement of our
method is the use of centimeter-accurate RTK-GPS instead of standard GPS,
which could reduce the RMSE of module geocoordinates and stabilize the SfM
procedure. Similarly, accuracy and stability of the SfM procedure could be
improved by using visual videos instead of IR videos, as visual videos provide
a higher resolution, wider viewing angle, color information and exhibit lower
variation of image intensities [41]. However, this requires accurate temporal
synchronization and spatial registration of the visual and IR stream, which is
a challenging task. Another future direction is the correlation of the
obtained temperature distribution with electrical data, such as power and
yield, which could provide additional insights into the health state of a PV
plant. Finally, our method could be extended for augmented reality
applications by rendering a more immersive $3$D model of the plant with
overlaid textures, module images and interactive reports for each module.

## V. Acknowledgements

This work was financially supported by the State of Bavaria via the project
PV-Tera (No. 446521a/20/5) and by BMWi via the project COSIMA (FKZ: 032429A).
We sincerely thank the N-Ergie Nürnberg and PV Service Pro Kollnburg for
supporting the project. The authors have declared no conflict of interest.

## References

  * [1] Fabrizio Bizzarri, Silvia Nitti and Gabriele Malgaroli  “The use of drones in the maintenance of photovoltaic fields”  In _E3S Web of Conferences_ 119, 2019  DOI: 10.1051/e3sconf/201911900021
  * [2] Yahya Zefri, Imane Sebari, Hicham Hajji and Ghassane Aniba  “Developing a deep learning-based layer-3 solution for thermal infrared large-scale photovoltaic module inspection from orthorectified big UAV imagery data”  In _International Journal of Applied Earth Observation and Geoinformation_ 106.5, 2022, pp. 102652  DOI: 10.1016/j.jag.2021.102652
  * [3] Roberto Pierdicca et al.  “Automatic Faults Detection of Photovoltaic Farms: solAIr, a Deep Learning-Based System for Thermal Images”  In _Energies_ 13, 2020, pp. 6496  DOI: 10.3390/en13246496
  * [4] Chris Henry, Sahadev Poudel, Sang-Woong Lee and Heon Jeong  “Automatic Detection System of Deteriorated PV Modules Using Drone with Thermal Camera”  In _Applied Sciences_ 10, 2020, pp. 3802  DOI: 10.3390/app10113802
  * [5] Vincenzo Carletti, Antonio Greco, Alessia Saggese and Mario Vento  “An intelligent flying system for automatic detection of faults in photovoltaic plants”  In _Journal of Ambient Intelligence and Humanized Computing_ 11, 2019, pp. 2027–2040  DOI: 10.1007/s12652-019-01212-6
  * [6] Sara Gallardo-Saavedra, Luis Hernández-Callejo and Oscar Duque-Perez  “Technological review of the instrumentation used in aerial thermographic inspection of photovoltaic plants”  In _Renewable and Sustainable Energy Reviews_ 93, 2018, pp. 566–579  DOI: 10.1016/j.rser.2018.05.027
  * [7] F. Grimaccia, S. Leva and A. Niccolai  “PV plant digital mapping for modules’ defects detection by unmanned aerial vehicles”  In _IET Renewable Power Generation_ 11.10, 2017, pp. 1221–1228  DOI: 10.1049/iet-rpg.2016.1041
  * [8] G. Francesco, L. Sonia and N. Alessandro  “A semi-automated method for Defect Identification in large Photovoltaic power plants using Unmanned Aerial Vehicles”  In _IEEE Power Energy Society General Meeting_ , 2018, pp. 1–5
  * [9] M. Aghaei, S. Leva and F. Grimaccia  “PV power plant inspection by image mosaicing techniques for IR real-time images”  In _IEEE 43rd Photovoltaic Specialists Conference_ , 2016, pp. 3100–3105
  * [10] Lukas Bommes et al.  “Computer vision tool for detection, mapping, and fault classification of photovoltaics modules in aerial IR videos”  In _Progress in Photovoltaics: Research and Applications_ 29.12, 2021, pp. 1236–1251  DOI: 10.1002/pip.3448
  * [11] Alessandro Niccolai, Francesco Grimaccia and Sonia Leva  “Advanced Asset Management Tools in Photovoltaic Plant Monitoring: UAV-Based Digital Mapping”  In _Energies_ 12.24, 2019, pp. 4736  DOI: 10.3390/en12244736
  * [12] P. Addabbo et al.  “UAV system for photovoltaic plant inspection”  In _IEEE Aerospace and Electronic Systems Magazine_ 33.8, 2018, pp. 58–67  DOI: 10.1109/MAES.2018.170145
  * [13] Marco Nisi et al.  “EGNSS High Accuracy System Improving Photovoltaic Plant Maintenance using RPAS integrated with Low-cost RTK Receiver”  In _Global Wireless Summit_ , 2016
  * [14] Dong Ho Lee and Jong Hwa Park  “Developing Inspection Methodology of Solar Energy Plants by Thermal Infrared Sensor on Board Unmanned Aerial Vehicles”  In _Energies_ 12.15, 2019, pp. 2928  DOI: 10.3390/en12152928
  * [15] Yahya Zefri, Achraf ElKettani, Imane Sebari and Sara Ait Lamallam  “Thermal Infrared and Visual Inspection of Photovoltaic Installations by UAV Photogrammetry—Application Case: Morocco”  In _Drones_ 2.4, 2018, pp. 41  DOI: 10.3390/drones2040041
  * [16] Ioannis (John) Tsanakas, Long Ha and Franck Al Shakarchi  “Advanced inspection of photovoltaic installations by aerial triangulation and terrestrial georeferencing of thermal/visual imagery”  In _Renewable Energy_ 102 (Part A), 2016, pp. 224–233  DOI: 10.1016/j.renene.2016.10.046
  * [17] Johannes L. Schönberger and Jan-Michael Frahm  “Structure-from-Motion Revisited”  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4104–4113
  * [18] Sameer Agarwal et al.  “Building Rome in a day”  In _IEEE International Conference on Computer Vision_ , 2009, pp. 72–79
  * [19] Lukas Bommes et al.  “Anomaly detection in infrared images of photovoltaic modules using supervised contrastive learning”  In _Progress in Photovoltaics: Research and Applications_ , 2022  DOI: 10.1002/pip.3518
  * [20] K. He, G. Gkioxari, P. Dollár and R. Girshick  “Mask R-CNN”  In _IEEE International Conference on Computer Vision_ , 2017, pp. 2980–2988
  * [21] Duane C. Brown  “Decentering Distortion of Lenses”  In _Photogrammetric Engineering_ 32.3, 1966, pp. 444–462
  * [22] G. Bradski  “The OpenCV library”  In _Dr. Dobb’s Journal of Software Tools_ 120, 2000, pp. 122–125
  * [23] Wolfgang Torge and Jürgen Müller  “Geodesy”  De Gruyter, 2012  DOI: 10.1515/9783110250008
  * [24] Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary Bradski  “ORB: An efficient alternative to SIFT or SURF”  In _International Conference on Computer Vision_ , 2011, pp. 2564–2571
  * [25] Mapillary  “OpenSfM”  In _GitHub repository_ , https://github.com/mapillary/OpenSfM, 2021
  * [26] Krystian Mikolajczyk and Cordelia Schmid  “An Affine Invariant Interest Point Detector”  In _European Conference on Computer Vision_ , 2002, pp. 128–142
  * [27] D. Nister  “An efficient solution to the five-point relative pose problem”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 26.6, 2004, pp. 756–770  DOI: 10.1109/TPAMI.2004.17
  * [28] O.. Faugeras and F. Lustman  “Motion and structure from motion in a piecewise planar environment”  In _International Journal of Pattern Recognition and Artificial Intelligence_ 02.03, 1988, pp. 485–508  DOI: 10.1142/S0218001488000285
  * [29] Vincent Lepetit, Francesc Moreno-Noguer and Pascal Fua  “EPnP: An Accurate O(n) Solution to the PnP Problem”  In _International Journal of Computer Vision_ 81.2, 2008, pp. 1573–1405  DOI: 10.1007/s11263-008-0152-6
  * [30] Rainer Kümmerle et al.  “G2o: A general framework for graph optimization”  In _IEEE International Conference on Robotics and Automation_ , 2011, pp. 3607–3613
  * [31] DWD Climate Data Center (CDC)  “Recent $10$-minute station observations of air temperature, global radiation, wind speed and wind direction for Germany, quality control not completed yet, version recent, last accessed: 08/10/2021”
  * [32] Frank Diggelen and Per K.  “The world’s first GPS MOOC and worldwide laboratory using smartphones”  In _28th International Technical Meeting of the Satellite Division of The Institute of Navigation_ , 2015, pp. 361–369
  * [33] Parvathy Rajendran and Howard Smith  “Modelling of solar irradiance and daylight duration for solar-powered UAV sizing”  In _Energy Exploration & Exploitation_ 34.2, 2016, pp. 235–243  DOI: 10.1177/0144598716629874
  * [34] Ryohei Nomura, Toru Harigai, Yoshiyuki Suda and Hirofumi Takikawa  “Second by second prediction of solar power generation based on cloud shadow behavior estimation near a power station”  In _AIP Conference Proceedings_ 1807.1, 2017, pp. 020024  DOI: 10.1063/1.4974806
  * [35] Kari Lappalainen and Seppo Valkealahti  “Analysis of shading periods caused by moving clouds”  In _Solar Energy_ 135, 2016, pp. 188–196  DOI: 10.1016/j.solener.2016.05.050
  * [36] Quanxing Wan, Benjamin Brede, Magdalena Smigaj and Lammert Kooistra  “Factors Influencing Temperature Measurements from Miniaturized Thermal Infrared (TIR) Cameras: A Laboratory-Based Approach”  In _Sensors_ 21.24, 2021  DOI: 10.3390/s21248466
  * [37] Janine Denz et al.  “Quantitative assessment of the power loss of silicon PV modules by IR thermography and its practical application in the field”  In _37th European Photovoltaic Solar Energy Conference and Exhibition_ , 2020, pp. 1542–1547
  * [38] Izhak Golan and Ran El-Yaniv  “Deep Anomaly Detection Using Geometric Transformations”  In _Advances in Neural Information Processing Systems (NIPS)_ , 2018, pp. 9781–9791
  * [39] Liron Bergman, Niv Cohen and Yedid Hoshen  “Deep Nearest Neighbor Anomaly Detection”  In _arXiv preprint arXiv:2002.10445_ , 2020  arXiv: http://arxiv.org/abs/2002.10445
  * [40] Dan Hendrycks, Mantas Mazeika and Thomas Dietterich  “Deep Anomaly Detection with Outlier Exposure”  In _International Conference on Learning Representations (ICLR)_ , 2019
  * [41] Shih-Schon Lin  “Review: Extending Visible Band Computer Vision Techniques to Infrared Band Images”  In _Technical Reports (CIS)_ , 2001

##  Appendix A Appendix

### A. Additional Georeferencing Results

(a) Plant A

(b) Plant C

(c) Plant D

(d) Plant B

(e) Plant E

Figure 12: Georeferencing results for all PV plants in our dataset.