# Conditional Visualization for Statistical Models:
An Introduction to the condvis Package in R

Mark O’Connell
Maynooth University Catherine B. Hurley
Maynooth University Katarina Domijan
Maynooth University<EMAIL_ADDRESS>
Mark O’Connell, Catherine B. Hurley, Katarina Domijan Conditional
Visualization for Statistical Models: An Introduction to the condvis Package
in R condvis: Conditional Visualization for Statistical Models The condvis
package is for interactive visualization of sections in data space, showing
fitted models on the section, and observed data near the section. The primary
goal is the interpretation of complex models, and showing how the observed
data support the fitted model. There is a video accompaniment to this paper
available here. This is a preprint version of O’Connell _et al._ (2016).
interactive, graphics, regression, classification, blackbox models
interactive, graphics, regression, classification, blackbox models Mark
O’Connell
Department of Mathematics & Statistics
Logic House
Maynooth University
Co. Kildare, Ireland
E-mail:

##  1 Introduction

When a model consists of a single continuous predictor and a single response,
the fitted model is simply visualized as a curve in two dimensions. When a
model involves two predictors, it may be visualized as a surface in three
dimensions; either as a contour plot or a perspective mesh. When a model
involves more than two predictors, there is no direct way to visualize the
model behaviour (see Figure 1). Clearly, there is a need for producing low-
dimensional visualizations of models in high-dimensional space. One approach
is conditional visualization.

In a geometric sense, conditional visualization means taking a section.
Consider a simple model with two predictors, relating mpg to wt and hp in the
mtcars data in R (R Core Team, 2016). The fitted model may be visualized as a
surface as in Figure 2(a). If we want to visualize the modelled effect of wt
conditional on hp, we take a section. The intersection of the fitted model and
the section is then a curve in two dimensions as in Figure 2(b). In this way,
conditional visualization offers a way to produce low-dimensional
visualizations of models in high-dimensional space. It is important to note
that such sections typically have no observed data lying on them, and so it is
difficult to understand how the observed data support the fitted model. In
condvis, we choose to display observed data points which are deemed to be
_near_ the section (discussed further in Section 2.4).

1 predictor 2 predictors …more predictors

Figure 1: Visualizing fitted models. With one predictor, fitted model may be
visualized as a curve. With two predictors, fitted model may be visualized as
a surface in three dimensions. What can we do for more predictors?

(a)

(b)

Figure 2: Visualizing a section. (a) Visualizing a 2 predictor model, and
showing a section at hp = 200. (b) Visualizing the section through model at hp
= 200.

###  1.1 Example: Forced expiratory volume (FEV) data

The FEV dataset in Kahn (2005) provides some useful discussion material for
conditional relationships in statistical models. It originally appears in an
earlier edition of Rosner (2010); we use the copy provided in the covreg (Niu
and Hoff, 2014) package. We use this data to launch into an example of the use
of condvis. The dataset concerns the relationship between lung health and
smoking in children. The response is forced expiratory volume (FEV, the amount
of air an individual can exhale in the first second of a forceful breath, used
as a proxy for lung health), and the predictors are gender, age, height and
smoking status (binary).

In making a plot of FEV versus smoking status, we get our first surprise (see
Figure 3).

Figure 3: Boxplot of FEV versus smoking status. Suggests smoking is associated
with higher FEV values.

In the marginal view, it seems as though smoking is associated with better
lung health! To illustrate the use of condvis, we fit a support vector machine
(Smola and Vapnik, 1997) from the e1071 (Meyer _et al._ , 2015) package. R>
library("e1071") R> m1 <\- svm(fev gender + smoke + age + height, data = fev)
We then call ceplot and start looking at sections through the fitted model,
investigating the modelled effect of smoking on FEV, conditional on the other
predictors. (There is a  video demonstration of this example, and a Shiny
application demo.) R> ceplot(data = fev, model = m1, sectionvars = "smoke",
type = "separate")

Taking a section around age = 14, height = 67, with either gender, shows a
more sensible result (see Figure 5). In these parts of the predictor space,
the fitted model suggests that smoking is associated with slightly lower FEV
values. This is an example of Simpson’s paradox, where the modelled
conditional association is of opposite sign to the apparent marginal
association. The observed data near these sections also seem to support the
fitted model, although it is worth noting that, for each section, there are
consistently more observations in the non-smoking group compared to the
smoking group.

Taking a section around age = 6, height = 55, gender = female, the model is
suggesting that smoking is related to higher FEV values (see Figure 5),

Figure 4: Section showing the modelled effect of smoking on FEV conditional on
height = 67, age = 14, gender = male. Suggests that smoking is associated with
lower FEV values, and observed data near this section seem to support this.
Figure 5: Section showing the modelled effect of smoking on FEV conditional on
height = 55, age = 6, gender = female. Model suggests that smoking is
associated with higher FEV values, but there are no observations in the
smoking group near this section.

as in the marginal view before! Why is this? On examining the section, we see
there are no observed data points in the smoking group in this part of the
data space. It is not surprising that there are no 6 year old smokers. Such a
prediction for 6 year old smokers should clearly be considered an
extrapolation and held in suitable suspicion accordingly. This example
demonstrates how ‘black-box’ models can mislead analysts just as easily as
more rigid linear models. A good defence against this is to take sections
through the model and see how nearby observed data support the model.

###  1.2 Outline

The remainder of the article proceeds as follows: Section 2 gives an overview
of the condvis package, explaining the basic ideas behind it. Section 3 gives
more detail on how to use condvis, describing available options and showing
some code examples. Section 4 briefly discusses some other approaches to
conditional visualization in the literature and in R. Section 5 concludes;
giving a short summary, some strengths and limitations of condvis, and finally
an outlook on further work. The appendix provides information on the
supplementary materials.

##  2 Overview

###  2.1 Basic workflow

The condvis package is available from CRAN and can be installed using R>
install.packages("condvis") The source code is also hosted at Github, where
bug reports and feature requests are welcome.

The main function in condvis is ceplot, which produces a visualization
consisting of two main parts; a section in data space, and condition selector
plots for interactively choosing the section. The default behaviour is to
place all graphics on a single device with base R graphics, but options exist
to place the section and condition selectors on separate devices, and to
produce a Shiny (Chang _et al._ , 2016) web application with the same displays
in a web browser.

Any usage of ceplot begins with a dataset and a fitted model object in R (see
Figure 6). This is minimally sufficient to call ceplot if model terms can be
extracted from the model object. In most cases, we will want to specify the
response and the predictor(s) of interest, the predictor(s) along which we
want to take a section. These are specified as the character name of the
variable in the dataframe.

Figure 6: Basic workflow with ceplot.

For a very simple example of the workflow in condvis, we can fit a linear
model to the mtcars data, and then call ceplot, specifying hp as the predictor
along which to take sections. R> m2 <\- lm(mpg wt + hp, data = mtcars) R>
ceplot(data = mtcars, model = m2, sectionvars = "hp") In this case, the
response variable can be extracted from the fitted model, and so does not need
to be specified.

Sections through fitted models are evaluated using the generic predict
function, so this currently works for models with standard predict methods.
Models of class lm, glm, gam, svm (Meyer _et al._ , 2015), and randomForest
(Liaw and Wiener, 2002) are examples of such models. For basis expansions in
linear models, the AsIs function (for example, I(x^2)) must be used in the
formula, unless the user explicitly wants to investigate the model in the
expanded data space. The same goes for interactions, which should be specified
using a*b or a:b as necessary, rather than adding constructed variables to the
dataframe. If a fitted model object does not have a suitable predict method, a
S3 wrapper object may be used to standardize the model object’s external
behaviour under a predict call.

###  2.2 Choosing sections: Condition selector plots

In condvis, choosing a section is treated as choosing a point in the space of
the conditioning predictors. This could be accomplished by keyboard input or
sliders for continuous predictors, and a dropdown menu for categorical
predictors. Such a simple approach would quickly succumb to the curse of
dimensionality however, and it would be difficult to choose sensible sections.
By sensible, we mean that a section is roughly in a part of the data space
where there are observed data points, and not an unwitting extrapolation.

We give three ways to graphically choose a section.

  * •

Univariate and bivariate displays: histograms, barplots, scatterplots,
boxplots, and spineplots (see Figures 9 and 9). These are the default
condition selectors, because they are easy to use and provide some way to
avoid the worst extrapolations. The scatterplot condition selector is changed
to a 2-D histogram when there are more than 2,000 observations and the gplots
(Warnes _et al._ , 2016) package is installed (see Figures 13 and 14).

  * •

Full scatterplot matrix (see Figure 9). This provides every possible bivariate
view of the data. It provides more views in which to detect extrapolation, but
can be very cumbersome. Factors are coerced to integers for visualization.

  * •

Parallel coordinates plot (see Figure 9). Relatively neat and easy to
understand, but spotting extrapolations is not straightforward. Factors are
coerced to integers for visualization.

###  2.3 Ordering condition selector plots

For each approach to graphically choosing sections, we have the choice to
order or group the conditioning predictors to provide for effective
exploration of the predictor space. For bivariate displays, we arrange the
conditioning predictors in pairs using a greedy algorithm that aims to reduce
the chance of unwitting extrapolation (when compared to using univariate
views).

For example, in comparing a scatterplot option to the alternative of two
histograms, we consider the ratio of the area of the convex hull of the data
in the bivariate view to the area of the bounding rectangle. If this ratio is
near 1, the scatterplot does not offer any extra help over two histograms in
avoiding extrapolations. As this ratio gets smaller, it becomes more important
to take account of the bivariate relationship. In the same manner, when
comparing a spineplot option to the alternative of two barplots, we consider
the ratio of the number of observed factor combinations to the total possible
number of factor combinations.

This is a simple approach, intended only to give a default ordering –
independent of the model or response – for users who have not supplied an
ordering. We anticipate that individual users will provide their own ordering,
based on variable importance measures or pre-existing knowledge of the
variables. For example, with a motor insurance pricing model, a salesperson
may wish to have quick access to the effect of voluntary excess or engine size
(inputs which may be changed) in negotiating a policy, but may not have any
need to understand the effect of age or claims history (inputs which cannot be
changed).

Figure 7: Univariate condition selector plots.

Figure 8: Bivariate condition selector plots.

Figure 9: Full scatterplot and parallel coordinates condition selector plots.

###  2.4 Visualizing sections

We limit ourselves to conditioning on all but one or two predictors, so we
visualize the intersection of sections and fitted models as either curves or
surfaces (categorical responses are shown as colours, or converted to integers
for visualization). The section visualizations used are summarized in Figure
10.

response cont cat 1 cont 1 cat 2 cont 2 cat 1 cont 1 cat section predictors

Figure 10: Visualizing sections through the fitted model.

The visualizations are chosen according to the type of response, and the type
of predictors we are interested in investigating. Two of the cells in Figure
10 are left blank because with binary logistic regression, the probability
(continuous) of class membership might actually be a more suitable response to
visualize rather than the expected class. If we are only interested in
expected class membership, then it is better to visualize a section along two
predictors. In Section 1.1, we show an example of a section with a continuous
response and a categorical predictor of interest.

After we visualize fitted models on the section, we must choose which observed
data to display. We choose observed data according to their distance from the
section.

As in Section 2.2, we consider both the section and observed data as points in
the space of the conditioning predictors. As a result, we can take a
dissimilarity measure defined between two points as the dissimilarity between
the section and an observed data point

$\displaystyle
d(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})=\lvert\lvert\boldsymbol{x}_{i}-\boldsymbol{x}^{\prime}\rvert\rvert_{p}+\lambda
M(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})$

where $\boldsymbol{x}_{i}$ represents the $i$th observation on the
conditioning predictors, and $\boldsymbol{x}^{\prime}$ gives the current
section in the conditioning predictors.
$\lvert\lvert\boldsymbol{x}_{i}-\boldsymbol{x}^{\prime}\rvert\rvert_{p}$ is
the generalized Minkowski distance on the numeric elements,
$M(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})$ is the number of mismatches on
the categorical elements, and $\lambda$ is a scaling constant (argument lambda
to ceplot). This distance is calculated after standardizing the continuous
elements of the conditioning predictors to have zero mean and unit variance.

We then have a function which assigns colours to observations based on their
distance from the section:

$\displaystyle
K(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime},\sigma)=\left\\{\begin{array}[]{ll}\text{black}&\hskip
14.93768pt\mbox{if }0\leq
d(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})<0.3\sigma\\\ \text{dark
grey}&\mbox{if }0.3\sigma\leq
d(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})<0.7\sigma\\\ \text{light
grey}&\mbox{if }0.7\sigma\leq
d(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})<\sigma\\\ \text{do not
plot}&\hskip 14.22636pt\mbox{if }\sigma\leq
d(\boldsymbol{x}_{i},\boldsymbol{x}^{\prime})\end{array}\right.$

where $\sigma$ is a threshold parameter given by the user (argument threshold
to ceplot). Letting $\sigma$ equal 0 gives exact conditioning, that is, we
only plot points on the section. Increasing $\sigma$ gives more approximate
conditioning. Setting $\lambda>\sigma$ means that only observations which
match the section on categorical predictors may be considered to be _near_ the
section. The correct way to achieve this with ceplot is to leave threshold =
NULL, which requires all factor levels to match but is more computationally
efficient.

The condvis package currently implements two special cases of the Minkowski
distance, namely maximum norm and Euclidean distance. The maximum norm is the
limit of the Minkowski distance as $p$ tends to infinity

$\displaystyle\lvert\lvert\boldsymbol{x}_{i}-\boldsymbol{x}^{\prime}\rvert\rvert_{\infty}=\max_{j}\lvert\boldsymbol{x}_{ij}-\boldsymbol{x}^{\prime}_{j}\rvert$

The maximum norm results in conditioning very much like that of trellis
(Becker _et al._ , 1996), where intervals on conditioning predictors are used
to choose observations to plot on a single panel. Euclidean distance is the
Minkowski distance with $p$ equal to 2. See Figure 11 for an example of both
distance measures with two conditioning predictors.

(a)

(b)

Figure 11: Choosing data to display using distance between section and
observations. Condition on x1 and x2 by taking a section defined by x1 = x2 =
0.4. There are no points exactly at these coordinates, so we take points
nearby. (a) Maximum norm distance. (b) Euclidean distance.

###  2.5 Interactive graphics platform

The interactive graphics in condvis have been implemented using base R
graphics. Extensive use is made of getGraphicsEvent and related functions in
grDevices (R Core Team, 2016), which were introduced in R 2.1.0, released in
April 2005. The decision to develop condvis on this platform was motivated
chiefly by the desire to keep software dependencies to a minimum. The main
non-standard dependency of condvis is the XQuartz (www.xquartz.org) device on
the Mac OS.

The secondary benefit of developing interactive graphics in R is the huge
wealth of static graphics which are already available in both the standard R
distribution and its extension packages. If we can interact with the R
graphics device, we do not need to port these plots to another
language/platform in order to interact with them. In this package alone, we
take advantage of the following graphics: histogram, barplot, scatterplot,
boxplot, spineplot, 2-D histogram, 3-D perspective mesh, and parallel
coordinates plot.

The Shiny web applications implemented in condvis provide an alternative to
the default implementations, and allow the graphics to be deployed to a web
page for others to use. On Mac OSX, Shiny becomes the default for ceplot in
the absence of XQuartz. See Section 4.2 for further discussion of interactive
graphics.

##  3 Using the package

###  3.1 Graphic type and layout

There are three layout options for the interactive graphics created by ceplot:

  * •

The default option is to place the section and condition selector plots on one
device, with the section being the largest graphic on the left, and the
condition selector plots arranged in columns on the right.

  * •

The separate111Special care should be taken when using this option with R
version $<$ 3.2.2, as closing the non-interactive device (the one showing the
section) can cause a crash (see Bug 16438 at bugs.r-project.org) option is set
using type = "separate" when calling ceplot. This places the section on one
device, and the condition selector plots on a second device. This option gives
the flexibility to use the scatterplot matrix and parallel coordinates plots
for condition selectors.

  * •

The Shiny option is set using type = "shiny" when calling ceplot. This gives
an arrangement similar to the default above, but is implemented as a web
application. This option allows some extra interactivity, such as changing the
distance function type and threshold value, and the ability to deploy your
interactive graphic to the internet for sharing.

###  3.2 Interacting with the graphics

The main user interaction with condvis is through clicking on condition
selector plots in order to select sections to visualize. When the section
being visualized is a three-dimensional perspective mesh, it may be rotated
using the arrow keys. In the default plot, the perspective mesh may also be
rotated by clicking and dragging with the mouse. In Shiny, the rotation may
instead be adjusted using sliders. When using the default or separate plots, a
snapshot of the current visualizations may be taken by pressing the ‘s’ key. A
pdf snapshot is then saved to the working directory. In Shiny, the button for
this is labelled “Download snapshot (pdf)”.

###  3.3 Examples

The following examples use the FEV data again, as well as the powerplant data
from Tüfekci (2014), and the wine data from Aeberhard _et al._ (1992).

#### FEV data

We use the FEV dataset from Section 1.1 for another example here. This time,
we are not addressing any new question, rather we are just exploring a few
different fitted models. This example shows the default plot arrangement on a
single device, and the comparison of several models on a 2-D section. R>
library("randomForest") R> library("mgcv") R> m3 <\- list( \+ RF =
randomForest(fev ., data = fev), \+ lm = lm(fev ., data = fev), \+ gam =
mgcv::gam(fev smoke + gender + s(age) + s(height), \+ data = fev)) R>
ceplot(data = fev, model = m3, sectionvars = "smoke") See Figure 12 for a
snapshot of the interactive graphic resulting from this call to ceplot.

Figure 12: Snapshot comparing the modelled effect of smoking on FEV from three
different models.

#### Powerplant data

The powerplant data were collected for the purpose of predicting full load
electrical power output of a powerplant. The underlying processes are well
understood, but require the difficult computation of differential equations in
order to model them. Tüfekci (2014) suggests machine learning techniques as an
alternative and proceeds to fit several different ‘black-box’ models. One
interesting aspect of this article is that, in presenting these complex models
to an arguably non-statistical audience, there are no graphics produced to
visualize a predictor effect.

We fit two models to the powerplant data – a support vector machine with a
radial kernel, and an additive model with smoothing splines for each
predictor.

R> data("powerplant") R> library("e1071") R> library("mgcv") R> m4 <\- list(
\+ svm = svm(PE ., data = powerplant), \+ gam = gam(PE s(AT) + s(V) + s(AP) +
s(RH), \+ data = powerplant)) The first call to ceplot shows how the support
vector machine fits the data, showing how some of the modelled effect
curvature occurs in regions of the data space with no observed data. R>
ceplot(data = powerplant, model = m4["svm"], sectionvars = "AT", \+ type =
"separate") The next call to ceplot compares the support vector machine to the
additive model (Figure 13, or Shiny demo), showing how both models fit the
data in a similar way in regions of the data space near observed data. R>
ceplot(data = powerplant, model = m4, sectionvars = "AT", \+ type =
"separate", threshold = 0.5) The final call to ceplot demonstrates a 3-D
section. R> ceplot(data = powerplant, model = m4["svm"], \+ sectionvars =
c("AT", "V"), type = "separate", view3d = TRUE, \+ threshold = 0.2) See Figure
14 for a snapshot of this example. There is a video demonstration of this
example here.

Figure 13: Snapshot showing the modelled effect of AT given some fixed values
of V, AP and RH, for two different models. The two models look quite
different, except in the region where observed data are nearby.

Figure 14: Snapshot showing the modelled effect of V and AT given some fixed
values of RH and AP. Note the condition selector is a two-dimensional
histogram rather than a scatterplot.

#### Wine data

The wine data are the results of a chemical analysis of wines grown in the
same region in Italy but derived from three different cultivars. The basic
task is to produce a classifier which can take the chemical measurements and
identify the correct cultivar. For illustration purposes, we fit a
randomForest (Liaw and Wiener, 2002) classifier on six of the predictors.
(Shiny application demo.) R> library("randomForest") R> data("wine") R>
wine$Class<-as.factor(wine$Class) R> m5 <\- randomForest(Class Alcohol + Malic
+ Ash + Magnesium + \+ Phenols + Flavanoids, data = wine) It is difficult to
comprehend how a classifier assigns regions to different classes in six
dimensions, but visualizing this on a two-dimensional section is
straightforward. We first take sections along the predictors Alcohol and
Phenols, using the option to create a Shiny web application. R> ceplot(data =
wine, model = m5, sectionvars = c("Alcohol", "Phenols"), \+ type = "shiny")
Next, we visualize sections on the same predictors, but use the parallel
coordinates condition selector. R> ceplot(data = wine, model = m5, sectionvars
= c("Alcohol", "Phenols"), \+ type = "separate", selectortype = "pcp",
threshold = 2)

Figure 15: Snapshot showing a section through a random forest classifier in
six dimensions. Shiny application here.

##  4 Related work

###  4.1 Related work in the literature

Trellis graphics (Becker _et al._ , 1996) are separate plots visualizing
subsets of the data. Conditioning on continuous variables is achieved by the
intersection of intervals on each variable. Nason _et al._ (2004) apply the
trellis technique to the conditional visualization of fitted models, calling
the resulting plots CARTscans. Embedded plots (Grolemund and Wickham, 2015)
are a collection of graphs organized in a larger graphic, that can display
more complex relationships than typically possible. Furnas and Buja (1994)
discuss the use of sections and projections in exploring high-dimensional
space.

Partial residual plots (Larsen and McCleary, 1972; Ezekiel, 1924) produce a
single plot showing the effect of one or two predictors, conditional on the
remaining predictors and a fitted additive model. Cook (1993, 1996) notes that
the model used in the construction of a partial residual plot must be
reasonably well specified for the conditioning predictors in order to have a
valid plot. Partial residual plots are also known as component + residual
plots.

ICE plots (Goldstein _et al._ , 2015) show sections through a fitted model at
observed data points. The main aim is to allow interpretation of ‘black-box’
model effects, and give an impression of extrapolation behaviour. Partial
dependence plots (Friedman, 2001) predate ICE plots, but can be interpreted as
the average of the sections shown in an ICE plot.

Trellis offers a simple, interpretable approach, but it is difficult to apply
it to fitted models, especially with continuous predictors. Partial residual
plots are limited by their dependence on a fitted additive model. ICE plots
can visualize more complicated non-additive models, but run into problems of
overplotting and do not explicitly show the observed data.

###  4.2 Related work in R

#### Conditional visualization

The lattice (Sarkar, 2008) package is an implementation of trellis graphics in
R. The coplot function in graphics (R Core Team, 2016) also provides a
conditioning plot following the trellis method. The effects (Fox, 2003)
package produces partial residual plots and related graphics for additive
models. The visreg (Breheny and Burchett, 2016) package produces partial
residual plots for additive models as well as extensions to the non-additive
case. The plot method plot.gam in both gam (Hastie, 2015) and mgcv (Wood,
2016) allows the production of two-dimensional and three-dimensional partial
residual plots for models of class gam. The ICEbox (Goldstein _et al._ , 2015)
package implements ICE plots. These are all static visualizations.

#### Interactive graphics

Many approaches to interactive graphics in R port the graphics to another
language for the interactive functionality, for example: iplots (Urbanek and
Wichtrey, 2013), shiny (Chang _et al._ , 2016), ggvis (Chang and Wickham,
2016). The main approach in condvis is to interact directly with graphics
produced by R code, displayed on standard R graphics devices. This is achieved
using functions contained in the grDevices package (R Core Team, 2016), which
is distributed with the basic R installation. The use of grDevices for
interactive graphics can also be seen in the Association Navigator of Buja _et
al._ (2010), and the sudoku (Brahm _et al._ , 2014) package. The supplementary
material contains three R scripts to demonstrate the use of grDevices in
producing basic interactive graphics with R.

###  4.3 Contribution

The condvis package takes the approach of visualizing the model in data space
(see discussion by Wickham _et al._ (2015)). Out of the related methods
mentioned previously, condvis is most similar to CARTscans (Nason _et al._ ,
2004) in the sense of applying conditional visualization to fitted models.
Rather than using trellis graphics for conditioning, we take a more flexible,
interactive approach to conditioning. Instead of producing multiple plots
representing ‘fat’ sections or subregions of the data space, we produce one
section at a time and attempt to show data which are near this section. This
means that fitted models are represented exactly (no need for integration or
averaging of the fitted model), and we also display observed data to give
context to the section.

##  5 Summary and outlook

###  5.1 Summary

The condvis package allows the user to interactively take 2-D and 3-D sections
in data space and visualize fitted models where they intersect the section,
and observed data if sufficiently near the section according to a distance
measure.

###  5.2 Strengths and limitations

The strength of condvis lies in creating low-dimensional visualizations of
fitted models in high dimensional space. This allows interpretation of complex
‘black-box’ models, and shows how observed data actually support the fitted
model in data space. The software has been tested and shown to work on
Windows, Mac and Linux, with the following model classes in R: lm, glm, gam,
svm (e1071), rpart, randomForest.

This method of interactively choosing sections is, however, limited to dealing
with no more than, say 20 or 25 conditioning predictors. The condition
selectors would begin to crowd the screen at this point, and exploring the
data space with only the help of bivariate data visualizations would become
difficult. This method also does not suit situations where categorical
predictors have more than 4 or 5 levels. The barplots and spineplots in such
cases may not represent these predictors well for the purpose of choosing
conditions.

###  5.3 Further work

This work has already opened up some avenues of further research. It becomes
obvious after a short time using condvis that it can sometimes be difficult to
choose interesting sections, or indeed to feel confident that the data space
can be explored effectively when choosing sections by hand. This naturally
suggests an automated approach to producing sections throughout the data
space.

Such an automated approach would likely result in a very large number of
sections, the direct visualization of which would not be humanly possible. One
solution to this may be to apply the ‘cognostics’ approach as discussed in
Tukey and Tukey (1982). This involves using a computer to rank plots according
to some criteria, and subsequently choosing a subset of plots for examination
by a human user.

Another approach would be to order the sections according to their location in
the data space, and then produce an animation which transitions between the
sections, resulting in a sort of conditional tour of the space (as in a tour
of sections, rather than a tour of projections as in Asimov (1985)). This
feature is currently in development for condvis (see ?condtour).

So far, we have discussed conditional visualization as a one-way process;
choose a section/condition and then investigate the section. The inverse
problem might also be interesting, for example, what areas of the predictor
space have high values of expected response, or the highest curvature of the
fitted model in a certain direction?

We have taken a very strict approach to conditioning in this discussion. We
have only considered conditions where predictors are set to a single value. If
we consider conditions more flexibly over regions of the predictor space, the
fitted model can no longer be visualized directly. However, it is possible to
use approximate integrals of the fitted model over such regions to make
informative visualizations.

Finally, only some of our section visualizations provide for model comparison.
As it stands, none of our 3-D sections allow model comparison, and so this is
something we would like to include in further versions of condvis.

## References

  * Aeberhard _et al._ (1992) Aeberhard S, Coomans D, de Vel O (1992).  “Comparison of Classifiers in High Dimensional Settings (92-02).”  _Technical report_ , Dept. of Computer Science and Dept. of Mathematics and Statistics, James Cook University of North Queensland.
  * Asimov (1985) Asimov D (1985).  “The Grand Tour: A Tool for Viewing Multidimensional Data.”  _SIAM Journal on Scientific and Statistical Computing_ , 6(1), pp. 128–143.  ISSN 0196-5204.  URL http://dx.doi.org/10.1137/0906011.
  * Becker _et al._ (1996) Becker RA, Cleveland WS, Shyu MJ (1996).  “The Visual Design and Control of Trellis Display.”  _Journal of Computational and Graphical Statistics_ , 5(2), pp. 123–155.  ISSN 10618600.  URL http://www.jstor.org/stable/1390777.
  * Brahm _et al._ (2014) Brahm D, Snow G, Seeliger C, Bengtsson H (2014).  _sudoku: Sudoku Puzzle Generator and Solver_.  R package version 2.6, URL http://CRAN.R-project.org/package=sudoku.
  * Breheny and Burchett (2016) Breheny P, Burchett W (2016).  _visreg: Visualization of Regression Models_.  R package version 2.2-0, URL http://CRAN.R-project.org/package=visreg.
  * Buja _et al._ (2010) Buja A, Krieger A, George E (2010).  “A Tool for Mining Large Correlation Tables: The Association Navigator.”  _Report to the Simons Foundation Autism Research Initiative_.  URL http://stat.wharton.upenn.edu/~buja/PAPERS/.
  * Chang _et al._ (2016) Chang W, Cheng J, Allaire J, Xie Y, McPherson J (2016).  _shiny: Web Application Framework for R_.  R package version 0.13.2, URL http://CRAN.R-project.org/package=shiny.
  * Chang and Wickham (2016) Chang W, Wickham H (2016).  _ggvis: Interactive Grammar of Graphics_.  R package version 0.4.3, URL http://CRAN.R-project.org/package=ggvis.
  * Cook (1993) Cook RD (1993).  “Exploring Partial Residual Plots.”  _Technometrics_ , 35(4), pp. 351–362.  ISSN 00401706.  URL http://www.jstor.org/stable/1270269.
  * Cook (1996) Cook RD (1996).  “Added-Variable Plots and Curvature in Linear Regression.”  _Technometrics_ , 38(3), pp. 275–278.  ISSN 00401706.  URL http://www.jstor.org/stable/1270611.
  * Ezekiel (1924) Ezekiel M (1924).  “A Method of Handling Curvilinear Correlation for any Number of Variables.”  _Journal of the American Statistical Association_ , 19(148), pp. 431–453.  ISSN 01621459.  URL http://www.jstor.org/stable/2281561.
  * Fox (2003) Fox J (2003).  “Effect Displays in R for Generalised Linear Models.”  _Journal of Statistical Software_ , 8 (15), pp. 1–27.  URL http://www.jstatsoft.org/v08/i15/.
  * Friedman (2001) Friedman JH (2001).  “Greedy Function Approximation: A Gradient Boosting Machine.”  _The Annals of Statistics_ , 29(5), pp. 1189–1232.  ISSN 00905364.  URL http://www.jstor.org/stable/2699986.
  * Furnas and Buja (1994) Furnas GW, Buja A (1994).  “Prosection Views: Dimensional Inference Through Sections and Projections.”  _Journal of Computational and Graphical Statistics_ , 3, 323–385.
  * Goldstein _et al._ (2015) Goldstein A, Kapelner A, Bleich J, Pitkin E (2015).  “Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation.”  _Journal of Computational and Graphical Statistics_ , 24(1), pp. 44–65.  URL http://dx.doi.org/10.1080/10618600.2014.907095.
  * Grolemund and Wickham (2015) Grolemund G, Wickham H (2015).  “Visualizing Complex Data with Embedded Plots.”  _Journal of Computational and Graphical Statistics_ , 24(1), 26–43.  10.1080/10618600.2014.896808.
  * Hastie (2015) Hastie T (2015).  _gam: Generalized Additive Models_.  R package version 1.12, URL http://CRAN.R-project.org/package=gam.
  * Kahn (2005) Kahn M (2005).  “An Exhalent Problem for Teaching Statistics.”  _Journal of Statistics Education_ , 13(2).  URL http://www.amstat.org/publications/jse/v13n2/datasets.kahn.html.
  * Larsen and McCleary (1972) Larsen WA, McCleary SJ (1972).  “The Use of Partial Residual Plots in Regression Analysis.”  _Technometrics_ , 14(3), pp. 781–790.  ISSN 00401706.  URL http://www.jstor.org/stable/1267305.
  * Liaw and Wiener (2002) Liaw A, Wiener M (2002).  “Classification and Regression by randomForest.”  _R News_ , 2(3), 18–22.  URL http://CRAN.R-project.org/doc/Rnews/.
  * Meyer _et al._ (2015) Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F (2015).  _e1071: Misc Functions of the Department of Statistics (e1071), TU Wien_.  R package version 1.6-7, URL http://CRAN.R-project.org/package=e1071.
  * Nason _et al._ (2004) Nason M, Emerson S, LeBlanc M (2004).  “CARTscans: A Tool for Visualizing Complex Models.”  _Journal of Computational and Graphical Statistics_ , 13(4), pp. 807–825.  ISSN 10618600.  URL http://www.jstor.org/stable/27594079.
  * Niu and Hoff (2014) Niu X, Hoff P (2014).  _covreg: A simultaneous regression model for the mean and covariance_.  R package version 1.0, URL http://CRAN.R-project.org/package=covreg.
  * O’Connell _et al._ (2016) O’Connell M, Hurley CB, Domijan K (2016).  “Conditional Visualization for Statistical Models: An Introduction to the condvis Package in R.”  _Journal of Statistical Software_ , p. to appear.
  * R Core Team (2016) R Core Team (2016).  _R: A Language and Environment for Statistical Computing_.  R Foundation for Statistical Computing, Vienna, Austria.  URL http://www.R-project.org/.
  * Rosner (2010) Rosner B (2010).  _Fundamentals of Biostatistics_.  7th edition. Cengage Learning.  ISBN 9780538733496.
  * Sarkar (2008) Sarkar D (2008).  _lattice: Multivariate Data Visualization with R_.  Springer-Verlag, New York.  ISBN 978-0-387-75968-5, URL http://lmdvr.r-forge.r-project.org.
  * Smola and Vapnik (1997) Smola A, Vapnik V (1997).  “Support Vector Regression Machines.”  _Advances in Neural Information Processing Systems_ , 9, 155–161.
  * Tüfekci (2014) Tüfekci P (2014).  “Prediction of Full Load Electrical Power Output of a Base Load Operated Combined Cycle Power Plant Using Machine Learning Methods.”  _International Journal of Electrical Power & Energy Systems_, 60(Sep 2014), pp. 126–140.  ISSN 0142-0615.  URL http://www.sciencedirect.com/science/article/pii/S0142061514000908.
  * Tukey and Tukey (1982) Tukey JW, Tukey PA (1982).  “Some Graphics for Studying Four-Dimensional Data.”  In KW Heiner, RS Sacher, JW Wilkinson (ed.), _Computer Science and Statistics: Proceedings of the 14th Symposium on the Interface_ , pp. 60–66. Springer-Verlag.
  * Urbanek and Wichtrey (2013) Urbanek S, Wichtrey T (2013).  _iplots: iPlots - Interactive Graphics for R_.  R package version 1.1-7, URL http://CRAN.R-project.org/package=iplots.
  * Warnes _et al._ (2016) Warnes GR, Bolker B, Bonebakker L, Gentleman R, Liaw WHA, Lumley T, Maechler M, Magnusson A, Moeller S, Schwartz M, Venables B (2016).  _gplots: Various R Programming Tools for Plotting Data_.  R package version 3.0.1, URL http://CRAN.R-project.org/package=gplots.
  * Wickham _et al._ (2015) Wickham H, Cook D, Hofmann H (2015).  “Visualizing Statistical Models: Removing the Blindfold.”  _Statistical Analysis and Data Mining: The ASA Data Science Journal_ , 8(4), 203–225.  ISSN 1932-1872.  URL http://dx.doi.org/10.1002/sam.11271.
  * Wood (2016) Wood S (2016).  _mgcv: Mixed GAM Computation Vehicle with GCV/AIC/REML Smoothness Estimation_.  R package version 1.8-14, URL http://CRAN.R-project.org/package=mgcv.

##  Appendix A Supplementary material

The supplementary material consists of the source code for condvis, a
replication script for the examples in this article, and three further scripts
which demonstrate the kind of interactive graphics described in the paper.

  * •

condvis_0.3-4.tar.gz: source code for condvis, version 0.3-3 available on
CRAN.

  * •

replication.R: R script for replicating examples in this article

  * •

interactive-example-1.R: Example of interacting with a single R plot.

  * •

interactive-example-2.R: Example of interacting with a single R plot, and
propagating the change to another plot.

  * •

interactive-example-3.R: Example of interacting with multiple R plots,
propagating the changes to the other plots as required. This currently seems
to work only in Windows.


Shankar Ramharack and Sanjay Bahadoorsingh 11institutetext: The University of
the West Indies, St. Augustine, Trinidad and Tobago, 11email:
<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Connecting the Dots: A Comprehensive Literature Review on Low and Medium-
Voltage Cables, Fault Types, and Digital Signal Processing Techniques for
Fault Location

Shankar Ramharack  0000-0003-3759-7333 ** Sanjay Bahadoorsingh

###### Abstract

The review begins with an exploration of acceptable cable types guided by
local standards. It then investigates typical cable faults, including
insulation degradation, conductor faults, and ground faults, providing
insights into their characteristics, causes, and detection methods.
Furthermore, the manuscript surveys the latest publications and standards on
DSP techniques in fault location spanning various algorithms used. This review
provides a comprehensive understanding of low and medium-voltage cables, fault
types, and DSP techniques. The findings contribute to improved fault diagnosis
and localization methods, facilitating more accurate and efficient cable fault
management strategies

###### keywords:

distribution, fault location, reflectometry

##  1 Background

Cable fault diagnostics play a crucial role in ensuring the safe and reliable
operation of electrical power systems[1]. Low and medium-voltage cables are
vital components of power distribution networks, supplying electricity to
residential, commercial, and industrial consumers. However, over time, these
cables can experience various types of faults, such as insulation degradation,
conductor faults, and ground faults. These faults can disrupt power supply,
lead to equipment failures, and pose safety risks. Identifying and
understanding acceptable cable types enables engineers, contractors, and
installers to make informed decisions during cable selection and installation
processes. Similarly, conducting a literature review on technical publications
and standards concerning typical cable faults on low and medium-voltage cables
is of great significance as it is often neglected in fault location work[2].
By reviewing technical publications and standards, researchers and
practitioners gain access to collective knowledge and experiences in cable
fault diagnostics. This knowledge aids in developing effective maintenance
strategies, reducing downtime, and improving the overall reliability of power
networks. Furthermore, the literature review on modern digital signal
processing (DSP) techniques used in reflectometry and cable fault location
addresses the need for advanced and accurate fault detection methodologies.
DSP techniques, such as time-domain reflectometry (TDR), offer powerful tools
for analyzing cable faults and determining their locations[3]. By reviewing
the literature on these techniques, researchers can identify the latest
advancements, algorithms, and methodologies employed in fault detection and
localization. This knowledge contributes to the development of more precise
and efficient cable fault location systems, minimizing repair time, reducing
costs, and enhancing the overall reliability of power distribution networks.

###  1.1 Objectives

The objectives of this work are as follows

  1. 1.

To perform a literature review on types of low and medium-voltage cables that
are acceptable for installations as guided by the local and international
standards.

  2. 2.

To perform a literature review of technical publications and standards on
typical cable faults on low and medium-voltage cables.

  3. 3.

To perform a literature review of modern digital signal processing techniques
used in reflectometry and cable fault location.

##  2 Low and Medium-Voltage Cables: Acceptable Types as Guided by
International and Local Standards

The most widely used standards guiding LV and MV Cable Installations in North
America and the Caribbean are those issued by:

  1. 1.

The Aluminum Association (AA)

  2. 2.

American National Standards Institute (ANSI)

  3. 3.

American Society for Testing and Materials (ASTM)

  4. 4.

Canadian Standards Association (CSA)

  5. 5.

Insulated Cable Engineers Association (ICEA)

  6. 6.

National Electrical Manufacturers Association (NEMA)

  7. 7.

Association of Edison Illuminating Companies (AEIC)

  8. 8.

Rural Utilities Service (RUS)

  9. 9.

Underwriter’s Laboratories (UL)

  10. 10.

National Electrical Code (NEC)

Aerial cable is used occasionally for primary conductors in special situations
where clearances are too close for open-wire construction or where adequate
tree trimming is not practical. The type of construction more frequently used
consists of covered conductors (nonshielded) supported from the messenger by
insulating spacers of plastic or ceramic material[4]. The conductor
insulation, usually a solid dielectric such as polyethylene, has a thickness
of about 150 mils for a 15-kV class circuit and is capable of supporting
momentary contacts with tree branches, birds, and animals without
puncturing[5].

The conductor sizes most commonly used in underground primary distribution
vary from No. 4 AWG to 1000 kcmil[6]. Four-wire main feeders may employ 3- or
4-conductor cables, but single conductor concentric-neutral cables are more
popular for this purpose. The latter usually employ crosslinked polyethylene
insulation, and often have a concentric neutral of one-half or one-third of
the main conductor cross-sectional area.

The smaller-sized cables used in lateral circuits of Underground Residential
Distributions(URD) systems are nearly always single-conductor, concentric-
neutral, crosslinked polyethylene-insulated, and usually directly buried in
the earth. Insulation thickness is on the order of 175 mils for 15-kV-class
cables and 345 mils for 35-kV class with 100% insulation level[6]. Stranded or
solid aluminum conductors have virtually supplanted copper for new
construction, except where existing duct sizes are restrictive. With the
solid-dielectric construction, to limit voltage gradient at the surface of the
conductor within acceptable limits, a minimum conductor size of No. 2 AWG is
common for 15-kV-class cables, and No. 1/0 AWG for 35-kV class.

Primary voltage circuits(5-35kV) use paper-insulated, lead-covered (PILC)
three-conductor cables extensively. Single-conductor secondary cables with
rubber insulation and neoprene jacket are common. More recently, single-
conductor polyethylene-insulated(PE) cables are being used for both primary
and secondary[7]. Copper conductors predominated in the past, but aluminum has
nearly displaced copper in new installations, except where existing duct space
is limiting. In residential and suburban areas, new underground distribution
systems to serve commercial loads often employ direct-buried cables[7];
conduits may be provided in locations where subsequent excavation would be
excessively expensive or inconvenient. Aluminum conductors are almost
universal. For primary cables, solid dielectric insulation is used almost
exclusively, with cross-linked polyethylene(XLPE) and ethylene–propylene
rubber(EPR) insulations [5]. Concentric-neutral wires are common. Secondary
cables in these systems generally have aluminum conductors and solid-
dielectric insulation, with cross-linked polyethylene being the most common.
The secondary neutral is usually an insulated conductor, although there is
some use of bare copper neutrals.

Electric supply cables are insulated with a range of materials depending on
voltage ratings, type of service, installation conditions etc. The following
are commonly used:

  1. 1.

Rubber and rubber-like for 0 to 35kV

  2. 2.

Varnished cambric for 0 to 28kV

  3. 3.

Impregnated paper of the solid type for voltages up to 69kV and with
pressurized gas or oil up to 345kV or higher

For most distribution circuits in the 5-kV class or higher, the cables employ
a shielded construction[8]. Shielding is used on the outer surface of the
cable insulation or directly over the main conductor, or both. Outside
shielding, often in the form of metallic tapes, metallic sheaths, or
concentric wires, must be effectively grounded.

The aforementioned insulation systems usually require a sheath or suitable
jacket to prevent infiltration of moisture, loss of oil, gas, or impregnate
and to provide protection against corrosion and electrolysis. In some cases,
an armor overlay is used to provide mechanical protection. Single conductor
cables are used in single-phase primary systems and frequently used in 3-phase
direct buried primary systems. The 3 conductor primary cables are often used
in duct systems. At the present time,solid-dielectric insulating materials
such as tree retardant, cross-linked polyethylene and EPR are receiving the
widest application in Underground Distribution Systems(UDS)[9, 5], both direct
buried and duct systems.

From the Electric distribution handbook[5], cables used in underground systems
may either be concentric neutral cables or power cables(for utilities). The
jacket is usually made of Linear low-density polyethylene (LLDPE), PE or
Semiconductors. The insulation most used in the industry is PE, XLPE, PILC,
TR-XLPE & EPR. For URD applications, aluminum is the choice of conductor.

Caribbean countries and areas outside US and Europe follow similar
installation practices and cable selections such as in Trinidad. During a
consultation with the Trinidad and Tobago Electrical Commission (T&TEC),
Shielded Polyvinyl Chloride (PVC) and shielded Cross Linked Polyethene (XLPE)
cables are most used in public transmission systems, however, in private
distributions, shielded EPR has been recently adopted[8]. This is supported by
[10] who performed a survey of the cables used in LV and MV installations.
Furthermore, the cables used by LV and MV installations usually follow the
guidelines of the British Standards Institution. The standards recommend PVC
Jacket, Aluminium-Armoured XLPE insulated cables with stranded copper cores.
Other standards such as the IEC utilize similar cable configurations with
slight differences in the installation environment guidelines, thermal
requirements, and conductor sizing (British Standards Institute 2007).

The same shielding and sheathing practices that are done internationally are
done locally in Trinidad and Tobago per the TTS standards[11, 12] which build
upon the NEC Standards. For URD applications, copper is the choice of
conductor while aluminum is used for the shield.

##  3  Typical Cable Faults in Low and Medium-Voltage Cables

Numerous studies have shed light on the types of faults typically found in low
and medium-voltage cables, as well as their underlying causes. The authors of
[13] identified insulation breakdown as a prevalent fault type, often caused
by aging, thermal stress, or manufacturing defects. Mechanical stresses, such
as bending or crushing, were found to be a significant cause of faults in
medium-voltage cables [14]. Another common fault type is moisture ingress,
resulting from damaged cable sheaths or inadequate sealing, as highlighted in
the investigation by [15]. Furthermore, [16] discussed short circuits and open
circuits as faults in low-voltage cables, which can arise due to insulation
damage, conductor breakage, or loose connections. Other studies have also
explored specific fault types, such as insulation material aging [17],
weather-related faults [18], manufacturing defects [19], and partial discharge
[20]. These findings emphasize the importance of understanding and addressing
the diverse causes of cable faults to enhance the reliability and performance
of low and medium-voltage cable systems.

After interviewing the fault location personnel at T&TEC they revealed that
the most common faults found locally are high resistance faults and single
line-to-ground (SLG) faults. It was reported in [21, 22] that SLG faults and
Series faults are the most common in underground power systems. Series faults
are often caused by a cable layer losing continuity due to a force above
ground such as a collision. A SLG fault occurs when the insulation of one or
more conductors fails. These faults are often permanent faults. Intermittent
faults are not considered in most fault locator designs since most
transmission systems utilize monitoring equipment to briefly cut transmission
and allow the fault to clear themselves. Furthermore, [23] state that up to
90% of cable faults are SLG faults and hence intermittent faults fall into a
minority. Resistive faults are classified as high resistive if it is beyond
200$\Omega$. TDR cannot locate faults greater than this but as mentioned, they
are rare. Hence it is useful to limit fault resistance to 200 $\Omega$ in the
test cases.

##  4 Digital Signal Processing Techniques in Reflectometry and Fault Location

The authors of [24] performed a state-of-the art review of cable fault
location methods showing that the most common methods of fault distance
location are impedance-based methods, differential equation-based methods and
travelling wave methods. Impedance-based methods are shown to be cheap and
simple to perform such as the Murray and Varley Loop method, but they are
sensitive to the fault resistance. It is shown in [25, 26] that the loop
methods perform accurate for open and short circuit faults, however, for
extreme low and high resistance faults, the bridges are difficult to
stabilize. Box’s optimization algorithm is proposed in [27] to automate bridge
stabilization to balance the bridges of the loop methods. The optimization
methods perform very favourable in real life testing with a 0.8% error rate.
it is suggested in [28] that capacitance bridges should be used for open
faults, resistance bridges for short circuit faults but HV bridges should be
used to locate insulation faults and high impedance faults.

According to [24], differential equation models are costly and do not perform
well for long lines. In addition, both ends of the line may need to be
accessed requiring more manpower and resources than a single ended approach.
It is reported in [29] the distributed line model, the characteristic method
has the merit of being suitable for short, long, transposed and untransposed
lines and can be modified for three-terminal systems. The accuracy is
sensitive to the choice of the time window and limited by the sampling rate.
This may not be practical for low-cost location for distribution lines.

Travelling wave (TW) methods have also been widely mentioned in the literature
as covered in [30, 26, 31, 32, 33, 34]reports the TW method performs poorly
for 6-35kV networks which indicates it may not be feasible to use in this
project. Measurement of TW is made more difficult when there are taps on the
distribution line adding to the complexity. The problems of TW fault
location(TWFL) is the attenuation of waves when sent through underground
cables. TWFL can be done either single ended or on both ends. It is reported
in [32] that it is possible to achieve greater accuracy with the multi-end
methods compared to the traditional fault location methods.

It is recommended by Megger, KEP, [26, 35, 28] that reflectometry methods be
used for low resistance faults. For higher resistance faults(where fault
resistance is ¿ 200$\Omega$), MIM, or ICE techniques on a surge wave
generator(SWG) should be used. Decay or HV-Bridge methods should be used for
locating high impedance or sheath faults. For tapped cables, differential
multiple impulse response methods can be used to determine the fault
locations. It is suggested in [35] to use of decay methods for intermittent
fault location. After meeting with T&TEC, the local fault location
professionals reported that TDR is mostly used for fault location.

There are many reflectometry methods used in fault location. Reflectometry
methods are distinguished by their EM test signal and method of the
reflectogram analysis[36]. The most common reflectometry methods are Time
Domain Reflectometry (TDR), Frequency Domain Reflectometry (FDR), Time-
Frequency Domain Reflectometry (TFDR) and Spectrum Time Domain Reflectometry
(STDR) (Shi and Kanoun 2014). Each method is suited to a particular
application. Reflectometry works well for most low and medium voltage
applications[37]). Reflectometry methods work on the principle of applying an
excitation signal to a cable or transmission line and analysing the reflected
trace. It is a form of RADAR and relies on the change of the characteristic
impedance of the line to generate a reflection. The crux of reflectometry is
the reflectogram analysis. There have been several analyses of the
reflectograms generated.

Time domain reflectometry methods usually involve sending a step signal or
pulsed waveform on a cable and sampling the input where the signal was applied
for a reflection[38]. The sampled reflectogram is denoised via a matched
filter or wavelet denoising algorithm then the travel time between signal
inflexions are read off from a screen or graph and used to calculate the fault
distance. Common methods to automate this are bubble sort to determine the
wavefront points and peak detection[39]. Work has also been done on automated
TDR systems which utilize reflection detection via MCU capture interrupts to
time the echo for the first reflection of a travelling wave [40, 41]. There
has been no work found attempting to use time series segmentation to treat the
TDR problem as a CPD problem possibly due to the significant waveform
distortion present on the waves.

Frequency domain reflectometry is rarely seen in the literature for medium
voltage applications hence it is not explored. Furthermore, it does not
perform well on cables longer than 2km and expensive directional couplers are
required to sample the system and obtain the reflectogram[38].

Time-Frequency domain reflectometry is much more common and attempts to
address the shortcomings of both time and frequency domain reflectometry. In
this a waveform with good time-frequency localization is incident on the
cable. The reflected wave is sampled and cross-correlated with the incident
signal to determine the fault location [42]. In [43], the authors model a
chafe in an aircraft cable using scattering parameters [44]. The transfer
function of the cable is then derived in terms of a parameter set, $\theta$.
The parameter set is estimated using a statistical method of probability
inversion using the reflected trace as target and a initial parameter
combination. The initial parameter is updated in a Bayesian approach until it
reaches a distribution similar to the reflected trace. The final parameters
will have information about the fault distance, type and dimension.

Similar to [43], most TFDR work utilize the Gaussian Envelope Linear Chirp
(GELC) signal as the incident signal due to its good time-frequency
localization [45]. However, they differ by their means of signal processing.
In [42, 46] a dictionary is created of all possible transformations of the
parameterized incident signal in terms of phase shift, amplitude and
frequency. The reflected signal is projected onto the dictionary to find the
closest match. The closest match’s time shift parameter is used to determine
the fault location. Other works utilize matched filters to eliminate noise and
time correlators to determine the fault distance [47].

In [48] a statistical model-based detection and frequency identification
method is employed to calculate the fault distance. The GELC is used a the
incident signal. A Likelihood Ratio test (LRT) is used to detect the
reflection and a Hidden Markov Model Hang Over Scheme is used to avoid the LRT
from cutting the tail of the GELC which can happen in cases of severe
attenuation. The Hilbert transform is used to determine the instantaneous
phase and consequently the signal’s instantaneous frequency. The instantaneous
frequency is obtained as linear combination of the GELC carrier sinusoid and
the angular frequency sweep rate. Ambient noise is removed via a constrained
Kalman filter (CKF). The filtered signal is the incident and reflected
wavefronts which are used to calculate the distance by multiplying half the
delay by the velocity of propagation (VOP). Variations on the frequency
estimation and noise handling was done in [49, 50].

The test signal in STDR is a PN binary code [51] that is launched from the
test device and encounters partial reflection and transmission at each
impedance discontinuity in the system being tested. The STDR response is
created by cross-correlating the reflections that return to the test point
with a delayed copy of the event PN code. SSTDR generates a sine-like
correlated reflection signature using a square- or sine-wave modulated PN code
as the test signal. The reflected signal will be dispersed and attenuated if
the system is frequency-dependent or lossy. Furthermore, the method is robust
to noise and can be applied on a electrical apparatus [52], underwater
applications [53] and power cables [37].

Changepoint detection methods in [54] have been used in RADAR and can be used
to analyse the reflectogram automatically and determine the fault distance
without the need for reflectogram interpreters [55] or commercial TDR
equipment. At the time at writing, the author has not found any work done on
the application of changepoint detection in time domain reflectometry for
power cable applications.

The crux of reflectometry-based fault location is the algorithm used. Hence,
this work explores the DSP methods used to perform fault location.
Furthermore, a guide to the development of a prototype that uses the method
explored is shown in the Discussion. This guide will account for
instrumentation noise (which can be addressed with a LPF) and recommendations
on things to be done to improve the accuracy of the algorithm for the
application domain.

##  5 Discussion

The literature review has provided valuable insights into acceptable cable
types, typical faults, and digital signal processing (DSP) techniques used in
fault location for low and medium-voltage cables. The review identified that
common cable types include polyethylene-insulated solid dielectric cables,
concentric-neutral cables, and shielded constructions. Additionally, aluminum
conductors have become the preferred choice over copper due to cost and
practical considerations.

Regarding typical cable faults, the review highlighted that insulation
breakdown, mechanical stresses, moisture ingress, short circuits, and open
circuits are among the common faults encountered in low and medium-voltage
cables. Understanding the root causes of these faults is crucial for effective
fault diagnostics and maintenance strategies. The literature revealed high
resistance, hard faults are the most common. Furthermore, and single line-to-
ground (SLG) faults are prevalent in the local power distribution systems.

The literature review on DSP techniques for cable fault location revealed the
prominence of impedance-based methods, differential equation-based methods,
and traveling wave (TW) methods. Impedance-based methods, such as the Murray
and Varley Loop method, are simple and cost-effective but sensitive to fault
resistance. Differential equation models are more accurate but costly and may
require access to both ends of the cable. TW methods, while widely mentioned
in the literature, have limitations in accuracy and may not be practical for
low-cost fault location in distribution networks. The use of modern digital
signal processing techniques, such as Box’s optimization algorithm, has shown
promise in automating bridge stabilization and improving the accuracy of
impedance-based methods.

##  6 Conclusion

This comprehensive literature review has provided valuable insights into the
types of low and medium-voltage cables acceptable for installations as guided
by local and international standards. Typical cable faults in low and medium-
voltage cables were explored, including insulation breakdown, mechanical
stresses, moisture ingress, short circuits, and open circuits. Knowledge of
these fault types and their underlying causes is critical for effective fault
diagnostics and maintenance strategies, enabling power utilities to minimize
downtime and enhance the overall reliability of power networks. The review of
digital signal processing techniques used in cable fault location highlighted
the prominence of impedance-based methods, differential equation-based
methods, and traveling wave (TW) methods. While impedance-based methods offer
simplicity and cost-effectiveness, modern DSP techniques like Box’s
optimization algorithm show promise in improving accuracy. However, TW methods
may not be practical for low-cost fault location in distribution networks.

The findings of this review contribute to improved fault diagnosis and
localization methods. Further research and development in DSP techniques hold
the potential for enhancing cable fault location systems, reducing repair
time, lowering costs, and ultimately improving the overall reliability of
power distribution networks.

##  7 Availability of data and material

There materials surrounding the manuscript can be obtained by contacting the
authors.

##  8 Competing interests

The authors declare no competing interests

##  9 Funding

This work was funded by the University of the West Indies St. Augustine
Campus.

##  10 Authors’ contributions

The authors confirm contribution to the paper as follows: study conception and
design: Shankar Ramharack, Sanjay Bahadoorsingh; analysis and interpretation
of results: Shankar Ramharack; draft manuscript preparation: Shankar
Ramharack.All authors reviewed the results and approved the final version of
the manuscript

##  11 Acknowledgements

The authors would like to thank Mr. Veeresh Ramnarine for their guidance
during the project. Furthermore, the authors would like to thank Mr. Anil
Rambharat and Mr. Varma Ratan for their insights into fault location within
Trinidad and Tobago. Lastly, the author would like to thank Dr. Letitia
Addison for their assistance in exploring changepoint detection.

## References

  * [1] Furse, C.M., Kafal, M., Razzaghi, R., Shin, Y.J.: Fault diagnosis for electrical systems and power networks: A review.  IEEE Sensors Journal 21(2), 888–906 (2021).  10.1109/JSEN.2020.2987321
  * [2] da Silva, F.: Analysis and simulation of electromagnetic transients in hvac cable transmission grids.  Ph.D. thesis, Institut for Energiteknik, Aalborg Universitet (2011)
  * [3] Ibitoye, O.T., Onibonoje, M.O., Dada, J.O.: Machine learning based techniques for fault detection in power distribution grid: A review.  In: 2022 3rd International Conference on Electrical Engineering and Informatics (ICon EEI), pp. 104–107 (2022).  10.1109/IConEEI55709.2022.9972279
  * [4] Moore, G.F.: Electric cables handbook.  Blackwell (1999)
  * [5] Short, T.A.: Electric power distribution handbook.  Crc Press, Taylor & Francis Group (2014)
  * [6] Thue, W.A.: Electrical Power Cable Engineering.  CRC Press (2017)
  * [7] NFPA: NFPA 70: National electrical code 2023.  National Fire Protection Association (2023)
  * [8] IEEE: Ieee guide on shielding practice for low voltage cables.  IEEE Std 1143-2012 (Revision of IEEE Std 1143-1994) pp. 1–91 (2013).  10.1109/IEEESTD.2013.6471984
  * [9] B1.30, C.W.: Cable systems electrical characteristics (2013)
  * [10] 47th Minnesota Power Systems Conference (MIPSYCON): Underground Power Cable Considerations: Alternatives to Overhead (2011).  URL https://pdi2.org/wp-content/uploads/2021/03/149-undergroudpowercableconsiderationsalternativestooverhead.pdf
  * [11] Trinidad, of Standards, T.B.: Trinidad and tobago standard: Trinidad and tobago electrical wiring code ; part 1, low voltage installations (2015)
  * [12] Trinidad, of Standards, T.B.: Tts 171:part2:2002, trinidad and tobago electrical wiring code – part 2 – high voltage installations (2002)
  * [13] Li, X., Zhang, Y., Li, Y., Liu, Y., Wang, H.: Analysis of cable faults in low-voltage distribution networks.  IEEE Transactions on Power Delivery 33(3), 1085–1093 (2018).  10.1109/TPWRD.2017.2762403
  * [14] Wang, J., Zhang, B., Li, W., Wang, X.: Fault analysis of medium-voltage cables in industrial plants.  IET Electric Power Applications 13(10), 1617–1624 (2019).  10.1049/iet-epa.2018.5501
  * [15] Kacprzak, A., Zajac, P., Krysiak, R., Bogusz, M.: Investigation of cable faults in low-voltage underground distribution networks.  Przeglad Elektrotechniczny (Electrical Review) 92(7), 221–225 (2016)
  * [16] Gupta, P., Swarnkar, A.: Fault location methods for low-voltage cables: A review.  In: 2017 IEEE Innovative Smart Grid Technologies-Asia (ISGT-Asia), pp. 1–6 (2017)
  * [17] Alduaili, M., et al.: Aging effect on the performance of low-voltage cables.  In: 2017 International Conference on Electrical and Computing Technologies and Applications (ICECTA), pp. 1–5 (2017)
  * [18] Liu, G., et al.: Analysis of cable faults in low-voltage aerial bundled distribution networks.  In: 2015 International Conference on Condition Monitoring and Diagnosis (CMD), pp. 1097–1101 (2015)
  * [19] Alawaji, H., et al.: Impact of manufacturing defects on the performance of medium voltage cables.  In: 2019 IEEE International Conference on High Voltage Engineering and Application (ICHVE) (2019)
  * [20] Mokari, A., Ebrahimi, M.: Cable fault location in medium voltage networks using artificial neural networks.  In: 2018 6th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), pp. 1–6 (2018)
  * [21] Kulkarni, S., Allen, A.J., Chopra, S., Santoso, S., Short, T.A.: Waveform characteristics of underground cable failures.  In: IEEE PES General Meeting, pp. 1–8 (2010).  10.1109/PES.2010.5590158
  * [22] K.K. Kuan, K.W.: Real-time expert system for fault location on voltage underground distribution cables.  IEE Proceedings C (Generation, Transmission and Distribution) 139, 235–240(5) (1992).  URL https://digital-library.theiet.org/content/journals/10.1049/ip-c.1992.0036
  * [23] Gilany, M., Ibrahim, D.k., Tag Eldin, E.S.: Traveling-wave-based fault-location scheme for multiend-aged underground cable system.  IEEE Transactions on Power Delivery 22(1), 82–89 (2007).  10.1109/TPWRD.2006.881439
  * [24] Dashti, R., Daisy, M., Mirshekali, H., Shaker, H.R., Hosseini Aliabadi, M.: A survey of fault prediction and location methods in electrical energy distribution networks.  Measurement 184, 109,947 (2021).  https://doi.org/10.1016/j.measurement.2021.109947.  URL https://www.sciencedirect.com/science/article/pii/S0263224121008824
  * [25] Rynjah, B., Lyngdoh, F., Sun, M.M., Goswami, B.: Automatic cable fault distance locator using arduino.  ADBU Journal of Electrical and Electronics Engineering (AJEEE) 3, 26–30 (2019)
  * [26] Gajbhiye, S., Karmore, S.P.: Cable fault monitoring and indication: A review (2013)
  * [27] Nag, A., Yadav, A., Abdelaziz, A.Y., Pazoki, M.: Fault location in underground cable system using optimization technique.  In: 2020 First International Conference on Power, Control and Computing Technologies (ICPC2T), pp. 261–266 (2020).  10.1109/ICPC2T48082.2020.9071462
  * [28] Deltenre, R.W., Schwarz, J.J., Wagnon, H.J.: Underground cable fault location: a handbook to td-153. final report.  OSTI (1977).  10.2172/7233049.  URL https://www.osti.gov/biblio/7233049
  * [29] Bao Lian, Salama, M.: An overview of digital fault location algorithms for power transmission lines using transient waveforms.  Electric Power Systems Research 29(1), 17–25 (1994).  https://doi.org/10.1016/0378-7796(94)90044-2.  URL https://www.sciencedirect.com/science/article/pii/0378779694900442
  * [30] Borghetti, A., Bosetti, M., Nucci, C.A., Paolone, M., Abur, A.: Integrated use of time-frequency wavelet decompositions for fault location in distribution networks: Theory and experimental validation.  IEEE Transactions on Power Delivery 25(4), 3139–3146 (2010).  10.1109/TPWRD.2010.2046655
  * [31] Chen, P., Wang, K.: Fault location technology for high-voltage overhead lines combined with underground power cables based on travelling wave principle.  In: 2011 International Conference on Advanced Power System Automation and Protection, vol. 1, pp. 748–751 (2011).  10.1109/APAP.2011.6180498
  * [32] Elhaffar, A.M.: Power transmission line fault location based on current traveling waves (2008).  URL http://lib.tkk.fi/Diss/2008/isbn9789512292455
  * [33] Ferreira, K.J.: Fault location for power transmission systems using magnetic field sensing coils.  Core.ac.uk (2019).  ”,.  URL https://core.ac.uk/works/61701326
  * [34] Fedorov, A., Petrov, V., Afanasieva, O., Zlobina, I.: Limitations of traveling wave fault location.  In: 2020 Ural Smart Energy Conference (USEC), pp. 21–25 (2020).  10.1109/USEC50097.2020.9281153
  * [35] Bawart, M., Marzinotto, M., Mazzanti, G.: A deeper insight into fault location on long submarine power cables.  e&i Elektrotechnik und Informationstechnik 131(8), 355–360 (2014).  10.1007/s00502-014-0233-x.  URL https://doi.org/10.1007/s00502-014-0233-x
  * [36] Chung, Y.C.: A critical comparison of reflectometry methods for location of wiring faults.  Smart Structures and Systems 2(1), 25–46 (2006)
  * [37] Jones, S.B., Wraith, J.M., Or, D.: Time domain reflectometry measurement principles and applications.  Hydrological Processes 16(1), 141–153 (2002).  10.1002/hyp.513.  URL https://doi.org/10.1002/hyp.513
  * [38] Shi, Q., Troeltzsch, U., Kanoun, O.: Detection and localization of cable faults by time and frequency domain measurements.  In: 2010 7th International Multi- Conference on Systems, Signals and Devices. IEEE (2010).  10.1109/ssd.2010.5585506.  URL https://doi.org/10.1109/ssd.2010.5585506
  * [39] Xudong, S., Jianzhong, Z., Tao, J., Liwen, W.: Design of aircraft cable intelligent fault diagnosis and location system based on time domain reflection.  In: 2010 8th World Congress on Intelligent Control and Automation. IEEE (2010).  10.1109/wcica.2010.5554557.  URL https://doi.org/10.1109/wcica.2010.5554557
  * [40] Shi, Q., Kanoun, O.: A new algorithm for wire fault location using time-domain reflectometry.  IEEE Sensors Journal 14(4), 1171–1178 (2014).  10.1109/jsen.2013.2294193.  URL https://doi.org/10.1109/jsen.2013.2294193
  * [41] Ziwei, M., Xueye, W.: A portable railway signal cable fault detector.  In: 2020 5th International Conference on Information Science, Computer Technology and Transportation (ISCTT). IEEE (2020).  10.1109/isctt51595.2020.00021.  URL https://doi.org/10.1109/isctt51595.2020.00021
  * [42] Hong, J.C., Sun, K.H., Kim, Y.Y.: Waveguide damage detection by the matching pursuit approach employing the dispersion-based chirp functions.  IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control 53(3), 592–605 (2006).  10.1109/tuffc.2006.1610568.  URL https://doi.org/10.1109/tuffc.2006.1610568
  * [43] Schuet, S., Timucin, D., Wheeler, K.: A model-based probabilistic inversion framework for characterizing wire fault detection using TDR.  IEEE Transactions on Instrumentation and Measurement 60(5), 1654–1663 (2011).  10.1109/tim.2011.2105030.  URL https://doi.org/10.1109/tim.2011.2105030
  * [44] Kowalski, M.E.: A simple and efficient computational approach to chafed cable time-domain reflectometry signature prediction (2009).  URL https://ntrs.nasa.gov/citations/20090035829
  * [45] Gabor, D.: Theory of communication.  Journal of the Institution of Electrical Engineers - Part I: General 94(73), 58–58 (1947).  10.1049/ji-1.1947.0015.  URL https://doi.org/10.1049/ji-1.1947.0015
  * [46] Hong, J.C., Sun, K.H., Kim, Y.Y.: The matching pursuit approach based on the modulated gaussian pulse for efficient guided-wave damage inspection.  Smart Materials and Structures 14(4), 548–560 (2005).  10.1088/0964-1726/14/4/013.  URL https://doi.org/10.1088/0964-1726/14/4/013
  * [47] Han, J.J., Park, S.R., Jeon, J.C., Kim, T.H., Yoo, J.G.: Enhanced locating method for cable fault using wiener filter.  Universal Journal of Electrical and Electronic Engineering 3(4), 107–111 (2015).  10.13189/ujeee.2015.030401.  URL https://doi.org/10.13189/ujeee.2015.030401
  * [48] Lee, C.K., Kwak, K.S., Yoon, T.S., Park, J.B.: Cable fault localization using instantaneous frequency estimation in gaussian-enveloped linear chirp reflectometry.  IEEE Transactions on Instrumentation and Measurement 62(1), 129–139 (2013).  10.1109/tim.2012.2212514.  URL https://doi.org/10.1109/tim.2012.2212514
  * [49] Lee, C.K., Kwak, K.S., Yoon, T.S., Park, J.B.: Application of instantaneous frequency estimation of sweep signal for localizing faults in a power cable.  In: Proceedings of the International Conference of the Society for Control and Robot Systems, pp. 1915–1918 (2011)
  * [50] Shin, Y.J., Powers, E., Choe, T.S., Hong, C.Y., Song, E.S., Yook, J.G., Park, J.: Application of time-frequency domain reflectometry for detection and localization of a fault on a coaxial cable.  IEEE Transactions on Instrumentation and Measurement 54(6), 2493–2500 (2005).  10.1109/tim.2005.858115.  URL https://doi.org/10.1109/tim.2005.858115
  * [51] Smith, P., Furse, C., Gunther, J.: Analysis of spread spectrum time domain reflectometry for wire fault location.  IEEE Sensors Journal 5(6), 1469–1478 (2005).  10.1109/jsen.2005.858964.  URL https://doi.org/10.1109/jsen.2005.858964
  * [52] Roy, S., Hanif, A., Khan, F.: Aging detection and state of health estimation of live power semiconductor devices using SSTDR embedded PWM sequence.  IEEE Transactions on Power Electronics 36(5), 4991–5005 (2021).  10.1109/tpel.2020.3032996.  URL https://doi.org/10.1109/tpel.2020.3032996
  * [53] Google: Monitoring arrangement (2017).  URL https://patents.google.com/patent/US10267841B2
  * [54] Truong, C., Oudre, L., Vayatis, N.: Selective review of offline change point detection methods.  Signal Processing 167, 107,299 (2020).  10.1016/j.sigpro.2019.107299.  URL https://doi.org/10.1016/j.sigpro.2019.107299
  * [55] Tao, J., Shengbiao, Z., Xudong, S., Liwen, W.: Design of aircraft cable fault diagnose and location system based on aircraft airworthiness requirement.  Procedia Engineering 17, 455–464 (2011).  10.1016/j.proeng.2011.10.055.  URL https://doi.org/10.1016/j.proeng.2011.10.055