# Uncertainties in ROC (Receiver Operating Characteristic) Curves Derived from
Counting Data

M.P. Fewell ARC Centre of Excellence for Dark-Matter Particle Physics and
Department of Physics, The University of Adelaide, Adelaide SA 5005, Australia

(June 2024)

###### Abstract

The ROC (receiver operating characteristic) curve is a widely used device for
assessing decision-making systems. It seems surprising, in view of its history
dating back to World War Two, that the assignment of uncertainties to a ROC
curve is apparently not settled. This note returns to the question, focusing
on the application of ROC curves to the analysis of data from counting
experiments and taking a practical operational approach to the concept of
uncertainty.

##  1 Introduction

The ROC (receiver operating characteristic) curve is a tool for assessing and
tuning data-analysis procedures, in particle physics and much more widely; for
it is a fundamental construct of signal-detection theory, applying to
decision-making under uncertainty, whenever one seeks to balance the false-
alarm rate against the rate of missed detections. As commonly plotted, a ROC
curve shows the probability of a detection—the complement of the probability
of a missed detection—as a function of the probability of a false alarm (where
a detection is declared although the looked-for object or occurrence is in
fact absent).

ROC curves are constructed by comparing two classes of events: those in which
a desired signal is known to be present, for example by construction in a
simulation of the process producing the events, and those where it is known to
be absent. For each class, one defines the _efficiency_ as the fraction of
events in which a detection is declared. A good detection process has high
efficiency for events containing signals—a high detection probability—and
simultaneously low efficiency for non-signal events—a low false-alarm (or
false-positive) rate. The two efficiencies are the coordinates of a point on
the ROC curve. The full curve is the locus of such points as some process
parameter that affects the efficiencies is varied.

This note deals with ROC curves constructed from counting data, taking an
example from experimental elementary-particle physics. The process parameter
affecting the efficiencies is a threshold, with the same threshold being
applied to signal and non-signal data. Sections 2.1 and 3.1 develop an
example.

The assignment of uncertainties to ROC curves is important, for often one
wishes to compare ROC curves. It implies assigning uncertainties to the
efficiencies that are the coordinates of points on a ROC curve. This seems to
be an open question (e.g. [1]), perhaps surprisingly. The present note argues
that a previous treatments of the question in the context of counting data,
such as those by Ullrich and Xu [2]—supposed in [1] to be correct—or by
Casadei [3], deal with a different type of efficiency from that used in ROC-
curve construction, a difference which appears to impact the statistical
behaviour.

The calculation of the type of efficiency needed for ROC-curve construction
from counting data is described in §2.1. Section 2.2 computes the uncertainty
in such an efficiency using an operationally inspired approach that aims to
mirror the calculational method closely, having regard to the practical
application of the resulting uncertainties. The approach makes no assumption
about the statistical behaviour of efficiency—such as, for example, that it is
binomially distributed—nor does it require that a Bayesian prior be
postulated. The approach does depend on the statistical independence of two
particular quantities; this is argued in §2.3, which begins with a
conceptualisation of the nature, source and uses of uncertainty in the present
context.

Implications for interpreting the resulting uncertainties are set out in §3.
Section 4 considers specific criticisms by Ullrich and Xu [2], arguing that
they do not apply to “efficiency“ calculated as described in §2.1, and
concluding in §4.4 with the important observation that the points near the
ends of ROC curves are of negligible interest when comparing ROC curves.

##  2 Efficiency as Used in ROC Curves

###  2.1 Efficiency Calculation

Figure 1 shows a histogram of isolation-cone energy111Several definitions of
isolation-cone energy are used in ATLAS. The differences are unimportant for
present purposes: all definitions have the same underlying statistical
properties. It is also unimportant statistically that the cut is often applied
to the ratio of isolation-cone energy to transverse photon momentum, rather
than to isolation-cone alone. The histogram in Fig. 1 shows results of Monte-
Carlo simulation of the detection process, rather than data from actual
proton–proton collisions. This can have statistical implications, but is
unimportant for the illustrative purpose to which it is applied here. for
cones of a particular size surrounding photon candidates in the ATLAS detector
at CERN. To achieve isolation, one stipulates that the energy in the cone be
less than a certain value, called the _threshold_ or _cut_ ; an example is
marked in Fig. 1. Events falling below the cut are declared to pass the
isolation test. The efficiency $\varepsilon$ of the test is simply

$\varepsilon=\frac{P}{N},$ (1)

where $P$ is the number of events passing the test and $N$ is the total number
of events in the histogram.

Obviously $\varepsilon$ varies with the location of the cut, behaviour that is
foundational to the ROC-curve concept. This is developed in §3, but first we
consider the uncertainty in $\varepsilon$.

Figure 1: Histogram of energy in an isolation cone around a photon candidate
in the ATLAS detector. Values less than zero reflect the effect of detector
energy resolution on the computation isolation energy. The blue line shows an
example of a cut used to compute efficiency, here at 3.0 GeV. In this
application, events with energy less than the cut are declared to be isolated;
that is, such events pass the isolation test.

###  2.2 Uncertainty in Efficiency

For uncertainty determination, eq. (1) has the difficulty that $P$ and $N$ are
correlated. So, let us rewrite $\varepsilon$ as

$\varepsilon=\frac{P}{P+F}\,,$ (2)

where $F$ is the number of events failing the test. Section 2.3 argues that
$P$ and $F$ are statistically independent. Given that, the standard rule for
propagating uncertainties in quadrature can be applied: the uncertainty
$\Delta\varepsilon$ in the efficiency is given by

$(\Delta\varepsilon)^{2}=\left(\frac{\partial\varepsilon}{\partial
P}\right)^{2}(\Delta P)^{2}+\left(\frac{\partial\varepsilon}{\partial
F}\right)^{2}(\Delta F)^{2}.$ (3)

Carrying out the differentiations leads to

$(\Delta\varepsilon)^{2}=\frac{(1-\varepsilon^{2})(\Delta
P)^{2}+\varepsilon^{2}\,(\Delta F)^{2}}{N^{2}}\,.$ (4)

If the events are equally weighted,222This need not be the case for events
generated by Monte-Carlo simulation [4, §7.2.2]. Poisson-distributed and
sufficiently numerous for the Gaussian approximation to apply, then

$(\Delta P)^{2}=P,\qquad(\Delta F)^{2}=F,$ (5)

which results in

$(\Delta\varepsilon)^{2}=\frac{\varepsilon(1-\varepsilon)}{N}.$ (6)

Equation (6) implies that $\Delta\varepsilon$ goes to zero as
$\varepsilon\rightarrow 0$ and as $\varepsilon\rightarrow 1$, behaviour that
has been criticised as “absurd” and that “violates our reasonable expectation”
[2, §2.2]. It is argued in §4 that this is not so: the behaviour is reasonable
and ought to be expected for the type of efficiency under discussion here.

###  2.3 Statistical Independence of $P$ and $F$

From an operational perspective, the purpose of uncertainty is to indicate the
extent to which the result may change if the whole analysis is repeated: for a
histogram built from experimental data, if the experiment is repeated; for one
built from simulation, if the Monte-Carlo simulation is repeated from scratch.
An uncertainty should reflect the fluctuation in results among many such
repetitions. Statistical correlation of two quantities then refers to their
relationship from repetition to repetition: whether they tend to fluctuate
together, either in synchrony or contrary-wise (positive or negative
correlation respectively), or not (the case of independence).

For the numbers of events passing and failing the test—$P$ and $F$
respectively (§2.2)—the argument for their statistical independence proceeds
in two steps: (i) showing that bins in a histogram like that in Fig. 1 are
statistically independent of each other and (ii) establishing conditions for
the statistical independence of different sums over histogram bins.

  1. (i)

Statistical independence of individual histogram bins. Choose a bin in the
histogram—say at 2.0 GeV in Fig. 1—and, from many repetitions of the analysis
that produces this histogram, choose those in which the contents of this bin
exceeds the mean value over all repetitions. One asks: does the value in
another bin—at 5.0 GeV, say—also exceed the mean for that bin (or fall below
that mean) in the chosen histograms more often than not? If so, there is
correlation between the two bins; if not, not; i.e. they are statistically
independent. Now each count in any bin in Fig. 1 comes from a separate event:
from a collision between two protons in the case of data, or from a simulation
of such a collision in the other case. There is no memory from collision to
collision; there is nothing to cause a correlation. This applies between any
two bins in the histogram, establishing their statistical independence.

Ullrich and Xu [2] seem implicitly to assume the statistical independence of
individual histogram bins in the second-last sentence of their §1, so the
argument of the previous paragraph would appear to be uncontroversial.

  2. (ii)

Condition for statistical independence of different sums over histogram bins.
Since any two bins of a histogram are statistically independent, so will be
two sums over histogram bins if the ranges of summation do not overlap. This
must be so, for such a summation is no more than re-binning the histogram, and
individual bins in the re-binned histogram are just as statistically
independent of each other as are the bins of the original histogram.

If there is overlap, then bins in the region of overlap contribute to both
sums, thereby providing a mechanism for correlation. From the perspective of
re-binning, one never includes an original bin in more than one new bin; for
to do so would be double-counting.

By construction, the sums $P$ and $F$ in §2.2 have summation ranges that do
not overlap. Hence they are statistically independent. On the other hand, the
contrary applies to the quantities $P$ and $N$ in eq. (1): the summation range
for $P$ is contained within that for $N$.

##  3 ROC Curves

###  3.1 Construction and Uncertainties

Figure 2 shows an example of histograms from which a ROC curve can be
constructed. The construction consists in choosing locations of the cut—shown
in Fig. 2 at 3 GeV—over the full range of values and computing the fraction of
the histogrammed events passing the test at each cut location. That is, the
cut location parameterises the ROC curve. The result is Fig. 3, in which the
points show the cut locations actually chosen, some of which are labelled, and
the curve is a smooth interpolation.

Figure 2: Like Fig. 1, but including ”background” events known not to contain
a signal photon, but which for various reasons have been misclassified as
though they do. (The signal events are the same as in Fig. 1). Figure 3: ROC
curve constructed from Fig. 2 by selecting a range of locations for the cut
(shown at 3.0 GeV in Fig. 2) and computing the fractions of events, either
signal or background, passing the test at each position. The markers show the
cut positions chosen; the curve is a smooth interpolation. The error bars are
100 times the uncertainties given by eq. (6), with the $100\times$ scaling
applied to make the error bars visible.

The error bars in Fig. 3 are 100 times the uncertainties calculated according
to eq. (6). The scaling factor of 100 was needed to make the error bars
appreciably larger than the markers.

Concerning statistical independence: the uncertainties in the x and y
directions on any given marker are independent, but there is correlation from
marker to marker. That is, the x-direction uncertainties displayed in Fig. 3
are partially correlated with each other, as are the y-direction
uncertainties. This occurs because the coordinates of the markers in a given
direction are obtained through different cuts on the same histogram.

###  3.2 Metrics and their Uncertainties

The _area under the curve_ is a commonly used metric for comparing ROC curves
of different processes. It approaches unity as the system approaches perfect
performance, which is signified by the ROC curve passing through the point (0,
1): zero background efficiency simultaneous with unit signal efficiency. It is
a satisfactory metric, but the calculation of its uncertainty must take into
account the correlation between uncertainties described in the last paragraph
of §3.1. This awkwardness can be avoided by use of a different metric: minimum
distance $\mathscr{D}_{min}$ from perfection. Figure 4 illustrates. The value
of $\mathscr{D}$ for any point
$(\varepsilon_{\mathrm{bgd}},\,\varepsilon_{\mathrm{sig}})$ on the ROC curve
is simply

Figure 4: Like Fig. 3, but illustrating the “distance $\mathscr{D}$ from
perfection” of a point on the ROC curve (arrow). The trailing diagonal is also
shown.

$\mathscr{D}=\sqrt{(\varepsilon_{\mathrm{bgd}})^{2}+(1-\varepsilon_{\mathrm{sig}})^{2}}\,.$
(7)

In principle $\mathscr{D}$ could be as large as $\sqrt{2}$, but the ROC curves
of practical processes rarely fall below the trailing diagonal, so in practice
$\mathscr{D}\leq 1$ and $\mathscr{D}_{\mathrm{min}}\leq\sqrt{2}/2$. Figure 5
shows the variation of $\mathscr{D}$ along the curve in Figs 3 and 4. The
minimum value is the metric of interest.

Figure 5: Variation of distance $\mathscr{D}$ from perfection along the ROC
curve of Fig. 4. The minimum value of this quantity is equivalent to “area
under the curve” as a metric if quality of the detection process that produced
the ROC curve.

Because $\mathscr{D}$ involves just one point on the curve, deriving an
expression for its uncertainty is straightforward. The efficiencies
$\varepsilon_{\mathrm{bgd}}$ and $\varepsilon_{\mathrm{sig}}$ are
statistically independent, so the equivalent of eq. (3) applies:

$(\Delta\mathscr{D})^{2}=\left(\frac{\partial\mathscr{D}}{\partial\varepsilon_{\mathrm{bgd}}}\right)^{2}(\Delta\varepsilon_{\mathrm{bgd}})^{2}+\left(\frac{\partial\mathscr{D}}{\partial\varepsilon_{\mathrm{sig}}}\right)^{2}(\Delta\varepsilon_{\mathrm{sig}})^{2},$
(8)

which, when eq. (6) applies, evaluates to

$(\Delta\mathscr{D})^{2}=\frac{\varepsilon_{\mathrm{bgd}}^{3}(1-\varepsilon_{\mathrm{bgd}})}{\mathscr{D}^{2}N_{\mathrm{bgd}}}+\frac{(1-\varepsilon_{\mathrm{sig}})^{3}\varepsilon_{\mathrm{sig}}}{\mathscr{D}^{2}N_{\mathrm{sig}}}\,.$
(9)

##  4 Discussion — Response to the Criticisms of Ullrich and Xu

Ullrich and Xu [2] derive eq. (6) through use of the binomial distribution. As
noted at the end of §2.2, they state that it is “absurd” to suppose the
uncertainty to be zero when $\varepsilon$ is zero or unity, that the result is
“defective” [2, Abstract], that it “violates our reasonable expectation” [2,
§2.2]. They also claim that the total number $N\;(=P+F)$ of events “is a fixed
quantity and not subject to any fluctuation” [2, §2.1]. The present section
responds to these claims in turn, then proposes a resolution.

###  4.1 Uncertainty at Zero and Unit Efficiency

The discussion in §3.1 indicates that, to obtain either zero or unit
efficiency $\varepsilon$, the cut must be positioned at infinity, either
positive or negative.333in principle. Real histograms, being discrete, do not
extend to infinity in practice, but the underlying distributions do, in
principle. Consider repetitions of the analysis such as described in §2.3 with
the cut at $+\infty$. Every repetition gives $\varepsilon=1$ without
exception; the uncertainty is indeed zero. This is appropriate in view of the
meaning ascribed to “uncertainty” in the first paragraph of §2.3.

A similar argument applies with the cut at $-\infty$: $\varepsilon=0$ for
every repetition, so the uncertainty is again zero.

One concludes not only that eq. (6) is correct when $\varepsilon=0$ or 1, but
also that any analysis giving a nonzero uncertainty for either of these
$\varepsilon$ values must be incorrect.

###  4.2 Statistical Nature of Total Number $N$ of Events

Ullrich and Xu state that $N$ “is a fixed quantity and not subject to any
fluctuation. It’s usually a well defined and known input quantity.” [2, §2.1]
Section 2.3 makes it clear that this cannot apply to the quantity $N$ as used
herein; for $N=P+F$ and not only do $P$ and $F$ fluctuate from repetition to
repetition, but also we have the result that they are statistically
independent, so their sum must fluctuate too.

Fixed $N$ is a feature of the binomial distribution. That is, for the binomial
distribution to apply in a given situation, not only must the result of an
event be binary, but the number of events in a trial must be constant and
specified. Repetition consists of repeated trials. The second condition does
not apply to counting data used for ROC-curve construction.

These considerations hint that the analytical situation underlying the
calculation of ROC curves is not the same as that considered by Ullrich and
Xu, at least as regards statistical behaviour. This suggestion is developed in
the next subsection.

###  4.3 Synthesis

Referring to the calculation of efficiency, Ullrich and Xu write [2, §1]

> …this procedure can be simplified to the comparison of two histograms. In
> histogram A, one plots the distribution of the quantity of interest for all
> the data of the sample; in histogram B one plots the distribution of the
> same quantity, but only for those satisfying the selection criteria, _i.e._
> , those data that pass the cuts. Intuition leads one to expect that the best
> estimate for the (unknown true) efficiency for each bin is just
> $k_{i}/n_{i}$ where $k_{i}$ is the number of entries in bin $i$ of histogram
> B and $n_{i}$ is the number of entries in bin $i$ of histogram A.

Clearly, the authors envisage the selection process—the definition of the
cuts— as being separate from the quantity being histogrammed: the selection is
determined by some other quantity or quantities. This is different from the
selection process involved in the construction of a ROC curve. The difference
seems too slight to be material, but it is a second indication of a divergence
between analytical situations.

The efficiency treated by Casadei [3] has the same feature, as he states
several times, for example in the first sentence of his §4: “… the events had
been selected by an independent process whose efficiency is completely
uncorrelated with respect to the efficiency of the process under study …”.

For a third example, Hollitt [5, Appendix A] derives the uncertainty in
“purity” independently of either Ullrich and Xu or Casadei and, unlike those
two studies, without recourse to Bayes’s rule, so without the need to
postulate a prior probability. This purity appears similar to the others’
efficiency. Hollitt obtains the same expression for uncertainty as Ullrich and
Xu—their eq. (19)—describing the result of eq. (6) above as “the Frequentist
prediction”, which is fair: the arguments in §2.3 are explicitly frequentist
in nature (see e.g. [4, §5.1.1]), as is appropriate for the analytical
situation herein and for the intended use of the resulting uncertainties.

Adoption of a frequentist approach is supported by a recent study of ROC-curve
uncertainties in the context of machine learning [6]; in their §9.1, the
authors write that the ”baseline frequentist approach … has correct
[statistical] coverage by construction.”

###  4.4 Aside — The Ends of a ROC Curve

Inaccuracies in the argument of §2.2 enter not through use of eq. (3) but
rather eq. (5), which is the Gaussian approximation to a Poisson distribution,
applying when $P$ and $F$ are large.

In the ROC curve shown in Fig. 3, which was constructed from the data in Fig.
2, the point closest to $(0,0)$ has $P=1199$ for the signal and $P=75$ for the
background. Both are large enough for the Gaussian approximation to be
sufficient; the lower limit is commonly cited to be about 10 (e.g. [6,
§9.2.3]. For the point closest to $(1,1)$, values of $F$ are many times
larger.

The question of how to deal with very small values of $P$ and $F$ is not
problematic for ROC-curve construction because the points $(0,0)$ and $(1,1)$
constitute opposite ends of every ROC curve, so they could, for example, be
added by hand. From the point of view of comparing ROC curves, these points
carry no information, precisely because they lie on all ROC curves.

##  5 Conclusion

Motivated by a desire to determine uncertainties on ROC (receiver operating
characteristic) curves constructed from counting data, the calculation of
uncertainty in efficiency is revisited; for the two coordinates of a point on
a ROC curve are the efficiencies for signal and non-signal data. The
derivation herein makes no assumption about the statistical nature of
efficiency, nor does it require that a Bayesian prior be postulated. Rather,
it relies on the conventional prescription for propagating uncertainties
through a computation, which is no more than an adaptation of the standard
expansion of a total derivative in terms of partial derivatives.

The expression obtained, eq. (6), has been derived before, likely many times,
but some previous treatments (e.g. [1, 2, 3, 5]) criticise the result
deprecatingly. This note argues to the contrary: that the result is correct
for the type of efficiency used in the construction of a ROC curve from
counting data. The authors cited use the term “efficiency” in other contexts;
none mentions ROC curves. It seems that their efficiencies have a different
statistical nature from those involved in ROC-curve construction. It is
suggested that in this difference lies the source of the contention.

The authors of [1, 2, 3] express concern over the accuracy of eq. (6) when the
number of events either passing or failing the test is small. These conditions
correspond to the two ends of a ROC curve. Uncertainties—even values—near
these locations are unimportant from the point of view of distinguishing one
ROC curve from another because all ROC curves have the same end-points.

## References

  * Heelan [2009] L. Heelan, “Calculating efficiency uncertainties”, ATLAS Collaboration internal presentation, Carleton ATLAS Group Meeting, 14 Aug 2009; available to members of the Collaboration at https:// indico.cern.ch/event/ 66256/contributions/2071577/attachments/1017176/1447814/ EfficiencyErrors.pdf
  * Ullrich & Xu [2007] T. Ullrich & Z. Xu, “Treatment of errors in efficiency calculations”, arXiv:physics/0701199v1. This version is time-stamped 17 Jan 2007 but the text referenced herein carries the date 24 Oct 2018. (A version 2, time-stamped 11 May 2012, was later withdrawn.)
  * Casadei [2012] D. Casadei, “Estimating the selection efficiency”, arXiv:0908.0130v8, 2012.
  * Buckley et al. [2021] A. Buckley, C. White & M. White, _Practical Collider Physics_ , Bristol UK, IOP Publishing, 2021.
  * Hollitt [2019] S.E. Hollitt, “B-Physics in lattice QCD and at Belle II”, PhD thesis, University of Adelaide, Oct 2019, Appendix A: “Error in purity calculations”; available at https://hdl.handle.net/2440/123637
  * Urlus et al. [2023] R.E.Q. Urlus, M.A. Baak, S. Collot & I. Fridman Rojas, “Pointwise sampling uncertainties on the precision–recall curve”, Proc. 26th Internat. Conf. on Artificial Intelligence and Statistics (AISTATS), Valencia, Spain, 2023.


# Exploring the Use of Generative AI in the Search for Extraterrestrial
Intelligence (SETI)

John Hoang *ref1  Zihe Zhengref2  Aiden Zelakiewiczref3  Peter Xiangyuan
Maref4  and Bryan Brzyckiref5  ref1SCIPP, California, USA; e-mail:
<EMAIL_ADDRESS>ref2Yale University, Connecticut, USA; e-mail:
<EMAIL_ADDRESS>ref3The Ohio State University, Ohio, USA; e-mail:
<EMAIL_ADDRESS>ref4Department of Mathematics, University of Toronto;
e-mail<EMAIL_ADDRESS>ref5Department of Astronomy, University of
California Berkeley; e-mail<EMAIL_ADDRESS>
###### Abstract

The search for extraterrestrial intelligence (SETI) is a field that has long
been within the domain of traditional signal processing techniques. However,
with the advent of powerful generative AI models, such as GPT-3, we are now
able to explore new ways of analyzing SETI data and potentially uncover
previously hidden signals. In this work, we present a novel approach for using
generative AI to analyze SETI data, with focus on data processing and machine
learning techniques. Our proposed method uses a combination of deep learning
and generative models to analyze radio telescope data, with the goal of
identifying potential signals from extraterrestrial civilizations. We also
discuss the challenges and limitations of using generative AI in SETI, as well
as potential future directions for this research. Our findings suggest that
generative AI has the potential to significantly improve the efficiency and
effectiveness of the search for extraterrestrial intelligence, and we
encourage further exploration of this approach in the SETI community.
(Disclosure: For the purpose of demonstration, the abstract and title were
generated by ChatGPT and slightly modified by the lead author.)

##  1 Introduction

The Breakthrough Listen project has been searching for technosignatures in our
universe using powerful radio telescopes around the world, including the Green
Bank Telescope, Parkes Telescope, Allen Telescope Array, and MeerKAT. The most
popular technique in radio SETI involves looking for narrow-band signals in
the time-frequency spectrogram data (often referred to as dynamic spectra or
waterfall plots in existing literature). In recent years, machine learning
(ML) algorithms have been employed to classify image-like spectrograms [1],
and setigen, an open-source Python library, was created to synthesize mock
labelled radio SETI training data set [2]. Since setigen is meant to be a
general-purpose heuristic framework, there are rooms for improvement if one
wishes to improve the speed of the algorithm in specialized cases. The method
used in this work is Generative Adversarial Network (GAN) [3].

A GAN consists of two competing deep neural networks: the generative network
and the discriminator network. The generator generates images from random
noise, and the generated images are fed into the discriminator along with real
images. The discriminator then classifies the images as either real or fake
and both the generator and the discriminator models are updated according to
the result. At equilibrium, the generator will generate images that look
similar to the training data and as a consequence, the discriminator will no
longer be able to distinguish between generated and real images. The output of
the generator will determine whether an image is similar to the training data
or not, which can be useful in determining whether the image is normal or
anomalous. Therefore, GAN can also be used to detect outliers. In astronomy,
GANs have been used to simulate galaxy images [4], gamma-ray Cherenkov
airshower signals [5], detecting outliers [6], to name but a few. GANs and its
family of other generative frameworks such as AutoEncoder are parts of what is
colloquially known as DeepFake, a portmanteau of Deep learning and Fake.

##  2 Experiments

Figure 1: 4 narrow-band data sets used for training in this work with
increasing complexity. Top row: narrow-band signals with a constant drift
rate, narrow-band signals with 6 drift rates. Bottom row: ON-OFF cadence
observations with narrow-band ETI-like signals in only ON observation, ON-OFF
cadence observations with narrow-band ETI-liked signal in only ON observation
and random narrow-band RFI-like in OFF observation.

We employ the setigen software to generate waterfall plots used for training.
Each generated set includes 15000 waterfall plots, each having 128 pixels in
time and 128 pixels in frequency. Background noise is randomly chosen from
either Gaussian noise or Chi-squared noise, with mean equals to 10 and
standard deviation (only for Gaussian noise) equals to 1. Injected narrow-band
signals are specified by the starting point, drift rate and width in each data
set. The width of the signals all range from 50 to 60 pixels. Random walk
signals are injected in some of the data sets to simulate radio-frequency
interference (RFI), which is frequently seen in real observations. These 4
sets used in the experiments are shown in Fig.1.

###  2.1 Simple drifting signals

We first test GAN’s ability to generate waterfall plots containing narrow-band
signals with a single drift rate. As seen from Fig.2, GAN successfully
reproduces this data set after training.

Figure 2: Vanilla GAN is able to reproduce the training data from setigen
(left): right plot contains GAN-generated waterfall plots with random
background noise level

###  2.2 Simple drifting signals with varying drift rates

We then expand the variety of data by changing the drift rate and starting
points of signals in the waterfall plots, which result in the 6-type dataset.
We train the same vanilla GAN network architecture on the 6-type data set
shown in Fig.1 for 1000 epochs. However, the network fails to produce
realistic waterfall plots.

Conditional GAN [7] is then selected as our next candidate architecture among
several variations of GAN since it allows the network to generate images
conditioned on the class of data. Conditional GAN achieved this idea by adding
labels for data according to their drift rates, which are fed in the network
along with the plot themselves. We labeled the 6-type data set with 0, 1, 2,
3, 4, 5 according to 6 different drift rates, and trained the conditional GAN
on the data set for 1000 epochs. The generator from the conditional GAN is
able to generate waterfall plots with a singular signal with various starting
points in each class. The results in comparison with traditional GAN are shown
in Fig.3.

Figure 3: Vanilla GAN struggles to learn the distinct features among different
drift rates. However, conditional GAN solves the issue by artificially
condition training data on the slope label.

###  2.3 Cadence signals

Due to the prevalence of radio frequency interference, a typical radio SETI
observation consists of pointing the telescope alternatively to ON-OFF sky
locations, and a potential narrow-band ETI signal candidate should only show
up in the ON observations. To simulate this, three horizontal noisy band of
size 21*128, 21*128 and 23*128 pixels are added to each waterfall plots to
simulate the ON-OFF cadence in real observations. Successful results are shown
in Fig.4

Figure 4: Conditional GAN is able to learn the more complex ON-OFF signal
patterns mimicking typical SETI observations.

###  2.4 Cadence signals with RFI

We finally test our generator on its resistance to RFI in the dynamic spectra.
We trained our conditional GAN model on the Cadence-RFI dataset, which
contains injected RFI noise. The result is shown in Fig.5. The generator
ignores the presence of RFI and does not produce them in the OFF observations.
In addition, the generator also avoids a common failure mode called mode-
collapse whereby GAN only learns to output a particularly plausible output and
nothing else. Fig.5 shows a degree of variability within a single class:
although all the waterfall plots in the same column have the same drift rate
(label), they vary with intensity and location within the frame.

Figure 5: Conditional GAN is able to ignore RFI in the training data. For
clarity, in the right plot, waterfall plots in each column have the same label
(drift rate).

###  2.5 Generator’s scores

Figure 6: Distribution of scores for different data sets by the discriminator.

Because the discriminator learns to classify realistic waterfall plots and
unrealistic waterfall plots during training, the trained discriminator can be
used subsequently as a classifier. We train our conditional GAN on the cadence
data set in Fig.1. To test the trained discriminator, we used the 4 data sets
shown in Fig.6, each containing 15000 waterfall plots: only noise background,
only cadence background, GAN-produced data, and the cadence-RFI data. It is
clear that waterfall plots that are less similar to training data receive
lower scores from the discriminator than those are similar to the training
data set. The blue and green distribution have similar means, confirming that
both generator and discriminator disregard RFI in the training data.

###  2.6 Timing

To illustrate the advantage of using pre-train model, we show in Tab.1 the
time it took our GAN to generate and write to storage $10^{2}$, $10^{3}$,
$10^{4}$, $10^{5}$ waterfall plots. Comparing with the clearly linear time of
setigen, GAN’s speed is superior when a large data set is involved, and we are
only limited by the availability of GPU RAM and storage I/O.

Table 1: CPU/GPU time to generate (G) and write (W) to storage as a function of number of waterfall plots, using both heuristic setigen with CPU and generative GAN with GPU. GAN excels when a large number of waterfall plots are involved, improving the speed by more than 2 orders of magnitude. |  | SETIGEN | GAN
---|---|---|---
# | File Size | G (s) | G&W (s) | G (s) | G&W (s)
$10^{2}$ | 12.5 MB | 0.578 | 0.613 | 3.335 | 3.378
$10^{3}$ | 125.3 MB | 5.09 | 6.287 | 3.784 | 4.181
$10^{4}$ | 1.2 GB | 51.279 | 55.723 | 3.807 | 7.724
$10^{5}$ | 12.2 GB | 515.802 | 563.833 | 4.358 | 95.597

##  3 Possible extensions

It is possible to further modify GANs into a more advanced bi-directional
architecture so that we can reverse-engineer images into meaningful latent
vectors. The proposed Bidirectional Conditional GAN (BiCoGAN) [8] includes an
additional encoder network that is trained simultaneously with the generator
and the discriminator, and can provide inverse mappings of data samples to
both intrinsic and extrinsic vectors. The benefit of BiCoGAN is two-fold: the
reverse-engineered latent vectors of hitherto unseen signals can be used to
compare with those of known signals to identify anomalies, and anomalous
latent vectors can be slightly perturbed and subsequently fed into the
generator to synthesize new anomalous signals useful for further training.
This remains a work in progress, and an example of hand-written digits
generated by BiCoGAN is shown in Fig.7.

Figure 7: MNIST images synthesized by a BiCoGAN’s generator with random latent
vectors (top row), comparing with original training data (middle row) and with
reconstructed latent vectors from the encoder network (bottom row).

##  4 Limitations

For the purpose of demonstrating the limitations of Generative AI, let us
examine this paper’s title and abstract, which were largely produced by
ChatGPT - Chat Generative Pre-trained Transformer. While reads convincingly at
first, the generated abstract certainly contains errors. Notably, the
relationship between natural language processing and radio data analysis makes
little sense, if at all. Nonetheless, upon a handful of minor edits by an
expert, the hybrid version of the abstract sounds even more convincing (see
Fig.8). In a similar vein, radio spectra produced by a GAN are realistic
overall, but they require more careful benchmark when high precision is
required or when statistics are low.

Figure 8: Response by ChatGPT from a prompt by the author, and with subsequent
edits to either improve language usage or correct factual knowledge.

Fundamentally, by the virtue of the Universal Approximation Theorems, neural
networks are functional approximator (often polynomial approximation if simple
activation functions are used) of a multidimensional distribution to arbitrary
precision. In essence, it is analogous to the Weierstrass Approximation
Theorem, which is as follows: Suppose f(x) is a continuous real-valued
function defined on the real interval [a,b], then:

$\forall\epsilon>0,\exists\ p(x)\quad\mathrm{s.t.}\quad\forall
x\in\mathclose{[}a,b\mathclose{]}\,\ ,\,\ |f(x)<p(x)|<\epsilon$ (1)

In the case of ChatGPT or any Generative AI, the interval [a,b] is the domain
in which the training data f(x) is contained. Since the domain of the function
is a specific closed interval instead of the entire real number line, the
generator can approximate or interpolate very well the distribution within the
domain using a polynomial p(x). In other words, the output is a sufficiently
good polynomial approximation of the training data contained therein. In the
case of this work’s abstract, since ChatGPT most likely obtained training data
from publications, it outputs "paper" instead of the more naturally sounded
"work" in the context of an abstract for a conference presentation. However,
outside the domain, the extrapolated output is sometimes only tangential, and
requires further scrutiny. The ideal training data set should includes both
theory-based data as well as real-life examples, although this is hardly
achievable in many cases whereby new phenomenon are being investigated.

##  5 Conclusions

We have explored the potential of deep generative networks for generating
dynamic spectra and their ability to detect outliers. The main advantage of
such generative AI is that they can encode the complex representations of the
training data and rapidly decode them at a rapid speed by leveraging GPU
usage. The work is relevant not only to radio SETI, and can be readily extend
to SETI search in other parts of the electromagnetic spectrum. Furthermore,
since the AI models used in this work are still far from the industrial grade
models such as one used by ChatGPT, there is much for further experimentation
with more sophisticated AI architecture. However, as in the case with other
applications of Generative AI, we caution against the over-reliance on their
outputs, as sometimes they are only tangential to the reality at best, and
nonsensical at worst. The code used in this work is publicly available at
https://github.com/zzheng18/SETIGAN.

## Acknowledgements

We thank the Breakthrough Prize Foundation and the University of California,
Berkeley, for their support. Special thanks to Andrew Siemion, Steve Croft,
and Yuhong Chen for several fruitful discussions. This work is funded by the
Breakthrough Listen Initiative and the National Science Foundation.

## References

  * [1] P. X. Ma et al., “A deep-learning search for technosignatures from 820 nearby stars,” Nature Astronomy, pp. 1–11, Jan. 2023, doi: https://doi.org/10.1038/s41550-022-01872-z.
  * [2] B. Brzycki et al., “Setigen: Simulating Radio Technosignatures for SETI,” The Astronomical Journal, vol. 163, no. 5, p. 222, May 2022, doi: 10.3847/1538-3881/ac5e3d.
  * [3] I. J. Goodfellow et al., “Generative Adversarial Networks,” arXiv.org, 2014. https://arxiv.org/abs/1406.2661
  * [4] M. Dia, E. Savary, M. Melchior, and F. Courbin, “Galaxy Image Simulation Using Progressive GANs,” arXiv:1909.12160 [astro-ph, stat], Sep. 2019, Accessed: Jan. 25, 2023. [Online]. Available: https://arxiv.org/abs/1909.12160
  * [5] J. Dubenskaya, A. Kryukov, and A. Demichev, “PoS(ICRC2021)874 Fast Simulation of Gamma/Proton Event Images for the TAIGA-IACT Experiment using Generative Adversarial Networks PoS(ICRC2021), Accessed: Jan. 25, 2023. [Online]. Available: https://pos.sissa.it/395/874/pdf
  * [6] B. Margalef-Bentabol et al., “Detecting outliers in astronomical images with deep generative networks,” Monthly Notices of the Royal Astronomical Society, vol. 496, no. 2, pp. 2346–2361, Jun. 2020, doi: 10.1093/mnras/staa1647
  * [7] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” arXiv.org, 2014. https://arxiv.org/abs/1411.1784
  * [8] MA. Jaiswal, W. AbdAlmageed, Y. Wu, and P. Natarajan, “Bidirectional Conditional Generative Adversarial Networks,” arXiv:1711.07461 [cs, stat], Nov. 2018, Accessed: Jan. 25, 2023. [Online]. Available: https://arxiv.org/abs/1711.07461


# Some pros and cons of implementing parallel and block teachings for
mathematics modules

N. Karjanto1 and S. T. Yong2
1Department of Applied Mathematics 2Department of Foundation in Engineering
Faculty of Engineering, The University of Nottingham Malaysia Campus
Jalan Broga, Semenyih 43500, Selangor, Malaysia

###### Abstract

The Department of Applied Mathematics at the University of Nottingham Malaysia
Campus has a responsibility for delivering mathematics modules for engineering
students. Due to the significantly large number of students, two methods of
teaching delivery–parallel teaching and block teaching–have been implemented.
This article discusses some pros and cons between these two methods,
particularly for the Foundation programme and the first year of the
Undergraduate programme in Engineering. Whether parallel teaching or block
teaching is implemented, feedback comments from the students indicate that
some areas need to be paid attention to.

Keywords: parallel teaching; block teaching; Foundation programme;
Undergraduate programme; mathematics modules.

##  1 Introduction

This article reveals some pros and cons when implementing different methods of
teaching delivery in several undergraduate and foundation mathematics modules
at our university. Our campus houses four different faculties and our
department belongs to the Faculty of Engineering. Although the Department of
Applied Mathematics does not really offer any study programme in math, it
plays an important role as a service department not only for the Undergraduate
programme but also for the Foundation programme in Engineering. Currently,
there are four engineering departments housed in the Faculty of Engineering:
Electrical and Electronic Engineering, Chemical and Environmental Engineering,
Mechanical, Materials and Manufacturing Engineering and Civil Engineering.

Since our university is a branch campus of the University of Nottingham
system, we adopt the English higher education system in our curricula, where
the undergraduate level is divided into three years [1], in contrast to the
North American system where the students will normally spend four years in
their undergraduate study. This means that the students who enter Year 1 have
an entry requirement of A (advanced)-level or a similar qualification.
However, a significant number of our potential students posses only an O
(ordinary)-level qualification. In order to accommodate them, the faculty
arranges a special programme known as the Foundation in Engineering. This
programme lasts for three semesters which normally starts around April or July
and ends in May of the following year. The purpose of this programme is to
equip the students with merely an O-level qualification to enter the
Undergraduate programme in Engineering. In this case, at the end of the third
semester of their Foundation study, they would have a similar level to the
students with an A-level qualification. After completing the Foundation
programme and satisfying the progression rules, the students may proceed to
choose one study programme offered by the four engineering departments.

Our department also services a number of higher level undergraduate and
graduate mathematics modules, such as for Year 3 and Year 4 (MSc and MEng)
students. These modules are usually offered as elective modules and generally,
the number of students who take such optional modules is relatively small in
comparison to the compulsory math modules. Keeping this in mind, the math
modules discussed in this article will refer to the modules in both Foundation
and Year 1 of the Undergraduate programmes.

This article is organized as follows. The following section will briefly
mention the number of students we currently have and we expect to have in the
future. It will also explain how these large numbers of students are organized
into groups. Section 3 will discuss the difference between the two methods of
teaching delivery, the parallel teaching and block teaching. In Section 4, we
will reveal some pros and cons between parallel teaching and block teaching
methods, both from lecturers’ and students’ perspectives. Finally, Section 5
draws a conclusion and provides remark to our discussion, with a balanced view
regarding the two teaching methods.

##  2 Class organization

Due to a significantly large number of students and a limited number of
academic staff in our department, methods of teaching delivery are of a great
concern to both sides of the teaching activity, the students and the
lecturers. Depending on the academic year, the number of students may vary
from one cohort to the other. The number of students currently registered at
the Foundation programme is around 250. However, we may anticipate this number
to increase up to 300 students or more in the coming academic years. In Year 1
of the Undergraduate programme, the total number of students has grown to
around 360, due to additional students registered straight away to Year 1
instead of passing through the Foundation stage. These newcomers normally have
done their A-level qualification somewhere else. Similarly, we might also want
to anticipate an increase in this number to around 400 students or more in the
future. It should be taken into account that those who are taking mathematics
modules in Year 1 may also include the students from another faculty, for
instance from the Business School, whereby the students take these as their
elective modules.

Regarding the class arrangement, the Year 1 students of the Undergraduate
programme have been clustered into their own respective departments. For the
students in the Foundation programme, however, there is not really a clear
mechanism regarding the arrangement of the class. In this case, we adopt the
grouping system that has been arranged by our colleagues at the School of
English Language Education, Faculty of Arts and Education. Whatever grouping
system we adopt, the same principle applies, i.e. the Foundation students are
divided into three different groups with the same number of students. Thus,
each group contains around 80 students. However, we are not able to maintain a
similar proportion anymore for Year 1 students. This is caused by the fact
that each department decides the number of students it is willing to admit and
some trends that develop among the Foundation students regarding which
department they would enrol. Thus, the four groups of Year 1 students could
range from 50 to 140 students.

Until the recent academic year, it is compulsory for the students enrolled in
the Foundation programme to take three mathematics modules. In Semester 1, the
module is HG1BMT Basic Mathematical Techniques (10 credit points). At this
stage, the students are enrolled at two different intakes (April and July) and
the number of students apparently is not so large. In Semester 2 and 3, the
module is HG1FND Foundation Mathematics (20 credit points), which is delivered
as a year-long module, where each semester carries 10 credit points. In
Semester 3, another math module is added, i.e. HG1M02 Applied Algebra (10
credit points). All Year 1 Engineering students must take HG1M11 Engineering
Mathematics 1 and HG1M12 Engineering Mathematics 2 in the Autumn and Spring
semesters, respectively. Both modules have the weight of 10 credit points
each. It is essential to note that although the Year 1 Engineering students
have chosen their own respective departments, the contents of the material in
both modules mentioned above are identical.

In the following section, we discuss two methods of teaching delivery in
connection to the mathematics modules mentioned above.

##  3 Teaching delivery

According to [2], parallel and block teaching (also referred to as ‘station
teaching’) are two of the five types of co-teaching delivery for secondary
school level. The other three are ‘lead and support’, ‘alternative teaching’
and ‘team teaching’.

In the context of this article, ‘parallel teaching’ refers to the teaching
delivery given by one lecturer handling one group of students only for the
whole semester, or even the whole year. Although the lecturers jointly plan
for the teaching material so that they will deliver the contents in a similar
way, it is inevitable that some variations will occur from one group to the
other since every lecturer has his/her own teaching style, method and
approach.

On the other hand, the ‘block teaching’ refers to the teaching delivery given
by two or more different lecturers to each group of students. This means that
within one semester, each lecturer will handle only certain topics that cover
a couple weeks of material for the whole cohort of students. So in one
semester the students will enjoy being taught by several lecturers. Some
examples of block teaching plans are given in Appendix A. In our example,
three lecturers are involved in one particular module. Block teaching by two
lecturers is common too, but we have never implemented four lecturers or more
in the block teaching.

For each module that employs the block teaching plan, there is one lecturer
assigned as a module convener. He/she will organize the other members of the
group on who will teach which weeks. In our example, we run an 11-week
semester. Since the final week is normally used for a test, then there will be
only 10 weeks of teaching activities. In this case, one lecturer, normally the
module convener, will handle four weeks of lectures, while the other two will
handle three weeks of class. In addition, the module convener also distributes
the workload to the other members of the group. For instance, it has to be
determined who will contribute which problems for quizzes, tests or the final
exam.

For every 10 credit points module, there are a two-hour lecturing session and
a one-hour tutorial session. During the two-hour lecturing session, the
lecturer covers the specified topics of material. Depending on the
availability of the time, during this session, the lecturer might also employ
students’ participation, for example by encouraging them to solve some
relevant problems. During the one-hour tutorial session, which is usually
conducted on a different day than the lecture, the lecturer discusses the
problem sheets given earlier to the students. During this session, the
students have ample opportunity to practice their problem-solving skills. The
lecturer gives a measure of guidance to the students when they attempt to
solve more challenging problems and even might demonstrate a step by step
explanation in solving a certain mathematical problem. It is good to note that
the tutorial session is not necessarily conducted by the same lecturer as the
one who conducts the lecturing session.

##  4 Pros and cons of parallel and block teaching

We have implemented both parallel and block teaching in delivering several
mathematics modules during the past couple of years. In order to find out the
general opinions, including some pros and cons between these two teaching
methods, we have conducted a number of interview sessions with several of our
lecturer colleagues and a number of students from the Foundation programme in
Engineering.

After implementing parallel teaching delivery, we received a number of reports
from the students regarding the different contents of material they received
from one group to another. It is observed that the students tend to compare
the teaching quality of their lecturers and the worked examples that they
receive with their own friends from another class. When the students find out
that they receive less worked examples or different ones, they might feel that
they have been treated sufficiently unfairly that they have grounds for
complaint. With this in mind, in the current academic year, we implemented
block teaching for all mathematics modules for both Foundation and Year 1
levels.

The following are some pros and cons regarding the two methods of teaching
delivery. The parallel teaching delivery allows the lecturer to get to know
the students better during the semester period. Although we always deal with a
large number of students, knowing the ability of each student is particularly
useful in smaller class size when guiding and helping them through their
difficulties. From the lecturer’s viewpoint, particularly the newly appointed
staff who lack some teaching experience, parallel teaching is also useful as
an opportunity to learn the module materials for the complete semesters,
instead of merely a few topics only. Nevertheless, despite the efforts around
jointly planning the content delivery in a similar way, some variations are
simply inevitable, particularly when the lecturers have a wide range of
teaching experience.

On the other hand, the block teaching allows identical deliveries to all
students regardless of the group in which they are sitting. By implementing
block teaching, all groups of students will receive the same lecture notes,
worked examples, example sheets and exercises and also will experience the
same teaching style from a particular lecturer. The block teaching delivery
exposes the students to many different lecturers in our department and allows
them to taste the diversity of different teaching styles.

However, the block teaching delivery is not without shortcomings. We again
receive some reports from the students mentioning that they find a difficulty
in adjusting to different styles of teaching delivery from one lecturer to the
another. In particular, the Foundation students suffer the most since the one-
hour tutorial for discussing example sheets and exercises is usually being
handled by a different lecturer.

It is common that different lecturers use different notation to express
certain concepts or definitions. This is not surprising since when we check
several mathematics textbooks, we will discover that different authors will
use different mathematical symbols to express certain quantities. For example,
many textbooks have used a standard notation for a vector using a boldface,
i.e. $\mathbf{u}$. However, there are some variations to express a vector. For
blackboard notation, it is acceptable to denote a vector using an arrow on top
of the letter, i.e. $\vec{\textmd{\,u}}$. Some, however, use an underline or
an undertilde notation to denote a vector, i.e. u or $\utilde{\textmd{u}}$.

It has also been reported that some tutorial lecturers often have a completely
different approach in solving certain problems compared to the main lecturer
who is teaching them in the two-hour lecture session. As a consequence, many
students get confused regarding which methods they should use in solving the
problems in the exams. Some examples of these differences are given in
Appendix B. The difference in notations and techniques used across mathematics
modules in the context of engineering lectures has also been observed in an
Australian university [3]. The author gives an example that Cramer’s rule
appears to be widely used to solve systems of linear equation, but only
Gaussian elimination and matrix inversion are taught in the mathematics
subjects.

Another major concern when adopting the block teaching method is in the
context of composing a timetable. One colleague who handles the timetable has
said that block teaching poses more restrictions to certain hours. For
instance, Foundation Mathematics module is scheduled at 9 am–11 am on Monday
for Group A, on Tuesday for Group B and on Wednesday for Group C. Block
teaching will require two or three different time slots for different days
since one lecturer may only handle one group at each time. On the contrary,
parallel teaching only requires one teaching slot for the whole groups, let
say on Monday 9 am–11 am, and the two or three lecturers simply go to their
own respective groups. This timetabling problem can be solved by implementing
a certain scheduling program. An example on building university timetabling
using certain mathematical programming is given in [4]. For school timetable
construction, the interested readers may consult [5].

##  5 Conclusion and Remark

In this article, we have discussed some pros and cons regarding two different
methods of delivering mathematics modules to engineering students at the
University of Nottingham Malaysia Campus. Implementing the parallel teaching
method has given a good opportunity for both sides of lecturers and students
in getting to know each other better so that the lecturers could monitor the
progress many of their students, if not all, until the semester ends. On the
other hand, parallel teaching gives room for more variation in teaching
techniques and delivery and in turn, the students may feel that they are not
treated sufficiently fairly since they do not receive the same materials as
their counterparts in the other classes do.

With this in mind, the block teaching has been introduced in the past academic
year. The number of complaints has been reduced after implementing this
method. However, a number of shortcomings of block teaching has also been
identified. Students find difficulties in adjusting the teaching style of one
lecturer to the other within a short period of time, i.e. a couple of weeks
period. Some students confuse with different notations being used within the
same module and different techniques introduced by different lecturers to
solve certain mathematical problems.

Whether we would like to implement parallel teaching or block teaching
delivery in the future, it is important that all lecturers who are involved
sit down and discuss the matter beforehand. Hopefully, this will prevent a
significant variation in the teaching materials. Furthermore, in order to
reduce students’ confusion, special attention must be given to the notations
and techniques used in the modules. Finally, by implementing a particular
scheduling program, it will reduce the trouble that comes when arranging the
timetable for block teaching delivery.

## Acknowledgement

We would like to thank Dr. Rohaizan Osman as the Director of Studies of
Department of Applied Mathematics, Mr. Chiang Choon Lai as the timetable
coordinator for Foundation Studies and a number of students at the Faculty of
Engineering who are willing to express their opinions regarding the different
methods of teaching delivery.

## References

  * [1] The framework for higher education qualifications in England, Wales and Northern Ireland. Available online at https://www.ncl.ac.uk/ltds/assets/documents/qsh-progapp-fheqsummary.pdf. Accessed .
  * [2] Friend, M., Reising, M., & Cook, L. (1993). Co-teaching: An overview of the past, a glimpse at the present, and considerations for the future. Preventing School Failure, 37(4): 6–10.
  * [3] Varsavsky, C. (1995). The design of the mathematics curriculum for engineers: A joint venture of the mathematics department and the engineering faculty. European Journal of Engineering Education, 20(3): 341–345.
  * [4] Guéret, C., Jussien, N., Boizumault, P. and Prins, C. (1996). Building university timetables using constraint logic programming. Practice and Theory of Automated Timetabling, Springer Berlin/Heidelberg.
  * [5] Willemen, R. J. (2002). School timetable construction–algorithms and complexity, PhD Thesis, Technical University Eindhoven, The Netherlands.
  * [6] Anton, H. (2005). Elementary Linear Algebra, 9${}^{\textmd{th}}$ edition, John Wiley & Sons, Inc.
  * [7] http://en.wikipedia.org/wiki/Rule_of_Sarrus. Accessed .
  * [8] Smith, R. T. and Minton, R. B. (2008). Calculus, 3${}^{\textmd{rd}}$ edition, McGraw-Hill.

### Appendix A Some examples of block teaching plan

Table 2: Block teaching plan for Foundation Mathematics (Spring) Week |  Topics | Lecturers
---|---|---
1 |  Applications of derivative 1 |
2 |  Applications of derivative 2 | Dr. V. Algebraist
3 |  Curve sketching 1 |
4 |  Curve sketching 2 |
5 |  Introduction to integral | Prof. Z. Mathematician
6 |  Techniques of integration | (Module convener)
7 |  Application of integral 1 |
8 |  Application of integral 2 |
9 |  Numerical integration | Dr. W. Statistician
10 |  Binomial theorem for any rational index |
11 |  Test |
Table 3: Block teaching plan for Engineering Mathematics 1 Week |  Topics | Lecturers
---|---|---
1 |  Coordinates system |
2 |  Algebra of complex numbers | Prof. G. Geometer
3 |  Matrix algebra | (Module convener)
4 |  System of equations and eigenvalue problems |
5 |  Application of matrix algebra |
6 |  Functions and their properties | Dr. N. Number-Theorist
7 |  Advanced differential calculus |
8 |  Taylor series |
9 |  Curve sketching | Dr. Q. Numerical-Analyst
10 |  Integral calculus of one variable |
11 |  Test |
Table 4: Block teaching plan for Engineering Mathematics 2 Week |  Topics | Lecturers
---|---|---
1 |  Function of several variables |
2 |  Partial derivatives and their interpretations | Dr. M. Mathematical-Physicist
3 |  Chain rule and total derivatives |
4 |  First order ordinary differential equations |
5 |  Exact differential equations | Prof. G. Geometer
6 |  Boundary and initial conditions | (Module convener)
7 |  Second-order linear constant coefficient |
|  ordinary differential equations |
8 |  Vector spaces |
9 |  Scalar and vector products | Dr. T. Graph-Theorist
10 |  Applications of vector methods in engineering |
11 |  Test |

### Appendix B Some examples of problems solved by different methods

#### Example 1 Vector (cross) product

To evaluate the vector product (or vector product) between two vectors, we
need to calculate the determinant of a 3 $\times$ 3 matrix. See [6] or any
other books on linear algebra for more complete descriptions on matrix
determinant. There are at least two ways to calculate this determinant, i.e.
using a combination of three 2 $\times$ 2 determinants and the other one is
using Sarrus’ rule or Sarrus’ scheme. To see the difference between these
methods, let us see an example. Let
$\mathbf{a}=a_{1}\mathbf{i}+a_{2}\mathbf{j}+a_{3}\mathbf{k}$ and
$\mathbf{b}=b_{1}\mathbf{i}+b_{2}\mathbf{j}+b_{3}\mathbf{k}$, then the cross
product of $\mathbf{a}$ and $\mathbf{b}$ reads

$\mathbf{a}\times\mathbf{b}=\left|\begin{array}[]{ccc}\mathbf{i}&\mathbf{j}&\mathbf{k}\\\
a_{1}&a_{2}&a_{3}\\\ b_{1}&b_{2}&b_{3}\\\ \end{array}\right|.$

Using the first method, the cross product is given by

$\displaystyle\mathbf{a}\times\mathbf{b}$
$\displaystyle=\left|\begin{array}[]{cc}a_{2}&a_{3}\\\ b_{2}&b_{3}\\\
\end{array}\right|\mathbf{i}-\left|\begin{array}[]{cc}a_{1}&a_{3}\\\
b_{1}&b_{3}\\\
\end{array}\right|\mathbf{j}+\left|\begin{array}[]{cc}a_{1}&a_{2}\\\
b_{1}&b_{2}\\\ \end{array}\right|\mathbf{k}$
$\displaystyle=(a_{2}b_{3}-a_{3}b_{2})\mathbf{i}-(a_{1}b_{3}-a_{3}b_{1})\mathbf{j}+(a_{1}b_{2}-a_{2}b_{1})\mathbf{k}.$

Sarrus’ rule is a method of evaluating the determinant of a 3 $\times$ 3
matrix. This method is named after the 19${}^{\textmd{th}}$-century French
mathematician Pierre Frédéric Sarrus (1798-1861). The scheme is given as
follows. Repeat the first two columns of the matrix behind the third column,
so that we have five columns in a row. Then add the products of the diagonal
components from top to bottom and subtract the products of the diagonal
components going from bottom to top [7]. Thus, using Sarrus’ rule, the cross
product is given by

$\displaystyle\mathbf{a}\times\mathbf{b}$
$\displaystyle=\left|\begin{array}[]{cccccc}\mathbf{i}&\mathbf{j}&\mathbf{k}&\mid&\mathbf{i}&\mathbf{j}\\\
a_{1}&a_{2}&a_{3}&\mid&a_{1}&a_{2}\\\ b_{1}&b_{2}&b_{3}&\mid&b_{1}&b_{2}\\\
\end{array}\right|=a_{2}b_{3}\mathbf{i}+a_{3}b_{1}\mathbf{j}+a_{1}b_{2}\mathbf{k}-b_{1}a_{2}\mathbf{k}-b_{2}a_{3}\mathbf{i}-b_{3}a_{1}\mathbf{j}$
$\displaystyle=(a_{2}b_{3}-a_{3}b_{2})\mathbf{i}+(a_{3}b_{1}-a_{1}b_{3})\mathbf{j}+(a_{1}b_{2}-a_{2}b_{1})\mathbf{k}.$

We can observe that in fact, both rules produce the same result. A confusion
might occur at the final expressions. Using Sarrus’ rule, all terms have
positive signs while the earlier method places a negative sign in front of the
second term.

#### Example 2 Definite integral by substitution

Another example of implementing different methods is to evaluate a definite
integral using substitution method. See [8] or any other Calculus books for
more detailed explanations and examples. Suppose we want to evaluate

$I=\int_{-\sqrt{\pi/6}}^{\sqrt{2\pi/3}}\;\frac{7}{5}x\cos\left(x^{2}+\frac{\pi}{6}\right)\,{\rm
d}x.$

Using the substitution method, we let $u=x^{2}+\pi/6$, so ${\rm d}u=2x\,{\rm
d}x$ or $x\,{\rm d}x=\frac{1}{2}{\rm d}u$. But we have also to take care the
boundary of integration, i.e. for $x=-\sqrt{\pi/6}$, $u=\pi/3$ and for
$x=2\pi/3$, $u=5\pi/6$. Thus the integral becomes

$I=\int_{\pi/3}^{5\pi/6}\frac{7}{10}\cos u\,{\rm d}u=\frac{7}{10}\sin
u\bigg{|}_{\pi/3}^{5\pi/6}=\frac{7}{20}(1-\sqrt{3}).$

A similar method is without introducing a new variable $u$ and therefore there
is no need to change the boundaries of integration. We can write the definite
integral above as follows:

$I=\int_{-\sqrt{\pi/6}}^{\sqrt{2\pi/3}}\;\frac{7}{10}\cos\left(x^{2}+\frac{\pi}{6}\right)\;{\rm
d}\left(x^{2}+\frac{\pi}{6}\right)=\left.\frac{7}{10}\sin\left(x^{2}+\frac{\pi}{6}\right)\right|_{-\sqrt{\pi/6}}^{\sqrt{2\pi/3}}=\frac{7}{20}(1-\sqrt{3}).$

It is observed that the students have a better grasp on the method of
substitution by introducing a new variable $u$. However, many of them often
forget to replace the boundaries of integration in term of the new variable.
On the other hand, the latter method is useful since the students do not have
to worry about the boundaries of integration. However, rewriting the integral
in the desired form as the above can be quite challenging to many students in
general.