# Designing Multi–Functional Metamaterials

J. R. Capers<EMAIL_ADDRESS>Department of Physics and Astronomy,
University of Exeter, Stocker Road, Exeter, EX4 4QL  S. J. Boyes  DSTL, Porton
Down, Salisbury, Wiltshire, SP4 0JQ  A. P. Hibbins  Department of Physics and
Astronomy, University of Exeter, Stocker Road, Exeter, EX4 4QL  S. A. R.
Horsley  Department of Physics and Astronomy, University of Exeter, Stocker
Road, Exeter, EX4 4QL

###### Abstract

The ability to design passive structures that perform different operations on
different electromagnetic fields is key to many technologies, from
beam–steering to optical computing. While many techniques have been developed
to optimise structure to achieve specific functionality through inverse
design, designing multi–function materials remains challenging. We present a
semi–analytic method, based on the discrete dipole approximation, to design
multi–functional metamaterials. To demonstrate the generality of our method,
we design a device that operates at optical wavelengths and beams light into
different directions depending on the source polarisation and a device that
works at microwave wavelengths and sorts plane waves by their angle of
incidence.

##  I Introduction

Much of modern technology is enabled by the control of electromagnetic waves.
One way to manipulate electromagnetic radiation is to structure materials in
space Munk2000 ; Leonhardt2006 ; Pendry2006 , and more recently time
Galiffi2022 ; Ptitcyn2022 , to control electromagnetic fields. Metamaterials,
man–made materials that have electromagnetic properties associated with their
structure rather than chemistry, have demonstrated extraordinary control over
all types of wave Kadic2019 . Typically metamaterials are built from
‘meta–atoms’, sub–wavelength resonant scattering elements with scattering
properties that depend upon their structure. By tuning the geometry of the
meta–atoms, almost any wave scattering effect can be realised. Of particular
interest is the ability to design multi–functional metamaterials: materials
that scatter differently in response to different input fields. Being able to
passively perform different operations upon different waves is key to several
problems across physics; from multiplexing output beams Pande2020  to mode
sorting Frellsen2016 ; Piggott2015  and beam steering Li2019 . Furthermore,
metamaterials comprising several discrete meta-atoms that can be arranged
anywhere in space, and that are explicitly aperiodic, are typically designed
using genetic algorithms Wiecha2017 . These are extremely effective at
exploring large search spaces, but are numerically expensive and produce
results that can be difficult to interpret Yeung2020 .

To understand why the design of multi–functional materials is challenging, we
consider an electric field $\vec{\phi}(\boldsymbol{r})$ of a fixed frequency
$\omega=ck_{0}$, with wave number $k_{0}$, in a material with a spatially
varying permittivty $\varepsilon(\boldsymbol{r})$. This wave obeys the vector
Helmholtz equation,

$\nabla\times\nabla\times\vec{\phi}(\boldsymbol{r})+k_{0}^{2}\varepsilon(\boldsymbol{r})\vec{\phi}(\boldsymbol{r})=0.$
(1)

The difficulty in the design of multi–functional materials is the problem of
finding a single material distribution (here the permittivity, $\varepsilon$)
that performs two (or more) desired wave transformations. This means that both
of the desired wave behaviours, $\vec{\phi}_{1}(\boldsymbol{r})$ and
$\vec{\phi}_{2}(\boldsymbol{r})$ must be solutions to the same Helmholtz
equation,

$\displaystyle\nabla\times\nabla\times\vec{\phi}_{1}(\boldsymbol{r})+k_{0}^{2}\varepsilon(\boldsymbol{r})\vec{\phi}_{1}(\boldsymbol{r})$
$\displaystyle=0,$
$\displaystyle\nabla\times\nabla\times\vec{\phi}_{2}(\boldsymbol{r})+k_{0}^{2}\varepsilon(\boldsymbol{r})\vec{\phi}_{2}(\boldsymbol{r})=0.$
(2)

From this statement, we can find a condition upon the two wave–fields for this
to be possible. Multiplying the first of these by
$\vec{\phi}_{2}(\boldsymbol{r})$ and the second by
$\vec{\phi}_{1}(\boldsymbol{r})$, then taking the difference eliminates the
material properties such that

$\vec{\phi}_{2}(\boldsymbol{r})\cdot\left[\nabla\times\nabla\times\vec{\phi}_{1}(\boldsymbol{r})\right]-\vec{\phi}_{1}(\boldsymbol{r})\cdot\left[\nabla\times\nabla\times\vec{\phi}_{2}(\boldsymbol{r})\right]=0.$
(3)

Integrating this over all space, then using Green’s vector identity
Stratton2007 , we find that

$\oint_{\partial
V}\left[\vec{\phi}_{2}(\boldsymbol{r})\times\nabla\times\vec{\phi}_{1}(\boldsymbol{r})-\vec{\phi}_{1}(\boldsymbol{r})\times\nabla\times\vec{\phi}_{2}(\boldsymbol{r})\right]\cdot
d\boldsymbol{S}=0.$ (4)

This places a stringent condition on the two wave–fields if they are to be
supported by the same material, which can be used to derive fundamental bounds
on the performance of multi–functional devices Miller2021 . Eq. (4) is a
generalization of Poynting’s theorem, representing the conservation of the
norm of the system modes; ensuring for example, their orthogonality. To better
understand the connection between the above and energy conservation, consider
the special case where we demand the same permittivity distribution supports
the solution $\vec{\phi}_{1}(\boldsymbol{r})=\boldsymbol{E}$ and its complex
conjugate (time reverse) $\vec{\phi}_{2}(\boldsymbol{r})=\boldsymbol{E}^{*}$.
The surface integral (4) can then be re–written using the divergence theorem,

$\nabla\cdot\left(\boldsymbol{E}\times\nabla\times\boldsymbol{E}^{*}-\boldsymbol{E}^{*}\times\nabla\times\boldsymbol{E}\right)=0.$
(5)

Applying Maxwell’s equations to convert the curls into magnetic fields
$\nabla\times\boldsymbol{E}=-i\omega\eta_{0}\boldsymbol{H}$, where $\eta_{0}$
is the impedance of free space, we find

$\nabla\cdot\left(\boldsymbol{E}\times\boldsymbol{H}^{*}+\boldsymbol{E}^{*}\times\boldsymbol{H}\right)=0,$
(6)

which is the usual expression for energy conservation expressed in terms of
the Poynting vector $\boldsymbol{S}=\frac{1}{2}{\rm
Re}\left[\boldsymbol{E}\times\boldsymbol{H}^{*}\right]$.

A direct application of Eq. (4) to design multi–functional materials is
generally difficult. Instead several other design methodologies have emerged
recently Molesky2018 . Where the function of the metasurface, a 2D
metamaterial, is to impart a known phase and amplitude offset, the
Gerchberg–Saxton algorithm Gerchberg1972  is commonly used Liu2021 . While
simple and efficient this method can struggle to capture the coupling between
elements of the metamaterial, requiring either strong field confinement
within, or large spacing between the elements. As well as this, several
full–wave simulations are required to build up a library of phase and
amplitude changing meta–atoms. For problems where a continuous permittivity
distribution is required, ‘topology optimisation’ Bendsoe2003  can be used to
design structures that sort waveguide modes Frellsen2016  or perform
wavelength–dependant behaviour Piggott2015 . This method typically uses a
full–wave solver to find the fields then gradient descent optimisation to
design a structure that optimises a figure of merit. For large structures this
can be computationally demanding, however the adjoint method can be used to
optimise a figure of merit by converting a shape derivative for the entire
design into two field calculations Lalau-Keraly2013 . Even with this greatly
improved numerical efficiency, many full–wave simulations are still required
over the course of the optimisation.

We present a general framework to design multi-functional metamaterials in
electromagnetism in order to overcome some of the aforementioned difficulties.
Based on the discrete dipole approximation Purcell1973 , the framework
proposed in this paper is numerically efficient while still being easily
applicable to a wide range of electromagnetics problems. The discrete dipole
approximation is briefly described in Section II then a summary of how
metamaterials can be designed within this approximation is provided in in
Section III. Considering the problem of shaping the far–field of a dipole
emitter while also increasing its efficiency, this is then extended to design
multi–functional materials in Section IV. Several examples of utilising this
framework are then shown in Section V. The first device we present operates at
optical wavelengths $\lambda=550$ nm, with silicon nanospheres used as the
scattering elements, and beams light into different directions based upon the
polarisaion of the source. The second device we show works at microwave
wavelengths $\sim 20$ mm (15.5 GHz), using ‘metacubes’ Powell2020  as the
scatterers, and sorts signals by their incidence direction.

##  II Modelling Metamaterials

Before trying to design the scattering properties of a metamaterial, it is
necessary to characterise the effect of the material upon an electromagnetic
wave. Maxwell’s equations for a fixed frequency $\omega=ck_{0}$, where $k_{0}$
is the wave–number, can then be written as

$\begin{pmatrix}\nabla\times\nabla\times&\boldsymbol{0}\\\
\boldsymbol{0}&\nabla\times\nabla\times\end{pmatrix}\begin{pmatrix}\boldsymbol{E}\\\
\boldsymbol{H}\end{pmatrix}-k_{0}^{2}\begin{pmatrix}\boldsymbol{E}\\\
\boldsymbol{H}\end{pmatrix}=\begin{pmatrix}\boldsymbol{E}_{\rm inc}\\\
\boldsymbol{H}_{\rm
inc}\end{pmatrix}+\begin{pmatrix}\omega^{2}\mu_{0}&i\omega\mu_{0}\nabla\times\\\
-i\omega\nabla\times&k_{0}^{2}\end{pmatrix}\begin{pmatrix}\boldsymbol{P}\\\
\boldsymbol{M}\end{pmatrix}$ (7)

where the material is characterised by the polarisation density
$\boldsymbol{P}$ and magnetisation density $\boldsymbol{M}$ and the incident
field is $(\boldsymbol{E}_{\rm inc},\boldsymbol{H}_{\rm inc})^{T}$. The
incident field could be due to an emitter near the structure, or might be a
plane wave. Assuming that the scatterers are sub–wavelength in size means that
the field can be treated as constant over the scatterer, and so the
scatterer’s distribution can be written as a delta–like point at its center
$\boldsymbol{r}_{n}$. The scatterer then acquires an electric and magnetic
dipole moment in response to the applied fields

$\displaystyle\boldsymbol{P}$
$\displaystyle=\sum_{n}\boldsymbol{\alpha}_{E}\boldsymbol{E}(\boldsymbol{r}_{n})\delta(\boldsymbol{r}-\boldsymbol{r}_{n})$
and $\displaystyle\boldsymbol{M}$
$\displaystyle=\sum_{n}\boldsymbol{\alpha}_{H}\boldsymbol{H}(\boldsymbol{r}_{n})\delta(\boldsymbol{r}-\boldsymbol{r}_{n}),$
(8)

where $\boldsymbol{\alpha}_{E}$ is the electric polarisability tensor and
$\boldsymbol{\alpha}_{H}$ is the magnetic polarisability tensor. The
wave–equation with delta–like source terms can be solved with the appropriate
Green’s function Schwinger1950 ; Tai1993  as

$\phi(\boldsymbol{r})=\phi_{\rm
inc}(\boldsymbol{r})+\sum_{n=0}^{N}G(\boldsymbol{r},\boldsymbol{r}_{n})\phi(\boldsymbol{r}_{n}).$
(9)

We have adopted the compact notation

$\displaystyle\phi(\boldsymbol{r})$
$\displaystyle=\begin{pmatrix}\boldsymbol{E}(\boldsymbol{r})\\\
\eta_{0}\boldsymbol{H}(\boldsymbol{r})\end{pmatrix}\ \text{and}$
$\displaystyle G(\boldsymbol{r},\boldsymbol{r^{\prime}})$
$\displaystyle=\begin{pmatrix}\xi^{2}\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\boldsymbol{\alpha}_{E}&i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\boldsymbol{\alpha}_{H}\\\
-i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\boldsymbol{\alpha}_{E}&\xi^{2}\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\boldsymbol{\alpha}_{H}\end{pmatrix},$
(10)

where

$\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})=\left[\boldsymbol{1}+\frac{1}{\xi^{2}}\nabla\otimes\nabla\right]\frac{e^{i\xi|\boldsymbol{r}-\boldsymbol{r^{\prime}}|}}{4\pi|\boldsymbol{r}-\boldsymbol{r^{\prime}}|}$
(11)

is the Dyadic Green’s function and $\xi$ is a dimensionless wave–number. To
completely specify the field solution (9), the fields applied to each
scatterer $\phi(\boldsymbol{r}_{n})$ must be determined. This includes the
source field, as well as all orders of multiple scattering interactions
between the scatterers themselves. The applied fields
$\phi(\boldsymbol{r}_{n})$ can be found through the self–consistency condition
Foldy1945

$\boldsymbol{R}_{mn}\boldsymbol{\phi}_{m}=\boldsymbol{\phi}_{i,n},$ (12)

with
$\boldsymbol{R}_{nm}=\boldsymbol{1}\delta_{nm}-G(\boldsymbol{r}_{n},\boldsymbol{r}_{m})$,
$\boldsymbol{\phi}_{m}=\phi(\boldsymbol{r}_{m})$ and
$\boldsymbol{\phi}_{i,n}=\phi_{i}(\boldsymbol{r}_{n})$. This forms a linear
system that can be solved for the fields applied to the scatterers
$\boldsymbol{\phi}_{m}$ using standard matrix methods. Once these are found,
the fields (9) are fully specified.

##  III Designing Metamaterials

The scattering properties of a metamaterial made of several sub–wavelength
scatterers can be designed using the methodology presented in Capers2021 . We
briefly review this here, before extending the method to multi–functional
devices. By Taylor expanding the delta function sources in the wave equation
(8) under small changes in the position of a scatterer yields,

$\delta(\boldsymbol{r}-\boldsymbol{r}_{n}-\delta\boldsymbol{r}_{n})=\delta(\boldsymbol{r}-\boldsymbol{r}_{n})-(\delta\boldsymbol{r}_{n}\cdot\nabla)\delta(\boldsymbol{r}-\boldsymbol{r}_{n})+\frac{1}{2}(\delta\boldsymbol{r}_{n}\cdot\nabla)^{2}\delta(\boldsymbol{r}-\boldsymbol{r}_{n})+\ldots,$
(13)

and by keeping only terms linear in $\delta\boldsymbol{r}_{n}$ an expression
for how the field $\phi(\boldsymbol{r})$ changes due to a small change in the
location of scatterer $n$ can be deduced,

$\delta\phi(\boldsymbol{r})=G(\boldsymbol{r},\boldsymbol{r}_{n})\nabla\phi(\boldsymbol{r}_{n})\cdot\delta\boldsymbol{r}_{n}.$
(14)

A figure of merit $\mathcal{F}$ can be written as a functional of the field
$\mathcal{F}=\mathcal{F}[\phi]$. Under small changes in the field as a result
of a small change in the position of a scatterer, the figure of merit is
changed by a small amount. Given a particular figure of merit, an analytic
expression for this change $\delta\mathcal{F}[\phi,\delta\phi]$ can be
derived. Since this will be linear in the change in position of a scatterer
$\delta\boldsymbol{r}_{n}$, the resulting expression allows for the derivative
of the figure of merit with respect to the scatterer locations
$\partialderivative{\mathcal{F}}{\boldsymbol{r}_{n}}$ to be calculated
analytically. As an example, say the figure of merit is the amplitude squared
of the field at a particular location $|\phi(\boldsymbol{r}_{\star})|^{2}$.
Expanding under small changes in the field we can find how the figure of merit
changes,

$\displaystyle\mathcal{F}$ $\displaystyle=|\phi(\boldsymbol{r}_{\star})|^{2},$
(15) $\displaystyle\delta\mathcal{F}$ $\displaystyle=2{\rm
Re}\left[\phi^{*}(\boldsymbol{r}_{\star})\delta\phi(\boldsymbol{r}_{\star})\right].$
(16)

Substituting into this in the variation of the field (13) and diving by
$\delta\boldsymbol{r}_{n}$ we find that

$\displaystyle\partialderivative{\mathcal{F}}{\boldsymbol{r}_{n}}$
$\displaystyle=2{\rm
Re}\left[\phi^{*}(\boldsymbol{r}_{\star})G(\boldsymbol{r}_{\star},\boldsymbol{r}_{n})\nabla\phi(\boldsymbol{r}_{n})\right].$
(17)

This approach gives an analytic expression for the derivative of a figure of
merit that can be evaluated for all of the scatterers at the same time then
used in a gradient descent optimisation Shwartz2014

$\boldsymbol{r}_{n}^{i+1}=\boldsymbol{r}_{n}^{i}+\gamma\partialderivative{\mathcal{F}}{\boldsymbol{r}_{n}},$
(18)

where $\gamma$ is the learning rate and $i$ is the iteration number. While the
derivatives of the fields still need to be evaluated, the derivative of the
figure of merit, which is more numerically expensive, does not.

Extending this to apply to multi–functional metamaterials, requires one to
seek to increase some set of figures of merit
$\left\\{F_{1},F_{2},F_{3}\ldots\right\\}$. A composite figure of merit can be
constructed that is a weighted sum of these,

$\mathcal{F}=\sum_{i}w_{i}F_{i},$ (19)

where $w_{i}$ are the weights for each figure of merit. This composite figure
of merit can be optimised in the same way as a single figure of merit, where
the gradient in equation (18) is now

$\partialderivative{\mathcal{F}}{\boldsymbol{r}_{n}}=\sum_{i}w_{i}\partialderivative{F_{i}}{\boldsymbol{r}_{n}}.$
(20)

Key to the success of this method is a sensible choice of the weights,
$w_{i}$. An appropriate choice can be informed by considering the desired
properties of the resulting device, as we explore in the next section.

##  IV Multi–Objective Optimisation Considerations

Figure 1: An example of a multi–objective optimisation problem. We seek to
shape the radiation pattern of a dipole emitter pointing out of the plane
along the $z$ axis, while also increasing its efficiency. To achieve this, we
consider placing a single isotropic scatterer near the emitter and explore the
effect this has upon the figures of merit; the power emission and the
radiation pattern. a) Shows how placing a scatterer at a particular point
changes the power emission and b) shows how the overlap integral is changed.
These represent the search spaces of the problem. The additional difficulty
posed by _multi_ , rather than single, objective problems is shown in c). A
scatterer can only be placed in the green regions, where both figures of merit
are enhanced. Each point in the search spaces correspond to a point in
solution space, shown in d). The blue triangle and red star and shown in both
search and solution spaces to demonstrate this. Also shown in the solution
space, d), is the diagonal ‘line of equal enhancement’ representing equal
performance of each figure of merit and the Pareto front Hwang1979 , which
represents the acceptable solutions to the multi–objective problem.

Consider a simple example problem, shown in Figure 1. The goal is to
distribute scatterers around a point emitter at location
$\boldsymbol{r^{\prime}}$ with polarisation
$\boldsymbol{p}=\boldsymbol{\hat{z}}$ such that _two_ figures of merit are
simultaneously maximised. More specifically, the goal is to re–shape the
radiation pattern of an emitter, while simultaneously increasing the
efficiency. The first figure of merit is the power emission of the emitter,

$F_{1}=P=\frac{\omega}{2}{\rm
Im}[\boldsymbol{p}^{*}\cdot\boldsymbol{E}(\boldsymbol{r^{\prime}})].$ (21)

The second is the overlap integral between the angular distribution of the
Poynting vector in the far–field, $|\boldsymbol{S}(\theta)|$ and the desired
angular distribution $\psi_{T}(\theta)$

$F_{2}=I=\frac{\int
d\theta|\boldsymbol{S}(\theta)|\psi_{T}(\theta)}{\sqrt{\int
d\theta|\boldsymbol{S}(\theta)|^{2}}\sqrt{\int d\theta\psi_{T}^{2}(\theta)}},$
(22)

where the angle $\theta$ is in the same plane as the metasurface. In the
following examples,the target distribution is

$\psi_{T}(\theta)=\begin{cases}\cos^{2}\theta\
270^{\circ}<\theta<90^{\circ},\\\ 0\ \text{otherwise}.\end{cases}$ (23)

Both of these can be expanded to first order to find the gradient of the
figure of merit with respect to the scatterer locations Capers2021 . These
expansions are given in Appendix A. For convenience, both will be normalised
by their free–space values, $P_{0}$ and $I_{0}$: these are the values of the
figures of merit without any scatterers present. Considering the effect of a
single scatterer upon these figures of merit, Figure 1 a) and b) show how
placing the scatterer in a particular location increases or decreases each
figure of merit. These maps define the search space for the problem. For
multi–functional problems, however, there is the additional constraint that a
scatterer should only be placed where _both_ figures are merit are increased.
This is shown in Figure 1c; it is clear that multi–functional problems are
significantly constrained and have complex search spaces. Each scatterer
location in the search spaces Figure 1 a) and b) corresponds to a point in the
solution space, shown in Figure 1d. To demonstrate this correspondence, the
red star and blue triangle are shown in the search and solution spaces. Each
point in solution space corresponds to a configuration of scatterers, which in
our example is one, but could be any number. The solution space, Figure 1d,
has a few interesting features. The Pareto front Hwang1979 , shown as a red
line, are all acceptable solutions to the multi–objective optimisation problem
(i.e. where one figure of merit cannot be improved without sacrificing the
other). Along the diagonal, the dashed magenta line, enhancement of the two
figures of merit is equal, which is often the desired outcome. It would not be
very useful to select a solution point where the power emission is large but
the overlap is small, even if it lies on the Pareto front. This observation
informs the choice of weights in the optimisation procedure. The weights are
chosen to be proportional to the figure of merit itself,

$\displaystyle w_{i}\propto\frac{1}{F_{i}}\ \text{, and normalised so
that}\sum_{i}w_{i}=1.$ (24)

Choosing the weights to be proportional to $1/F_{i}$ means that when the
figure of merit is small the contribution of the gradient associated with that
figure of merit to the sum (20) is large, but when the figure of merit is
large the contribution is suppressed. Note that the figures of merit must be
normalised so that their magnitudes can be meaningfully compared, for example
by division by a free space value. Choosing the weights in this way allows for
the design of multi–functional metamaterials built from discrete scatterers,
for a variety of applications. A few examples are offered in the following
section.

##  V Multi–functional Devices

Continuing with the example developed in the previous section and shown in
Figure 1, the multi–objective problem of designing a structure that re–shapes
the radiation pattern of an emitter in the plane of the metasurface, while
also increasing efficiency, is addressed. We work at a wavelength of
$\lambda=550$ nm and the scatterers are small silicon spheres of radius 65 nm.
For this system, the polarisability tensor can be found analytically from the
Mie $a_{1}$ and $b_{1}$ coefficients. Our figure of merits are the power
emission (21) and the overlap with the desired radiation pattern (22) and we
choose the weights according to (24). Analytic expression for the gradients of
these figures are merit can be found analytically. Using these gradients,
weights (24) and the gradient descent method (18), we design the structures
shown in Figure 2.

Figure 2: Multi–objective optimisation solutions, seeking to increase the
emitted power from a dipole while also shaping the far–field radiation pattern
into the desired double–lobed shape. For comparison, the single–objective
case, where only the radiation pattern is shaped is shown. a) shows the
far–field radiation pattern in the plane of the multi–functional structure
(red) as well as the target radiation pattern (black dashes). The case where
only radiation pattern is shaped is shown in blue. In b), the paths in
solution space of the single and multi–objective cases are shown. It is clear
that our choice of weightings works well: both figures of merit undergo
similar enhancements from their starting values. When only radiation pattern
is controlled (blue line) power emission changes little over the optimisation,
however when it is part of the composite figure of merit (red line) clear
enhancement is seen at the same time as the overlap integral is increased. The
single–objective structure is shown in c) and the multi–objective structure is
show in d), with the emitter polarised out of the page at the origin. In this
example, we work at $\lambda=550$ nm and the scatterers are silicon spheres of
radius 65 nm.

For comparison, we also consider the single–objective case where only the
far–field radiation pattern is shaped. The resulting far–field radiation
patterns are shown in Figure 2a, with the path of the optimisation in solution
space shown in Figure 2b and the two resulting structures shown in Figure 2c
and d. Examining first the radiation pattern, we note that the multi–objective
optimisation produces a slightly worse match to the target distribution than
the single–objective case. This is due to the trade–off between the two
figures of merit we seek to optimise. In solution space, Figure 2b, we see
that in the case where only the radiation pattern is shaped (blue line) only a
very small change in emitted power is seen. Conversely, when both power and
radiation pattern are optimised, the emitted power approximately doubles while
the overlap integral also increases. Unlike the case for a single scatterer
shown in Figure 1, it is impossible to plot the whole search space and
determine the location of the Pareto front. The scatterers have a diameter
$\sim\lambda/4$ and our solution box has size $8\lambda$, meaning that there
are $32^{2}=1024$ possible ‘pixels’ a scatterer could occupy. This means that
for $N$ scatterers, the number of possible solutions is $1024!/(N!(1024-N)!)$.
For $N=64$, this is $\approx 10^{21}$. Despite the exceedingly large search
space, which even a genetic algorithm would explore only a very small portion
of, our method has found a solution that performs well. A comparison between
solving this problem using the method we present and a genetic algorithm is
given in Appendix B.

The second example we consider is manipulating the radiation pattern based on
source polarisation. Again, we work at $\lambda=550$ nm and use 65 nm silicon
spheres as the scatterers. We aim to create beams at angles $\theta_{i}$,
associated with source polarisation $\boldsymbol{p}_{i}$. Our figures of merit
are therefore

$F_{i}=|\boldsymbol{S}(\theta_{i})|.$ (25)

The expansion of this to find analytically the gradient is given in Appendix
A. We consider the source polarisation being either left or right circularly
polarised, i.e.

$\boldsymbol{p}=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\\ \pm i\\\
0\end{pmatrix}.$ (26)

The Poynting vector can then be expanded to first order to find the
derivatives of the figures of merit for the optimisation procedure.

Figure 3: Solution to the multi–objective problem of beaming in different
directions based on the polarisation of the source. We work at optical
wavelengths $\lambda=550$ nm, using silicon spheres of radius $65$ nm as the
scatterers. The far–field Poynting vector is shown a) in the plane of the
metamaterial and b) the full far–field sphere defined by the in–plane angle
$\theta$ and the polar angle $\phi$ for left and right handed circularly
polarised sources. The aim was for a right handed source to beam into the 0∘
direction and for a left handed source to beam into the 100∘ direction. The
path in solution space of the optimisation, c), shows that over the
optimisation both figures of merit are enhanced equally, due to our choice of
weights. The multi–functionality condition (4) is verified in panel d), by
computing the integrals numerically. This yields 10-8, which is small enough
to be considered zero within the numerical error associated with evaluating
the surface integral.

Figure 3a shows the radiation patterns of the designed structure excited by
each of the two different sources we consider. For a right-handed source, the
target angle is $\theta=0^{\circ}$ and for a left–handed source,
$\theta=100^{\circ}$. The far–field Poynting vector, Figure 3b, also shows
clear peaks at the desired locations. The path in solution space, Figure 3c,
shows that the choice of weighting has ensured that the performance of both
figures of merit remain similar over the optimisation and in the final result.
The multi–functionality condition (4) is considered in Figure 3d. Forming the
vector fields $\vec{\phi}_{i}\times\nabla\times\vec{\phi}_{j}$ on the surface
of a sphere enclosing the structure, integrating over the surface and taking
the difference yields a result of the order $10^{-8}$, within expected
numerical error.

The third and final example we consider is designing a device for beam
sorting. Working at 15.5 GHz and using ‘metacubes’ Powell2020  as the
scattering element. The metacubes, formed of six metal faces joined by three
connecting spokes, exhibit a strong dipole resonance at 15.5 GHz. Due to their
complexity the polarisability tensor cannot be found analytically. Instead,
one can model a single scatterer under plane–wave incidence using a full–wave
solver such as COMSOL comsol  and integrate over the currents to find the
electric and magnetic dipole moments Arango2013 ; Liu2016  that may be
converted to polarisability tensors. Optimisation of a structure of many
complex scatterers using such full–wave methods quickly becomes intractable.
Our method presents the key benefit of being able to model large systems of
potentially complicated scatterers, provided they can be approximated as
dipolar, although it is possible to include higher order multipoles into the
formalism Raab2005 ; Evlyukhin2011 ; Evlyukhin2013 . The validity of using the
discrete dipole approximation to describe these systems is verified by
comparison with full–wave solution in Appendix C. We seek a structure of
metacubes that takes plane waves from different directions and focuses them to
distinct points. A device of this sort could be used, for example, to detect
from which direction a signal is coming. The figures of merit for this problem
are,

$\displaystyle F_{1}$ $\displaystyle=|\boldsymbol{E}_{1}(\boldsymbol{r}_{1})|\
\text{, and}$ $\displaystyle F_{2}$
$\displaystyle=|\boldsymbol{E}_{2}(\boldsymbol{r}_{2})|,$ (27)

where $\boldsymbol{r}_{i}$ denotes the location to focus the wave at for
incident direction $i$ and $\boldsymbol{E}_{i}$ is the electric field produced
by the structure under incidence from direction $i$. The gradients of these
figures of merit are given in Appendix A. The structure resulting from this
optimisation is shown in Figure 4.

Figure 4: Design of a multi–functional device, with its operation depending
upon the direction of incidence of a plane wave. If the wave is incident from
$\pm 20^{\circ}$, the wave is focused to different locations. The performance
of the device under $-20^{\circ}$ incident is shown in a) and under
$20^{\circ}$ incidence in b), with direction of incidence shown as a white
arrow. We work at 15.5 GHz using metacubes, shown inset in a), as the
scatterers. Panel c), shows cuts of the fields under the two different
incidence angles, demonstrating peaks at the target positions. The path in
solution space, d), shows that the two figures of merit progress at the same
rate over the optimisation, leading to a structure with roughly equal
performance for each figure of merit. The multi–functionality condition (4) is
verified in e) by evaluating the surface integrals. In this case, the result
is $10^{-4}$. The numerical error here is larger than the results in Figure 3
due to the strongly oscillatory nature of the integrand, making evaluation of
the surface integral more sensitive.

Operation of the device when driven by a TE plane wave incident at
$20^{\circ}$ is shown in Figure 4a and for a plane wave at $-20^{\circ}$ in
Figure 4b. The two different focus points are clearly visible in the fields.
The path in solution space is shown in Figure 4c, where again the choice of
weighting has ensured roughly equal performance of $F_{1}$ and $F_{2}$. Slices
of the fields from Figure 4a,b are shown in Figure 4d, indicating the large
main peaks at the desired focus locations. The validity of the
multi–functionality condition is shown in Figure 4e. Here, the numerical error
is larger due to the highly oscillatory nature of the integrands.

##  VI Conclusions & Outlook

Beginning from general considerations of vector fields, we have derived a
condition that the fields must satisfy if they are to be supported by the same
material distribution. Interestingly, this can be expressed as a surface
integral so only the fields on a boundary are needed to determine whether a
particular multi–functional device is feasible. While interesting, this
condition is difficult to solve in general so instead we present an efficient
and versatile semi–analytic method for designing multi–functional
metamaterials made from discrete scattering elements. Demonstrating the
generality of our formulation, we apply our method to design multi–functional
devices at both optical and microwave wavelengths. We show structures that: i)
enhance the efficiency of an emitter while shaping its radiation pattern; ii)
beam in different directions based on the source polarisation; and iii) sort
waves by their incidence direction. In addition to the design method being
simple, the structures we design are easy to fabricate as permittivity does
not need to be graded. Instead many identical resonators must be distributed
in space.

Our approach could be utilised to design very wide classes of multi–functional
devices. For example, the inclusion of higher–order multipoles could allow
more complex resonators to be used, proving more degrees of freedom. Due to
the generality of the two central ideas behind our framework: expanding
figures of merit analytically to avoid expensive numerical derivatives and the
formulation of the multi–objective problem in Section IV, a wide range of
devices could be designed using this approach. Graded index structures as well
as propagation in waveguides and fibre optical cables could all be engineered
to depend upon polarisation or wavelength for communications, cloaking or
sensing applications using the methodology we propose here.

## Acknowledgements

J.R.C would like to thank Dean Patient for many useful discussions and Alex
Powell for providing the COMSOL model for the metacubes.

We acknowledge financial support from the Engineering and Physical Sciences
Research Council (EPSRC) of the United Kingdom, via the EPSRC Centre for
Doctoral Training in Metamaterials (Grant No. EP/L015331/1). J.R.C also wishes
to acknowledge financial support from Defence Science Technology Laboratory
(DSTL). S.A.R.H acknowledges financial support from the Royal Society
(RPG-2016-186). All data and code created during this research are openly
available from the corresponding author, upon reasonable request.

## Author Contributions

J. R. C. and S.A.R.H conceived the idea. J.R.C derived the analytic
expressions, performed the numerical simulations and wrote the manuscript.
S.A.R.H, A.P.H. and S.J.B supervised the project. All authors commented on the
manuscript.

## Competing Interests

The authors declare no competing interests.

##  Appendix A Figure of merit expansions

In this section we give expansions of the figures of merit used in the main
text and the gradients that result from these expansions. To write these, we
will need to use the expressions for the changes in the fields that are given
by (14) in the main text. Unpacking the notation, we have,

$\displaystyle\delta\boldsymbol{E}(\boldsymbol{r})$
$\displaystyle=\left[\xi^{2}\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{E}\nabla\boldsymbol{E}(\boldsymbol{r}_{n})+i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{H}\nabla\boldsymbol{H}(\boldsymbol{r}_{n})\right]\delta\boldsymbol{r}_{n},\text{and}$
(28) $\displaystyle\delta\boldsymbol{H}(\boldsymbol{r})$
$\displaystyle=\left[\xi^{2}\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{H}\nabla\boldsymbol{H}(\boldsymbol{r}_{n})-i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{E}\nabla\boldsymbol{E}(\boldsymbol{r}_{n})\right]\delta\boldsymbol{r}_{n}.$
(29)

###  A.1 Emitted power

Beginning from the usual expression for emitted power, expanding under small
changes in the fields and then substituting (28) gives the gradient of the
power emission with respect to scatterer positions analytically,

$\displaystyle P$ $\displaystyle=\frac{\omega}{2}{\rm
Im}\left[\boldsymbol{p}^{*}\cdot\boldsymbol{E}(\boldsymbol{r^{\prime}})\right],$
(30) $\displaystyle\delta P$ $\displaystyle=\frac{\omega}{2}{\rm
Im}\left[\boldsymbol{p}^{*}\cdot\delta\boldsymbol{E}(\boldsymbol{r^{\prime}})\right],\text{and}$
(31) $\displaystyle\nabla_{\boldsymbol{r}_{n}}P$
$\displaystyle=\frac{\omega}{2}{\rm
Im}\left[\boldsymbol{p}^{*}\cdot\left(\xi^{2}\boldsymbol{G}(\boldsymbol{r^{\prime}},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{E}\nabla\boldsymbol{E}(\boldsymbol{r}_{n})+i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r^{\prime}},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{H}\nabla\boldsymbol{H}(\boldsymbol{r}_{n})\right)\right].$
(32)

###  A.2 Overlap Integral

The figure of merit used to manipulate the shape of the Poynting vector in the
far–field is the normalised overlap integral between the current angular
distribution of the Poynting vector $|\boldsymbol{S}(\theta)|$ and the target
angular distribution $\psi_{T}(\theta)$

$I=\frac{\int d\theta|\boldsymbol{S}(\theta)|\psi_{T}(\theta)}{\sqrt{\int
d\theta|\boldsymbol{S}(\theta)|^{2}}\sqrt{\int d\theta\psi_{T}^{2}(\theta)}}.$
(33)

The modulus of the Poynting vector can be expanded as

$\delta|\boldsymbol{S}|=2{\rm
Re}\left[\boldsymbol{S}^{*}\cdot\delta\boldsymbol{S}\right],$ (34)

where

$\delta\boldsymbol{S}=\delta\boldsymbol{E}\times\boldsymbol{H}^{*}+\boldsymbol{E}\times\delta\boldsymbol{H}^{*}.$
(35)

Using these expressions to expand both the numerator and denominator in (33),
we find an expression for how the overlap integral changes when the fields
change by a small amount,

$\displaystyle\delta I=\frac{1}{2\sqrt{\int
d\theta|\boldsymbol{S}(\theta)|^{2}\int d\theta^{\prime}\phi_{\rm
T}^{2}(\theta^{\prime})}}\left[\int\frac{d\theta}{|\boldsymbol{S}(\theta)|}{\rm
Re}\left\\{\boldsymbol{S}^{*}(\theta)\cdot\left[\delta\boldsymbol{E}\times\boldsymbol{H}^{*}+\boldsymbol{E}\times\delta\boldsymbol{H}^{*}\right]\right\\}\phi_{\rm
T}(\theta)\right.$ (36) $\displaystyle\left.-\frac{\int d\theta\phi_{\rm
T}(\theta)|\boldsymbol{S}(\theta)|}{\int
d\theta|\boldsymbol{S}(\theta)|^{2}}\int d\theta\ {\rm
Re}\left\\{\boldsymbol{S}^{*}(\theta)\cdot\left[\delta\boldsymbol{E}\times\boldsymbol{H}^{*}+\boldsymbol{E}\times\delta\boldsymbol{H}^{*}\right]\right\\}\right].$

Substituting into this the expressions of the variations of the fields (28,
29), the gradient of the overlap integral can be written analytically.

###  A.3 Modulus of Poynting Vector in the Far–Field

When designing the beaming device, shown in Figure 3, the figure of merit is
the modulus of the Poynting vector in the far–field when the structure is
excited by a source of polarisation $\boldsymbol{p}_{i}$. In this example, we
have chosen the two circular polarisations. For each polarisation, we write
the figure of merit as is,

$F_{i}=|\boldsymbol{S}(\theta_{i})|^{2}.$ (37)

This means that for different source polarisations, power will be beamed into
different directions. The expansion of this follows the same procedure as the
expansion of the modulus of the Poynting vector did in the overlap integral,
giving the result,

$\displaystyle\delta F_{i}$ $\displaystyle=2{\rm
Re}\left[\boldsymbol{S}^{*}(\theta_{i})\cdot\delta\boldsymbol{S}(\theta_{i})\right]$
(38) $\displaystyle=2{\rm
Re}\left[\boldsymbol{S}^{*}(\theta_{i})\cdot\left(\delta\boldsymbol{E}(\theta_{i})\times\boldsymbol{H}^{*}(\theta_{i})+\boldsymbol{E}(\theta_{i})\times\delta\boldsymbol{H}^{*}(\theta_{i})\right)\right].$
(39)

Substituting into this the expressions for the field variations (28, 29) gives
analytic expressions for $\nabla_{\boldsymbol{r}_{n}}F_{i}$.

###  A.4 Modulus of Electric Field

For the ‘lensing’ problem, demonstrated in Figure 4, we seek to increase the
modulus of the electric field at particular positions
$\boldsymbol{r}_{\star}$,

$F=|\boldsymbol{E}(\boldsymbol{r}_{\star})|^{2}.$ (40)

The positions are different for each of the incidence directions. This figure
of merit can be easily expanded to find its gradient analytically,

$\displaystyle\delta F$ $\displaystyle=2{\rm
Re}\left[\boldsymbol{E}^{*}(\boldsymbol{r}_{\star})\cdot\delta\boldsymbol{E}(\boldsymbol{r}_{\star})\right],$
(41) $\displaystyle\nabla_{\boldsymbol{r}_{n}}F$ $\displaystyle=2{\rm
Re}\left[\boldsymbol{E}^{*}(\boldsymbol{r}_{\star})\cdot\left(\xi^{2}\boldsymbol{G}(\boldsymbol{r}_{\star},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{E}\nabla\boldsymbol{E}(\boldsymbol{r}_{n})+i\xi\nabla\times\boldsymbol{G}(\boldsymbol{r}_{\star},\boldsymbol{r}_{n})\boldsymbol{\alpha}_{H}\nabla\boldsymbol{H}(\boldsymbol{r}_{n})\right)\right].$
(42)

##  Appendix B Comparison with a Genetic Algorithm

We compare the results of our optimisation for both power emission and
directivity with the results of a genetic algorithm solving the same problem.
Using the differential evolution algorithm Storn1997 , with a population size
of 20, and a maximum allowed iterations of 5000. The differential weight
parameter is $F=0.5$ and the crossover probability is CR = 0.7. This genetic
algorithm was run several times and the best solution selected. The comparison
between this result and the result of our local optimisation is shown in
Figure 5.

Figure 5: A comparison of the results of our optimisation and a genetic
algorithm seeking to shape a far–field radiation pattern while also improving
efficiency. The far–field radiation patterns are compared in a), and the
solution space paths are shown in b). The progress of our method is shown in
red, and the progress of the genetic algorithm as green dots. Each dot
represents a single population member. The final result of the genetic
algorithm is shown as a blue star. The resulting structures are shown in c)
and d).

The genetic algorithm produces a slightly higher power emission but a slightly
lower value of overlap integral. From the scatter of the solutions generated
by the genetic algorithm in solution space, shown in Figure 5, it is evident
that the genetic algorithm explores more of the search space than our local
optimisation. However, due to the size of the search space for
multi–functional problems, this does not provide much advantage.

##  Appendix C Validity of the Discrete Dipole Approximation

To verify the validity of the discrete dipole approximation, we compare our
results with full–wave solutions using a finite element method numerical
solver (COMSOL Multiphysics) comsol .

The beaming device shown in Figure 3 has been validated by considering the
nearest 20 scatterers to the source, due to memory considerations. For this
reduced system, the comparison between the discrete dipole approximation and
COMSOL is shown in Figure 6.

Figure 6: Comparison between our analytic results, based on the discrete
dipole approximation, and full–wave simulations in COMSOL. Considering only
the 20 (of 64) scatterers nearest to the source of the device shown in Figure
3 of the main paper, we compare a) the far–fields and b),c) the near–fields.
For these scatterers, 65 nm radius silicon spheres, good agreement with the
discrete dipole approximation is shown.

The scatterers here are silicon spheres of radius 65 nm, for which the
electric and magnetic polarisabilities can be found analytically.

Validation of the ‘lensing’ device shown in Figure 4 of the main paper is
shown in Figure 7. We consider a TE plane wave incident upon a small number of
metacubes.

Figure 7: Comparison between the discrete dipole approximation and COMSOL for
several metacubes under EM polarised plane wave incidence. Both a) the
scattered field in the far–field and b)-e) the near–fields are shown.
Excellent agreement between the analytics and full–wave simulations is found.

Comparing the near and far fields in Figure 7 the main difference is in the
field at the location of the scatterers, where the discrete dipole
approximation is not valid anyway. The PEC boundary condition on the metal
cubes in COMSOL ensures that the field inside the metacubes is zero. However,
in the analytics, the field on one of the scatterers is proportional to
$\boldsymbol{G}(\boldsymbol{r}_{n},\boldsymbol{r}_{n})$. The real part of this
expression diverges, while the imaginary part remains finite. The divergence
of the real part is what cases the difference in the fields at the scatterer
locations.

## References

  * (1) Munk, B. A.  _Frequency Selective Surfaces: Theory and Design_ (John Wiley and Sons, 2000).
  * (2) Leonhardt, U. & Philbin, T. G.  General relativity in electrical engineering.  _New J. Phys._ 8, 247 (2006).  URL https://doi.org/10.1088/1367-2630/8/10/247.
  * (3) Pendry, J. B., Schurig, D. & Smith, D. R.  Controlling electromagnetic fields.  _Science_ 312, 5781 (2006).  URL https://doi.org/10.1126/science.1125907.
  * (4) Galiffi, E. _et al._ Photonics of time-varying media.  _Adv. Photonics_ 4, 014002 (2022).  URL https://doi.org/10.1117/1.AP.4.1.014002.
  * (5) Ptitcyn, G. _et al._ Scattering from spheres made of time-varying and dispersive materials (2021).  URL https://arxiv.org/abs/2110.07195.
  * (6) Kadic, M., Milton, G. W., van Hecke, M. & Wegener, M.  3d metamaterials.  _Nat. Rev. Phys._ 1, 198–210 (2019).  URL https://doi.org/10.1038/s42254-018-0018-y.
  * (7) Pande, D., Gollub, J., Zecca, R., Marks, D. L. & Smith, D. R.  Symphotic multiplexing medium at microwave frequencies.  _Phys. Rev. Applied_ 13, 024033 (2020).  URL https://doi.org/10.1103/PhysRevApplied.13.024033.
  * (8) Frellsen, L. F., Ding, Y., Sigmund, O. & Frandsen, L. H.  Topology optimized mode multiplexing in silicon-on-insulator photonic wire waveguides.  _Opt. Express_ 24, 16866–16873 (2016).  URL https://doi.org/10.1364/OE.24.016866.
  * (9) Piggott, A. Y. _et al._ Inverse design and demonstration of a compact and broadband on-chip wavelength demultiplexer.  _Nature Photon._ 9, 374–377 (2015).  URL https://doi.org/10.1038/nphoton.2015.69.
  * (10) Li, S.-Q. _et al._ Phase-only transmissive spatial light modulator based on tunable dielectric metasurface.  _Science_ 364, 1087–1090 (2019).  URL https://doi.org/10.1126/science.aaw6747.
  * (11) Wiecha, P. R. _et al._ Evolutionary multi-objective optimization of colour pixels based on dielectric nanoantennas.  _Nature Nanotech._ 12, 163–169 (2017).  URL https://doi.org/10.1038/nnano.2016.224.
  * (12) Yeung, C. _et al._ Elucidating the behavior of nanophotonic structures through explainable machine learning algorithms.  _ACS Photonics_ 7, 2309–2318 (2020).  URL https://doi.org/10.1021/acsphotonics.0c01067.
  * (13) Stratton, J. A.  _Electromagnetic Theory_ (John Wiley and Sons, 2007).
  * (14) Shim, H., Kuang, Z., Lin, Z. & Miller, O. D.  Fundamental limits to multi-functional and tunable nanophotonic response.  _arXiv:2112.10816v1_ (2021).  URL https://arxiv.org/pdf/2112.10816.
  * (15) Molesky, S. _et al._ Inverse design in nanophotonics.  _Nature Photon._ 12, 659–670 (2018).  URL https://doi.org/10.1038/s41566-018-0246-9.
  * (16) Gerchberg, R. W. & Saxton, W. O.  A practical algorithm for the determination of the phase from image and diffraction plane pictures.  _Optik_ 35, 237–246 (1972).  URL http://www.u.arizona.edu/~ppoon/GerchbergandSaxton1972.pdf.
  * (17) Liu, M. _et al._ Multifunctional metasurfaces enabled by simultaneous and independent control of phase and amplitude for orthogonal polarization states.  _Light Sci. Appl._ 10, 107 (2021).  URL https://doi.org/10.1038/s41377-021-00552-3.
  * (18) Bendsøe, M. P. & Sigmund, O.  _Topology optimization: Theory, methods and applications_ (Springer, 2003).
  * (19) Lalau-Keraly, C. M., Bhargava, S., Miller, O. D. & Yablonovitch, E.  Adjoint shape optimization applied to electromagnetic design.  _Opt. Express_ 21, 21693–21701 (2013).  URL https://doi.org/10.1364/OE.21.021693.
  * (20) Purcell, E. M. & Pennypacker, C. R.  Scattering and absorption of light by nonspherical dielectric grains.  _Astrophys. J._ 186, 705 (1973).  URL https://doi.org/10.1086/152538.
  * (21) Powell, A. W. _et al._ Strong, omnidirectional radar backscatter from subwavelength, 3d printed metacubes.  _IET Microw. Antennas Propag._ 14, 1862–1868 (2020).
  * (22) Levine, H. & Schwinger, J.  On the theory of electromagnetic wave diffraction by an aperture in an infinite plane conducting screen.  _Commun. Pure Appl. Math_ 3, 355–391 (1950).
  * (23) Tai, C.-T.  _Dyadic Greens Functions in Electromagnetic Theory_ (IEEE Press, New York, 1993 (second edition)).
  * (24) Foldy, L. L.  The Multiple Scattering of Waves. I. General Theory of Isotropic Scattering by Randomly Distributed Scatterers.  _Physical Review_ 67, 107 (1945).  URL https://doi.org/10.1103/PhysRev.67.107.
  * (25) J. R. Capers, A. P. H., S. J. Boyes & Horsley, S. A. R.  Designing the collective non-local responses of metasurfaces.  _Communications Physics_ 4, 209 (2021).  URL https://doi.org/10.1038/s42005-021-00713-1.
  * (26) Shalev-Shwartz, S. & Ben-David, S.  _Understanding Machine Learning: From Theory to Algorithms_ (Cambridge University Press, 2014).
  * (27) Hwang, C.-L. & Masud, A. S.  _Multiple Objective Decision Making: Methods and Applications_ (Springer–Verlag, 1979).
  * (28) Comsol multiphysics v. 6.0, http://www.comsol.com.
  * (29) Arango, F. B. & Koenderink, A. F.  Polarizability tensor retrieval for magnetic and plasmonic antenna design.  _New J. Phys._ 15, 073023 (2013).
  * (30) Liu, X.-X., Zhao, Y. & Alù, A.  Polarizability tensor retrieval for subwavelength particles of arbitrary shape.  _IEEE Trans. Antennas Propag._ 64, 2301–2310 (2016).
  * (31) Raab, R. E. & de Lange, O. L.  _Multipole Theory in Electromagnetism_ (Oxford University Press, Oxford, 2005).
  * (32) Evlyukhin, A. B., Reinhardt, C. & Chichkov, B. N.  Multipole light scattering by nonspherical nanoparticles in the discrete dipole approximation.  _Phys. Rev. B_ 84, 235429 (2011).
  * (33) Evlyukhin, A. B., Reinhard, C., Evlyukhin, E. & Chichkov, B. N.  Multipole analysis of light scattering by arbitrary-shaped nanoparticles on a plane surface.  _J. Opt. Soc. Am. B_ 30, 2589–2598 (2013).
  * (34) Storn, R. & Price, K.  Differential evolution – a simple and efficient heuristic for global optimization over continuous spaces.  _Journal of Global Optimization_ 11, 314–359 (1997).  URL https://doi.org/10.1023/A:1008202821328.


11institutetext:  Dept. of Computer Science and Eng., Chalmers $\mid$
University of Gothenburg, Gothenburg, Sweden 22institutetext: Dept. of
Mechanics and Maritime Sciences, Chalmers University of Technology,
Gothenburg, Sweden 33institutetext: Zenseact AB, Gothenburg, Sweden

# Continuous Experimentation and Human Factors

An Exploratory Study

Amna Pir Muhammad 11 0000-0001-8328-4149 Eric Knauss 11 0000-0002-6631-872X
Jonas Bärgman 22 0000-0002-3578-2546 Alessia Knauss 33 0000-0003-4857-7784

###### Abstract

In today’s rapidly evolving technological landscape, the success of tools and
systems relies heavily on their ability to meet the needs and expectations of
users. User-centered design approaches, with a focus on human factors, have
gained increasing attention as they prioritize the human element in the
development process. With the increasing complexity of software-based systems,
companies are adopting agile development methodologies and emphasizing
continuous software experimentation. However, there is limited knowledge on
how to effectively execute continuous experimentation with respect to human
factors within this context. This research paper presents an exploratory
qualitative study for integrating human factors in continuous experimentation,
aiming to uncover distinctive characteristics of human factors and continuous
software experiments, practical challenges for integrating human factors in
continuous software experiments, and best practices associated with the
management of continuous human factors experimentation.

###### Keywords:

Continuous Experimentation Human Factors Human Factors Experiments Continuous
Human Factors Experimentation

##  1 Introduction

In today’s fast-paced software development environments, characterized by
competitive and unpredictable markets, there is a need to deliver and improve
products rapidly [31]. This urgency is intensified by complex customer
requirements and rapid technological advancements. Consequently, many software
companies have embraced or are transitioning toward continuous experimentation
[25, 35].

Continuous software experimentation 111Key terms of this study are defined in
Table 1. involves iteratively gathering user feedback and observing user
interactions[6]. With the growing significance of software in complex and
automated systems, continuous experimentation has become increasingly
prevalent across various industries. These systems require robust and
continuously evolving software [19]. Researchers have acknowledged that the
design for such systems is inherently complex and that a more comprehensive
understanding of the real world can be achieved by actively looking at the
system from a human factors perspective and not only a technical perspective
[2, 11].

In order to ensure the effectiveness, safety, and reliability of systems,
particularly complex software systems, it is desirable to provide more
holistic knowledge on human factors in continuous experimentation. Especially
for safety-critical systems, a human factors perspective may provide crucial
in-depth insights. Therefore, integrating human factors experimentation into
the continuous experimentation process promises to be a game changer [30, 16].
Human factors refer to the various aspects of individuals, including their
physical, cognitive, social, and emotional elements, all of which can
significantly influence their performance and interactions with systems [12].
Human factors experiments prioritize studying user behavior and involve
experiments with humans as participants [8]. We acknowledge that the concepts
of continuous software experimentation and human factors experiments overlap
to some extent (i.e., the latter can be a component of the former, and vice
versa), but in this study, we discuss them as separate entities as they come
from different domains and are likely to complement each other. However, to
understand whether HF experiments fit the continuous software experiment
practices, one needs to understand in detail where they differ, where they
overlap, and in what they can be integrated.

While the significance of human factors has been widely recognized [32, 24]
and continuous software experimentation methodologies are widespread in
industry and have received extensive research attention [7, 13], there remains
a research gap when it comes to incorporating human factors experiments into
the well established continuous software experimentation processes [30].
Consequently, further investigation is required to bridge this gap [22].

This research aims to address differences, associated challenges, and best
practices for integration of human factors experiments within the context of
continuous experimentation. The following research questions (RQs) are used to
guide our research:

RQ1:

What are main differences when comparing human factors experiments with
continuous software experimentation?

RQ2:

What are main practical challenges when managing human factors experiments in
continuous software experimentation?

RQ3:

What are best practices for managing human factors in continuous
experimentation?

The findings for RQ1 reveal that while both human factors and software
experimentation emphasize the significance of understanding user behavior and
needs, they differ in their approach. RQ2 highlights the challenges in
managing human factors experiments, pointing to complexities like GDPR
compliance, data collection issues, additional costs, and an industry scarcity
of experts. RQ3 focuses on best practices in this domain, emphasizing the need
to prioritize research based on product timelines, invest in actionable
metrics, maintain robust experimental infrastructure and documentation, and
including or transferring human factors knowledge.

The rest of this paper is structured as follows: We start with an overview of
definitions for key terms used in this paper in Section 2, which covers
background knowledge and related work as well. Section 3 presents the research
methodology, and Section 4 outlines the findings. Section 5 presents the
discussion and potential threats to validity. Finally, Section 6 concludes the
paper.

##  2 Background and related work

Key terms of this study can be interpreted differently depending on the
domain. Hence, for the scope of this study, we use the definitions provided in
Table 1.

Table 1: Definitions of key terms used in this study Term |  Definition
---|---
Continuous
(software)
experimentation |  An approach to support software development, where research and development activities are guided by iteratively conducting experiments, collecting user feedback, and observing the interaction of users with the system or services under development. The goal of continuous software experimentation is to evaluate features, assess risks, and drive evolution [6, 13, 35].
Human factors in
development |  The field that aims to inform developers by providing fundamental knowledge about human capabilities and limitations throughout the design cycle so that products will meet specific quality objectives. These capabilities and limitations include cognitive, physical, behavioral, psychological, social, effective, and motivational aspects [21, 12].
Human factors
experiments |  Investigations that focus on how human capabilities and limitations affect specific quality objectives during the interaction between humans and the system, service, or product under development. Thus, humans are part of human factors experiments and their behavior and perception/opinions (of, e.g., the system, service, or product under assessment) can impact the result and consequently the design of the system [28, 9].
Continuous human
factors experimentation |  An iterative approach in software development that evaluates how human capabilities and limitations impact specific quality objectives during user interactions. It involves ongoing experiments, user feedback, and observations to inform the design process and enhance user experience.

_Continuous Software Experimentation._ Agile development methodologies have
gained widespread popularity in software development due to their iterative
and collaborative nature [1]. These methodologies emphasize continuous
experimentation, which involves constantly testing and validating hypotheses
to make data-driven decisions throughout development [35]. This approach has
proven effective in optimizing software products and services.

Continuous experimentation is primarily applied in web-based systems, allowing
developers to analyze and deploy changes based on real-world data and user
preferences, rather than relying solely on simulations or the opinion of the
highest-paid person’s opinion (HiPPO) [14]. Leading technology companies like
Microsoft, Google, Facebook, and Booking.com utilize online controlled
experiments, also known as A/B tests, to evaluate the impact of changes made
to their software products and services [13, 7, 5].

Despite the numerous advantages of wide-ranging continuous software
experimentation, there are still several challenges that need to be addressed
during its implementation. Some of the major hurdles include cultural shifts
within development teams, slow development cycles, product instrumentation,
and the identification of appropriate metrics for measuring user experience
[15, 17]. Rissanen and Münch [26] confirmed these challenges and also found
that capturing and transferring user data becomes challenging due to legal
agreements.

_Human Factors and Experimentation._ By including human factors experiments
from the outset, it becomes possible to ensure system reliability and evaluate
the system considering real-world human constraints [28]. Human factors
experiments aim to understand how people interact with technology, products,
and systems to optimize usability, user experience, and overall performance
[12]. They commonly evaluate aspects such as user interface design, cognitive
workload, situation awareness, and user behavior [29, 10, 27, 33].

_Continuous Human Factors Experimentation._ In terms of testing and
experiments, there have been some initial efforts to integrate usability
testing and user-centered design practices into agile development, like for
example the approach proposed by Nakao et al. [23] to incorporate usability
testing throughout the agile development process. Despite these efforts,
research has emphasized the need for new processes and tools that empower
practitioners of human factors to promote usable and effective products in the
agile development environment [30] and the integration of human factors into
the well established continuous software experimentation practices used in
agile development [22].

Note that our research does not center around the impact of human factors on
employees or developers involved in the development processes, as mentioned in
[34]. Instead, our focus is primarily on the product itself. By conducting and
analyzing a series of semi-structured interviews, we aim to explore the
integration of human factors experiments within the context of continuous
experimentation in software development.

##  3 Methodology

Sampling: We conducted interviews with eight professionals (P1-P8). We aimed
for a broad sample of expert participants with high experience in human
factors, continuous experimentation, ideally in both fields. This criteria
however limits the number of available subjects. Thus, we accepted lower
participant numbers than initially planned and focused on interviewing a
smaller selection of leading experts in their respective fields for this
exploratory study.

We focused on recruiting industry participants from renowned organizations
such as Microsoft. Targeting those known for their impactful success stories,
to ensure a significant impact and obtain high-quality input. Our academic
interviewees have extensive experience collaborating closely with industry,
and their credentials include thousands of citations (h-index $>$ 35 in four
cases), providing them with a good overview of practices in the field that
supports our exploratory study goal.

Table 2 presents each participant’s role and experience level.

Table 2: Interviewees’ roles and relevant work experience (Experience level: Low= 0–5 years, Medium=5–10 years, High= More than 10 years).  ID |  Role |  Main
Domain |  Continuous experimentat. experience |  Human Factors experience
---|---|---|---|---
P1 |  SE Researcher |  Academia |  High |  Low
P2 |  Human Factors Researcher |  Academia |  Low |  High
P3 |  Human factors Engineer |  Industry |  High |  High
P4 |  UX Expert |  Industry |  High |  High
P5 |  Data Scientist |  Industry |  High |  Low
P6 |  SE/Human Factors Researcher |  Academia |  High |  High
P7 |  CS Researcher and IT Consultant |  Industry & Academia |  High |  High
P8 |  Human Factors Researcher |  Academia |  Low |  High

#####  Data Collection:

To gather comprehensive information for our study, we used a qualitative study
design inspired by Maxwell [20]. Our data collection involved conducting a
series of semi-structured interviews, following a predefined set of open-ended
questions while allowing flexibility to include additional follow-up questions
when necessary. The interview questions used can be found here.

The interviews were conducted online through Zoom, with each session lasting
around one hour. We obtained permission from the interviewees to record the
sessions, which we later transcribed and anonymized for analysis.

The interview questions were organized into three main categories. The first
set aimed to collect demographic information from the interviewees, as well as
confirming their experience working with continuous experimentation and human
factors. The second set focused on exploring the management of experimentation
in both software and human factors contexts. We used these question to get a
better understanding of the participants background, how and which experiments
they use and generally of the topic under study. Finally, we asked specific
questions related to human factors in continuous experimentation. We used the
entire data in our analysis and to answer our research questions.

We initiated each interview by providing a brief overview of the study to
establish a shared understanding and create a comfortable environment. We also
presented the basic terms and definitions relevant to the study topic, seeking
agreement from the interviewees. This approach aimed to establish a common
foundation for our discussions, minimize potential confusion, and ensure a
consistent standpoint when gathering participants’ perspectives. Notably, all
participants expressed agreement with our definitions, offering no suggestions
for improvement or indicating any discrepancies between their own
understanding and our proposed definitions as outlined in Table 1.

##### Data Analysis:

For the qualitative analysis, we employed the thematic analysis approach [4]
to identify themes and analyze the content. This approach consists of six key
steps. Initially, we comprehensively reviewed all the interview notes and
generated research-related memos. To facilitate the process, we employed Nvivo
initially and later transitioned to using the Miro board for enhanced
visualization. These tools allowed us to assign codes or labels to the text.
Through an iterative process, we refined the coding scheme to uncover
significant ideas and viewpoints. The codes were then analyzed and grouped
together to identify common patterns, thereby defining the themes.
Subsequently, we thoroughly reviewed and verified the themes that emerged from
the coding process, ensuring clarity, consistency, and addressing any
ambiguities, contradictions, or omissions.

##  4 Findings

We present our findings for each research question with primary themes and
their related sub-themes. Figure 1 gives an overview of the main themes.

Figure 1: Overview of key high-level themes identified from the interview
analyses.

### RQ1: What are main differences when comparing human factors experiments
with continuous software experimentation?

####  4.0.1 F1.1: Contextual Factors

##### Human Behavior vs. Technical Aspects:

Both software developers and human factors professionals recognize the
importance of an intuitive user perspective. They acknowledge that users have
varying levels of technical proficiency and may not be inclined to explore
complex features. However, human factors experts go a step further by
emphasizing the need to understand the underlying reasons for potential user
challenges. For example, these challenges could include over-trusting software
or avoiding it altogether due to fear or apprehension. To address these
concerns, human factors experiments are conducted to gain insights into human
behaviors, needs, and experiences. These experiments prioritize the user
perspective and strive to optimize user satisfaction and safety. On the other
hand, software experiments typically have a more technical development-centric
focus. This discrepancy in approach highlights the importance of adopting a
human-centric understanding of user behavior and needs, which may differ from
the primary focus of developers on technical functionalities.

> _“They can develop and test and design and maybe it doesn’t need to involve
> human, then it works fine, as soon as you add human, a whole set of
> questions & requirements come into place which needs to be considered.” _ —
> P8

Human factors experts primarily focus on observing and analyzing human
behaviors to collect data using different interaction metrics. Such an
environment poses inherent challenges due to numerous uncontrollable variables
at play. For instance, humans exhibit a learning effect that can significantly
impact the experimental results. Moreover, interpersonal communication and
feedback loops among participants may also influence their responses to the
experiments.

Conversely, continuous software experimentation primarily focuses on
monitoring system behavior rather than directly observing human behavior. Such
experiments collect data from performance indicators, system logs, issue
reports, or user interactions documented by the software. They are often
conducted under controlled conditions, emphasizing variables like reaction
time, resource usage, scalability, or software stability. We believe that
these differences are brought to a point by the following exemplary quote:

> _“The main difference between human factor and traditional experiments, for
> instance, is that humans have much more of a learning effect.”_ — P7

####  4.0.2 F1.2: Methodological Approach

##### Diverse Approaches in Experimentation:

The methodology for both human factors and continuous software experiments
varies depending on the nature and scope of the feature being tested. Various
techniques can be employed for both software and human factors experiments.

> _“If it’s a very small audience, then product teams can also choose actually
> to do some surveys and interviews they invite customers in. So it really
> depends on like what is the scope of the feature that you’re testing.”_ — P5

While some methodologies, such as surveys and interviews, can be utilized for
both software and human factors experiments, there are some notable
differences in how the results are analyzed and interpreted. We found that
while A/B experimentation is a dominant method in continuous software
experimentation, it is often only one of many methods used in human factors
experiments.

##### Qualitative and Quantitative:

Much like software experiments, human factors involve qualitative and
quantitative data analysis. However, the analysis of human factors experiments
leans more towards qualitative methods due to the complexity of measuring and
interpreting human behavior. Therefore, conducting effective human factors
experiments necessitates practitioners with a strong foundation in qualitative
methodologies and empirical work involving human participants. Such
practitioners are able to capture the rich and nuanced aspects of human
behavior and user experience. In contrast, continuous software experiments
often adopt a more quantitative approach, aiming to establish causal
relationships between independent and dependent variables, allowing for
statistical analysis. That said, a substantial part of human factors
experiments still involve collecting quantitative data, such as eye-tracking
data and performance data (e.g., in the automotive domain in terms of measures
of lane keeping, time gaps, etc., or task completion times considering desktop
software tools).

> _“If you have a background in quantitative experiments with technical
> systems, I would think you cannot do [human factors experiments] in a good
> way. You need some kind of background in doing empirical work with humans.”_
> — P6

### RQ2: What are main practical challenges when managing human factors
experiments in continuous software experimentation?

####  4.0.3 F2.1: Complexity in Sampling

##### Controlled vs. Uncontrolled Variables:

One aspect is the presence of a higher amount of uncontrolled variables in
human factors experiments. Numerous contextual factors cannot be fully
controlled, which poses challenges in ensuring comparability and measuring
variables. Lack of control over contextual factors also complicates the
analysis, as there may be numerous variables that cannot be fully controlled
or accounted for in the experiment.

> _“The other issue is control. I think you will look at situations where
> there are just a lot of context factors, there is just no way to control
> everything.”_ — P1

##### Statistical Analysis:

One challenge lies in the statistical analysis of the data. In certain cases,
conducting a rigorous statistical analysis may not be feasible due to the
nature of the human factors experiment. For instance, the research goal might
involve observing how people react in a particular situation without
quantifiable metrics, so conducting a traditional statistical analysis becomes
challenging.

> _“It might not be possible to do a proper statistical analysis because you
> might want to expose people to a certain situation and see what happens.”_ —
> P2

##### Participant Scarcity:

Another challenge in human factors experiments is the limited availability of
participants. Getting enough people to participate can be difficult, and the
scarcity of eligible participants further complicates the process. In
contrast, continuous software experiments, especially those conducted online,
can be performed on a larger scale. While involving as many participants as
possible is generally advised, practical limitations may hinder this goal.

> _“Often these studies are fairly small regarding the number of subjects.”_ —
> P6

####  4.0.4 F2.2: Nature of Experiments

##### Personal Information and GDPR Issues:

When conducting experiments, the collection of personal information can be
crucial for understanding human behavior and software performance. In
experimental research, collecting personal information is pivotal for
understanding both human behavior and software performance. This is
particularly evident in human factors experiments, where insights into how
individuals from varied backgrounds interact with technology are essential.
However, collecting this in-depth personal information presents challenges,
mainly due to privacy and ethical issues. The requirements of GDPR regulations
amplify these concerns, necessitating meticulous attention. While software
experiments might occasionally need such information, the emphasis is much
greater in human factors experiments.

> _“It is a bit hard. Like with the GDPR and everything. How to store stuff
> actually? It makes it a bit more complicated.”_ — P4

##### Prototype vs. Real Environment:

Our interviewees mentioned that, although experiments are typically carried
out using prototypes or simulators, human factors experts also advocate for
conducting experiments in the actual environment where the product will be
finally be used. Experiments conducted in real environments offer a more
realistic and authentic representation of how participants interact with the
product or system in their natural settings. Unlike prototype experiments,
where external factors can be tightly controlled, real environment experiments
expose participants to multiple variables and contextual factors that can
significantly impact human performance and behavior.

> _“Having design prototypes is one approach so that people get the vision
> behind. But testing in real cars, it makes it so difficult, which is, but
> also important, to go in that direction or to get more research done.”_ — P3

##### Expensive:

Human factors experiments are often perceived as more costly compared to
continuous software experiments. This perception stems from the direct
involvement of real humans participating in real-time scenarios. For instance,
experts in human factors often need to recruit participants for their studies,
compensating them for their time and effort, which can be a significant
expense. On the other hand, many continuous software experiments can gather
data online, reducing the need for physical presence and direct human
interaction, and direct payment. While continuous software experiments do have
associated costs—such as development, deployment, and server
infrastructure—these expenses are generally lower than those of human factors
experiments.

> _“We have to pay for this for facilities, we have to pay participants
> because we get people from the real world, and the preparations is quite
> prolonged.”_ — P8

####  4.0.5 F2.3: Integration in Development Lifecycle

##### Execution Time:

Managing and executing human factors experiments in agile development can be
challenging due to their inherent time-consuming nature. Unlike continuous
software experiments that typically run for at least a week, human factors
experiments often require more time to obtain meaningful results. The duration
of such experiments is influenced by the desired change in a metric being
measured. Obtaining timely results from human factors, that can be integrated
into ongoing projects without significant delays can be difficult, especially
in agile, short sprint-based, work flows.

> _“You do a sprint and then you need results to run it and assume you need
> these kind of results quickly. So not in three months. And that’s, I would
> say that’s the problem for integrating these kind of things.”_ — P2

##### Infrastructure Needs:

One challenge involves obtaining the necessary tools and setup to conduct the
desired tests. Ensuring that the basic infrastructure is in place to
facilitate the experiments can be a significant hurdle.

> _“If there’s getting the right tools and right setup, like the basics in
> place to even be able to test what you wanna test. That could be a
> challenge.”_ — P4

##### Too Few Human Factors Experts:

Many companies struggle with insufficient human factors expertise and limited
resources, which can hinder their ability to improve user experience. This
deficiency often leads to a few outliers (or even the development team itself)
having a disproportionate impact on the final product design. This concern
arises from the fact that there are too few human factors experts available,
which limits comprehensive evaluations and increases the risk of biased
results.

> _“So I think that’s what, what other companies are lacking actually: Enough
> human factors, people doing that kind of work.”_ — P3

##### Lack of Motivation:

Another challenge is that many individuals with a technical mindset often
overlook the importance of understanding human behaviors. This lack of
motivation can hinder the collection of relevant data and make it difficult to
address the complexities involved in studying human subjects.

> _“How can you influence people? I think that’s the number one thing.”_ — P1

### RQ3: What are best practices for managing human factors in continuous
experimentation?

####  4.0.6 F3.1: Research Methods and Practices

##### Prioritizing Hypotheses/Research Questions:

Prioritizing research questions and hypotheses based on the product timetable
and development sprints is a crucial aspect in agile development. By
identifying the experiments that have the most impact on design decisions and
user experience improvement, organizations can allocate resources efficiently
and gain valuable insights.

> _“The number of experiments that you can do is basically infinite. So the
> hardest part in running experiments is how do I prioritize running the most
> valuable experiments first. And, I think that’s where many companies
> struggle.”_ — P7

##### Metrics and Measurement Instrumentation:

Based on our interviewees, to enable informed decision-making it is essential
to invest in the development of meaningful metrics that align with the desired
outcomes. While simple interaction metrics like clicks or selections are
useful, it is important to go beyond them and capture success metrics related
to user sessions and product features. As one of our interviewees pointed out,
the value of experiments ultimately relies on having good metrics and making
significant investments in their development. Without such metrics,
experiments become less valuable as they fail to provide actionable results
for decision-making. It was also emphasised that developing and validating
such metrics can (and must be allowed to) take substantial time.

Another critical aspect is the measurement of various metrics that provide
insights into different aspects of the product under evaluation. It is worth
noting that interviewees stressed the significance of using proper measurement
methods to obtain valuable results for making informed decisions. To measure
different aspects of the product or system being evaluated, multiple metrics
should be computed simultaneously. These metrics should align with the goals
of the experiment and help determine what is reasonable to measure and what
constitutes a good outcome.

> _“But at the end of the day about experiments, it all boils down to metrics.
> If you don’t have good metrics and you don’t invest significantly into
> metrics, your experiments will not be valuable.”_ — P5

##### Results and Lessons Learned:

When determining whether to reuse or evolve experiments, the organization may
take several factors into consideration. These factors include the importance
of the findings, potential influencing factors, and information indicating
changes in the validity of previous results. The relevance of the results and
their impact on decision-making are carefully evaluated when planning
subsequent experiments. It was also mentioned that the decision to reuse
experiments is often driven by the interest and initiative of individuals
involved in the projects, rather than being a formalized process.

> _“There are sometimes factors that are influencing what’s factors that may
> confound the outcomes from one experiment such that we need to rerun it in
> order to make sure that the thing is still true.”_ — P7

####  4.0.7 F3.2: Infrastructure Needs

##### Experimental Setup:

The infrastructure should support the setup and integration of different
components required for the experiment. This includes ensuring that the
necessary tools and setups are in place to conduct the experiments
effectively. It may involve creating prototypes, simulating scenarios, or
integrating various hardware and software components to enable the desired
testing environment. Careful planning of the experiment is crucial.

> _“If there’s getting the right tools and getting the right setup or the
> right HMI, like the basics in place even to be able to test what you wanna
> test.”_ — P4

##### Traceability and Documentation:

Maintaining traceability and documentation throughout the experimental process
is important. This includes preserving initial design proposals that led to
the ideas being tested. Having a clear traceability trail helps in
understanding the decision-making process during the experiment and provides
valuable insights for product teams. Utilizing an experimentation platform
that incorporates this traceability is essential.

> _“So having some traceability on the decisions that led to what is being
> tested would be very helpful, I think, for product teams. And that should be
> part of the experimentation platform.”_ — P5

##### Collaboration and Management Support:

Our interviewees highlighted that infrastructure should facilitate
collaboration among different teams involved in the experiments. It should
provide a platform for coordinating activities, managing participants, and
ensuring the smooth execution of the experiments. Additionally, management
buy-in, support, and drive are also important factors to overcome obstacles
and successfully implement the infrastructure needed for human factors
experiments.

> _“Main obstacle is kind of like management, high management buy-in, and
> support and then like knowledge on how to design and collect it. So, to me,
> infrastructure would be something they [practitioners] would know how to
> solve that.”_ — P6

####  4.0.8 F3.3: Research skills

##### Roles and Responsibilities:

Our findings indicate that experiment management becomes a collaborative
effort within cross-functional teams in an agile environment. These teams
typically include data scientists, engineers, product managers, program
managers, and user researchers. Our findings also highlight the pivotal role
of data scientists in continuous software experiments and the need for
technical support from engineers in human factors experiments. Moreover,
considering a single role for responsibility, product managers are crucial in
deciding which experiments to run and ensuring that relevant metrics are
effectively measured. We learned that while the responsibility for managing
continuous human factors experiments can be shared within a team or primarily
held by the manager, it is crucial to recognize that specialized knowledge and
expertise are often necessary. Having human factors specialists in human
factors experimentation can greatly benefit the planning and management of
human factors experiments. Human factors specialists bring specialized
knowledge and expertise in research methodology, data analysis, and
experimental design to guide the team and ensure precise and accurate
experiments.

> _“I really think that it should be less of a single responsibility and more
> of a team responsibility.”_ — P7

##### Knowledge and Training:

A solid foundation of knowledge, theory, and models is essential to design and
evolve effective human factors experiments. Furthermore, establishing an
infrastructure to disseminate this knowledge and provide comprehensive
training to researchers and teams is crucial. Agile teams can conduct human
factors experiments with appropriate training and methodologies.

> _“A bit with training. If you follow a specific procedure, then I think it’s
> not a problem.”_ — P4

The training should cover experimental design, research methodology, human
factors principles, biases, usability evaluation methods, and research
methods. Although individuals inherently possess some understanding of human
behavior, training will help broaden their perspective.

##  5 Discussion

Continuous experimentation for web-based systems has received extensive
research attention [7, 13], however, the human factors aspect remains
relatively underexplored. This study explores the idea to bridge this gap by
discussing the integration of human factors experiments with continuous
experimentation. This promises to enable continuous experimentation even in
the domain of safety critical systems to a larger extent. Integrating human
factors experiments into continuous experimentation presents both benefits and
challenges [18]. For instance, these experiments can shed light on usability,
user experience, and decision-making [28]. Yet, they also pose challenges,
such as the need to execute experiments in real environments with real human
participants [3].

We confirm challenges highlighted by previous studies [15, 17, 26] that have
investigated challenges in continuous experimentation in general (e.g.,
cultural shifts and appropriate identification of metrics) also for the
integration of human factors into continuous experimentation. On top of that,
our findings introduce additional complexities when human factors are
integrated into the mix.

Moreover, our findings indicate that the integration of human factors in
continuous experimentation is currently lacking. One of the contributing
factors to this gap is the shortage of human factors experts available to
collaborate with teams engaged in continuous experimentation [21]. While these
teams conduct experiments tailored to their specific system components, they
often lack input from human factors specialists. Another factor is the usually
higher complexity of human factors experiments. On the fast pace of continuous
experimentation, this affects options for data collection and appears to cause
human factors experiments leaning towards qualitative data collection in this
context.

To effectively integrate human factors experiments into continuous
experimentation, companies should consider including human factors experts
within teams and raising awareness among developers about the importance of
incorporating human factors. The successful execution of human factors
experiments by teams requires developers to be skilled in empirical study
methods, enabling them to conduct impactful human factors experiments.

####  5.0.1 Threats to validity:

The interdisciplinary nature and vast scope of the fields involved introduces
a threat to Construct Validity in that various definitions exist for the same
terms, such as “human factors”. Consequently, different individuals may have
different interpretations. We have included clear definitions of the key
concepts in interviews and report to mitigate this threat and ensure a common
understanding of the fundamental concepts used in this study. Additionally,
experienced authors were involved in the study to address the risk of
construct validity. Their expertise assisted the first author in developing an
interview guide that effectively aligned with the study’s research objectives.
For Internal Validity, we implemented measures to reduce bias and confounding
variables, such as having multiple authors conduct each interview to minimize
personal bias. Due to the specialized scope and high demands on participant
expertise (human factors and continuous experimentation), we had to rely on
convenience sampling, taking into account both the profile and availability of
potential subjects. Consequently, the low number of participants introduces a
threat to External Validity. We aimed to mitigate this threat by aiming for
covering a wide range of roles, domains, and cultural backgrounds. Finally, to
ensure Reliability, we implemented various measures. Throughout the
interviews, we had multiple researchers present to enhance the reliability of
our data. Additionally, we provided used materials and a detailed analysis
process, enabling other researchers to replicate our methodology in diverse
contexts. Moreover, the authors actively engaged in discussions to maintain
consistency in the coding results. However, despite our efforts, we
acknowledge the possibility of some subjectivity in our analysis.

##  6 Conclusion

This qualitative exploratory study investigates the integration of human
factors with continuous experimentation. To effectively integrate human
factors experiments in continuous experimentation, there’s a pressing need for
upgraded infrastructure, improved developers’ awareness about the importance
of human factors, and training developers in empirical study methods essential
for effective human-centric experimentation.

By fostering interdisciplinary collaboration and promoting the integration of
human factors considerations into continuous experimentation, organizations
can enhance the user experience, and improve the quality of software and
systems. Future research should focus on developing frameworks and detailed
guidelines for effectively incorporating human factors into continuous
experimentation processes, leading to the creation of more user-centric, safe,
and acceptable systems.

##### Acknowledgements:

The authors express their gratitude to the interviewees for their valuable
time and insights. The project has received funding from the Marie Skłodowska-
Curie grant agreement 860410 under the European Union’s Horizon 2020 research
and innovation program.

## References

  * [1] Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile software development methods: Review and analysis. arXiv preprint arXiv:1709.08439 (2017)
  * [2] Boy, G.A.: Human-centered design of complex systems: An experience-based approach. Design Science 3, e8 (2017)
  * [3] Charlton, S.G., O’Brien, T.G.: Handbook of human factors testing and evaluation. CRC Press (2019)
  * [4] Clarke, V., Braun, V., Hayfield, N.: Thematic analysis. Qualitative psychology: A practical guide to research methods 3, 222–248 (2015)
  * [5] Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch, J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). pp. 770–780. IEEE (2017)
  * [6] Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The right model for continuous experimentation. Journal of Systems and Software 123, 292–305 (2017)
  * [7] Feitelson, D.G., Frachtenberg, E., Beck, K.L.: Development and deployment at facebook. IEEE Internet Computing 17(4), 8–17 (2013)
  * [8] Franklin, A.D.: What makes a’good’experiment? The British Journal for the Philosophy of Science 32(4), 367–374 (1981)
  * [9] Gandevia, S.: A human factor in ‘good’experiments. The British Journal for the Philosophy of Science 37(4), 463–466 (1986)
  * [10] Hancock, P., Caird, J.K.: Experimental evaluation of a model of mental workload. Human factors 35(3), 413–429 (1993)
  * [11] Hancock, P.A.: Some pitfalls in the promises of automated and autonomous vehicles. Ergonomics 62(4), 479–495 (2019)
  * [12] Human Factors and Ergonomics Society: Definitions of human factors and ergonomics (2023), https://www.hfes.org/About-HFES/What-is-Human-Factors-and-Ergonomics, accessed on 17 Feb, 2023
  * [13] Kevic, K., Murphy, B., Williams, L., Beckmann, J.: Characterizing experimentation in continuous deployment: a case study on bing. In: 39th International Conference on Software Engineering:(ICSE-SEIP). IEEE (2017)
  * [14] Kohavi, R., Henne, R.M., Sommerfield, D.: Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In: Proceedings of the 13th ACM SIGKDD. pp. 959–967 (2007)
  * [15] Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Henne, R., Ferres, J.L., Melamed, T.: Online experimentation at microsoft. Data Mining Case Studies 11(2009), 39 (2009)
  * [16] Lee, J.D., Seppelt, B.D.: Human factors in automation design. Springer handbook of automation pp. 417–436 (2009)
  * [17] Lindgren, E., Münch, J.: Software development as an experiment system: A qualitative survey on the state of the practice. In: Agile Processes in Software Engineering and Extreme Programming: 16th International Conference, XP. Springer (2015)
  * [18] Madni, A.M.: Integrating humans with and within complex systems. CrossTalk 5 (2011)
  * [19] Maruping, L.M., Matook, S.: The evolution of software development orchestration: current state and an agenda for future research. European Journal of Information Systems 29(5), 443–457 (2020)
  * [20] Maxwell, J.A.: Qualitative research design: An interactive approach. Sage publications (2012)
  * [21] Muhammad, A.P., Knauss, E., Bärgman, J.: Human factors in developing automated vehicles: A requirements engineering perspective. Journal of Systems and Software p. 111810 (2023)
  * [22] Muhammad, A.P., Knauss, E., Bärgman, J., Knauss, A.: Towards challenges and practices with managing human factors in automated vehicle development. In: 31st IEEE International Requirements Engineering Conference, (RE’23). IEEE (2023)
  * [23] Nakao, Y., Moriguchi, M., Noda, H.: Using agile software development methods to support human-centered design. NEC Technical Journal 8(3), 37–40 (2014)
  * [24] Norman, D.: Design of everyday things, revised and expanded: Basic books (2013)
  * [25] Olsson, H.H., Alahyari, H., Bosch, J.: Climbing the “stairway to heaven”–a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 2012 38th euromicro conference on software engineering and advanced applications. pp. 392–399. IEEE (2012)
  * [26] Rissanen, O., Münch, J.: Continuous experimentation in the b2b domain: a case study. In: 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering. pp. 12–18. IEEE (2015)
  * [27] Royer, M., Houser, K., Durmus, D., Esposito, T., Wei, M.: Recommended methods for conducting human factors experiments on the subjective evaluation of colour rendition. Lighting Research & Technology 54(3), 199–236 (2022)
  * [28] Sætren, G.B., Hogenboom, S., Laumann, K.: A study of a technological development process: Human factors—the forgotten factors? Cognition, Technology & Work 18, 595–611 (2016)
  * [29] Shneiderman, B.: Human factors experiments in designing interactive systems. Computer 12(12), 9–19 (1979)
  * [30] Steinberg, R., Grumman, N.: Human factors at the speed of relevance for agile engineering. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. vol. 66. SAGE Publications Sage CA: Los Angeles, CA (2022)
  * [31] Verhoef, P.C., Broekhuizen, T., Bart, Y., Bhattacharya, A., Dong, J.Q., Fabian, N., Haenlein, M.: Digital transformation: A multidisciplinary reflection and research agenda. Journal of business research 122, 889–901 (2021)
  * [32] Wickens, C.D., Gordon, S.E., Liu, Y., Lee, J.: An introduction to human factors engineering, vol. 2. Pearson Prentice Hall Upper Saddle River, NJ (2004)
  * [33] Williams, K.W.: Impact of aviation highway-in-the-sky displays on pilot situation awareness. Human Factors 44(1), 18–27 (2002)
  * [34] Yaman, S.G.: Initiating the transition towards continuous experimentation: empirical studies with software development teams and practitioners. Series of Publications A (2019)
  * [35] Yaman, S.G., Munezero, M., Münch, J., Fagerholm, F., Syd, O., Aaltola, M., Palmu, C., Männistö, T.: Introducing continuous experimentation in large software-intensive product and service organisations. Journal of Systems and Software 133, 195–211 (2017)