# Deep Reinforcement Learning for Multi-Objective Optimization: Enhancing Wind
Turbine Energy Generation while Mitigating Noise Emissions

Martín de Frutos<EMAIL_ADDRESS>Oscar A. Marino  David Huergo  Esteban
Ferrer  ETSIAE-UPM-School of Aeronautics, Universidad Politécnica de Madrid,
Plaza Cardenal Cisneros 3, E-28040 Madrid, Spain  Center for Computational
Simulation, Universidad Politécnica de Madrid, Campus de Montegancedo,
Boadilla del Monte, 28660 Madrid, Spain

###### Abstract

We develop a torque-pitch control framework using deep reinforcement learning
for wind turbines to optimize the generation of wind turbine energy while
minimizing operational noise. We employ a double deep Q-learning, coupled to a
blade element momentum solver, to enable precise control over wind turbine
parameters. In addition to the blade element momentum, we use the wind turbine
acoustic model of Brooks Pope and Marcolini. Through training with simple
winds, the agent learns optimal control policies that allow efficient control
for complex turbulent winds. Our experiments demonstrate that the
reinforcement learning is able to find optima at the Pareto front, when
maximizing energy while minimizing noise. In addition, the adaptability of the
reinforcement learning agent to changing turbulent wind conditions,
underscores its efficacy for real-world applications. We validate the
methodology using a SWT2.3-93 wind turbine with a rated power of 2.3 MW. We
compare the reinforcement learning control to classic controls to show that
they are comparable when not taking into account noise emissions. When
including a maximum limit of 45 dB to the noise produced (100 meters downwind
of the turbine), the extracted yearly energy decreases by 22%. The methodology
is flexible and allows for easy tuning of the objectives and constraints
through the reward definitions, resulting in a flexible multi-objective
optimization framework for wind turbine control. Overall, our findings
highlight the potential of RL-based control strategies to improve wind turbine
efficiency while mitigating noise pollution, thus advancing sustainable energy
generation technologies.

###### keywords:

Wind turbine , Deep Reinforcement Learning, Q-learning , Blade Element
Momentum Theory , Aeroacoustic , Brooks Pope and Marcolini , multi-objective
optimization , torque-pitch control

##  1 Introduction

In recent years, the aerodynamic design of wind turbines has undergone
significant advances, reaching near-optimal efficiency through substantial
investments in aerodynamic optimization, as well as advancements in
manufacturing techniques and materials. Consequently, the focus has shifted
towards addressing the issue of noise generated by wind turbines, which is
emerging as a competitive factor within the wind energy industry. Numerous
studies have explored the correlation between wind turbine sound power levels
and public-reported perceptions of annoyance (Wolsink et al., 1993; Pedersen
et al., 2009). Accurate prediction of wind turbine noise under real
operational and atmospheric conditions is crucial to design quieter turbines
and complying with imposed noise regulations (Davy et al., 2018). This
necessity underscores the importance of fast turn-around methods to
incorporate noise calculations into design and optimization processes, as well
as to assess noise in real time during operation. Such efforts not only
optimize wind resource utilization, but also minimize the impact on the
quality of life of nearby communities and wildlife.

Aerodynamic noise poses a significant limitation to further exploiting wind
energy resources. This type of noise results from the turbulent flow
interacting with the airframe, necessitating a detailed resolution of the flow
for accurate far-field noise prediction. However, computational fluid dynamics
(CFD) solvers, while capable of simulating the flow field, incur a high
computational cost which escalates further when resolving the acoustic field.
Consequently, numerical approaches to wind turbine noise prediction remain
challenging. Therefore, most noise prediction models for wind turbines are
based on aeroacoustic semi-empirical models rather than numerical simulations
(Wagner et al., 1996). Despite these obstacles, wind turbines remain an
essential component in the generation of clean and renewable energy. However,
effective control strategies are imperative to optimize their performance
under variable wind conditions.

Wind turbine control systems are designed to maximize energy generation while
ensuring structural integrity and safe operation (Njiri and Söffker, 2016;
Novaes Menezes et al., 2018; Andersson et al., 2021). Given the dynamic nature
of wind, adaptive control strategies are essential, with classic mechanisms
including adjustments to yaw angle and rotational speed, as well as blade
pitch angle modulation. Leveraging real-time wind measurements, turbine
dynamics, and advanced control algorithms enables simultaneous adjustments to
rotor speed, and pitch, enhancing energy generation, reducing fatigue loads,
and extending turbine lifespan.

The emergence of reinforcement learning (RL) presents novel opportunities for
wind turbine control by enabling data-driven adaptive decision-making (Le
Clainche et al., 2023; Garnier et al., 2021). RL, a machine learning approach,
involves an agent learning to make decisions in an environment to maximize
cumulative rewards over time (Sutton and Barto, 1998). Applied to wind
turbines, RL offers autonomous learning of control inputs to maximize power
generation by capturing complex non-linear relationships between wind
conditions, turbine states, and actions. RL-based control methods adapt in
real-time to changing wind conditions, offering significant advantages in wind
turbine operation.

This paper reviews recent advances in RL-based control strategies for wind
turbines, focusing on pitch angle and rotor speed modulation. Previous studies
have proposed RL algorithms with comprehensive reward definitions, showcasing
their efficacy in optimizing wind turbine performance under varying wind
conditions. For example Chen et al. (2020) proposed a RL pitch controller to
maintain the nominal power using an adaptive dynamic programming algorithm,
reducing the energy consumption of the pitch actuator. Xie et al. (2023)
developed a data-driven model to perform a torque-pitch controller, modeling
the dynamics and using RL to control the wind turbine. Sierra-Garcia et al.
(2020) discussed different reward definitions for wind turbine controllers,
while Puech and Read (2023) and Saenz-Aguirre et al. (2019) developed RL
methods for yaw control avoiding control parameter tuning. Kushwaha et al.
(2020) and Wei et al. (2016) employed Q-learning RL methods for maximum power
point tracking (MPPT) control of the generator speed. Overall, these studies
demonstrate the adaptability of RL systems to realistic wind conditions,
thereby enhancing overall energy generation and efficiency of wind farms.

In this paper, we introduce a reinforcement learning-based dynamic control
method designed to maximize power output while adhering to specified maximum
decibel levels. The paper is structured as follows. First, we summarize the
methodology in Section 2. There, we include the wind turbine model, validating
the aeroacoustic model with three different wind conditions. Additionally, the
multi objective reinforcement learning strategy is explained, we provide
details on the reward, the neural network architecture and the training
procedure. Second, in Section 3 we validate the controller with simple steady
winds to later challenge the method with turbulent wind conditions obtained
from experimental measurements. We end with conclusions and outlooks.

##  2 Methodology

In this section, we detail the methodology for integrating Deep Reinforcement
Learning (DRL) with the dynamic control of a wind turbine. We begin by
describing the model of the wind turbine, focusing on how both the power
output and the noise levels are computed, and validate the methodology using
field measurements for a SWT2.3-93 wind turbine with a rated power of 2.3 MW.
Subsequently, we detail the setup of the DRL algorithm, which is designed to
maximize power generation within specified noise constraints, demonstrating
the application of advanced machine learning techniques to real-world energy
optimization challenges.

###  2.1 Wind turbine modeling using OpenFAST

One critical requirement for incorporating a wind turbine solver into the DRL
control framework is the ability to perform rapid evaluations, as the DRL
training process requires a large number of simulations. To meet this need, we
have chosen to employ an efficient Blade Element Momentum Theory (BEMT)
solver. Specifically, we use OpenFAST (National Renewable Energy Laboratory,
2024), a well-known open-source software for simulating wind turbine dynamics
and acoustic predictions.

BEMT, is known for its efficiency and offers a simple yet accurate method for
estimating the aerodynamic forces and energy generation of wind turbines. Its
ability to perform rapid function evaluations is crucial for training and
validating the agent within a reasonable timeframe. In the realm of BEMT, the
wind turbine blade is segmented into smaller sections along its span. The
aerodynamic forces exerted on each section are computed based on the local
wind conditions and the airfoil’s geometry. These local flow conditions,
defined for every section and time step, encompass the wind’s speed and
direction, along with the turbulence intensity. Polar curves for each airfoil
section are used to compute the aerodynamic forces (lift, drag and moment
coefficients). By integrating the forces along the span of the blades, we can
derive the overall power and thrust generated by the wind turbine.

Additionally, OpenFAST includes an aeroacoustic module that enables the
computation of noise levels generated by the wind turbine at specific observer
locations. To determine the aerodynamic noise sources from wind turbine
blades, various semi-empirical noise models are included (Zhu et al., 2005)
and we select the Brooks Pope and Marcolini model. The sound pressure level
(SPL) for each blade segment is calculated based on individual noise
mechanisms. The cumulative effect of these mechanisms yields the noise source
for each airfoil. Finally, the noise sources from all blade segments are
combined as uncorrelated noise sources, contributing to the overall
computation of the wind turbine’s sound power level. The essential aspect of
this process is the precise identification and modeling of the various noise
mechanisms associated with each blade section. These mechanisms can be
categorized into two groups: turbulent inflow noise and airfoil self-noise.
OpenFAST implements the turbulent inflow model presented by Moriarty et al.
(2004) and, among the airfoil self-noise models described by Brooks et al.
(1989), we have specifically selected: turbulent boundary layer trailing edge
noise and tip vortex formation noise.

####  2.1.1 Validation of OpenFAST with a SWT2.3-93 wind turbine

The onshore wind turbine selected for the study is based on the SWT2.3-93,
which performs a rated power of 2.3 MW. This turbine has undergone extensive
in field experimental testing and complete details on its geometry and
benchmark data are available in the open access repository Zenodo (through the
work of the European project Christophe et al. (2022)). The airfoil polar
curves are available in the airfoil catalog compiled by Bertagnolio et al.
(2001). More details can be found in Matthew (2012).

All the open-source information enables us to create a SWT2.3-93 OpenFAST
model. Additionally, the benchmark results from the Zenodo dataset can be
utilized to validate the model. Figure 1 presents the validation of both the
power output and the sound pressure level (SPL) of the wind turbine. Figure
1(a) compares the experimental power curve (from the Zenodo database) with
that generated by the OpenFAST solver, showing good agreement. Meanwhile,
Figure 1(b) shows the one-third octave SPL for frequencies ranging 10Hz to
10kHz, comparing the Zenodo dataset with results from OpenFAST. These
comparisons cover three different operational conditions, detailed in table 1.
The Zenodo acoustic results are computed for an observer positioned 100 meters
downstream from the wind turbine, at ground level. We observe good agreement
with the experimental data for the three operating conditions. We conclude
that OpenFAST, with the acoustic model of Brooks Pope and Marcolini, provides
accurate predictions of power generation and acoustics, and is therefore a
valid tool to perform multi-objective optimization.

(a) Power curve for the SWT2.3-93 wind turbine. The manufacturer curve is
obtain from the Zenodo open access repository.

(b) One-third octave SPL diagram for three different operational points. The
measurements of the SPL spectrum are obtained from the Zenodo open access
repository.

Figure 1: Comparison of Zenodo dataset benchmarks and OpenFAST modeling of the SWT2.3-93 wind turbine. Operation point |  $\bm{U_{\infty}}$ (m/s) |  $\bm{\Omega}$ (rpm) |  $\bm{\theta}$ (degrees)
---|---|---|---
OP1 | 6 | 13 | 3
OP2 | 8 | 14 | $-2$
OP3 | 9.5 | 17 | 5
Table 1: Operational conditions studied on the Zenodo aeroacoustic dataset.
The operation point is defined by the wind speed $U_{\infty}$, the rotational
speed $\Omega$ and the blade pitch angle $\theta$.

####  2.1.2 Sensitivity to control parameters

The selected parameters to control the wind turbine power and noise include
the rotational speed $\Omega$ and the blade pitch angle $\theta$. Before
discussing the RL setup, it is crucial to illustrate the sensitivity of these
parameters for the two performance metrics: the power coefficient and the
overall sound pressure level.

In Figure 2 the sensitivity analyses for both rotational speed and blade pitch
angle are displayed for a single incoming wind speed ($U_{\infty}$). The
values for one-third octave SPL, power coefficient and overall sound pressure
level are shown for different operational conditions. Figure 2(a) depicts the
influence of $\Omega$, increasing the rotational speeds makes both the sound
pressure level and the power coefficient rise, highlighting the trade-off
between maximizing power and minimizing noise. The entire SPL spectrum
increases uniformly when increasing $\Omega$, due to a higher relative
velocity in the blades and leading to a rise in SPL regardless of the noise
mechanism. A similar analysis is presented in Figure 2(b) for the blade pitch
angle. Although the general conclusion about the opposition between power and
noise remains valid, the SPL spectrum behaves differently across frequencies.
The pitch angle mainly affects trailing edge noise leading to changes at
relatively low frequencies ranging from 10Hz to 1kHz.

(a) Sensitivity analysis for rotational speed.
Operational point : $U_{\infty}=12$ m/s and $\theta=-1^{\circ}$.

(b) Sensitivity analysis for pitch.
Operational point : $U_{\infty}=12$ m/s and $\Omega=16.5$ rpm.

Figure 2: Sensitivity analyses for control parameters in relation to OSPL,
power coefficient ($C_{p}$), and one-third octave SPL (dB A) spectra.

###  2.2 Design of a reinforcement learning control

Reinforcement Learning is a branch of machine learning that focuses on how
agents should take actions in an environment to maximize cumulative reward.
Unlike supervised learning, where the model learns from a labeled dataset, RL
is driven by agent-environment interactions. The agent takes actions based on
the current state of the environment and receives feedback in the form of
rewards. The state represents the situation of the environment at a given
time, while the actions are the possible moves the agent can make. The reward
is the feedback indicating the immediate benefit or cost of an action, guiding
the agent toward better actions over time. In particular, in this work we use
Q-learning RL, which is detailed in the next section.

####  2.2.1 Reinforcement Learning for Multi Objective Control

Q-learning is a widely recognized reinforcement learning algorithm (Watkins
and Dayan, 1992). It is categorized under model-free RL algorithms, implying
that it operates without the necessity for prior knowledge or explicit models
that represent the dynamics of the system. The fundamental component of
Q-learning is the Q-value, which quantifies the anticipated cumulative reward
for executing a specific action in a given state. The Q-value is updated
iteratively via the Bellman equation, which formulates the optimal action-
value function in terms of the maximum expected future reward. During the
learning process, the wind turbine interacts with the environment, transitions
between states, and takes actions according to its current policy. The
Q-learning algorithm employs an $\epsilon$-greedy exploration-exploitation
trade-off to strike a balance between exploring new actions ($\epsilon$ times)
and exploiting current knowledge ($1-\epsilon$ times) to maximize cumulative
rewards. In RL the cumulative reward is computed taking into account that a
reward received immediately is worth more than a reward received in the
future, specifically, each time step the reward is discounted by $\gamma$, the
discount rate.

Initially, the Q-values are arbitrarily initialized. As the wind turbine
explores the environment and receives feedback in the form of rewards, the
Q-values are updated using the temporal difference error. The temporal
difference error represents the discrepancy between the observed reward and
the predicted reward based on the Q-values. Through repeated iterations, the
Q-learning algorithm gradually converges to an optimal policy. In this state,
the wind turbine learns the best actions to take in different states, thereby
maximizing power generation while minimizing noise. In our case, this agent-
environment interactions for the wind turbine control are illustrated on
Figure 3.

StateActionObservationReward$s_{t}$(wind conditions, control
variables)$r_{t}$(Power, SPL)$a_{t}$$Q(s,a)$AgentEnvironmentOpenFAST+Wind
conditions Figure 3: Flow diagram of the reinforcement learning control
methodology.

Deep Q-Network (DQN) is a variant of Q-learning that employs a deep neural
network to estimate Q-values (Mnih et al., 2013). It replaces the traditional
lookup table with a neural network, enabling generalizations across states to
handle large state spaces efficiently. In this study, a Double Deep Q-Learning
(DDQN) is employed. DDQN is an extension of DQN that uses two neural networks:
the primary network and the target network. The primary network select the
action and the target network evaluates its Q-value. This way of decoupling
the action selection and evaluation addresses the overestimation of Q-values,
often observed in DQN algorithms due to the maximization bias, (Van Hasselt et
al., 2016). The weights of the primary network are obtained by minimizing the
following loss function:

$\mathcal{L}(\phi)=\mathbb{E}_{(s,a,r,s^{\prime})}\left[\left(r+\gamma
Q_{\phi^{\prime}}\left(s^{\prime},\arg\max_{a}Q_{\phi}(s^{\prime},a)\right)-Q_{\phi}(s,a)\right)^{2}\right],$
(1)

where $r$ is the reward, $s$ is the state of the environment, $a$ denotes a
possible action that the agent can take, $Q(s,a)$ is the Q-Function and $\phi$
and $\phi^{\prime}$ are the set of weights of the primary and target network,
respectively. The loss function $\mathcal{L}(\phi)$ quantifies the residual of
the Bellman equation, which defines formally the optimal values of the
Q-values, (Sutton and Barto, 1998). The set of weights from the target network
, $\phi^{\prime}$, is updated using a soft update rule to enhance the
stability of the learning process, (Lillicrap et al., 2015).

$\phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime}.$ (2)

To train the DDQN, an experience replay buffer is utilized. During the
training phase, the agent interacts with the environment and stores the
experiences (state, action, reward, next state) in the replay buffer.
Subsequently, random batches of experiences are sampled from the replay buffer
to train the network and update its weights. This process helps to break the
correlation between consecutive samples and improves stability during the
learning process.

An additional consideration in solving this reinforcement learning problem is
the need to balance maximizing power output with minimizing noise impact.
These objectives are inherently conflicting, placing this problem within the
Multi-Objective Reinforcement Learning (MORL) framework. MORL extends
traditional reinforcement learning to handle problems involving multiple,
often conflicting, objectives.

Various strategies exist for addressing MORL problems. One of the simplest
methods is to define the reward using a scalarized function that combines the
rewards for each objective into a single global reward, thus transforming the
problem into a single-objective reinforcement learning task (Van Moffaert et
al., 2013). Another approach involves Pareto optimization, which aims to find
a set of optimal policies that lie on the Pareto Front, where no other policy
is superior in all objectives (Van Moffaert and Nowé, 2014). There are already
methods that apply these MORL approaches using deep learning implementations
(Mossalam et al., 2016). In this work, a scalarized method is adopted to
define a reward that balances the two objectives of maximizing power and
minimizing noise.

####  2.2.2 State-action structure

The state of the agent must include all the necessary information about the
environment to enable the agent to take the best possible action. If the state
lacks relevant information, the agent may not be able to achieve optimal
performance. The state of the agent is defined by the incoming wind
conditions, specifically the wind speed, $U_{\infty}$, along with the control
variables of the wind turbine, which are the rotational speed, $\Omega$, and
the blade pitch angle, $\theta$. To fit within the DDQN framework, the state
space $\mathcal{S}$ needs to be bounded. Some variables (rotational speed and
pitch) are bounded by mechanical/structural limitations, whereas the wind
speed is bounded by physical range. Note that these can be tuned for specific
wind turbines and geographic sites. We include an additional constraint on the
tip speed ratio $\lambda=\frac{\Omega R}{U_{\infty}}$, with the blade radius
$R=46.5$ m, to ensure the correct behavior of the BEMT solver. The specific
values of all the constraints are outlined below:

  * 1.

$U_{\infty}\in$ [4,16] m/s,

  * 2.

$\Omega\in[6,18]$ rpm,

  * 3.

$\theta\in[-5,10]$ degrees,

  * 4.

$\lambda\in[3,12]$.

The actions available to the agent involve either increasing or decreasing the
control variables. Since Q-learning is defined for a discrete action space
$\mathcal{A}$, the control variables can only be adjusted by fixed increments.
Five distinct actions are defined: two for each control variable (one for
increasing and one for decreasing), and one for maintaining the current state
(doing nothing). The specific fixed increment for each possible action is
determined based on the sensitivity analysis detailed in Section 2.1.2. Since
the rotational speed is more sensitive to both power generation and sound
pressure level compared to the pitch angle, the incremental adjustments for
each variable has been designed so that their corresponding actions have
effects of the same magnitude. The actions that the agent can take are
specified as follows:

  * 1.

$a_{1}$: increase $\Omega$ by 0.5 rpm,

  * 2.

$a_{2}$: decrease $\Omega$ by 0.5 rpm,

  * 3.

$a_{3}$: increase $\theta$ by 1 degree,

  * 4.

$a_{4}$: decrease $\theta$ by 1 degree,

  * 5.

$a_{5}$: do nothing.

It is important to note that the transition between states is not
deterministic a priori. Although we can freely adjust the control variables,
the wind conditions depend on the environment and are beyond our control. This
motivates the use of a model-free reinforcement learning method, as model-
based approaches only guarantee convergence if the transition function between
states is known.

####  2.2.3 Reward definition

The reward function is key when defining the RL algorithm, as it is the only
feedback to quantify how successful are the actions taken by the agent.
Therefore, the reward must be carefully crafted for each specific problem to
learn an appropriate policy. As mentioned in Section 2.2.1, this is a multi-
objective optimization problem (or MORL), requiring a specific strategy to
address the two conflicting objectives: maximizing power extraction while
minimizing noise generation. In this work, we choose to blend the two
objectives through a linear function, to define the overall reward. The reward
can be expressed as follows:

$r=r_{\text{PW}}+r_{\text{SPL}},$ (3)

where $r_{\text{PW}}$ denotes the reward associated to the power objective and
$r_{\text{SPL}}$ the one related to the SPL one.

As we already discussed in our previous work Soler et al. (2024), the reward
power component should encourage the agent to obtain the highest energy
generation possible, regardless of the wind conditions. To achieve this, we
use the power coefficient $C_{p}$ of the wind turbine. We set this reward to
increase linearly from 0 to 1, with 1 corresponding to the maximum possible
value of the power coefficient within the state space, $C_{p,\text{nom}}$.
Therefore, the reward power component reads,

$r_{\text{PW}}=\frac{C_{p}}{C_{p,{\text{nom}}}}.$ (4)

The reward term related to sound generation, $r_{\text{SPL}}$, is highly
dependent on the specific problem being modeled. First, we need to select the
observer locations where the SPL is computed, typically in critical areas
where noise mitigation is a priority. In this work, we decide to set one
observer 100 m downstream the wind turbine, see Figure 5. Next, we decide how
to penalize the sound generation (SPL) in the reward function. We opt to use a
ReLU activation function that begins penalizing once the SPL exceeds a certain
threshold, $\text{SPL}_{\text{thr}}$. Below this threshold, the agent focuses
solely on maximizing power. Additionally, we define a $\Delta$dB value that
specifies how much the SPL threshold can be exceeded before the reward becomes
$-1$. Beyond this point, no matter how much power the agent generates, the
total reward will be negative. Therefore, $\text{SPL}_{\text{thr}}+\Delta$dB
serves as an effective noise limit. For this specific application, we defined
$\text{SPL}_{\text{thr}}=45$ dB and $\Delta$dB = 5 dB, but note that these
values can be adapted to specific sites or regulations. The reward noise
component can be seen in Figure 5 and reads as follows:

$r_{\text{SPL}}=-\text{ReLU}\left(\frac{\text{SPL}-\text{SPL}_{\text{thr}}}{\Delta\text{dB}}\right).$
(5)

Figure 4: Directivity map of Sound Pressure Level (SPL) generated by OpenFAST.
The wind direction is oriented along the positive $x$ axis (left to right),
perpendicular to the wind turbine rotor. The operational conditions are
$U_{\infty}=12$ m/s, $\Omega=16.5$ rpm and $\theta=-1$ degree.
The observer location ’$\times$’ is situated 100 m downwind. |  $-1$$\text{\small{SPL}}_{\text{thr}}$$\Delta$dBSPL$r_{\text{SPL}}$ Figure 5: Reward noise component.
---|---

In addition, we need to include the bounds of $\mathcal{S}$ in the reward. To
make the agent learn the limits, it receives punishments whenever it performs
a forbidden action, that is, an action that leads to a state
$s_{t+1}\notin\mathcal{S}$. In such cases, the agent receives a negative
reward with a value of $r=-3$ and the action is revoked so that the control
variables remain the same. The punishment is set to $-3$ to differentiate it
from the possible negative reward of $r_{\text{SPL}}$. This distinction is
made because exceeding the $\mathcal{S}$ limits is considered worse than
generating noise above the threshold. Finally, the reward function for the
agent is the following

$r(s_{t},a_{t},s_{t+1})=\begin{cases}\displaystyle\frac{C_{p}(s_{t+1})}{C_{p,nom}}-\text{ReLU}\left(\frac{\text{SPL}(s_{t+1})-\text{SPL}_{\text{thr}}}{\Delta\text{dB}}\right),\\\
\qquad-3\qquad\text{if }s_{t+1}\notin\mathcal{S}.\\\ \end{cases}$ (6)

####  2.2.4 Neural Network architecture

When using DQN, neural networks (NN) are employed to approximate the
Q-Function. Typically, the NN is designed to approximate the Q-Vectors,
$\bm{q}(s)$, which represent the Q-Values in the state $s$ for all possible
actions. That is, $\bm{q}(s)_{i}=Q(s,a_{i})$. This approach is used because
$\mathcal{A}$ is a discrete space, and encoding these discrete actions as
inputs can be problematic; it is more convenient to create a mapping between
real subspaces. The neural network map is defined as
$\bm{q}_{\phi}(\bm{s}):\mathcal{S}\subset\mathbb{R}^{3}\to\mathbb{R}^{5}$,
where $\phi$ denotes all the NN weights, the output space dimension is
$|\mathcal{A}|=5$ and $\bm{s}$ denotes the state vector, which is
$\bm{s}=[U_{\infty},\Omega,\theta]^{T}$.

The neural network architecture employs a Multi-Layer Perceptron structure,
consisting of two dense hidden layers with Rectified Linear Unit (ReLU)
activation functions. The final layer, uses a linear activation function
instead of ReLU. This allows the Q-values to take on any sign, rather than
being restricted to positive numbers. The number of dense layers and their
sizes were determined through extensive trial and error. Ultimately, two
layers with 128 and 64 neurons, were found to be sufficient to accurately
represent the Q-Function. The architecture of the Q-Network used to train the
DDQN agent is shown in Figure 6.

$\vdots$$\vdots$$\vdots$$\vdots$$\bm{s}$128 neurons64
neurons$\bm{q}_{\phi}(\bm{s})$$Q(s,a_{1})$$Q(s,a_{2})$$\vdots$$Q(s,a_{5})$
Figure 6: Q-Network architecture.

####  2.2.5 Training of the Wind Turbine DDQN Agent

The main ideas of Q-Learning have already been explained in Section 2.2.1.
However, here the specific details of the DDQN training to design the wind
turbine controller are included. During the training phase, the agent faces
random steady wind conditions during short episodes of 20 time steps. This
allows the agent to adapt to virtually any wind, even if the wind speed
changes faster than 20 time steps. This adaptability is achieved because
experiences are stored in the replay buffer, and batches are selected
randomly. Consequently, the specific temporal evolution of states during the
agent’s experience is not critical, provided that the stored transitions
comprehensively represent all actual transitions in the system.

1: Initialize primary network $Q_{\phi}$ with random weights $\phi$

2: Initialize target network $Q_{\phi^{\prime}}$ with weights
$\phi^{\prime}=\phi$

3: Initialize replay buffer $\mathcal{R}$

4: Set hyperparameters: $\alpha$ (learning rate), $\gamma$ (discount factor),
$\epsilon$ (exploration probability), $\tau$ (soft update parameter), $N$
(number of training iterations), $n$ (number of steps taken every training
step, $m$ (period of update of the target network).

5: Initialize random state $s$

6: for iter=1 to N do

7: for n steps do

8: Select action $a$ using $\epsilon$-greedy policy from $Q_{\phi}$

9: Take action $a$, compute reward $r$ and new state $s^{\prime}$ using the
wind turbine solver

10: Store transition $(s,a,r,s^{\prime})$ in replay buffer $\mathcal{R}$

11: if last step of the episode then

12: Initialize random state $s$

13: else

14: Update current state: $s\leftarrow s^{\prime}$

15: end if

16: end for

17: Sample random mini-batch of transitions
$(s_{j},a_{j},r_{j},s_{j}^{\prime})$ from $\mathcal{R}$

18: Estimate loss function from eq. 1 with the mini-batch transitions

19: Update the primary set of weights $\phi$ using a gradient optimizer.
$\phi\leftarrow\phi-\alpha\nabla_{\phi}\mathcal{L}$

20: each m iterations do Soft update of the target network weights
$\phi^{\prime}$ using eq. 2 rule.

21: end for

Algorithm 1 Double Deep Q-Learning Algorithm

The Double Deep Q-Network (DDQN) agent is trained using the hyperparameters
listed in Table 2. To illustrate the influence of each parameter on the
training process, a pseudocode for the DDQN training is presented in Algorithm
1. The effectiveness of the learning progress during training is assessed by
displaying the Q-values of the state-action pairs encountered by the agent, as
shown in Figure 7. As the agent learns, the Q-values of the actions taken at
each state are expected to increase, as depicted in the figure. For the
implementation, we utilized OpenAI Gym (Towers et al., 2023) to create the
environment, serving as a bridge between the reinforcement learning
formulation and the OpenFAST wind turbine solver. TF-Agents (Guadarrama et
al., 2018) was employed to develop the agent and manage the entire training
process. All neural networks were constructed using the Keras API (Chollet et
al., 2015), and the Adam optimizer was used for training (Kingma and Ba,
2014).

Figure 7: Average Q-Values taken during the training process of the DDQN
agent.

Parameter | Value
---|---
Environment interactions | 200k
Steps of the environment per training iteration | 5
Maximum capacity of the replay buffer | 50k
Batch size | 64
Learning rate | 5e-4
Discount factor | 0.95
Epsilon value for the epsilon greedy policy | 0.50
Tau soft update parameter | 0.1
Period of update of the target network | 20

Table 2: Training parameters used for the DDQN agent.

##  3 Results

The agent’s performance is assessed under various wind conditions. First, we
validate the operational point that the agent reaches under constant wind
conditions, assessing its optimality via a Pareto front. Next, we evaluate the
agent’s ability to adapt to turbulent wind conditions, comparing its control
strategy against a classic controller. Finally, we estimate the agent’s annual
energy production and compare it against a classic control strategy designed
to maximize energy extraction.

###  3.1 Steady wind validation

The simplest test for evaluating the agent is to assess its performance under
steady wind conditions. In this scenario, with unchanging wind conditions, the
agent should identify, reach and maintain the state that maximizes the
cumulative reward. This optimum state does not depend on the initial
conditions of pitch and rotor speed, as the wind conditions are steady.
However, it is important to note that the agent’s actions are discrete,
limiting its ability to reach every possible state.

To validate the performance and robustness of the agent, a Pareto diagram is
used. The agent is tested for different initial conditions (with the same
steady wind speed) to determine if it can consistently reach optimal states
(at the pareto front), regardless of the initial state. To illustrate the
agent’s trajectory (sequence of state-action-reward) for each initial
condition, these trajectories are displayed on a power coefficient - sound
pressure level diagram, along with values for 1000 random states from
$\mathcal{S}$. Figure 8 presents the Pareto diagram, illustrating the agent’s
trajectory from four different initial conditions. It is clear that,
regardless of the initial condition, the agent successfully achieves high
power outputs up to the maximum permissible decibel level. Furthermore, the
agent demonstrates robust performance by consistently avoiding the maximum
limit of SPL (dB A) while remaining close to the limit to maximize power. It
can be seen that the RL does not always reach the same final state, but that
the optima are relatively close to each other. This suggests the existence of
local optimum. In addition, the discrete nature of the Q-learning actions, may
not allow the agent to reach certain optima, since not all states are
reachable from an initial state. Despite these issues, the agent consistently
avoids acoustic penalties and achieves high power outputs, with power
coefficients ranging from 0.26 to 0.30.

For completness, Table 3 shows the initial conditions of the control variables
for the trajectories displayed on Figure 8, as well as the final state control
variables.

Figure 8: Pareto diagram at a wind speed of 10 m/s, showing the power
coefficient $C_{p}$ and sound pressure level SPL (dB A) computed for 1000
random states within the state space. Different agent trajectories are
displayed on the Pareto diagram. The numbering of the initial conditions
correspond to Table 3.

Case |  $\mathbf{\Omega_{0}}$ (rpm) |  $\mathbf{\Omega_{f}}$ (rpm) |  $\mathbf{\theta_{0}}$ (°) |  $\mathbf{\theta_{f}}$ (°)
---|---|---|---|---
Case 1 | 9.61 | 11.11 | 9.94 | 5.93
Case 2 | 17.98 | 10.98 | 3.87 | 3.87
Case 3 | 8.43 | 10.93 | -1.25 | 2.75
Case 4 | 16.54 | 11.04 | 8.37 | 5.37

Table 3: Four different cases evaluated under steady wind speed of 10 m/s. The
initial condition is defined by the initial control variables $\Omega_{0}$ and
$\theta_{0}$ and the agent get to the optimum defined by the control variables
$\Omega_{f}$ and $\theta_{f}$.

Overall, the agent robustness has been tested on a simple steady wind
scenario. The agent is able to reduce the wind turbine noise to admissible
levels while maximizing power. Furthermore, the agent finds optimum
operational conditions regardless of the initial condition, showing the
robustness of the algorithm. In other words, the neural network which
approximates $Q(s,a)$ has covered all his input space
$\mathcal{S}\times\mathcal{A}$.

###  3.2 Control Strategy for Experimental Winds

The agent capabilities are now tested over experimental wind conditions. We
compare the energy extraction between our agent and two controllers that are
designed solely to maximize power. By doing so, we can demonstrate how much
power we need to sacrifice to keep the wind turbine at acceptable decibel
levels. The performance of the three controllers are going to be compared.
These controllers include:

  * 1.

Classic wind turbine controller: Standard wind turbine controller designed to
reach the power curve of the wind turbine, using torque or pitch control
depending on whether the wind speed is above or below rated wind speed.
Details can be seen in B.

  * 2.

Power DDQN: Agent designed to maximize solely power. It is trained with no
noise penalization, that is only the power reward is included, see eq. 4.

  * 3.

Quiet DDQN: Agent designed to maximize power without producing more that 45 dB
decibel levels at 100 m downwind of the rotor. It is trained with the complete
reward definitions including power and noise, see eq. 6.

The wind data used to validate the control performance under real wind
conditions was obtained from the measurement and instrumentation data center
(MIDC) of NREL, see Jager and Andreas (1996). These daily wind measurements
are available as open-source. For this study, wind speed and wind direction
measurements at an 80 m height from June 1, 2023, to June 1, 2024, are
selected. Figure 9(a) displays a wind rose illustrating the wind speed and
wind direction of this dataset. Since this study concentrates on torque-pitch
control, it is assumed that the incoming wind is consistently aligned with the
wind turbine, a condition typically managed by yaw control. Therefore, we
assume perfect alignment and only the wind speed distribution is used in the
subsequent results.

Note that the Power DDQN agent considers power optimization uniquely.
Therefore, it is only applicable to the below rated wind speed region defined
on B. When the wind speed exceeds the rated value, the control strategy
maintains nominal power rather than maximizing it. To achieve this behavior
with a reinforcement learning agent, the reward function would need to be
modified. Consequently, the wind speed distribution used to validate the agent
under turbulent wind conditions is restricted to below-rated wind speeds,
enabling a meaningful comparison between the three controllers.

The control performance of the three agents is analyzed in detail over an
8-hour time span, using the wind speed distribution shown in Figure 9(b). The
RL controllers are allowed to control each minute. In the next section, we
will estimate the annual energy production for all controllers.

(a) Wind rose for one year of data.

(b) Wind speed distribution for 8 hours.

Figure 9: Wind conditions on the experimental wind environment.

Figure 10 shows the results from the different controllers over the first 8
hours of the yearly dataset. Figures 10(a) and 10(b) display the control
parameters evolution while Figures 10(c) and 10(d) illustrates the power and
the noise generated 100 m downwind. It is noted that the Power agent matches
the power extraction achieved by the classic control strategy, essentially
implementing the same control approach but with the discrete actions defined
for the reinforcement learning agent. Since neither of these controllers is
designed to consider the acoustics of the wind turbine, both generate high
levels of noise when the wind speed is sufficiently high. In contrast, the
Quiet agent can match the power generation of the power-oriented controllers
when the wind speed is moderate. When wind speeds are higher, it extracts as
much power as possible while keeping noise levels below the threshold value.
Moreover, all three controllers maintain a constant pitch value to maximize
power extraction. However, the Quiet agent adjust the pitch angle to reduce
the noise levels when the wind speeds get higher.

(a) Rotational speed control.

(b) Pitch control.

(c) Power generation.

(d) Sound Pressure Level of the wind turbine.

Figure 10: Control results for the three different agents on the 8 hour
experimental wind environment.

This test shows the flexibility of the RL strategy for control and highlights
the possibility of including multi-objectives. In addition, we see that there
is no need to have an a priori knowledge of the turbine performance (e.g., the
power curve or rated maximum power) since the RL will learn these
characteristics when trained.

###  3.3 Annual wind energy estimation

There are different methodologies to obtain an estimate of the annual
generation of wind energy, (Garcia-Bustamante et al., 2009). The standard
procedure is based on decoupling the wind turbine from the wind distribution
of the particular site. It considers the observed wind speed frequency
histogram to fit a theoretical probability density function (PDF) for the wind
speed. It also requires a transfer function that models the relation between
power output and wind speed. Typically, the Weibull distribution is used to
fit the wind speed frequency histogram. García-Bustamante et al. (2008) showed
that although the Weibull distribution may not be substantiated for most
sites, it does not include important errors on the energy estimations. The
probability density function of the Weibull distribution is given by:

$f_{U}(u)=\left(\frac{k}{c}\right)\cdot\left(\frac{u}{c}\right)^{k-1}\exp{\left(-\left(\frac{u}{c}\right)^{k}\right)},$
(7)

where $U$ denotes the random variable that models the wind speed. The fit of
the shape and scale parameters $k$ and $c$ are established from the mean and
variance of the wind speed, $\mu_{U}=\mathbb{E}[U]$ and
$\sigma_{U}^{2}=\mathbb{V}[U]$. The specific relations can be seen in Spiru
and Simona (2024) work and are the following:

$k=\left(\frac{\sigma_{U}}{\mu_{U}}\right)^{-1.086},$ (8)

and

$c=\frac{\mu_{U}}{\Gamma\left(1+\frac{1}{k}\right)},$ (9)

where $\Gamma$ denotes the special gamma function. This formulation can be
employed to obtain the Weibull probability density function that represents
the one-year experimental data reported by Jager and Andreas (1996). This is
illustrated in Figure 11.

Figure 11: Wind speed frequency histograms of one year experimental data and
PDF of the adjusted Weibull distribution. The shape and scale parameters are
$k=1.195$ and $c=4.837$ respectively.

Regarding the transfer function between power and wind speed, various
strategies exist (Abolude and Zhou, 2018). The Theoretical Power Curve (TPC)
does not account for control mechanisms. Furthermore, since our wind turbine
control strategy considers acoustic generation, the wind turbine will exhibit
a significantly different Effective Power Curve (EPC). It is necessary to
compute an EPC that accurately represents the transfer function between power
and wind speed for our specific control scenario. The EPC can be computed
using simulations of the wind turbine control. The agent is faced against a
turbulent wind that covers all the range of interest of wind speed, mainly
between cut-in and cut-off wind speed. This turbulent wind must be
representative of the turbulent nature of the wind that the wind turbine is
going to face during operation. Once the simulation is done, all the pairs of
data points $(U_{\infty},C_{p})$ can be used to obtain a transfer function for
the power coefficient $C_{p}(U_{\infty})$. A subset of 100 hours of the
experimental wind measurements from the MIDC (Jager and Andreas, 1996) has
been used to obtain the EPC of the SWT2.3-93 wind turbine using the Classic
Control and the Quiet DDQN agent already introduced on Section 3.2. Figure 12
illustrates the results of this simulations, showing the operational laws of
control for each agent on Figures 12(a) and 12(b) and the SPL and power
associated on Figures 12(d) and 12(c) respectively. The behavior is as
expected, the Quiet Agent does not increase the rotational speed above 10.5
rpms to avoid surpassing the SPL threshold and uses the pitch to reduce noise
if needed, which explains the high variance bars on the pitch (see fig. 12(b))
and low ones in the rotational speed (see fig. 12(a)). Meanwhile the classic
control can increase the rotational speed freely and the pitch is only use in
above-rated wind speed scenarios, see B. The large standard deviations on the
classic control pitch are due to the PID control, which is dynamically
adjusting to the turbulent wind. In Figure 12(c) it is illustrated how the
classic control matches on average the TPC. However, it is not able to adjust
perfectly to the turbulent wind, showed by its high variance on the above-
rated wind speed region. The Quiet agent achieves less power than the classic
one but is able to maintain the sound pressure level below the specified
threshold of 45 dB A, see fig. 12(d).

(a) Rotational speed empiric operational curve.

(b) Pitch empiric operational curve.

(c) Effective Power Curve (EPC) of both control strategies against the
Theoretical Power Curve (TPC) from the manufacturer.

(d) Sound Pressure Level of the wind turbine observed at 100 m downwind for
both control strategies against the SPL threshold.

Figure 12: Average values of the control variables for wind speed bins,
obtained from simulating two different control agents over a 100 hour
experimental wind speed period. The error bars denote the standard deviation.

Traditional approaches use only the average value or polynomial fits of the
historical/simulated data to construct the EPC. All these methods do not
capture the variance of the data in the model. To account for this variability
on the EPC model we introduce a statistical method. For simplicity, we model
the power coefficient $C_{p}(U_{\infty})$, which is obtained by non-
dimensionalizing the EPC data. A Gaussian Process Regression (MacKay et al.,
1998) can be employed to model the power coefficient at each wind speed as a
Gaussian probability distribution,
$C_{p}(U_{\infty})\sim\mathcal{N}[\mu_{C_{p}}(U_{\infty}),\sigma_{C_{p}}(U_{\infty})]$.
Figure 13(a) shows the power coefficients points obtained after the wind
turbine control simulation. This data is used to fit the Gaussian Process (GP)
model and obtain the mean and standard deviation of the power coefficient as
functions of the wind speed, this fit is also included in Figure 13(a). The
$C_{p}$ distribution for specific values of the wind speed is illustrated on
Figure 13(b) where it is compared with the histogram of the power coefficient
from the data.

(a) Power coefficient EPC. The data points represent all the observed
$(U_{\infty},C_{p})$ pairs observed during the simulation performed to obtain
the EPC. The Gaussian Process Regression fit, mean and confidence interval is
included.

(b) Comparison of the $C_{p}$ frequency histograms and the Gaussian
probability distribution ( ) obtained with the GP for different wind speed
values. The wind speed values for each histogram are specified in Figure 13(a)
with vertical colored lines. The order is the following:
Top Left:  , Top Right:  , Bottom Left:  , Bottom Right:  .

Figure 13: Power coefficient obtained from simulating the Quiet DDQN
controller over a 100 hour experimental wind speed period.

Assuming the Weibull probability distribution for the wind speed $U_{\infty}$
combined with the GP model for the $C_{p}$ distribution, the estimation of the
annual wind energy can be performed by computing the expectation of the wind
energy $E_{w}$. Mathematical details on the statistical distributions are
provided in A. Table 4 presents the annual energy estimation for each control
strategy. It is important to note that the Power DDQN controller is applicable
only in the below-rated wind speed region. Therefore, when computing the
annual wind energy generation with this control, we impose nominal power for
wind speeds above the rated value. The Power DDQN controller is included in
the comparison to ensure that it remains competitive with the Classical
Control within the below-rated wind speed range. The Quiet DDQN controller is
able to control the wind turbine without surpassing the sound pressure level
threshold selected and obtains an 78% of the annual energy production compared
to the Classical Control. Additionally, in Table 4 it is shown the average
standard deviation across the wind speed for the power coefficient GP fit.
There, it is shown that the Quiet agent exhibits the control strategy with the
least variance.

Controller |  $\mathbf{E_{w}}$ [MWh] | $\mathbf{\hat{\sigma}_{C_{p}}}$
---|---|---
Quiet DDQN | 2722 | 0.024
Power DDQN | 3541 | 0.042
Classic Controller | 3458 | 0.057
Table 4: Annual wind energy generation for the three different controllers.
Additionally, the average standard deviation across wind speed for each
controller is presented,
$\hat{\sigma}_{C_{p}}=\int_{U_{\text{in}}}^{U_{\text{off}}}\sigma_{C_{p}}(u)f_{U}(u)du$,
details on A at eq. 14.

##  4 Conclusions

In conclusion, integration of reinforcement learning with wind turbine control
holds promise for optimizing energy generation and efficiency while minimizing
acoustic environmental impact. A DDQN reinforcement learning agent can
replicate the control strategy of a standard wind turbine controller without
prior explicit knowledge of the wind turbine, relying solely on a wind turbine
solver for experiential learning. Moreover, advanced control strategies can be
readily implemented by modifying the reward function. In this work, an RL
controller is defined to maximize power output while maintaining acceptable
decibel levels, thereby incorporating acoustic effects into the control
strategy. This demonstrates that MORL is capable of dynamically balancing two
different objectives effectively.

An effective power curve is computed from control simulations of turbulent
wind data. This allow to characterize the reinforcement learning control
strategy, obtaining the operational laws and obtaining an annual wind energy
estimation. The methodology is validated using a SWT2.3-93 wind turbine with a
rated power of 2.3 MW. We evaluate the yearly energy production for a
realistic site. The DDQN reinforcement learning control provides similar
energy production that a traditional control. The methodology presented allows
for the inclusion of noise limits leading to a 22% reduction in the annual
energy extraction when activating a maximum allowed noise of 45 dB (100 meters
downwind downwind of the turbine).

Further research directions include investigating Multi-Agent Reinforcement
Learning algorithms for cooperative control of wind turbines within farms,
which could enhance overall system performance while controlling noise at the
farm level.

##  Appendix A Statistical details for the EPC model

Let us consider a two dimension random variable $\mathbb{X}=(C_{p},U)$. This
random variable model the probability of obtaining a certain wind speed with a
certain power coefficient. The wind speed marginal distribution accounts for
the global wind speed distribution of the localization of the wind turbine.
Meanwhile, the power coefficient distribution measures the performance of the
wind turbine at different wind speeds.

There exist an a priori unknown joint probability density $f(c_{p},u)$. The
power generated by the wind turbine, $P$, is a function of this random
variable, so it is itself a random variable,

$P=\frac{1}{2}\rho AC_{p}U^{3}.$

The wind energy, $E_{w}$, that the wind turbine extracts from the wind for a
given period can be written as $E_{w}=\int_{0}^{T}Pdt$. However, this
statistical model does not include information about the temporal evolution of
$C_{p}$ and $U$. We can compute the expectation of the wind energy using the
expectation of the power over a period of time.

$\mathbb{E}[E_{w}]=\Delta T\mathbb{E}[P]=\Delta T\int\frac{1}{2}\rho
Ac_{p}u^{3}f(c_{p},u)dc_{p}du,$ (10)

where $\Delta T$ denotes the period of time. Notice that this only make sense
if the unknown wind speed evolution $U_{\infty}(t)$ fits in the annual
distribution modeled by the random variable $U$.

Although we do not know the joint PDF, we know that the wind speed random
variable $U$ follows a Weibull distribution. Therefore, the marginal
probability density function of $U$, $f_{U}(u)$ is a weibull PDF that follows
eq. 7.

On the other hand we can obtain the distribution of power coefficient for each
wind speed value. This would be the conditioned power coefficient PDF, that is
$f_{C_{p}}(c_{p}|U=u)$. From this two PDF we can obtain the joint PDF, using
the following relation:

$f_{C_{p}}(c_{p}\mid U=u)=\frac{f(c_{p},u)}{f_{U}(u)}$ (11)

In this work, the conditional power coefficient PDF is obtained using a
Gaussian Process Regression algorithm. Therefore, its density function is the
following:

$f_{C_{p}}(c_{p}\mid
U=u)=\frac{1}{\sqrt{2\pi\sigma_{C_{p}}(u)^{2}}}\exp{\left(\displaystyle\frac{c_{p}-\mu_{C_{p}}(u)}{\sigma_{C_{p}}(u)}\right)^{2}},$
(12)

where the mean $\mu_{C_{p}}(u)$ and standard deviation $\sigma_{C_{p}}(u)$ are
obtained from the wind turbine control simulations.

Finally, the expectation of the wind energy can be compute as follows:

$\mathbb{E}[E_{w}]=\Delta T\mathbb{E}[P]=\left(\frac{1}{2}\Delta T\rho
A\right)\int_{U_{\text{in}}}^{U_{\text{off}}}\left(\int_{0}^{C_{p,\text{nom}}}c_{p}f_{C_{p}}(c_{p}|U=u)dc_{p}\right)u^{3}f_{U}(u)du.$
(13)

Notice that the inner integral is the expectation of the conditional
distribution, $\mathbb{E}[C_{p}|U=u]=\mu_{C_{p}}(u)$. Hence, the expectation
of the wind energy only requires the mean of the distribution fitted by the
GP. The variance of the control, $\sigma_{C_{p}}(u)^{2}$, has no influence on
the estimation of the wind energy. However, it gives us information about the
control and can be useful to measure, this can be done computing the
expectation of the variance.

$\mathbb{E}[\sigma_{C_{p}}(U)]=\int_{U_{\text{in}}}^{U_{\text{off}}}\sigma_{C_{p}}(u)f_{U}(u)du$
(14)

##  Appendix B Standard Wind Turbine Control Strategy

The control strategy for variable-speed horizontal-axis wind turbines can be
divided into four regions based on wind speed. Although each region definition
may vary depending on the specific control design, the fundamental objectives
within each region are as follows:

  * 1.

Region I: When the wind speed is below the cut-in value, the turbine cannot
operate.

  * 2.

Region II: At wind speeds above the cut-in threshold but below the rated
speed, the primary objective is to optimize power generation. This is achieved
by adjusting the rotor speed to align with the power curve of the wind
turbine, utilizing a predetermined lookup table.

  * 3.

Region III: When wind speeds exceed the rated value, the focus shifts to
maintaining a consistent rotor speed across a broad range of wind velocities.
This is typically achieved through adjustment of the blade pitch, commonly
implemented using a proportional-integral-derivative (PID) control strategy,
although there are more sophisticated approaches (Gambier and Yunazwin
Nazaruddin, 2018).

  * 4.

Region IV: When the wind speed surpasses the cut-off value, the turbine must
be shut down for safety.

The transition between regions II and III, sometimes referred to as Region II
$\frac{1}{2}$, is characterized by maintaining a constant rotor speed.
Although there are different options depending on the specific control design.
Figure 14 illustrates these regions on the power curve of the wind turbine.
Further details on classical wind turbine control strategies can be found in
the works of Burton et al. (2011) or Åström and Hägglund (2006).

The controller module in OpenFAST facilitates the customization of
controllers. In this study, the wind turbine controller is derived from the
OpenFAST implementation from Mulders and Van Wingerden (2018), tailored to
suit the characteristics of the SWT wind turbine.

Rated power$U_{\infty,\text{rated}}$$U_{\infty,\text{cut-
in}}$$U_{\infty,\text{cut-off}}$$U_{\infty}$$P$Region IRegion IIRegion II
1/2Region IIIRegion IV Figure 14: Control strategy regions for a variable-
speed, horizontal-axis wind turbine on the power curve.

## Acknowledgments

Esteban Ferrer and Oscar A. Marino would like to thank the support of Agencia
Estatal de Investigación for the grant ”Europa Excelencia” for the project
EUR2022-134041 funded by MCIN/AEI/10.13039/501100011033) and the European
Union NextGenerationEU/PRTR and also the funding received by the Grant DeepCFD
(Project No. PID2022-137899OB-I00) funded by MICIU/AEI/10.13039/501100011033
and by ERDF, EU. This research has been cofunded by the European Union (ERC,
Off-coustics, project number 101086075). Views and opinions expressed are,
however, those of the author(s) only and do not necessarily reflect those of
the European Union or the European Research Council. Neither the European
Union nor the granting authority can be held responsible for them.

## References

  * Abolude and Zhou (2018) Akintayo Temiloluwa Abolude and Wen Zhou.  Assessment and performance evaluation of a wind turbine power output.  _Energies_ , 11(8):1992, 2018.
  * Andersson et al. (2021) Leif Erik Andersson, Olimpo Anaya-Lara, John Olav Tande, Karl Otto Merz, and Lars Imsland.  Wind farm control - part i: A review on control system concepts and structures.  _IET Renewable Power Generation_ , 15(10):2085–2108, 2021.  doi: https://doi.org/10.1049/rpg2.12160.  URL https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/rpg2.12160.
  * Åström and Hägglund (2006) Karl Johan Åström and Tore Hägglund.  _Advanced PID Control_.  ISA - The Instrumentation, Systems and Automation Society, 2006.  ISBN 978-1-55617-942-6.
  * Bertagnolio et al. (2001) Franck Bertagnolio, Niels Sørensen, Jeppe Johansen, and Peter Fuglsang.  Wind turbine airfoil catalogue.  2001\.
  * Brooks et al. (1989) Thomas F Brooks, D Stuart Pope, and Michael A Marcolini.  Airfoil self-noise and prediction.  Technical report, 1989.
  * Burton et al. (2011) T. Burton, N. Jenkins, D. Sharpe, and E. Bossanyi.  _Wind Energy Handbook_.  Wiley, 2011.  ISBN 9781119993926.  URL https://books.google.es/books?id=dip2LwCRCscC.
  * Chen et al. (2020) Peng Chen, Dezhi Han, Fuxiao Tan, and Jun Wang.  Reinforcement-based robust variable pitch control of wind turbines.  _IEEE Access_ , 8:20493–20502, 2020.
  * Chollet et al. (2015) François Chollet et al.  Keras.  https://keras.io, 2015.
  * Christophe et al. (2022) J Christophe, S Buckingham, C Schram, and S Oerlemans.  zephyr-large on shore wind turbine benchmark.  https://zenodo.org/records/7323750, 2022.  Accessed : 2024-04-07.
  * Davy et al. (2018) John L. Davy, Kym Burgemeister, and David Hillman.  Wind turbine sound limits: Current status and recommendations based on mitigating noise annoyance.  _Applied Acoustics_ , 140:288–295, 2018.  ISSN 0003-682X.  doi: https://doi.org/10.1016/j.apacoust.2018.06.009.  URL https://www.sciencedirect.com/science/article/pii/S0003682X17311428.
  * Gambier and Yunazwin Nazaruddin (2018) Adrian Gambier and Yul Yunazwin Nazaruddin.  Nonlinear pid control for pitch systems of large wind energy converters.  In _2018 IEEE Conference on Control Technology and Applications (CCTA)_ , pages 996–1001, 2018.  doi: 10.1109/CCTA.2018.8511531.
  * García-Bustamante et al. (2008) E García-Bustamante, JF González-Rouco, PA Jiménez, J Navarro, and JP Montávez.  The influence of the weibull assumption in monthly wind energy estimation.  _Wind Energy: An International Journal for Progress and Applications in Wind Power Conversion Technology_ , 11(5):483–502, 2008.
  * Garcia-Bustamante et al. (2009) E Garcia-Bustamante, JF González-Rouco, PA Jiménez, J Navarro, and JP Montávez.  A comparison of methodologies for monthly wind energy estimation.  _Wind Energy: An International Journal for Progress and Applications in Wind Power Conversion Technology_ , 12(7):640–659, 2009.
  * Garnier et al. (2021) Paul Garnier, Jonathan Viquerat, Jean Rabault, Aurélien Larcher, Alexander Kuhnle, and Elie Hachem.  A review on deep reinforcement learning for fluid mechanics.  _Computers & Fluids_, 225:104973, 2021.  ISSN 0045-7930.  doi: https://doi.org/10.1016/j.compfluid.2021.104973.  URL https://www.sciencedirect.com/science/article/pii/S0045793021001407.
  * Guadarrama et al. (2018) Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Efi Kokiopoulou, Luciano Sbaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo.  TF-Agents: A library for reinforcement learning in tensorflow.  https://github.com/tensorflow/agents, 2018.  URL https://github.com/tensorflow/agents.  [Online; accessed 06-April-2024].
  * Jager and Andreas (1996) D. Jager and A. Andreas.  Nrel national wind technology center (nwtc): M2 tower; boulder, colorado (data).  http://dx.doi.org/10.5439/1052222, 1996.  NREL Report No. DA-5500-56489.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kushwaha et al. (2020) Ashish Kushwaha, Madan Gopal, and Bhim Singh.  Q-learning based maximum power extraction for wind energy conversion system with variable wind speed.  _IEEE Transactions on Energy Conversion_ , 35(3):1160–1170, 2020.
  * Le Clainche et al. (2023) Soledad Le Clainche, Esteban Ferrer, Sam Gibson, Elisabeth Cross, Alessandro Parente, and Ricardo Vinuesa.  Improving aircraft performance using machine learning: A review.  _Aerospace Science and Technology_ , 138:108354, 2023.  ISSN 1270-9638.  doi: https://doi.org/10.1016/j.ast.2023.108354.  URL https://www.sciencedirect.com/science/article/pii/S1270963823002511.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * MacKay et al. (1998) David JC MacKay et al.  Introduction to gaussian processes.  _NATO ASI series F computer and systems sciences_ , 168:133–166, 1998.
  * Matthew (2012) JC Matthew.  A method for designing generic wind turbine models representative of real turbines and generic siemens swt-2.3-93 and vestas v80 specifications, 2012.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Moriarty et al. (2004) Patrick Moriarty, Gianfranco Guidati, and Paul Migliore.  Recent improvement of a semi-empirical aeroacoustic prediction code for wind turbines.  In _10th AIAA/CEAS aeroacoustics conference_ , page 3041, 2004.
  * Mossalam et al. (2016) Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson.  Multi-objective deep reinforcement learning.  _arXiv preprint arXiv:1610.02707_ , 2016.
  * Mulders and Van Wingerden (2018) SP Mulders and JW Van Wingerden.  Delft research controller: an open-source and community-driven wind turbine baseline controller.  In _Journal of Physics: Conference Series_ , volume 1037, page 032009. IOP Publishing, 2018.
  * National Renewable Energy Laboratory (2024) National Renewable Energy Laboratory.  OpenFAST.  https://github.com/OpenFAST/openfast, 2024.  Accessed: 2024-04-07.
  * Njiri and Söffker (2016) Jackson G. Njiri and Dirk Söffker.  State-of-the-art in wind turbine control: Trends and challenges.  _Renewable and Sustainable Energy Reviews_ , 60:377–393, 2016.  ISSN 1364-0321.  doi: https://doi.org/10.1016/j.rser.2016.01.110.  URL https://www.sciencedirect.com/science/article/pii/S1364032116001404.
  * Novaes Menezes et al. (2018) Eduardo José Novaes Menezes, Alex Maurício Araújo, and Nadège Sophie Bouchonneau da Silva.  A review on wind turbine control and its associated methods.  _Journal of Cleaner Production_ , 174:945–953, 2018.  ISSN 0959-6526.  doi: https://doi.org/10.1016/j.jclepro.2017.10.297.  URL https://www.sciencedirect.com/science/article/pii/S0959652617326021.
  * Pedersen et al. (2009) Eja Pedersen, Frits Van Den Berg, Roel Bakker, and Jelte Bouma.  Response to noise from modern wind farms in the netherlands.  _The Journal of the Acoustical Society of America_ , 126(2):634–643, 2009.
  * Puech and Read (2023) Alban Puech and Jesse Read.  An improved yaw control algorithm for wind turbines via reinforcement learning.  pages 614–630, Cham, 2023. Springer Nature Switzerland.  ISBN 978-3-031-26419-1.
  * Saenz-Aguirre et al. (2019) Aitor Saenz-Aguirre, Ekaitz Zulueta, Unai Fernandez-Gamiz, Javier Lozano, and Jose Manuel Lopez-Guede.  Artificial neural network based reinforcement learning for wind turbine yaw control.  _Energies_ , 12(3), 2019.  ISSN 1996-1073.  doi: 10.3390/en12030436.  URL https://www.mdpi.com/1996-1073/12/3/436.
  * Sierra-Garcia et al. (2020) J. Enrique Sierra-Garcia, Matilde Santos, and Ravi Pandit.  Wind turbine pitch reinforcement learning control improved by pid regulator and learning observer.  2020\.  doi: https://doi.org/10.1016/j.engappai.2022.104769.  URL https://www.sciencedirect.com/science/article/pii/S0952197622000598.
  * Soler et al. (2024) Daniel Soler, Oscar Mariño, David Huergo, Martín de Frutos, and Esteban Ferrer.  Reinforcement learning to maximize wind turbine energy generation.  _Expert Systems with Applications_ , 249:123502, 2024.  ISSN 0957-4174.  doi: https://doi.org/10.1016/j.eswa.2024.123502.  URL https://www.sciencedirect.com/science/article/pii/S0957417424003671.
  * Spiru and Simona (2024) Paraschiv Spiru and Paraschiv Lizica Simona.  Wind energy resource assessment and wind turbine selection analysis for sustainable energy production.  _Scientific Reports_ , 14(1):10708, 2024.
  * Sutton and Barto (1998) Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  MIT Press, 1998.  URL http://www.cs.ualberta.ca/~sutton/book/the-book.html.
  * Towers et al. (2023) Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis.  Gymnasium, March 2023.  URL https://zenodo.org/record/8127025.
  * Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30, 2016.
  * Van Moffaert and Nowé (2014) Kristof Van Moffaert and Ann Nowé.  Multi-objective reinforcement learning using sets of pareto dominating policies.  _The Journal of Machine Learning Research_ , 15(1):3483–3512, 2014.
  * Van Moffaert et al. (2013) Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé.  Scalarized multi-objective reinforcement learning: Novel design techniques.  In _2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL)_ , pages 191–199. IEEE, 2013.
  * Wagner et al. (1996) Siegfried Wagner, Rainer Bareiß, and Gianfranco Guidati.  Noise mechanisms of wind turbines.  _Wind Turbine Noise_ , pages 67–92, 1996.
  * Watkins and Dayan (1992) Christopher JCH Watkins and Peter Dayan.  Q-learning.  _Machine learning_ , 8:279–292, 1992.
  * Wei et al. (2016) Chun Wei, Zhe Zhang, Wei Qiao, and Liyan Qu.  An adaptive network-based reinforcement learning method for mppt control of pmsg wind energy conversion systems.  _IEEE Transactions on Power Electronics_ , 31(11):7837–7848, 2016.
  * Wolsink et al. (1993) Maarten Wolsink, Maarten Sprengers, Armin Keuper, Torben Holm Pedersen, and Chris A Westra.  Annoyance from wind turbine noise on sixteen sites in three countries.  In _European community wind energy conference_ , volume 271, page 276, 1993.
  * Xie et al. (2023) Jingjie Xie, Hongyang Dong, and Xiaowei Zhao.  Data-driven torque and pitch control of wind turbines via reinforcement learning.  _Renewable Energy_ , 215:118893, 2023.  ISSN 0960-1481.  doi: https://doi.org/10.1016/j.renene.2023.06.014.  URL https://www.sciencedirect.com/science/article/pii/S0960148123007905.
  * Zhu et al. (2005) Wei Jun Zhu, Nicolai Heilskov, Wen Zhong Shen, and Jens Nørkær Sørensen.  Modeling of aerodynamically generated noise from wind turbines.  2005\.


The OEIS: A Fingerprint File for Mathematics

N. J. A. Sloane111N. J. A. Sloane is President of The OEIS Foundation, Inc.,
and is a Visiting Scholar at the Math. Dept., Rutgers University, New
Brunswick, NJ. His email address is<EMAIL_ADDRESS>
##  1 The OEIS

The On-Line Encyclopedia of Integer Sequences or OEIS [4] is a free website
(https://oeis.org) containing information about 350,000 number sequences. You
will probably first encounter it when trying to identify a sequence that has
come up in your work. If your sequence is recognized, the response will tell
you the first 100 or sometimes 10,000 terms, give a definition, properties,
formulas, references, links, computer programs, etc., as appropriate. If it is
not recognized, the system will invite you to submit it if you think it is of
general interest, so that the next person who looks it up will find it - and
be grateful to you.

Ron Graham called the OEIS a “fingerprint file for mathematics”. It has also
been described as, “pound for pound, one of the most useful mathematics sites
on the web”. If you have been struggling with a sequence, and the OEIS tells
you what it is, you will understand why it is so popular. Of course, if it
tells you that your problem was solved forty years ago, you may be unhappy,
but it is better to find out now rather than later. The database has been
around, in one form or another, for 57 years, so if your sequence is not yet
included, there is a moderate chance it is new (of course this is not a
proof).

##  2 The OEIS as a source of problems

The entries are constantly being updated. Every day we get a hundred or so
submissions of new sequences, and another hundred comments on existing entries
(new formulas, references, additional terms, etc.).

The new sequences are often sent in by non-mathematicians, and are a great
source of problems to work on. You can see the current submissions at
https://oeis.org/draft. Often enough you will see a sequence that is so
interesting you want to drop everything else and work on it. Well, go ahead!
Many research papers have been born in this way. The “Yellowstone Permutation”
A098550 is one of my favorite examples: it was quite a challenge to prove that
it is a permutation of the positive integers, and the entry has a link to a
paper [1] that a group of us wrote analysing it.

Some of the things you might work on after seeing an interesting sequence on
the drafts stack are:

  * •

Is the sequence well-defined?

  * •

Does it contain infinitely many terms?

  * •

How fast does it grow? The “graph” button in every entry is helpful here.

  * •

Is there a formula or generating function?

  * •

Or you may see a conjecture or question that you think you can answer.

  * •

How would you program it to generate more terms?

Studying the “drafts” stack is an endless source of fun (and sleepless
nights).

##  3 The OEIS as a reference work

Even when you know what the sequence is, you may still look it up, to find out
what is presently known about it. We try to make sure the sequences are well-
supplied with references, especially any recent articles. The coverage is
broad: besides the obvious fields like combinatorics, graph theory, group
theory, number theory, computer science, recreational mathematics, there are
large numbers of sequences from physics and chemistry.

This makes the OEIS extremely useful as a reference work. Earlier this year a
senior group theorist told us he had been working on a problem for 20 years,
but when he looked it up in the OEIS he found a reference that he was not
aware of.

Another great resource are the computer programs. An important sequence will
have programs to generate it in C, Java, MAGMA, Maple, Mathematica, PARI,
Python, Sage, etc. This is very helpful for numerical investigations.

##  4 Submitting your sequence or comment

If you have an interesting number sequence that is not in the OEIS, you should
definitely submit it. Having a sequence in the OEIS is something you can be
proud of. You will be joining an enterprise that has been running for almost
60 years, and to which over 12,000 people have already contributed.
Contributions come from almost every country, and we have been called one of
the most successful international collaborations.

You must register before you can contribute, using your real name, the name
you would use in a scientific publication (see Request Account). The OEIS is
not a “social media”. One of the reasons for the success of the database is
that we have high standards, all contributions are refereed by the editors,
and accuracy is of primary importance. See here for more about the submission
process. The OEIS Wiki also has a page showing examples of What not to submit!

##  5 The OEIS Wiki

The OEIS Wiki (https://oeis.org/wiki) has a great deal of useful information
for users, especially in the “Information” section. There is a general index
to the sequences, a style sheet, a Q&A page, an FAQ page, pages called How to
add a comment, more terms, or a b-file, …, Instructions for general users, The
multi-faceted reach of the OEIS, and so on.

##  6 Citations of the OEIS

An especially important part of the OEIS Wiki is the section (see
https://oeis.org/wiki/Works_Citing_OEIS) that lists citations of the OEIS in
the literature or on the web. There are now about 10,000 citations, which
often say things like “This theorem would not have been discovered without the
help of the OEIS”.

You can help keep these pages up-to-date. If you come across a paper that
mentions a number sequence, check if the sequence is in the OEIS, add it if it
isn’t, and make sure the entry has a reference to the paper. If the context
seems new, consider adding a comment saying “Arises in the spectral analysis
of cobweb singularities, see A. Spider et al. (2021).” If you come across an
article that references the OEIS, make sure the paper is listed in the “Works
Citing the OEIS” pages on the Wiki. And if you happen to see a comment that
“This sequence is not in the OEIS”, then add it at once.

When you write a paper yourself, using information from the OEIS, don’t forget
to mention us in your references list, typically by saying

The OEIS Foundation Inc., Entry A123456. Published electronically at
https://oeis.org, 2021,

and also mention it in any relevant OEIS entries. Many authors “forget” to do
this, but it will help your career by drawing attention to your paper.

##  7 “Link Rot”

Don’t get me started! The OEIS serves as a guide to the literature on a huge
number of subjects, and we have hundreds of thousands of links. And every day
many of these links break. System administrators feel they are not doing their
job if they don’t change all their URLs every couple of years. Or, as happened
last year, a major university will decide to delete all the faculty home
pages, with no warning. That broke several hundred of our links. Pages on
individual’s web sites are the most fragile of all.

If you are a frequent user of the OEIS you will often run into this problem.
You can help by locating a replacement URL if the site has simply moved, or by
adding a new link to a copy of the missing page on the wonderful Wayback
Machine, run by the Internet Archive [3]. A better solution is to ask the
author of the page for permission to put a local copy of the page on the OEIS
server. We have a strong reputation, and hope to be around for a long time.
Almost everyone we have asked has agreed. But it is a time-consuming business,
and you could help.

##  8 Other topics

“Superseeker”. This is a program that runs on our server, and tries very hard
to identify a sequence. It runs several programs that try to guess a formula
or recurrence (including the powerful Salvy-Zimmermann program gfun [5]). It
also transforms the sequence in a hundred ways and looks up the results in the
OEIS, hoping for a match. To use it, send an email to<EMAIL_ADDRESS>with a blank subject, containing a single line of the form

lookup 0 1 3 7 11 15 23 35 43 47

(with no commas). Since this uses many resources on our server, please use it
sparingly.

One of my current goals is to strengthen Superseeker. If you are interested in
helping, please contact me.

The Sequence Fans Mailing List. Any registered user of the OEIS can join, and
messages go out to a few hundred fellow sequence-lovers. This can be even more
powerful than Superseeker if you are really desperate to identify a sequence.
There is a link on the OEIS Wiki.

Triangles and arrays of numbers. The OEIS also includes triangles and arrays
of numbers. Triangles are read by rows. Pascal’s triangle becomes A007318:
$1,\leavevmode\nobreak\ 1,1,\leavevmode\nobreak\ 1,2,1,$
$1,3,3,1,\leavevmode\nobreak\ 1,4,6,4,1,$ $1,5,10,10,5,1,\ldots$. Arrays are
read by antidiagonals. The table of Nim-sums $m\oplus n$ [2],

$\begin{array}[]{ccccccc}0&1&2&3&4&5&\ldots\\\ 1&0&3&2&5&4&\ldots\\\
2&3&0&1&6&7&\ldots\\\ 3&2&1&0&7&6&\ldots\\\ 4&5&6&7&0&1&\ldots\\\
.&.&.&.&.&.&\ldots\\\ \end{array}$

becomes A003987, $0,\leavevmode\nobreak\ 1,1,\leavevmode\nobreak\ 2,0,2,$
$3,3,3,3,$ $4,2,0,2,4,\ldots$.

The OEIS aims for a broad coverage of integer sequences arising in science,
especially in mathematics (of course including several versions of the famous
subway stops sequence, since they have been published on tests; but not the
number of pages in the $n$th issue of these Notices since 1954, because no one
has mentioned it before). Our motto is accuracy and completeness, combined
with good judgment. We have a great many contributors, a great many entries,
and we hope you will join us at https:oeis.org.

## References

  * [1] David L. Applegate, Hans Havermann, Bob Selcoe, Vladimir Shevelev, N. J. A. Sloane, and Reinhard Zumkeller, The Yellowstone Permutation, _J. Integer Sequences_ , 18:6 (2015), #15.6.7.
  * [2] J. H. Conway and R. K. Guy, _The Book of Numbers_ , Springer, 1996; p. 295.
  * [3] Internet Archive, _The Wayback Machine_ , https://archive.org/web, 2021\.
  * [4] The OEIS Foundation Inc., _The On-Line Encyclopedia of Integer Sequences_ , https://oeis.org (2021).
  * [5] Bruno Salvy and Paul Zimmermann, GFUN: a Maple package for the manipulation of generating and holonomic functions in one variable, _ACM Trans. Math. Software_ , 20 (1994), 163–177.

2020 Mathematics Subject Classification: 00Axx, 05-00, 11-00, 11Bxx, 97K10