# RFRL Gym: A Reinforcement Learning Testbed for Cognitive Radio Applications

Daniel Rosen1, Illa Rochez1, Caleb McIrvin1, Joshua Lee1, Kevin D’Alessandro1,
Max Wiecek1, Nhan Hoang1,
Ramzy Saffarini1, Sam Philips1, Vanessa Jones1, Will Ivey1, Zavier Harris-
Smart2,
Zavion Harris-Smart2, Zayden Chin2, Amos Johnson2, Alyse M. Jones1, William C.
Headley1  1Virginia Tech National Security Institute, 2Morehouse College
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Radio Frequency Reinforcement Learning (RFRL) is anticipated to be a widely
applicable technology in the next generation of wireless communication
systems, particularly 6G and next-gen military communications. Given this, our
research is focused on developing a tool to promote the development of RFRL
techniques that leverage spectrum sensing. In particular, the tool was
designed to address two cognitive radio applications, specifically dynamic
spectrum access and jamming. In order to train and test reinforcement learning
(RL) algorithms for these applications, a simulation environment is necessary
to simulate the conditions that an agent will encounter within the Radio
Frequency (RF) spectrum. In this paper, such an environment has been
developed, herein referred to as the RFRL Gym. Through the RFRL Gym, users can
design their own scenarios to model what an RL agent may encounter within the
RF spectrum as well as experiment with different spectrum sensing techniques.
Additionally, the RFRL Gym is a subclass of OpenAI gym, enabling the use of
third-party ML/RL Libraries.

We plan to open-source this codebase to enable other researchers to utilize
the RFRL Gym to test their own scenarios and RL algorithms, ultimately leading
to the advancement of RL research in the wireless communications domain. This
paper describes in further detail the components of the Gym, results from
example scenarios, and plans for future additions.

###### Index Terms:

machine learning, reinforcement learning, wireless communications, dynamic
spectrum access, OpenAI gym

##  I Introduction

Due to the rapid growth of Radio Frequency (RF) spectrum usage, efficient use
of the spectrum has become increasingly difficult, leading to over-congestion
in the RF spectrum [1]. This largely is a consequence of the exponential
increase in wireless devices, with the CTIA citing a growth of 183 times its
size since 2010 in its 2021 Wireless Internet Survey [2], ultimately resulting
in decreased data rates from poor channel conditions and collisions [3]. These
collisions are a form of signal jamming, also known as interference, leading
to incomplete and corrupted transmissions. Interference can either be
intentional from an adversary or unintentional due to less-than-ideal
conditions in the RF spectrum. Furthermore, spectrum congestion makes it more
difficult to differentiate whether the collision was from an adversary or
over-congestion. Regardless, to avoid further loss to communication systems,
both interference types must be prevented. Thus, to prevent interference as
well as losses in data rate, there is a clear need for improvement in the
utilization of the RF Spectrum.

In recent years, Cognitive Radios (CR) have gained traction as a promising
solution to alleviating spectrum congestion [4] due to its ability to
intelligently determine which channels are currently occupied, and which are
not through Dynamic Spectrum Access (DSA)[5]. DSA was proposed to overcome the
shortcomings of traditional methods, such as Frequency Hopping Spread Spectrum
(FHSS)[6] and Direct Sequence Spread Spectrum (DSSS)[7], where both fail in a
dynamic RF spectrum from not adapting its method to where holes in the
spectrum are located. DSA, however, is able to do so through spectrum sensing,
where methods such as energy detection are utilized to identify which
frequency bands are in use [8], allowing a CR to switch to an empty band.
However, DSA alone is not enough to handle spectrum congestion since it cannot
predict the location of future spectrum holes, resulting still in accidental
collisions. However, by coupling DSA with predictive intelligence, a CR can
predict where and when other users will occupy the RF spectrum in the future.
One promising solution is Reinforcement Learning (RL).

RL is a Machine Learning method known for its ability to predict the best
decisions to achieve a desired outcome[9]. When applied to CRs, RL’s
predictive capabilities enable signals to preemptively respond to the
movements of other signals in the spectrum as well as changing channel
conditions such that the optimal frequency at which to transmit can be
identified. Significant effort has been invested into investigating RL-based
protocols for spectrum assignment and wireless communication enhancement,
where RL has shown to drastically reduce congestion in the spectrum [10, 11,
12, 13].

From these works, it is clear that RL Algorithms _must_ be trained in a
representative environment, in this case the RF spectrum. However, there is
currently an absence of a comprehensive, universal tool simulating the RF
spectrum that researchers can use to develop and test RL algorithms for CR-
specific applications. Conventionally, new tools have been developed for each
project and can not be easily used in other applications. Thus, this paper
introduces the RFRL Gym111https://github.com/vtnsiSDD/rfrl-gym, a training
environment of the RF spectrum designed to provide comprehensive
functionality, such as custom scenario generation, multiple learning settings,
and compatibility with third-party RL packages. As a result, the RFRL Gym can
be applicable to a wide range of wireless communications projects, where
researchers can utilize the tool to further the advancement of RL in wireless
communications. Additionally, through a gamified mode of the RF spectrum, this
tool can be used to teach novices about the fields of AI/ML and RF. The
remainder of this paper defines the current state of the RFRL Gym, its
capabilities, and planned future enhancements.

##  II Background

As mentioned, prior works [14, 15, 16, 17] have developed tools to achieve RL
for their own wireless communications applications. Table I describes how the
RFRL Gym differs from these existing tools through the following notable key
features.

Flexible Scenario Design: The ability to create, modify, or utilize different
scenarios to view how RL algorithms will react and interact in different
learning spaces. [16] was designed specifically for a particular use case of
wireless communications and contained no ability to create custom scenarios on
their framework. [17] did have some included scenarios within their gym, such
as for WiFi and cellular scenarios. However, it is unclear if custom scenarios
can be generated and if so, how difficult it would be. [14, 15] contain the
most flexibility in scenario generation of the three since its data is
generated through GNU Radio, a platform to design communication systems [18].
However, this process is not straightforward, as designing flow-graphs in GNU
Radio can be difficult for non-experts and those not familiar with the
software. Therefore, the RFRL Gym includes flexible scenario generation where
users can easily create custom scenarios according to their application
through input json files.

RL Package Compatibility: The ability to interface with other RL libraries to
expand a user’s access to different RL agents. Examples include Mushroom
RL[19] and Stable Baselines[20]. This feature removes the need to implement RL
algorithms manually if not desired. Of the tools considered, only the RFRL Gym
has explicitly tested and confirmed this feature. Additionally, custom
creation of RL algorithms is also possible within the RFRL Gym.

Spectrum Sensing Capabilities: The ability to locate spectrum holes through
spectrum sensing. To fully realize a DSA scenario, this is a requirement. [17]
does have capabilities for spectrum sensing, but to our knowledge, there is no
ability to create custom spectrum sensing methods.

Multi-Agent Capabilities: The ability to support several intelligent agents
(i.e. RL agents). [16] and [17] do have capabilities for multi-agent
reinforcement learning. It is our desire for future work to include this
functionality within the RFRL Gym.

Ease Of Use: The tool is designed with considerations for an easy user
experience, especially for those with little or no familiarity with wireless
communications and/or RL. This feature allows for ML/RL specialists to better
interact and interface with Wireless Communication specialists through easy
scenario generation, provided examples, and a gamified mode to learn the
wireless communications concepts at an abstracted level. By doing so, the RFRL
Gym suits the needs for both novices to learn RFRL concepts and experts to use
the gym to further their research.

Graphical User Interface: The ability to control the simulation (i.e. scenario
generation) through a Graphical User Interface (GUI) to make it easier for
users to interface with the RFRL Gym. Existing tools are all controlled
through scripts and a command line interface. If the user wants to make
scenario generation changes, they are forced to change it within the code.
However, a GUI bypasses this need such that users can make any changes to the
scenario without touching the code. Future plans for the GUI include
controlling RL hyperparemeters and environment parameters such that the entire
simulation can be run entirely from the GUI.

Hardware Compatible: The ability to operate over radio hardware in real-time,
rather than simulation alone, for a realistic implementation. [14, 15] are
hardware compatible since GNU Radio supports radio hardware. Additionally, a
primary contribution of [16] was to implement their system over embedded
systems in radio hardware. In future versions, the RFRL Gym will also aim for
hardware compatibility.

In comparison to existing tools described above, the RFRL Gym was designed to
serve as a universal, easy-to-use tool that both novices and experts within
this field can use to learn about AI/ML and RF as well as further
technological advancement of RFRL. For novices, in the gamified mode, the
complex features of wireless communications within the RFRL Gym are abstracted
so they can use the tool with limited knowledge of wireless communications and
be eased into more advanced topics, such as signal modulation and the
generation of IQ data. In addition, the gym has a built-in GUI system for
setting scenarios and parameters without changing the code as well as the
ability to support outside packages, such as Mushroom RL. Additionally, prior
works discussed in this section only allow for pre-set scenarios, whereas our
tool allows for scenario customization. These additions separate our tool from
others, emphasizing an easy-to-use environment for experimentation and
testing.

TABLE I: Comparison of existing OpenAI Gym RL tools for wireless communications Features |  RFRL Gym |  GrGym [14], [15] |  MR-iNet Gym [16] |  ns3-Gym [17]
---|---|---|---|---
Flexible Scenario Design |  ✓ |  ✓ |  |  ✓
RL Package Compatibility |  ✓ |  |  |
Spectrum Sensing Capabilities |  ✓ |  |  |  ✓
Multi-Agent Capabilities |  |  |  ✓ |  ✓
Ease of Use |  ✓ |  |  |
Graphical User Interface |  ✓ |  |  |
Hardware Compatible |  |  ✓ |  ✓ |
Figure 1: The RFRL Gym framework, illustrating how user-defined scenarios and
reinforcement learning algorithms interact with the gym environment. This
framework facilitates simulating an RF-based reinforcement learning
interaction loop of the RL agent taking in states (e.g. sensing results) and
rewards (e.g. receiver performance) and providing to the gym its determined
next action (e.g. transmitter tuning).

##  III The RFRL Gym Framework

A high-level overview of the structure of the RFRL Gym is provided by Figure
1. The framework consists of a series of interconnected modules that function
together in order to simulate RF spectrum scenarios. These modules follow the
Open AI Gym API to ensure the RFRL Gym is compatible with third party gym-
compatible RL software through its predefined functions and methods for a
Reinforcement Learning agent and the environment to communicate. As a result,
the RFRL Gym is able to exploit the “RL Cycle” training protocol. This
protocol is a cyclical passage of information between the Gym and RL agent
over a discrete set of training episodes, with each episode consisting of a
number of time-steps. The cycle begins at the beginning of each time-step when
the RL agent performs an action in the custom-made RF environment. This action
may be to occupy a channel or to not transmit in the next timestep. At this
point, Non-RL Entities will also select an action. Based on the actions of the
Non-RL Entities, the RL Agent’s action is evaluated and assigned a reward.
This reward and the new observation space are made accessible to the RL Agent
for training. This process repeats for all time-steps. The remainder of this
section describes the individual components of the RFRL Gym outlined in Fig.
1.

###  III-A Non-Player Entities

Several non-player entities are implemented in our environment. Non-Player
refers to an entity whose decision-making is not controlled by the primary
Reinforcement Learning algorithm. They will herein be referred to as
“Entities”. These Entities are crucial for testing as they simulate physical
interactions between signals and replicate real signals in the environment.
There are five entities that have been implemented into our code base at this
point, as described in Table II.

TABLE II: Non-player entity types within the RFRL Gym environment that can be included in scenario files, with descriptions and real-world use cases Entity Type |  Description | Real-world Example
---|---|---
Constant |  Remains in one channel | T.V. or Radio
Fixed Hopper |  Moves according to a user-defined pattern | Bluetooth
Stochastic Hopper |  Channel selected based on a set of user-defined probabilities for each channel | N/A
Agile Hopper |  Moves to an empty channel. Has perfect knowledge of other entities in the bandwidth. If multiple channels are empty, it selects one with uniform probability. Most similar to the RL agent but without the RL. | CR with DSA
Simple Jammer |  Has perfect knowledge of other entities in the environment, and moves to an occupied channel based on uniform distribution. If no channels are occupied, it moves to an unoccupied channel. | Intelligent Adversary
Custom |  Created by users of our codebase | N/A

###  III-B RL Entities

The purpose of the RFRL Gym is to train RL agents for RF applications. Users
can run simulations with their own custom RL algorithms or those built by
third parties. Multiple modes have been implemented for the observation and
reward to enable a diversity of scenarios on which to train RL agents.

####  III-B1 Reward Modes

The two reward types currently implemented in the environment for the RL agent
are DSA reward and Jamming reward. DSA mode can be used to simulate a
situation where an RL agent wants to avoid interference, either by avoiding a
channel with bad channel conditions or a channel with other users and/or
adversaries. Applications include commercial communications as well as
Electronic Warfare (EW). Alternatively, Jamming mode allows for the simulation
of situations where the agent wants to prevent others from successful
communications and use of the RF spectrum by acting as an adversary. This mode
is also applicable to EW. The optional dampening factor was added to consider
the computational cost of switching to model the desired real-world behavior
of energy conservation, where the ideal behavior is to move only when
necessary. This is especially important for small, energy-constrained devices,
such as IoT devices.

  * •

DSA Reward - Rewards the agent for avoiding channels containing other
entities.

$\textbf{DSA reward }=\begin{cases}1,&\text{no collision}\\\
-1,&\text{collision}\end{cases}$ (1)

  * •

Jamming Reward - Rewards the agent for occupying the same channel as a
particular entity in the environment. The entity to be jammed is selected by
the user.

$\textbf{Jamming reward = }\begin{cases}1,&\text{collision}\\\ -1,&\text{no
collision}\end{cases}$ (2)

  * •

Dampening Factor - The dampening factor will increase the reward if the agent
stays in the same channel and will reduce the reward if it changes channels.

$\textbf{reward}=\begin{cases}reward-0.5,&\text{movement made}\\\
reward+0.5,&\text{no movement made}\end{cases}$ (3)

  * •

DSA Reward \- Rewards the agent for avoiding channels containing other
entities. The agent is awarded a $1$ for no collision and $-1$ for a
collision.

  * •

Jamming Reward \- Rewards the agent for occupying the same channel as a target
entity in the environment. The entity to be jammed is selected by the user.
The agent is awarded a $1$ for a collision and $-1$ for no collision.

  * •

Dampening Factor \- The dampening factor will increase the reward if the agent
stays in the same channel and reduce the reward if it changes channels. The
agent is awarded an additional $0.5$ for no movement made and $-0.5$ for
movement made.

####  III-B2 Observation Modes

The gym contains two different observation modes for the agents: detect and
classify. These modes determine what information is sent to the agent as the
current state of the environment. In detect mode, the agent only knows if a
channel is occupied or not. In this mode, the RL agent does not know what type
of entities or how many entities occupy a single channel. In classify mode,
the agent receives information about the channel location, type, and number of
entities (i.e. which entities are in which channel or if there is a Multi-
Entity collision in a channel). These modes are realized through spectrum
sensing, where occupancy can be determined across each frequency based on the
amount of energy present. If the amount of energy is above the noise floor, it
is an indication that an entity is transmitting over that channel and/or the
channel has poor conditions.

###  III-C Rendering Modes

The RFRL Gym provides two rendering modes to visualize simulations, the state
of entities and their interactions at each time-step, as well as results:
terminal rendering mode and PyQt rendering mode. Both modes display the
location of the entities within the different channels across time as well as
the reward for each time step and cumulative reward across each episode.

####  III-C1 Terminal Rendering

The terminal rendering mode sends data to be displayed in Standard Output
using ascii text. Therefore, the data can be directed to the terminal or
redirected to a file. Depending on whether the gym is in classify or detect
mode, the visualization will change. In detect mode, occupied channels are
represented with a 1, while unoccupied channels are represented with a 0. In
classify mode, the location of each entity is shown with its corresponding
entity number. Lastly, the terminal mode runs efficiently, making it suitable
for more complex scenarios where a faster output is desired.

####  III-C2 PyQt Rendering

Enabled by the pyqtgraph package, the PyQt rendering mode displays the same
data as the terminal mode but with more detailed visuals, making it easier for
users to visually understand how entities move across time and frequency
within the RF spectrum. This mode also displays a plot of the cumulative
reward for each time-step in an episode. An example of visual results for this
mode can be seen in Section IV.

###  III-D Scenarios

A scenario is the set of parameters that define the environment in which the
agent is placed to perform its learning. The parameters fall into three
categories: _environment_ , _render_ , and _entities_. Scenarios are defined
in customizable JSON files (”scenario files”) that are fed into the RFRL Gym.

####  III-D1 Environment

This section defines the learning parameters needed to prepare the
environment, such as the number of channels available and number of time
steps. Additionally, the observation mode can be set to the “classify” or
“detect” mode, and similarly, the reward mode can be set to “dsa” for dynamic
spectrum access, or “jam” for jamming mode. For jamming mode, the target
entity is also set.

####  III-D2 Entities

Here, the user will determine which entities will exist in the environment. An
entity is designated by a user-defined name, which will appear in the rendered
output. Within an entity, various parameters can be set. For the gamified,
basic mode, these parameters include the channel(s) that the entity can
occupy, the on/off transmission pattern(“on” referring to a non-idle state,
“off” referring to idle), and the first and last steps at which the entity
transmits. For the advanced mode containing IQ, the entities’ modulation type
and order, center frequency, bandwidth, start time, and duration can also be
set.

Figure 2: GUI-based scenario generator where users can set up the environment,
render, and entities.

####  III-D3 Render

This section determines whether the render is displayed within the terminal,
or if it utilizes the render written using the pyqtgraph library. The render’s
speed, in frames per second, and the number of steps that the render displays
at a time are also included options.

To improve the user experience, a Scenario Selection GUI was created which
enables the modification and creation of scenario files without interfacing
with a JSON file directly. This simplifies the process of creating scenario
files and allows non-software-oriented users to produce scenarios. Fig. 2
shows the layout of the GUI.

##  IV Simulation Results and Discussion

In this section, multiple representative wireless communications scenarios are
presented in order to highlight the capabilities of the RFRL Gym for test and
evaluation of RL agents. For all scenarios, the environment consists of 10
wireless channels, assumes learning episodes consisting of 100 time-steps, 100
training episodes, a Q-learning agent balancing exploration and exploitation
through an epsilon greedy strategy, and the use of the MushroomRL software
package for training and evaluating the agent.

TABLE III: Scenario settings with entities, reward, dampening, and observation modes defined Scenario |  Entities | Reward | Dampening | Observation
---|---|---|---|---
1 |  1 Constant | Jamming | No | Classify
2 |  1 Fixed Hop | Jamming | No | Classify
3 |  1 Fixed Hop | DSA | No | Detect
|  1 Constant |  |  |
|  1 Stochastic |  |  |
4 |  1 Fixed Hop | DSA | No | Classify
|  1 Constant |  |  |
|  1 Stochastic |  |  |
|  1 Agile |  |  |

###  IV-A Scenario 1: Single Entity Jamming

This is a simple scenario that was designed to be solved easily and validate
that the RFRL Gym performs as expected and yields accurate results. It
consists of a single Constant Entity that is to be jammed. As expected, a
Q-Learning agent was able to converge to an optimal policy quickly within the
first five episodes, as demonstrated in Fig. 3.

Figure 3: Cumulative reward results for Scenario 1, showing optimal behavior
by converging to the optimal reward of 100

###  IV-B Scenario 2: Hop Frequency Jamming—Non-Markovian

This scenario consists of a single Hopping entity with a fixed hopping
pattern. It spends 2 time-steps in channel 4, 1 time-step in channel 5, and
repeats. The object of this scenario is to produce a _Non-Markovian_ problem,
meaning the problem does not follow the Markov Decision Process which holds
the assumption of the Markov property where the evolution of the Markov
process depends only on the current state. In this case, the current state is
not always sufficient information for the next time step, hence the agent
cannot achieve an optimal policy. In order for the agent to jam the Hopper
perfectly, it must have some sense of the previous location of the hopper. For
example, if the Hopper is in channel 4, the agent would need to know whether
the Hopper was in channel 4 or 5 in the previous time-step. Without that
information, our Q-Learning agent can not predict whether the entity will stay
or move. Therefore, it will converge to a sub-optimal policy.

Optimally, the agent would jam the Hopper in all 100 time-steps. In Fig. 4,
the agent can be seen choosing not to transmit in the second and third time-
steps of the hopping pattern. This is because it was unable to consistently
jam the Hopper during the training process due to Non-Markovian behavior.
These incorrect guesses yield a reward of $-1.0$ but non-transmission yields a
reward of $0.0$. Thus, the agent elects to accept the suboptimal reward of 33
and not risk the negative reward. Calculations below show this suboptimal
convergence to 33, rather than the optimal performance of 100.

$\textbf{Sub-Optimal Reward:
}100\Big{(}\frac{1}{3}(1)+\frac{2}{3}(0)\Big{)}=33$

The results produced by the RFRL Gym were aligned exactly with the expected
outcome. This scenario verifies that our Gym will perform as expected even
when an agent is unable to achieve an optimal policy. The dependability of the
Gym’s results make it a valuable tool for formulating and testing hypotheses.

###  IV-C Scenario 3: Solvable Multi-Entity DSA

This scenario contains a Fixed Hopper Entity moving through all channels non-
sequentially, a Constant Entity in channel 5, and a Stochastic Hopper that
moves to either channel $0$ or $9$ with equal likelihood. This scenario
exhibits the multi-Entity capability of our environment. As shown in Fig. 5,
the Q-Learning agent can handle avoidance of multiple Entities and achieve an
optimal policy. The RFRL Gym’s gamified rendering functionality enables easy
visualization of the agent’s optimal policy. The absence of collisions after
convergence can be seen easily by the lack of red blocks in Fig. 5(b), as
opposed to before convergence where collisions occurred frequently in Fig.
5(a).

A notable conclusion that can be drawn from this result as well is that a
Q-Learning agent in DSA mode is able to achieve an optimal policy when
entities behave according to a stationary distribution, meaning the
probability in which it chooses each channel stays constant. However, this is
not true for Agile Entities as will be shown in Scenario 4.

Figure 4: PyQt render results for scenario 2, shown in advanced mode where IQ
data is viewed. The diagonal signal is the RL agent, while the horizontal
signal is the non-RL entity the agent is targeting to jam.

(a) Render results showing non-convergent behavior before learning, where the
RL agent collides with other entities within the environment

(b) Render results showing convergent behavior after learning, where the RL
agent successfully avoids other entities within the environment

Figure 5: Scenario 3 render results, showing before and after convergence
behavior, in the gamified detect mode.

###  IV-D Scenario 4: Unsolvable Multi-Entity DSA

This scenario is similar to Scenario 3. The single difference is the addition
of an Agile Entity moving across all channels. The Stochastic Hopper Entity
still behaves according to a stationary distribution between each of its
channels since it only moves between the two channels with the highest
probabilities (channel 0 and 9), which the Q-learning agent learns to avoid.
On the other hand, the addition of the Agile Entity confuses the RL agent such
that it is unable to definitively predict where it will go next because the
Agile Entity has no deterministic pattern due to its non-stationary, time-
varying behavior caused by randomly choosing an empty channel according to
where other entities occupy other channels. Therefore, the RL agent will
occasionally collide with the Agile Entity, as can be seen in red in Fig. 6.
As a result, the agent is unable to converge to an optimal policy, as can be
seen by the noisy, non-stabilizing results in Fig. 7. This result implies that
more advanced Reinforcement Learning algorithms such as Deep Q Learning are
necessary to solve problems with non-stationary, time-varying behavior.

Figure 6: Render results showing non-convergent behavior due to collisions
with an Agile Entity. Render is in classification mode to show that the Agent
is colliding with the Agile Entity. Figure 7: Cumulative reward results for
Scenario 4, showing suboptimal behavior due to the Agile Entity

##  V Future Additions

###  V-A Multi-Agent RL functionality

In our gym, we are currently moving towards incorporating multi-agent
functionality through a separate, multi-agent-specific environment. As
realistic dynamic spectrum access scenarios may involve highly contested
bandwidths, it is important to consider and provide solutions for cases where
several RL agents interact in the same set of channels. As a result,
incorporating Multi-Agent Reinforcement Learning (MARL) learning strategies is
crucial to the success of our agents in realistic scenarios. These scenarios
may be divided into two separate classes - cooperative and adversarial.

####  V-A1 Cooperative

Cooperative MARL agents attempt to coordinate with other agents in order to
maximize total uncontested access to the spectrum.

####  V-A2 Adversarial

Adversarial agents attempt to prevent other agents from accessing the spectrum
by intentionally broadcasting on the channel already occupied by its target or
one the target is predicted to hop to in the future.

Incorporating MARL into the RFRL Gym could be made possible by utilizing MARL
libraries compatible with OpenAI Gym or similar to OpenAI Gym. One such
package worth investigating is PettingZoo [21], a MARL library designed with
OpenAI Gym’s simple API in mind in addition to the ability to create custom
MARL environments.

###  V-B Physical signal integration and Hardware compatibility

In real-world scenarios, perfect knowledge of the environment is unlikely.
Thus, the RFRL Gym will include functionality for converting raw signal data
into state information that can be processed by RL agents. Convolutional
neural networks (CNNs) are currently being developed to identify signal types
based on a signal’s structure [22]. The network will take in raw waveform IQ
data from an RF channel and classify the signal. This result can be integrated
into the observation space of RL agents to target specific entities.
Additionally, we are working towards developing the hardware functionality for
more realistic software simulation such that signals experience realistic
channels conditions in which fading and multi path propagation exist.

###  V-C GUI expansion

The next iteration of the GUI intends to connect the visualizations rendered
with PyQt to the pre-existing Scenario Selection GUI so that the user can run
simulations entirely from the Scenario Selection GUI. Additionally, we intend
to implement tabs that will guide the user through the scenario setup process,
rather than having them interact with all of the steps simultaneously. Also,
there are plans to implement an auto-sync feature. This would mean that
whenever a new rewarding type, observation mode, or non-player entity is
added, the visual output will automatically update to allow the user to see
their changes.

##  VI Conclusion

A sharp rise in wireless usage has caused significant congestion in the RF
spectrum. To remedy this, the predictive capabilities of Reinforcement
Learning are increasingly being exploited in Cognitive Radio applications.
Therefore, the existence of a resource like the RFRL Gym is imperative to
train algorithms in a representative environment.

Tools that serve similar purposes have been created, but the RFRL Gym
possesses various distinct advantages and includes more comprehensive
functionality. Firstly, the simulation functionality in our environment is
abstracted, so technical expertise in Wireless Communications is not necessary
to use the Gym. Secondly, our codebase is compatible with external libraries
including Mushroom RL and Stable Baselines. A third major advantage of our Gym
over comparable resources is the GUI which makes the operation of the
environment simple. Though the RFRL Gym currently possesses extensive
functionality, additions are still being made. Future iterations of the
environment will show MARL functionality, physical signal integration, radio
hardware compatibility, and simulations run entirely from the GUI.

All in all, we hope our work will aid in developing new reinforcement learning
strategies for cognitive radio applications. By providing a powerful tool with
a low barrier of entry, we seek to encourage the exploration of new pursuits
in the rapidly growing field of RFRL.

## References

  * [1] P. Tilghman, “Will rule the airwaves: A darpa grand challenge seeks autonomous radios to manage the wireless spectrum,” _IEEE Spectrum_ , vol. 56, pp. 28–33, 06 2019.
  * [2] “Summary of ctia’s annual wireless industry survey,” https://api.ctia.org/wp-content/uploads/2022/09/Summary-of-CTIAs-Wireless-Industry-Survey-2022.pdf.
  * [3] “Spectrum allocation in the united states 2022,” https://api.ctia.org/wp-content/uploads/2022/09/Spectrum-Allocation-in-the-United-States-2022.09.pdf.
  * [4] J. Mitola and G. Maguire, “Cognitive radio: making software radios more personal,” _IEEE Personal Communications_ , vol. 6, no. 4, pp. 13–18, 1999\.
  * [5] R. Awati, “What is cognitive radio (cr) and how does it work?” Jun 2021. [Online]. Available: https://www.techtarget.com/searchnetworking/definition/cognitive-radio
  * [6] M. Khatib, _Advanced Trends in Wireless Communications_. Rijeka: IntechOpen, Feb 2011. [Online]. Available: https://doi.org/10.5772/655
  * [7] B. P. Lathi and Z. Ding, _Direct Sequence Spread Spectrum_. Oxford University Press, 2019, pp. 702–707.
  * [8] A. G. Fragkiadakis, E. Z. Tragos, and I. G. Askoxylakis, “A survey on security threats and detection techniques in cognitive radio networks,” _IEEE Communications Surveys and Tutorials_ , vol. 15, no. 1, p. 428–445, 2013.
  * [9] R. S. Sutton and A. G. Barto, _Reinforcement learning: an introduction_. The MIT Press, 2018.
  * [10] S. Singh and A. Trivedi, “Anti-jamming in cognitive radio networks using reinforcement learning algorithms,” in _2012 Ninth International Conference on Wireless and Optical Communications Networks (WOCN)_ , 2012, pp. 1–5.
  * [11] F. Slimeni, Z. Chtourou, and A. B. Amor, “Reinforcement learning based anti-jamming cognitive radio channel selection,” in _2020 4th International Conference on Advanced Systems and Emergent Technologies (IC_ASET)_ , 2020, pp. 431–435.
  * [12] S. Machuzak and S. K. Jayaweera, “Reinforcement learning based anti-jamming with wideband autonomous cognitive radios,” in _2016 IEEE/CIC International Conference on Communications in China (ICCC)_ , 2016, pp. 1–5.
  * [13] Y. Huang, C. Xu, C. Zhang, M. Hua, and Z. Zhang, “An overview of intelligent wireless communications using deep reinforcement learning,” _Journal of Communications and Information Networks_ , vol. 4, pp. 15 – 29, 2019.
  * [14] A. Zubow, S. Rosler, P. Gawłowicz, and F. Dressler, “Grgym: When gnu radio goes to (ai) gym,” 2021.
  * [15] A. Zubow, S. Roesler, P. Gawlowicz, and F. Dressler, “Grgym: A playground for research on rl/ai enhanced wireless networks,” in _European Wireless 2022; 27th European Wireless Conference_ , 2022, pp. 1–4.
  * [16] J. Jagannath, K. Hamedani, C. Farquhar, K. Ramezanpour, and A. Jagannath, “Mr-inet gym: Framework for edge deployment of deep reinforcement learning on embedded software defined radio,” _WiseML_ , 2022.
  * [17] P. Gawlowicz and A. Zubow, “ns-3 meets openai gym: The playground for machine learning in networking research,” in _ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM)_ , November 2019. [Online]. Available: https://bits.informatik.hu-berlin.de/ zubow/gawlowicz19_mswim.pdf
  * [18] “Gnu radio - the free & open software radio ecosystem,” https://www.gnuradio.org/, 2023.
  * [19] C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters, “Mushroomrl: Simplifying reinforcement learning research,” _Journal of Machine Learning Research_ , vol. 22, no. 131, pp. 1–5, 2021. [Online]. Available: http://jmlr.org/papers/v22/18-056.html
  * [20] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines,” https://github.com/hill-a/stable-baselines, 2018.
  * [21] J. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. S. Santos, C. Dieffendahl, C. Horsch, R. Perez-Vicente _et al._ , “Pettingzoo: Gym for multi-agent reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 15 032–15 043, 2021.
  * [22] L. J. Wong, W. C. Headley, and A. J. Michaels, “Specific emitter identification using convolutional neural network-based iq imbalance estimators,” _IEEE Access_ , vol. 7, pp. 33 544–33 555, 2019.


# RAMO: Retrieval-Augmented Generation for Enhancing MOOCs Recommendations

Jiarui Rao

Jionghao Lin

Carnegie Mellon University  5000 Forbes Ave  Pittsburgh, PA 15213
<EMAIL_ADDRESS>Carnegie Mellon University  5000 Forbes Ave
Pittsburgh, PA 15213<EMAIL_ADDRESS>
###### Abstract

Massive Open Online Courses (MOOCs) have significantly enhanced educational
accessibility by offering a wide variety of courses and breaking down
traditional barriers related to geography, finance, and time. However,
students often face difficulties navigating the vast selection of courses,
especially when exploring new fields of study. Driven by this challenge,
researchers have been exploring course recommender systems to offer tailored
guidance that aligns with individual learning preferences and career
aspirations. These systems face particular challenges in effectively
addressing the “cold start” problem for new users. Recent advancements in
recommender systems suggest integrating large language models (LLMs) into the
recommendation process to enhance personalized recommendations and address the
“cold start” problem. Motivated by these advancements, our study introduces
RAMO (Retrieval-Augmented Generation for MOOCs), a system specifically
designed to overcome the “cold start” challenges of traditional course
recommender systems. The RAMO system leverages the capabilities of LLMs, along
with Retrieval-Augmented Generation (RAG)-facilitated contextual
understanding, to provide course recommendations through a conversational
interface, aiming to enhance the e-learning experience.

###### keywords:

Retrieval-Augmented Generation (RAG), Personalized Learning, Recommender
Systems, Artificial Intelligence

††titlenote: This paper underwent a rigorous review process and was officially
accepted on May 31, 2024, for presentation at the Educational Data Mining 2024
Workshop: Leveraging Large Language Models for Next Generation Educational
Technologies.

##  1 Introduction

Massive Open Online Courses (MOOCs) gently facilitate access to learning for a
diverse global audience [3]. By providing an extensive range of courses
through an easily accessible online platform, MOOCs not only enhance
individual learning and development but also enrich the broader educational
community [4]. However, the diverse categories of courses across disciplines
can often overwhelm students when they step into new fields of study [17].
Selecting the right courses that align with both personal interests and
academic requirements is crucial, as improper choices may lead to wasted time,
and resources, and a lack of fulfillment in one’s educational journey.

To resolve this, researchers have developed course recommender systems using
advanced algorithms to offer tailored guidance that aligns with individual
learning preferences [30]. Many existing implementations of recommendation
systems have demonstrated significant benefits, such as enhancing personalized
learning experiences and improving student engagement, as highlighted by a
recent study [11]. However, these systems also face critical limitations,
particularly the “cold start” problem, which occurs when trying to make
recommendations for new users with limited historical data [15]. Though
previous research proposed a more complex framework—a novel meta-learning
heterogeneous information networks approach [25]—to address the “cold start”
recommendation issue, the approach faces the challenge of high computational
complexity, which is not scalable for large-scale MOOCs platforms.

In response to address the limitations of prior work in recommendation
systems, where the recommendations lack sufficient personalization and
interaction with users, researchers have proposed integrating large language
models (LLMs) into course recommendations [18]. This approach enhances
recommendation accuracy and personalization by leveraging user history and
conversational prompts. Recent frameworks like GPT4Rec [21] and Chat-Rec [9]
demonstrated the potential of LLMs in improving course alignment with
learners’ interests and interaction. However, LLMs can sometimes generate
misleading or outdated information. To counteract these shortcomings, one
possible solution is the integration of Retrieval-Augmented Generation (RAG)
with LLMs [26].

Demo of the RAG facilitated course recommender system we developed

Figure 1: Interface of the Retrieval-Augmented Generation for MOOCs (RAMO)
system

RAG [10] is a process that optimizes the output of LLMs by extending their
robust capabilities to cater specifically to distinct domains or an
organization’s internal knowledge base, eliminating the need for retraining
the model [8]. The use of RAG in recommendation systems enhances the
adaptability of LLMs, ensuring that recommendations remain current and
contextually relevant [26]. This advancement paves the way for more precise
and targeted course recommendations that adapt to changes in educational
content and learner preferences. Despite these improvements, there is a
noticeable gap in research specifically focused on using LLMs in course
recommender systems, particularly in addressing the “cold start” problem where
the system lacks a user’s profile. Thus, our study aims to investigate the
potential of LLMs, particularly those enhanced by RAG, in providing course
recommendations tailored to individual user needs. We introduce a course
recommender system, RAMO (Retrieval-Augmented Generation for MOOCs), which
employs a RAG-based LLM model (refer to Figure 1). RAMO leverages RAG’s
advantage to improve the quality of course recommendations, addressing and
mitigating common issues associated with LLMs especially in “cold start”
problem.

##  2 Related Works

###  2.1 Course Recommender Systems

Course recommender systems are essential in educational technology, helping
students choose courses that align with their interests and academic goals.
Many prior studies have employed collaborative filtering methods to build
course recommender systems [2, 31, 19]. For instance, Schafer et al. [31]
proposed a recommender system that suggested courses based on the preferences
of similar users. A more recent example by Koren et al. [19] developed
advanced collaborative filtering techniques to enhance course recommendation
accuracy. However, a significant issue arises when recommending courses for
new users, as there is no historical data available for these individuals—this
is known as the “cold start” problem [35]. To address this challenge, a recent
study by Wu et al. [35] leveraged large language models (LLMs), which utilize
extensive pre-trained knowledge from web datasets, demonstrating potential in
overcoming the cold start problem. Despite the advancements in LLMs, their
integration into course recommendation systems remains largely unexplored,
presenting an opportunity for future research to innovate and improve student
course selection processes.

###  2.2 Large Language Models in Education

Large language models (LLMs) like ChatGPT, trained on extensive datasets, have
the ability to generate human-like text and respond to questions with
exceptional precision [12, 36]. Many studies have highlighted the potential of
LLMs in educational applications, leveraging their capabilities to enhance
various aspects of teaching and learning. For example, Kabir and Lin [16]
developed an adaptive practicing system utilizing ChatGPT to generate
personalized questions and feedback, demonstrating LLMs’ potential in
facilitating tailored educational interactions. Researchers investigated
multiple GPT models on their ability to generate tailored learning materials
and provide instant feedback on student errors, enhancing personalized
learning experiences [34]. Huber et al. [13] demonstrated the use of LLMs in
creating interactive, conversational systems that assist both students and
teachers by providing adaptive learning support and resources. Moreover, LLMs
are also used in generating automatic feedback for students [5, 6], handling
sparse learner performance data [37] from intelligent tutoring systems,
predicting learning performance [38], and supporting tutor training session
[23].

###  2.3 Retrieval-Augmented Generation in Education

Retrieval-augmented generationn (RAG) has emerged as a significant technique
to enhance the effectiveness of educational tools powered by LLMs. For
example, a study [20] integrated textbook content into LLM prompts via RAG
improved the quality of responses in interactive question-answering (QA)
scenarios for middle-school math students, and demonstrated that students
generally prefer responses generated by RAGs. RAG has also been employed in
programming education to generate improved feedback for student’s completion
of coding tasks [14], by incorporating transcriptions of lecture recordings
and using timestamps as meta-information, RAG reduces hallucinations and
ensures the use of accurate technical terms. Moreover, RAG has been utilized
to assess novice math tutors’ use of social-emotional learning strategies
[22], they proved that RAG-enhanced prompts demonstrated more accurate and
cost-effective performance compared to other prompting strategies by providing
relevant external content. This application highlights the potential of RAG in
developing personalized tutor training programs and enhancing the overall
effectiveness of tutored learning.

While traditional course recommender systems have laid the groundwork for
personalized education, the integration of LLMs and techniques such as RAG
offers unprecedented opportunities for enhancing educational experiences.
These advanced methods address limitations of earlier approaches and pave the
way for more sophisticated and effective educational tools, inspiring us to
utilize RAG in developing our course recommender system.

##  3 Method

###  3.1 Dataset

In this study, we utilized the “Coursera Courses Dataset
2021”111https://www.kaggle.com/datasets/khusheekapoor/coursera-courses-
dataset-2021 from Kaggle. The dataset, scraped from Coursera’s publicly
available information in September 2021, contains a variety of courses that
feature comprehensive details such as skill requirements, difficulty levels,
and direct course links. It provides a robust knowledge base for our RAMO
system, enabling it to suggest courses tailored to students’ specific skills
and educational needs. This dataset effectively supports our objective to
enhance accessibility and personalized learning through course
recommendations. We first cleaned the dataset to remove meaningless symbols
and duplicate rows, and it has 3,342 non-duplicate courses in total after
data-cleaning, with 6 columns:

  * •

Course Name: The title of the course.

  * •

University: The institution offering the course.

  * •

Difficulty Level: The level of complexity of the course content.

  * •

Rating: The average rating given by learners.

  * •

URL: The web address where the course can be accessed.

  * •

Description: A brief overview of what the course covers.

  * •

Skills: The specific abilities or knowledge areas that the course aims to
develop.

###  3.2 Recommendation System Design

####  3.2.1 Prompt Design

The “cold start” problem, where systems lack user historical data, is a
significant challenge in recommendation systems. Both traditional course
recommender algorithums like content-based and collaborative-filtering
algorithms and LLM-based system recommendation systems struggle with this
issue. However, our RAG-based solution addresses this by using a ‘prompt
template’ in the back-end. This template guides RAMO to generate relevant
responses even when no user-specific data is available, as detailed in Table
1. The RAMO system can provide meaningful recommendations from the outset,
unlike non-RAG-based recommender systems, which lack a retrieval process and
prompt-based customization. The prompt to our retriever (i.e., to retrieve the
relevant docs from the databases) is called the ‘prompt template’, which is
shown in Table 1. The prompt to our generator is composed with three parts: 1)
User Question, 2) Prompt Template, and 3) Search Results (the context of the
retrieved relevant documents). We also added the uplifting adverb ‘fantastic’
to the prompt template, to elevate it with Emotional Intelligence since
ChatGPT is designed to recognize patterns in language, including those
associated with emotions [33].

Table 1: Overview of interaction prompt structure Prompt Template
---
You are a fantastic Coursera course recommender. Use the following pieces of
context to answer the question and recommend relevant courses to the user. If
the user doesn’t specify their requirements, you can just recommend some
courses that are most popular in the system based on their ratings and
difficulty levels. You only need to provide the course title to the user.
Also, please pay attention to how many courses the user wants you to
recommend. If you don’t know the answer, just say “I don’t know”.
Context
Retrieved course data
User Question
User’s specific question to the generator

####  3.2.2 Integration of RAG approach

As shown in Table 2 below, we employed several LLMs to build our course
recommender system. We provide a list of the LLM models we used, along with
details on their associated costs and token limits. The token limit refers to
the maximum number of tokens (a token represents about 3/4 of a word or four
characters, according to Open AI [1]) that the model can process in a single
input. While some models, like Llama 2 and Llama 3, are free to use on small-
scale dataset, due to their open-source nature, others may incur costs based
on usage or subscription plans [27].

Table 2: Cost and token limit of models we used LLM Model | Output Cost | Token Limit
---|---|---
GPT-3.5 Turbo |  $0.50$ per 1M tokens | 4,096 tokens
GPT-4 |  $30.00$ per 1M tokens | 8,192 tokens
Llama-2 | Free | 4,096 tokens
Llama-3 | Free | 8,000 tokens

Workflow of the RAG facilitated course recommendation system we proposed

Figure 2: Workflow for the RAMO System

We then leveraged the RAG approach to enhance the system’s understanding of
the user context. As shown in Figure LABEL:fig:_diagram_of_RAMO, RAG consists
of two primary components: the retriever and the generator. The retriever aims
to enhance the prompt templates, which ‘augment’ the retrieval process,
tailoring it to specific user queries. The knowledge base used for the
retrieval process can contain any format of course data (e.g., csv, pdf, and
json), providing a flexible and rich source of information for generating
responses and we used the largest MOOC platform—coursera’s course dataset in
csv format as the knowledge base. The dataset was transformed into text
embeddings and stored in the vector database. These embeddings were then used
to find high-quality, relevant information, which was incorporated into the
prompt for the generator. Here we use OpenAI embedding model (text-embedding-
ada-002 [29]) to tokenize the course data and store the embeddings in vector
store, considering its advantage over BERT (Bidirectional Encoder
Representations from Transformers) [7], while OpenAI embeddings [29] offer
better generalization and contextual understanding [28], making them more
suitable for diverse educational content. The generator is powered by LLMs,
which generate the textual contents based on the engineered prompts. To
facilitate user’s interaction with the system, we make the recommendation
process to be completed via conversational manner.

The interface of our recommender system is shown in Figure 1, where we listed
5 default courses based on their ratings in the dataset on the web page to
make it more user-friendly. As for the implementation of the system, we use
GPT-3.5 Turbo, selected for its robust integration with the LangChain [32]
framework—a platform designed to streamline the implementation of language
models in application-specific contexts. This setup allows the system to
dynamically retrieve relevant documents and generate responses tailored to
user inputs, as illustrated in the workflow in Figure 2.

####  3.2.3 Comparative Analysis

To evaluate the performance of our system, we conducted a series of tests by
providing different prompts representing various user needs to RAMO. This
allowed us to explore its ability to deliver course recommendations based on
the outputs generated in response to varied user prompts.

LLM vs. Non-LLM. We explored both the relevance of the recommended courses to
the user’s interests and responding time (the time it takes to generate a
response) of the LLM-based recommender system compared to non-LLM course
recommender systems (e.g., course recommender system using collaborative
filtering and content-based approaches), focusing particularly on their
ability to address the “cold start” problem. This problem occurs when the user
lacks specific requirements on what skills they want to learn, and the system
lacks data on the new user.

LLM vs. LLM with RAG. We further examined the performance of a standard LLM
recommender system (without RAG and without using a dataset as a knowledge
base) versus an RAG-enhanced LLM recommender system by testing different
prompt templates for the retriever and various user queries for the generator
to ascertain improvements in system performance and recommendation
personalization.

To explore the performance of our course recommender system, we focused on
comparing the relevance of the recommended courses to different prompts by
varying prompt templates and user-specific requirements.

##  4 Results

###  4.1 LLM vs. Non-LLM

LLM output Vs Rag-Based Output for New User who want to learn something on
coursera

Figure 3: Sample output for a cold-start question on LLM vs RAG-LLM system

We compared RAMO with a traditional course recommendation system built by the
content-based and collaborative filtering using the same
dataset222https://www.kaggle.com/code/sagarbapodara/coursera-course-
recommendation-system-webapp. During this comparison, we focused on the “cold
start” problem. The “cold start” problem is especially pertinent in the
context of an e-learning platform for tutor training, such as tutor training
platform [24]. When new tutors join the platform, they are encouraged to
complete various training courses to enhance their tutoring skills. Given the
wide range of courses available, new tutors may feel overwhelmed when deciding
where to begin their learning journey. In such scenarios, they may ask general
questions such as, “What can I learn today since I am a new tutor onboarding
to this platform?” They do not have prior course completions or specific
learning preferences logged in the system, making it challenging for the
recommendation system to personalize suggestions based on historical data.
When prompted with “I am a new user”, the traditional recommender system
failed to generate a recommendation because its algorithm relies on the cosine
similarity of the descriptive texts of the user’s desired learning topic and
the database items, and there are no courses with similar title or description
as the phrase ‘new user’. In contrast, both our standard LLM and the RAG-
enhanced LLM system can provide relevant course suggestions for the new user,
with the LLM offering more detailed descriptions based on its internal
knowledge base and RAG offering more customized outputs based on its external
knowledge base and the prompt template we designed. The comparative results
for both the standard and RAG-based recommender systems are displayed in
Figure 3.

Regarding system performance, the traditional system typically took about 0.02
seconds longer than RAMO to generate responses according to the same user
interest—a certain topic the user wants to learn, and this delay increased
with the complexity of the user’s input regarding relevant skills.

###  4.2 LLM vs. LLM with RAG

To explore how well our LLMs can provide personalized course recommendations,
we used prompts that specified a particular skill to be learned. The non-RAG
LLM (based on GPT-3.5) delivered detailed suggestions for relevant courses
available on Coursera, utilizing its internal database of courses. In
contrast, the recommendations from the RAG-enhanced LLM varied according to
the specific prompt template used by the retriever. This adaptability allows
developers to tailor the quantity and detail of the courses recommended,
showcasing the flexibility of the RAG approach. The user interface and the
outcomes for a query focused on learning a specific skill are illustrated in
Figure 4.

User Entry and the Output part of the system we developed

Figure 4: Output for a specific user question

We modified the retrieval prompts and generation queries to test the
adaptability of our recommendation system. First, we conducted tests on
various user queries using the same prompt template to compare the variations
in output. The first module in Figure 5 illustrates the system’s response to a
“cold start” problem, while modules 2 through 6 demonstrate how the output
varies based on user questions about the number of courses recommended and the
level of detail provided, such as reasons for recommendations, URLs, and other
specifics. For example, when user asks question like “I want to learn python,
can you recommend me some courses?”, RAMO can give the output to the user:
“Sure! Here are some recommended Python courses for you: 1\. Introduction to
Python 2\. Crash Course on Python 3\. First Python Program 4\. Python Basics
These courses cover a range of topics from basic syntax to building
interactive applications. Happy learning!” When the user changes their mind
and decides to learn about another topic, RAMO can give relevant
recommendations. The outputs consistently matched the user requirements in
relevance, successfully retrieving the pertinent courses from the Coursera
dataset, more examples could be found at Figure 5.

Different User Questions and Responses of the Generator

Figure 5: User questions and related outputs

Different Prompt Templates and Responses of the Generator

Figure 6: Prompt templates and related outputs

We also utilized different retrieval prompt templates to explore how the
output varies based on different prompts. Specifically, we used the same user
question “I want to learn python”, and altered the prompt templates to specify
the number of recommended courses and the level of detail provided in the
output, ranging from mere course titles to comprehensive descriptions that
include titles, URLs, and rationales for each recommendation. The variations
in the prompt templates and their corresponding outputs are illustrated in
Figure 6. Here, red lines highlight changes in the number of courses
recommended, blue lines detail the content of the courses—such as the
inclusion of reasons for recommendations or just the course titles, ratings,
and URLs—while green highlights how we addressed the “cold-start” problem,
resulting in recommendations of the three most popular (based on course
ratings) and easiest courses (based on its difficulty level), as depicted in
the output module labeled 1 in Figure 6. The generated response in response to
varied prompts underscores the system’s robustness; for instance, when the
template specifies “recommend three courses at a time”, the output
consistently includes exactly three courses. Similarly, if the prompt contains
‘course URLs and titles’, the system reliably appends this information to each
recommended course, ensuring that the output meticulously adheres to the
specified criteria.

##  5 Conclusions

In this study, we have demonstrated the application of LLMs as course
recommender systems, particularly within MOOCs. Our findings confirm the
potential of LLMs to deliver personalized course recommendations based on
user’s different requirements. We initially compared four LLMs, including
GPT-3.5 Turbo and GPT-4. Ultimately, we selected GPT-3.5 as the back-end model
for the RAMO system due to its comparable performance to GPT-4 at a lower
cost. Although the Llama models are free to access, we found that the GPT
models were significantly faster. Specifically, GPT-3.5 had an approximate
response time of 3 seconds, whereas Llama 2 and Llama 3 took approximately 5
minutes and 8 minutes, respectively. Furthermore, the integration of RAG has
enhanced the quality of recommendation outputs, as evidenced by the generated
responses based on various user prompts, which are highly related to user’s
needs and all came from the knowledge base. Additionally, our system supports
conversational interaction with users, which could be seamlessly integrated
into numerous online educational platforms. Our use of open-source LLMs (e.g.
Llama 2 and Llama 3 [27]) has also been validated, proving to be a cost-
effective approach for broader deployment.

Limitations
As this study is ongoing, we have not yet conducted comprehensive evaluations
of our recommender systems, including human evaluations or user studies. This
is primarily due to the nascent stage of our research. Moreover, while many
research projects on recommendation systems employ benchmarks to evaluate
system adaptability, our study currently lacks such benchmarks because we do
not possess a test dataset. The Coursera dataset we utilized includes only
course data, lacking user profiles which are essential for evaluating the
effectiveness of recommender systems across different time periods. If we had
access to user data, including users’ past course learning histories and their
preferences, we could integrate this information with the course data to
enhance our retrieval process. This integration would allow us to personalize
recommendations more effectively, tailoring course suggestions to individual
learning patterns and preferences. Incorporating detailed user data would
enable RAMO to provide more accurate and relevant recommendations, improving
user satisfaction and engagement. It would also allow for longitudinal studies
to track how users’ interactions with the system evolve over time and how well
the recommendations align with their long-term learning goals.

Future Work
We plan to undertake several further steps to advance our research. Firstly,
we aim to conduct thorough evaluations and tests to validate the efficacy and
reliability of our recommender systems. This will involve integrating user
studies and utilizing real user data once our systems are deployed on our
e-learning platform. Such measures will enable us to robustly measure
performance and refine our approach. Secondly, we will focus on enhancing
system performance, considering scalability and the potential to expand our
technology to encompass a broader range of educational tools and platforms.
These efforts will ensure that our recommender systems not only meet current
educational needs but also adapt to future demands and technological
advancements. Thirdly, we could deploy RAMO on our own e-learning platform,
and then have the opportunity to gather comprehensive user data and utilize
our own course dataset rather than Coursera’s. This deployment would allow us
to conduct extensive testing and validation, further proving the eligibility
and effectiveness of the LLM for recommending courses. With access to real-
time user data, we could continuously refine our algorithms, making the system
more adaptive and responsive to users’ evolving needs.

To evaluate the effectiveness of our LLM-based course recommendation system,
we plan to conduct a comprehensive experiment that includes quantitative
metrics, user studies, and personalization improvements. Our experiment aims
to assess both the relevancy of the recommendations and the satisfaction of
the users with the recommended courses.

We will utilize several quantitative metrics to evaluate the performance of
the recommendation system. Key metrics include post-test performance, measured
by the improvement in students’ scores from pre-test to post-test after
tutoring sessions, and course completion rate, which compares the rate of
course completion between students who follow the system’s recommendations and
those who do not. Additionally, engagement rate will be tracked by monitoring
whether students continue engaging with the lesson without dropping out
midway. User satisfaction will also be assessed through feedback collected
after each lesson via a thumbs-up or thumbs-down system and detailed surveys.
To gather qualitative insights into the system’s effectiveness and user
experience, we will conduct user studies. These will involve satisfaction
surveys completed by students following each lesson to gauge their
satisfaction with the course content and the relevance of the recommendations,
as well as focus group discussions to explore students’ experiences in more
depth and gather suggestions for improvement.

##  6 Acknowledgments

We extend our sincere gratitude to Chenfei Lou, a current software engineer at
X (former twitter), for his invaluable guidance in developing our demo. We
also thank Sandy Zhao, a current master’s student in the CMU METALs program,
for her excellent assistance in generating the wonderful diagram.
Additionally, we appreciate Yuting Wang, an undergraduate student at CMU, for
her help in refining the design in this paper.

## References

  * [1] Maximum length - netdocuments, 2024.  Accessed: 2024-07-05.
  * [2] J. Basilico and T. Hofmann.  Unifying collaborative and content-based filtering.  In Proceedings of the twenty-first international conference on Machine learning, page 9, 2004.
  * [3] M. H. Baturay.  An overview of the world of moocs.  Procedia - Social and Behavioral Sciences, 174:427–433, 2015.  International Conference on New Horizons in Education, INTE 2014, 25-27 June 2014, Paris, France.
  * [4] N. M. Castillo, J. Lee, F. T. Zahra, and D. A. Wagner.  Moocs for development: Trends, challenges, and opportunities.  Information Technologies & International Development, 11(2):pp–35, 2015.
  * [5] W. Dai, J. Lin, H. Jin, T. Li, Y.-S. Tsai, D. Gašević, and G. Chen.  Can large language models provide feedback to students? a case study on chatgpt.  In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE, 2023.
  * [6] W. Dai, Y.-S. Tsai, J. Lin, A. Aldino, F. Jin, T. Li, D. Gasevic, et al.  Assessing the proficiency of large language models in automatic feedback generation: An evaluation study.
  * [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
  * [8] Z. Feng, X. Feng, D. Zhao, M. Yang, and B. Qin.  Retrieval-generation synergy augmented large language models.  In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11661–11665. IEEE, 2024.
  * [9] Y. Gao, T. Sheng, Y. Xiang, Y. Xiong, H. Wang, and J. Zhang.  Chat-rec: Towards interactive and explainable llms-augmented recommender system, 2023.
  * [10] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang.  Retrieval-augmented generation for large language models: A survey.  arXiv preprint arXiv:2312.10997, 2023.
  * [11] Z. Gulzar, A. A. Leema, and G. Deepak.  Pcrs: Personalized course recommender system based on hybrid approach.  Procedia Computer Science, 125:518–524, 2018.
  * [12] Hasan.  How does chatgpt generate human-like text?, November 2023.
  * [13] S. E. Huber, K. Kiili, S. Nebel, R. M. Ryan, M. Sailer, and M. Ninaus.  Leveraging the potential of large language models in education through playful and game-based learning.  Educational Psychology Review, 36(1):25, 2024.
  * [14] S. Jacobs and S. Jaschke.  Leveraging lecture content for improved feedback: Explorations with gpt-4 and retrieval augmented generation.  arXiv preprint arXiv:2405.06681, 2024.
  * [15] J. Jeevamol and V. Renumol.  An ontology-based hybrid e-learning content recommender system for alleviating the cold-start problem.  Education and Information Technologies, 26:4993–5022, 2021.
  * [16] M. R. Kabir and F. Lin.  An llm-powered adaptive practicing system.  In AIED 2023 workshop on Empowering Education with LLMs-the Next-Gen Interface and Content Generation, AIED, 2023.
  * [17] J. Knox.  Digital culture clash:“massive” education in the e-learning and digital cultures mooc.  Distance Education, 35(2):164–177, 2014.
  * [18] T. E. Kolb, A. Wagne, M. Sertkan, and J. Neidhardt.  Potentials of combining local knowledge and llms for recommender systems.  In # PLACEHOLDER_PARENT_METADATA_VALUE#, volume 3560, pages 61–64. CEUR-WS. org, 2023.
  * [19] Y. Koren, S. Rendle, and R. Bell.  Advances in collaborative filtering.  Recommender systems handbook, pages 91–142, 2021.
  * [20] Z. Levonian, C. Li, W. Zhu, A. Gade, O. Henkel, M.-E. Postle, and W. Xing.  Retrieval-augmented generation to improve math question-answering: Trade-offs between groundedness and human preference.  arXiv preprint arXiv:2310.03184, 2023.
  * [21] J. Li, W. Zhang, T. Wang, G. Xiong, A. Lu, and G. Medioni.  Gpt4rec: A generative framework for personalized recommendation and user interests interpretation, 2023.
  * [22] J. Lin, A. Gurung, D. R. Thomas, E. Chen, C. Borchers, S. Gupta, K. R. Koedinger, et al.  Improving assessment of tutoring practices using retrieval-augmented generation.  arXiv preprint arXiv:2402.14594, 2024.
  * [23] J. Lin, Z. Han, D. R. Thomas, A. Gurung, S. Gupta, V. Aleven, and K. R. Koedinger.  How can i get it right? using gpt to rephrase incorrect trainee responses.  arXiv preprint arXiv:2405.00970, 2024.
  * [24] J. Lin, D. R. Thomas, Z. Han, W. Tan, N. D. Nguyen, S. Gupta, E. Gatz, C. Tipper, and K. R. Koedinger.  Personalized learning squared (plus): Doubling math learning through ai-assisted tutoring.  2023\.
  * [25] Y. Lu, Y. Fang, and C. Shi.  Meta-learning on heterogeneous information networks for cold-start recommendation.  In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1563–1573, 2020.
  * [26] H. Lyu, S. Jiang, H. Zeng, Y. Xia, Q. Wang, S. Zhang, R. Chen, C. Leung, J. Tang, and J. Luo.  Llm-rec: Personalized recommendation via prompting large language models, 2024.
  * [27] Meta.  Llama: Large language model meta ai, July 2024.
  * [28] B. Moradiya.  The battle of language models: Openai vs. bert, October 2023.
  * [29] OpenAI.  New and improved embedding model, July 2024.
  * [30] C. Romero and S. Ventura.  Educational data mining: a review of the state of the art.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (applications and reviews), 40(6):601–618, 2010.
  * [31] J. B. Schafer, D. Frankowski, J. Herlocker, and S. Sen.  Collaborative filtering recommender systems.  In The adaptive web: methods and strategies of web personalization, pages 291–324. Springer, 2007.
  * [32] O. Topsakal and T. C. Akinci.  Creating large language model applications utilizing langchain: A primer on developing llm apps fast.  In International Conference on Applied Engineering and Natural Sciences, volume 1, pages 1050–1056, 2023.
  * [33] R. Vinay, G. Spitale, N. Biller-Andorno, and F. Germani.  Emotional manipulation through prompt engineering amplifies disinformation generation in ai large language models.  arXiv preprint arXiv:2403.03550, 2024.
  * [34] S. Wang, T. Xu, H. Li, C. Zhang, J. Liang, J. Tang, P. S. Yu, and Q. Wen.  Large language models for education: A survey and outlook.  arXiv preprint arXiv:2403.18105, 2024.
  * [35] X. Wu, H. Zhou, Y. Shi, W. Yao, X. Huang, and N. Liu.  Could small language models serve as recommenders? towards data-centric cold-start recommendation.  In Proceedings of the ACM on Web Conference 2024, pages 3566–3575, 2024.
  * [36] ZDNet.  What is chatgpt and why does it matter? here’s everything you need to know, July 2024.
  * [37] L. Zhang, J. Lin, C. Borchers, M. Cao, and X. Hu.  3dg: A framework for using generative ai for handling sparse learner performance data from intelligent tutoring systems.  arXiv preprint arXiv:2402.01746, 2024.
  * [38] L. Zhang, J. Lin, C. Borchers, J. Sabatini, J. Hollander, M. Cao, and X. Hu.  Predicting learning performance with large language models: A study in adult literacy.  arXiv preprint arXiv:2403.14668, 2024.

##  Appendix A DATASET WE USE

The 2021 coursera dataset we use is available at
https://www.kaggle.com/datasets/khusheekapoor/coursera-courses-dataset-2021.

##  Appendix B Link of the RAG system code

The code could be run
at:https://colab.research.google.com/drive/1wLwM5QphDoIctW9_EZt26D6RIpSmfaCD?usp=sharing,
including the data preprocessing and the RAG process.

##  Appendix C Link of the Chatbot DEMO

The demo of the chatbot could be accessed
at:https://huggingface.co/spaces/dinosaur-organization/coursera-
recommendation#/, you need to first obtain your Openai Api and enter it on the
left side.

##  Appendix D Link of the Medium Blog inspired us of the demo design

We got inspired by a blog published on medium on the designing of a book
recommendation chatbot, the article is available at:https://medium.com/data-
and-beyond/data-science-and-machine-learning-books-recommendation-
chatbot-83757cbb92f7#/