# ResearchBot: Bridging the Gap between Academic Research and Practical
Programming Communities

Sahar Farzanehpour, Swetha Rajeev, Huayu Liang, Ritvik Prabhu, Chris Brown
saharfarza, rswetha, huayu98, ritvikp<EMAIL_ADDRESS>Virginia
TechBlacksburgVirginiaUSA24060

(2024)

###### Abstract.

Software developers commonly rely on platforms like Stack Overflow for
problem-solving and learning. However, academic research is an untapped
resource that could greatly benefit industry practitioners. The challenge lies
in connecting the innovative insights from academia to real-world problems
faced by developers. This project introduces ResearchBot, a tool designed to
bridge this academia-industry gap. ResearchBot employs a modular approach,
encompassing understanding questions, curating queries to obtain relevant
papers in the CrossRef repository, summarizing paper content and finally
answering user questions based on paper summaries. The core objective of
ResearchBot is to democratize access to academic knowledge for industry
professionals. By providing concise summaries of cutting-edge research
directly in response to SE-related questions, ResearchBot facilitates the
application of academic insights to practical contexts. Ultimately, it aims to
bridge the gap between academia and industry, using research evidence to
support learning and decision-making in software development.
Demo URL: https://youtu.be/ny6TIoNAor8
GitHub: https://github.com/Sfrznp/ResearchBot
Data Set: https://tinyurl.com/y3cu9mvx

Software Engineering, Software Tools, Natural Language Processing (NLP)

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; October 17–01, 2024; Sacramento, CA††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Software and its engineering Software notations and tools

##  1\. Introduction

In software engineering (SE), there is a significant gap between academic
research and practical programming communities. This gap is caused by
challenges such as the accessibility, relevance, and credibility of research
when applied to the fast-paced field of software development (Winters, 2024).
Despite the rigorous methods and extensive analyses used in SE research,
evidence from scientific insights often fail to reach practitioners seeking
solutions. Community-driven platforms (e.g., Stack
Overflow111https://stackoverflow.com/) are frequently used to support learning
and problem-solving in SE (Barua et al., 2014). More recently, large language
models (LLMs, e.g., ChatGPT222https://chat.openai.com/) are capable of
providing solutions to SE-related tasks (Leung and Murphy, 2023). However,
prior work highlights issues regarding academic findings in Stack Overflow
(Huang and Chattopadhyay, 2024) and ChatGPT (Auer et al., 2023), emphasizing
the need for novel approaches to align research evidence with practical
community needs.

To address these challenges, we introduce ResearchBot, an early prototype of a
chatbot designed to make academic research more accessible and relevant to
programming practitioners. ResearchBot leverages advanced natural language
processing (NLP) techniques to automatically understand user questions,
retrieve relevant papers from the CrossRef
repository333https://www.crossref.org/—a comprehensive database containing
metadata about scholarly articles, conference proceedings, books, and
datasets, and provide concise summaries. Our preliminary evaluation explores
the capabilities of ResearchBot to provide evidence-based and human-like
responses to practical SE-related queries. By transforming theoretical
knowledge into practical contexts, ResearchBot aims to provide a first step in
bridging the gap between academia and industry—facilitating the application of
research evidence to real-world SE problems (Kitchenham et al., 2004).

##  2\. Implementation

Figure 1. Pipeline for the implementation of ResearchBot

The implementation of ResearchBot integrates two main modules: the Retrieval
Module and the Question-Answer (QA) Module, leveraging GPT-4’s superior
capabilities of natural language understanding. This system utilizes specific
prompts to guide the model in querying the Crossref repository and generating
informative responses based on retrieved research papers. Figure 1 illustrates
the pipeline of ResearchBot, showcasing the flow from query to response
generation.

###  2.1. Retrieval Module

The Retrieval Module of ResearchBot is responsible for the process of
retrieving relevant academic papers based on user queries.

####  2.1.1. Prompt used to instruct GPT-4 to curate a query for the Crossref
API:

    From now on I will pass in a prompt like "How do I
    perform document similarity using NLP" and I would
    like you to  curate an output that looks similar to
    this "nlp+natural+language+processing+document+
    similarity". This is what a search query would look
    like for the given prompt. It uses ’+’ which
    separates all the key words.

####  2.1.2. Significance of Prompt:

The prompt used in this module is beneficial because it guides GPT-4 on how to
transform a natural language question into a structured query format. This
ensures that the search query generated by the model is specific and optimized
for retrieving relevant academic papers. Without a clear prompt, the model
might struggle to understand the intended format of the query and could
produce less accurate results.

####  2.1.3. Workflow:

  1. (1)

ResearchBot receives a user question.

  2. (2)

GPT-4 generates a structured query based on the user question, formatted with
appropriate keywords for the CrossRef API

(e.g., ”nlp+natural+language+processing+document+

similarity”).

  3. (3)

The query is sent to the CrossRef API to retrieve the top 15 most relevant
papers.

  4. (4)

Abstracts of these papers are extracted for further processing.

###  2.2. QA Module

The QA Module of ResearchBot processes the retrieved paper abstracts and user
question to generate a comprehensive response.

####  2.2.1. Prompt used to instruct GPT-4 to answer the user’s question:

    From now on I will pass in a Software Engineering
    question and some relevant research papers as
    context for the answer. I would like you to answer
    the question with references made to the contexts
    (make sure that the focus of the answer is to
    answer the question, not the context itself).
    Be sure to also include the URL after the reference
    to the paper in the text. Use this format to cite
    in-text (title).

####  2.2.2. Significance of Prompt:

The prompt used in the QA Module is crucial as it provides GPT-4 with clear
instructions on how to structure its response. By specifying the focus on
answering the user question with reference to the provided context (research
paper abstracts), the prompt ensures that the generated response is relevant
and informative. Additionally, the prompt dictates the citation format,
enabling proper acknowledgment of the cited sources within the response.

####  2.2.3. Workflow:

  1. (1)

The user question and retrieved paper abstracts (along with their URLs) are
passed to GPT-4.

  2. (2)

GPT-4 generates a concise and informative response, utilizing the context
provided by the abstracts to address the user’s question.

  3. (3)

Citations to relevant papers are embedded within the response text using the
specified format.

###  2.3. Reasoning behind the use of LLMs

LLMs powered by generative artificial intelligence (AI), such as GPT-4, offer
several advantages for ResearchBot:

  * •

Natural Language Understanding: LLMs excel in understanding and generating
human-like text, enabling them to process complex queries and produce coherent
responses.

  * •

Contextual Awareness: GPT-4 can leverage context from multiple sources (e.g.,
abstracts of research papers) to provide informed answers tailored to specific
questions.

  * •

Flexibility and Adaptability: The model’s flexibility allows it to adapt to
various domains and tasks, making it suitable for bridging academic literature
with practical programming challenges.

##  3\. Methodology

###  3.1. Data Collection

For the purpose of this study, data was systematically collected from the
Software Engineering Stack Exchange
community.444https://softwareengineering.stackexchange.com/ This platform was
selected due to its extensive use by programming professionals, who regularly
engage in discussions concerning practical challenges and their solutions. Our
selection criteria focused primarily on technical inquiries, particularly
those pertaining to subjects such as programming languages and software
development. Recognizing the potential limitation in our bot’s ability to
accurately respond to coding-related questions, we avoided using inquiries
from Stack Overflow. However, we still included some coding questions from SE
Stack Exchange in our evaluation. Using these criteria, we collected 100
questions to analyse the functionality of ResearchBot.

###  3.2. Data Analysis

####  3.2.1. Semantic Textual Similarity

We utilized semantic textual similarity (STS), an NLP technique for
determining how similar two pieces of text are in terms of meaning, to test
and analyze our ResearchBot’s performance and accuracy. The primary goal of
this analysis is to compare the likeness in meaning between the user’s answer
to a question on SE Stack Exchange and ResearchBot’s generated response. The
higher their similarity, the better our tool answers the user’s question. To
achieve our goal, we used the Sentence-BERT 555https://sbert.net/ (SBERT)
model and the similarity method from spaCy
666https://spacy.io/usage/linguistic-features#vectors-similarity, an open-
source NLP library in Python.

####  3.2.2. LLM Comparison

To further assess ResearchBot’s performance, we compared its responses with
those generated by other LLM tools commonly used in SE: Google Gemini, GitHub
Copilot, and ChatGPT. This evaluation involved 50 top questions from Stack
Exchange. Upon manual review of the answers, we found responses from Gemini
and Copilot were unreliable and lacked quality references. ChatGPT offered
better resources; thus, we use ChatGPT as the state-of-the art platform to
compare against ResearchBot.

####  3.2.3. Analysis Process

Firstly, we selected 100 question-and-answer sets for testing and analysis. We
then wrote a script to generate corresponding answer outputs for the 100
questions by ResearchBot and ChatGPT. Next, we used the STS tools, SBERT and
spaCy, to compare the answers most upvoted from users (top answer) and answers
produced by ResearchBot (generated answer). We use the top answer to represent
the correct answer, most accepted by humans with similar inquiries. Finally,
we calculated each comparison set’s semantic similarity score and each STS
tool’s overall statistic measurement characteristics of data. We also employed
the STS tools to compare generated answers with those produced by ChatGPT.
This analysis may introduce biases, as it is based on the assumption that the
top answer is both correct and precise. This could lead to overestimating the
bot’s performance, as similarity with the top answer might not represent the
most accurate or comprehensive response. To enhance the validity of our
results, the incorporation of a broader range of benchmarks and metrics are
recommended.

##  4\. Results

This section will present the results of our data collection and analysis
approach. Table 1 and Table 2 present the statistical results for semantic
similarity comparing the performance of ResearchBot with Stack Exchange and
ChatGPT, respectively.

Table 1. Stack Exchange vs. ResearchBot Semantic Textual Similarity (STS) Tools | Maximum | Minimum | Average | Median | Std Dev
---|---|---|---|---|---
Sentence-BERT model | 0.9147 | 0.0341 | 0.6609 | 0.6788 | 0.1618
spaCy library | 0.9473 | 0.7908 | 0.9203 | 0.9335 | 0.0506

###  4.1. STS Results

The results presented in Table 1 demonstrate that ResearchBot performs
effectively in answering users’ questions as analyzed by spaCy, with an
average similarity score of 0.92. However, the performance analyzed by SBERT
shows a lower average of 0.66. This discrepancy is largely due to a notably
low minimum similarity score of 0.0341 recorded by SBERT. This result
highlights SBERT’s higher scrutiny evaluation criteria, contributing to higher
variability in scores, as indicated by a standard deviation of 0.16, compared
to spaCy. Such variability suggests a re-evaluation of spaCy’s data handling,
as it indicates methodological differences, suggesting a need for a deeper
comparative analysis of how each tool handles and interprets textual
variations. Further analysis could provide insights into the different
sensitivities of the tools to specific text features or contexts, thus
explaining the variations in their performance metrics. Moreover, similar to
recent efforts in sentiment analysis, future work can explore SE-specific NLP
tools for STS.

Overall, the answer outputs provide further insights, demonstrating
ResearchBot not only matches the top answers but also provides more
comprehensive and detailed responses with research-based evidence. This
ability to elaborate and expand on critical points makes the tool particularly
valuable. Consequently, while ResearchBot shows promise, there is room for
improvements of our tool and evaluation in answering SE-related questions.

###  4.2. Comparison Results

We compared ResearchBot with state-of-the-art ChatGPT in responses to SE-
related questions. We employed the STS tools to compare the answers generated
by ChatGPT and ResearchBot. The scores are detailed in Table 2. ResearchBot
achieved an average score of 0.9525 with spaCy and 0.8175 with SBERT,
indicating that its answers are quite similar to those from ChatGPT. However,
a closer examination revealed issues with the evidence provided in ChatGPT
responses. It is worth noting that, though some answers did reference academic
papers, we observed several instances of broken links and unreliable websites.
ResearchBot, in contrast, demonstrated superior reliability and resource
quality in its answers. To further assess the responses, we are conducting a
user study to determine user preferences between ChatGPT and ResearchBot.

Table 2. ChatGPT vs. ResearchBot Semantic Textual Similarity (STS) Tools | Maximum | Minimum | Average | Median | Std Dev
---|---|---|---|---|---
Sentence-BERT model | 0.9323 | 0.7137 | 0.8175 | 0.8195 | 0.0426
spaCy library | 0.9874 | 0.8432 | 0.9525 | 0.9666 | 0.0506

##  5\. Discussion

###  5.1. Implication and Impact of the Results

By utilizing STS tools and comparing against mainstream LLMs, we gained
valuable insights into how ResearchBot performs in generating accurate and
relevant responses to user inquiries. The results show a mean STS scores of
0.66 (SBERT) and 0.92 (spaCy), indicating accurate human-like responses to
practical SE-related questions. Further manual analysis shows that, despite
lower scores for SBERT, ResearchBot’s answers are not only correct but provide
further background to answer the user’s question. We also show our system
responses are comparable to ChatGPT while providing more reliable responses
based on research evidence. The implications of our study extend to practical
contexts, motivating future tools to facilitate theoretical knowledge (as in
non-coding questions—i.e., conceptual questions) transfer between academia and
industry. By enhancing ResearchBot’s ability to provide accurate and
informative responses based on academic knowledge, we contribute to bridging
the academic-industrial gap in SE.

###  5.2. Future Work

We envision the following future work to further extend and evaluate
ResearchBot:

##### User Study to Further Evaluate the Responses.

A limitation of our preliminary study is we rely on STS comparisons with top
answers from SE Stack Exchange and ChatGPT responses to evaluate the
capabilities of ResearchBot. As previously mentioned, we are currently
conducting a user study where participants are asked to choose between answers
to practical SE-related questions generated by ResearchBot and ChatGPT,
providing a rationale for their preference. This study aims to provide a
better understanding of the types of answers users prefer, motivate
enhancements to ResearchBot, and further promote evidence-based practices in
SE contexts.

##### Expansion of Academic Repositories.

Currently, ResearchBot only utilizes the Crossref repository to retrieving
academic papers for summarization in responses to practical SE questions.
Future iterations will incorporate additional repositories such as arXiv, ACM,
IEEE, or academic journals in specific SE-related domains. This expansion
would enrich the diversity of available research sources and broaden the scope
of insights provided to users.

##### User Interaction and Feedback Mechanisms.

The preliminary prototype of ResearchBot has limited features. Implementing
interactive features, such as feedback loops, personalized recommendations,
and the ability for follow-up inquiries and discussions in chat form similar
to other LLM-based systems, can enhance ResearchBot’s user experience. In
addition, integrating mechanisms for users to rate responses and provide
feedback will enable iterative improvements based on community input.

##### Code Understanding, Creation, and Correction.

A limitation of our chatbot is that it does not have the ability to generate
or understand code to provide responses to SE-related questions. Advancing
ResearchBot’s capabilities to understand, create, and/or correct code based on
extracted insights from academic research can empower users to not only obtain
theoretical knowledge but also practical code solutions aligned with research
findings, enhancing evidence-based practices in industry.

##### Integration with Programming Platforms.

We also envision exploring integration opportunities for our system in popular
programming platforms beyond SE Stack Exchange, such as GitHub or developer
forums. This can expand ResearchBot’s reach and utility within the broader SE
community. Further, integration into integrated development environments
(IDEs) can also facilitate insights from ResearchBot during software
development to answer questions and inform decision making.

##  6\. Conclusion

ResearchBot applies advanced AI techniques to connect academic SE research
with programming communities. By using NLP and intelligent summarization, our
tool helps developers access and use relevant research insights more
effectively. Our findings indicate that the early prototype of ResearchBot
provides accurate human-like responses to practical SE inquiries, comparable
to ChatGPT with more reliable research-based evidence supporting responses.
Research Bot has the potential to transform how programmers interact with
academic knowledge. The tool’s ability to interpret user queries, search
repositories, and summarize key findings from research papers can make
scientific knowledge more accessible to practitioners, meeting immediate needs
and helping tackle complex SE challenges with evidence-based responses.

## References

  * (1)
  * Auer et al. (2023) Sören Auer, Dante AC Barone, Cassiano Bartz, et al. 2023\.  The sciqa scientific question answering benchmark for scholarly knowledge.  _Scientific Reports_ 13, 1 (2023), 7240.
  * Barua et al. (2014) Anton Barua, Stephen W Thomas, and Ahmed E Hassan. 2014.  What are developers talking about? an analysis of topics and trends in stack overflow.  _Empirical software engineering_ 19 (2014), 619–654.
  * Huang and Chattopadhyay (2024) Run Huang and Souti Chattopadhyay. 2024.  A Tale of Two Communities: Exploring Academic References on Stack Overflow. In _Companion Proceedings of the ACM on Web Conference 2024_. 855–858.
  * Kitchenham et al. (2004) Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004.  Evidence-based software engineering. In _Proceedings. 26th International Conference on Software Engineering_. IEEE, 273–281.
  * Leung and Murphy (2023) Mira Leung and Gail Murphy. 2023.  On Automated Assistants for Software Development: The Role of LLMs. In _2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 1737–1741.
  * Winters (2024) Titus Winters. 2024.  Thoughts on applicability.  _Journal of Systems and Software_ 215 (2024), 112086.