0009-0005-4176-4861Hao-Cheng Lo

# From PARIS to LE-PARIS: Toward Patent Response Automation with Recommender
Systems and Collaborative Large Language Models

Jung-Mei Chu<EMAIL_ADDRESS>Hao-Cheng Lo<EMAIL_ADDRESS>Jieh
Hsiang<EMAIL_ADDRESS>Chun-Chieh Cho<EMAIL_ADDRESS>JCIPRNET,
Taipei, Taiwan  Department of Computer Science and Information Engineering,
National Taiwan University, Taipei, Taiwan  Department of Psychology, National
Taiwan University, Taipei, Taiwan

###### Abstract

[S U M M A R Y] In patent prosecution, timely and effective responses to
Office Actions (OAs) are crucial for acquiring patents, yet past automation
and AI research have scarcely addressed this aspect. To address this gap, our
study introduces the Patent Office Action Response Intelligence System (PARIS)
and its advanced version, the Large Language Model Enhanced PARIS (LE-PARIS).
These systems are designed to expedite the efficiency of patent attorneys in
collaboratively handling OA responses. The systems’ key features include the
construction of an OA Topics Database, development of Response Templates, and
implementation of Recommender Systems and LLM-based Response Generation. Our
validation involves a multi-paradigmatic analysis using the USPTO Office
Action database and longitudinal data of attorney interactions with our
systems over six years. Through five studies, we examine the constructiveness
of OA topics (studies 1 and 2) using topic modeling and the proposed Delphi
process, the efficacy of our proposed hybrid recommender system tailored for
OA (both LLM-based and non-LLM-based) (study 3), the quality of response
generation (study 4), and the practical value of the systems in real-world
scenarios via user studies (study 5). Results demonstrate that both PARIS and
LE-PARIS significantly meet key metrics and positively impact attorney
performance.

###### keywords:

patent office action response recommendation system large langauge model

##  1 Introduction

To address the increasing volume and complexity of inventions filed with the
United States Patent and Trademark Office (USPTO) over the last two decades
(USPTO, 2023), automation and language model (LM) technologies have been
extensively applied in patent drafting and prosecution, including tasks such
as patent classification (Lee and Hsiang, 2020b) and claim generation (Lee and
Hsiang, 2020a), as well as early stages of patent examination like prior art
search (Helmers et al., 2019) and patentability assessment (Lo and Chu, 2021).

Despite these efforts, automation has not thoroughly benefited the critical
aspect of the patent prosecution lifecycle: the Office Action (OA) and
response to OA process. This process involves intricate communication and
extensive technical and legal knowledge exchange among examiners, inventors,
and patent agents (attorneys). Such ongoing negotiation and effective
communication are instrumental in resolving technical and legal issues,
thereby enhancing the likelihood of patent approval (Tu, 2021).

The response process in patent prosecution is inherently time-sensitive. In
patent law, the importance of timeliness is underscored by statutory response
deadlines. Failure to respond promptly can render an application invalid,
adversely affecting the applicant’s patent rights (Tu, 2021). For inventors,
timeliness directly influences the market value of their innovations.
Expedient responses can accelerate the patent approval process, allowing
inventors to enter the market more swiftly and sustain market competitiveness
(Gaudry, 2012). Patent agents, snowed under multiple cases, must adhere to
time constraints to facilitate seamless case progression and avoid workload
overflow, impacting the quality of client service and case management
capabilities (De Rassenfosse et al., 2023). Lastly, examiners benefit from the
applicants’ timely responses in maintaining the review process’s pace and
efficiency (Lu et al., 2017).

Given the urgency to streamline and assist inventors or patent attorneys in
the response to OA, we introduce the Pattern Office Action Response
Intelligence System (PARIS). Developed in 2018 and continuously updated, PARIS
semi-automatically provides a case-oriented and user-need-based response
template recommender system, leveraging LM techniques. Its core objective is
to offer attorneys precise strategies to handle various OA scenarios, thereby
accelerating the patent examination process.

With the advent and advancement of Large Language Models (LLMs), we in 2023
further propose the LLMs Enhanced PARIS (LE-PARIS). LE-PARIS, building upon
the existing PARIS framework, incorporates the powerful capabilities of LLMs
to enhance the automation of response generation and the accuracy of language
processing. The system primarily introduces an LLM-based recommender system,
integrating response templates with external relevant resources and user-
defined remark of any style or language to directly generate responses that
meet the linguistic standards and requirements of the USPTO.

To evaluate the efficacy of the systems aforementioned, our research raises
several fundamental questions. For PARIS, we investigate whether the response
templates are well-constructed (Studies 1 and 2). We also examine the
performance of the recommendater system in terms of several accuracy metrics
(Study 3). For LE-PARIS, we explore whether the integration of LLMs enhances
the effectiveness of the recommendater system (Study 3) and the quality of the
generated responses (Study 4). Finally, we assess the practical utility of
both systems for users (Study 5). Hence, our contributions to the field are
three-fold:

  1. 1.

We pioneered the development of PARIS and LE-PARIS to facilitate the crucial,
yet underappreciated, process of responding to Office Actions.

  2. 2.

Both systems synergize response template recommendation with advanced LM
and/or LLM technologies, adopting a collaborative approach. This methodology
addresses complex legal and technical content in patents, moving beyond mere
automation with inaccuracies.

  3. 3.

We utilize a multi-paradigm approach for scientific validation of the systems’
effectiveness, not only proving their efficacy through scientific metrics but
also demonstrating their practical value beyond theoretical indicators.

##  2 Preliminaries and Background

For automating the OA Response process, we initially focus on the use of
response templates. Despite the complex legal and technical terms involved,
patterns can still be discerned under various rejection statutes, making
templates an effective framework or strategic guideline. Previous attempts
have highlighted the significant benefits of templates in organizing and
completing patent-related documents. For instance, provisional application
templates provide a structured outline guiding applicants to correctly
complete provisional patent applications, even without attorneys’ assistance
(Erickson, 2020). Moreover, templates have been widely used in drafting
various legal documents, such as contracts or confidentiality agreements. Such
templates can be customized to individual needs, ensuring consistency and
professionalism in documents while saving drafting time (Wernikoff, 2020).
Therefore, we construct high-quality templates for responding to OAs. These
templates will guide applicants or agents to cover all necessary information
and effectively organize response structures. The response templates will
feature fill-in-the-blank to facilitate customization, ensuring that responses
not only comply with patent legal requirements but also accurately reflect the
unique characteristics of each case.

While templates enhance the efficiency of patent attorneys in drafting
responses, the vast array of OA topics and their corresponding templates makes
finding an appropriate template a potential bottleneck in automation. To
address this, we consider the integration of a recommender system. This system
eases the way of decision-making and provides personalized suggestions based
on user behaviors and the OA content, primarily hybridising collaborative
filtering (CF) and content-based filtering (CB) approaches. CF recommends
items (i.e. response template) based on the preferences of similar users,
while CB makes recommendations based on item features (Su and Khoshgoftaar,
2009; Wu et al., 2021). In fact, recommender systems have been applied in
patent studies. For example, Chen and Deng (2023) employing deep learning
introduced an interpretable patent recommendation method, capable of
recommending patents based on their features. Thus, we utilize recommender
systems to assist patent attorneys in finding suitable response templates and
strategies to address various issues in OAs.

Given the fundamentally text-based nature of patent prosecution, the advent of
Natural Language Processing (NLP), LM, and LLM has opened new avenues in this
domain (Krestel et al., 2021; Lee, 2023). These advanced technologies exhibit
remarkable capabilities in contextual reasoning, precise text representation,
and efficient text generation (Zhao et al., 2023). These strengths are able to
provide substantial support in text processing, recommendation system
development, and response generation within our framework. The direct
application of these technologies in crafting OA responses, however, remains
an understudied area.

Since 2022, LLMs such as GPT-3, GPT-4, and LLaMA have shown distinctive in-
context learning abilities and ability eliciting (Zhao et al., 2023). While
these generative models have been employed in drafting patent claims and have
been pretrained on patent texts (Lee, 2023), producing complete and high-
quality patent content is still challenging. This challenge largely hinges on
precise user input and effective prompt engineering (i.e. garbage in, garbage
out). Therefore, we investigate the potential of a human-AI collaborative
approach (Wang et al., 2020), focusing on how patent attorneys can
synergistically utilize the technical prowess of LLMs to elevate the quality
and efficiency of OA responses.

##  3 System Overview

Figure 1: The overview of PARIS and LE-PARIS. (a) Construction of an OA Topics
and Legal Keywords (L-keywords) Database. (b) Development of Response
Templates Based on OA Topics and Legal Keywords. (c) Integration of the
Response Template Recommendation System. (d) Patent Attorney Workflow for
Responding to Office Actions. (T-keywords is short for technical keywords) (e)
Semi-Autocompletion Feature in OA Response System (f) Generative Components in
LE-PARIS System. (g) User Data Recording and Feedback Mechanism in System
Optimization.

We aim to harness the capabilities of NLP, LM, and LLMs to assist in the
categorization and extraction of response templates and legal keywords (Figure
1 (a)-(b)). Furthermore, we plan to utilize a hybrid recommender system to
suggest appropriate response strategies for patent attorneys upon receiving an
OA. This system enables the creation of suitable responses through attorney
customization or generative technologies (Figure 1 (c)-(f)). Ultimately, the
system is designed to record user workflows for iterative system updates
(Figure 1 (g)).

Firstly, as shown in Figure 1 (a), we are constructing a database of topics
and legal keywords. This database is designed to encapsulate topics and
corresponding legal terms extracted from previous OAs. Given the vast scale of
OA data, we employ a two-step classification approach: initially, a bottom-up
methodology using topic modeling techniques (Vayansky and Kumar, 2020) to
explore potential clusters and their legal keywords. Subsequently, an expert
panel employs a top-down approach for examining and annotating these clusters.
This preliminary assessment by the experts includes referencing external
manuals like MPEP 2100 to support their decision-making process. To ensure the
precision and consistency of these classifications, we implement the
convergent Delphi method (Hsu and Sandford, 2019), enlisting over a hundred
patent attorneys to evaluate these topics and legal keywords. The expert team
and attorneys iteratively refine the topics and legal keywords until a
consensus is reached (converged). Through these steps, we have established an
OA Topics and Legal Keywords Database, comprising 132 identified topics and
their corresponding keywords.

Following the establishment of the OA Topics and Legal Keywords Database, our
next step involves constructing response templates for each topic, guided by
the associated legal keywords. This process, as depicted in Figure 1 (b),
entails searching through OA and response database, Re-examination, or
MPEP-2100 for relevant data. For every topic, we identify all pertinent OAs
and their responses. Given the varying quality of responses, we employ patent
value analysis (Van Zeebroeck, 2011) to quantitatively assess the value of
each response. This analysis may include evaluating factors such as the number
of forward rejections, changes in claim, and law firm rankings (Falk and
Train, 2017; Osenga, 2011). When the weighted sum of a response’s normalized
value scores surpasses a certain threshold, it is incorporated into the
response template for the respective topic. This procedure enables the system
to extract specific and effective response templates for each topic, leading
to the creation of a corresponding database. The details regarding making
these templates customisable will be further elaborated in Figure 1 (e).
Overall, we have curated 15571 response templates in total.

Next, we progress to the Response Template Recommender System, depicted in
Figure 1 (c). This system integrates data from three primary sources: the User
Log Database, Response Template Database, and the Current OA received by
patent attorneys. It employs a hybrid recommendation approach, combining CB
and CF (Geetha et al., 2018). In the CB method, the system initially LM-
vectorizes the received Current OA and calculates its cosine similarity with
the vectorized corresponding OAs in the Response Template Database. This step
is designed to identify OAs most similar to the current one, thereby
recommending relevant response templates. For the CF aspect, the system
utilizes several techniques, such as ALS (Koren et al., 2009), BPR (Rendle et
al., 2012), and BiVAE (Truong et al., 2021), on User Log Database to recommend
response templates to the current patent attorney based on preferences of
similar attorneys. Ultimately, the system re-ranks the recommendations from
both content-based and collaborative filtering, providing a weighted
recommendation to the attorney. This combination of content-based and
collaborative filtering methodologies enables the creation of a more accurate
and personalized recommendation system, tailored to individual attorney
preferences (Kunaver and Požrl, 2017).

Upon receiving an OA, patent attorneys follow a systematic workflow as
depicted in Figure 1 (d). The process involves several steps: First, attorneys
receive recommended response templates and legal keywords from the
recommendation system. Alternatively, they may search directly in the Response
Template Database to find templates that align with their needs or interests.
Once templates are selected, the system auto-fills blanks with relevant
technical keywords or bibliographic information. Attorneys then manually
complete or edit the argument sections not autofilled by the system. The
system also automatically imports previous claims, which attorneys can amend
as needed. Finally, the system integrates the completed templates and amended
claims, formatting them to produce a comprehensive response to the OA.

The semi-autocompletion feature of our system, as illustrated in Figure 1 (d)
and detailed in Figure 1 (e), functions through a sequence of processes. When
an OA document is received, the system utilizes an NLP Parser to extract
relevant bibliographic information, such as claim numbers, cited case numbers,
inventor names, and cited drawings. This information is then used to search
the patent database for descriptions of the current patent and prior arts. The
system conducts an analysis to extract and compare technical keywords from
these documents, focusing on claims and descriptions. This includes
preprocessing steps like part-of-speech tagging and adjusting regular
expressions to suit patent-specific terminology (e.g. die), thereby
identifying technical keywords or components. The system then automatically
integrates these bibliographic details and technical keywords into the
response template. Notably, key sections of the response templates, including
bibliographic information and technical keywords, are structured as fillable
blanks, while the analysis and argument sections are left for manual
completion by the patent attorney.

The LE-PARIS system is a significant enhancement to the existing PARIS
framework, aimed at leveraging LLM technology to improve the patent response
process and address challenges faced by the original system. One of its core
innovations is the LLM module, which offers multi-language input support and
revisions in patent legal language, significantly reducing language barriers
for patent attorneys with limited English proficiency or unfamiliarity with
legal terminology. Additionally, LE-PARIS allows for deep customization of
response templates and automated referencing of external documents, providing
high-quality content references to strengthen the persuasiveness and accuracy
of responses.

In the LE-PARIS system, the content-based filtering aspect of the
recommendation system (as shown in Figure 1 (c)) has undergone significant
technical advancements. This includes the implementation of more sophisticated
semantic representation methods, such as OpenAI’s text-embedding-ada-002, for
enhanced precision. Furthermore, the LE-PARIS system has incorporated a
generative component as a fundamental part of its framework, particularly in
the stage outlined in Figure 1 (d).

The generative component is shown in Figure 1 (f). Initially, the system
integrates recommended or attorney-selected templates into a Template Cluster,
without the need for attorneys to fill additional blanks. Attorneys can also
add self-defined templates, which are stored in Template Cluster as well. The
Keywords Cluster stores legal and technical keywords related to response
templates and current OA, and attorneys can input custom technical or legal
keywords into this cluster. Subsequently, the system matches these keywords
with external resources like re-examination and MPEP 2100 to form Relevant
Documents for identifying crucial resources related to the current patent
response. Additionally, LE-PARIS allows attorneys to draft Response Drafts
based on the current OA and previous claims, enhancing response quality and
accuracy. These elements are weighted and arranged to form meaningful input,
ensuring the Response Draft receives the highest priority due to its
significance.

Finally, the component optimizes and generates responses. This stage includes
optimizing token counts using a token optimizer to fit LLM input limitations,
removing redundant or irrelevant information while retaining key content. The
system also employs prompting engineering to tailor the model for the current
response task. Ultimately, this information serves as input for the LLM,
generating results displayed on the user interface.

The final stage of both systems, as depicted in Figure 1 (g), focuses on the
recording and feedback of user data. The system logs all user activities on
the platform in the User Log Database. This includes choices of response
templates, data filled or submitted by users, results of the generated
response drafts, and records of multiple interactions with one OA. Collecting
these data not only enhances the accuracy of the recommendation system but
aids in analyzing the system’s overall effectiveness, enabling continuous
optimization and improvement of the system.

##  4 Study 1: Topic Modeling

In response to the challenge of evaluating whether response templates are
well-constructed in the context of a vast amount of OA-response data, our
study adopts a bottom-up methodology. This approach utilizes topic modeling
techniques (Vayansky and Kumar, 2020) to explore and identify broad clusters
and their associated legal keywords. This method enables us to sequentially
conduct a more efficient and precise manual analysis, ensuring the quality and
relevance of the response templates in our system.

###  4.1 Data Acquisition

The data for our study on OAs and responses was collected from two distinct
sources. The OAs were obtained from the USPTO Office Action Text Retrieval
API, comprising formal documents that detail the examination processes and
decisions, including reasons for refusals and pertinent citations. This
dataset encompasses approximately nine million records. In contrast, the
responses to these OAs were sourced from the Global Dossier’s examination
records. These responses, which complement the OAs, offer insights into the
argumentation and strategies employed by patent attorneys in response to USPTO
decisions. The paired OA-response documents in our collection, dating from
2001 to 2018, currently number over 6.6 million.

###  4.2 Data Prepocessing

For the response documents, which were in PDF format, we utilized Optical
Character Recognition (OCR) technology to convert them into text files. After
OCR, texts were corrected for common typographical errors. This was followed
by text preprocessing, where each OA-response pair underwent a series of steps
to enhance data quality. These steps included converting text into lowercase,
splitting text into individual words, and removing stop words, punctuation,
symbols, special characters, and self-defined common words in OA process, such
as regarding, et al., office action and so forth. The subsequent stage
involved transforming the cleaned text into a numerical representation. To
achieve this, the OA-response documents were converted into a document-term
matrix using Gensim (Rehurek et al., 2011), a library designed for processing
texts, conducting topic modeling, and document similarity analysis. This
conversion was crucial for facilitating further data analysis and topic
modeling.

###  4.3 Latent Dirichlet Allocation (LDA)

The LDA topic modeling technique (Blei et al., 2003) is utilized to analyze
the corpus of OA and response documents. The LDA model operates by
establishing the topic of each document based on probability distributions.
The document generation process within the LDA framework involves setting
parameters such as the total number of words, the number of topics per
document, and the total number of topics in the corpus. It then generates the
topic distribution for each document and the word distribution within each
topic. Words are sampled from these distributions, with the process repeating
until the total word count is reached. This method allows for a probabilistic
understanding of the thematic structure of the document corpus.

###  4.4 Results

Table 1: Perplexity and Coherence Across Different Topic Counts Number of OA Topics | Perplexity $\downarrow$ | Coherence $\uparrow$
---|---|---
10 | -5.32 | 0.21
30 | -4.77 | 0.49
50 | -6.21 | 0.55
80 | -7.24 | 1.32
120 | -7.01 | 1.10
200 | -7.56 | 1.26

Initially, we examined the performance of LDA over modeling various numbers of
OA topics, as detailed in Table 1. The effectiveness of the model was
evaluated using two key metrics: perplexity and coherence.

Perplexity, which measures the model’s predictive power, generally improves
(decreases) with a more accurate model. In our study, the perplexity decreased
as the number of topics increased, with the lowest perplexity observed at 200
topics. Nevertheless, the results between 80 to 200 topics showed
insignificant variance. On the other hand, coherence, which assesses the
semantic similarity between high-scoring words within each topic, peaked at 80
topics. This indicates that the topics generated at this level are the most
coherent and thematically consistent. Considering these findings, we opted for
an optimal set of 80 topics for our topic modeling, following the principle of
Occam’s razor.

Figure 2: Visualization of Results Topic Modeling. (a) Intertopic Distance
Map. (b) Word Cloud for a specific topic (which could involve 35 U.S. Code §
103).

Furthermore, the visualization of topic modeling, with an optimal set of 80
topics, is presented in Figure 2. The Intertopic Distance Map (a) illustrates
each topic as a bubble, sized according to the topic’s prevalence within the
dataset, with proximity indicating similarity between topics. The lexical
composition of a particular topic, possibly related to 35 U.S. Code § 103, is
depicted through a Word Cloud (b), featuring terms like "obvious" and "skill,"
highlighting connections to key legal concepts.

These results provide both a macro-level thematic clustering of the office
action texts and a micro-level identification of pertinent legal keywords,
aiding in the nuanced analysis of legal documents and informing following
studies.

##  5 Study 2: Panel Consensus

To further refine and enhance the practical applicability and precision of our
topic modeling results, we implemented an expert evaluation (panel consensus)
phase. In this stage, we utilized our proposed Convergent Delphi process,
aiming to achieve a more accurate consensus among patent practitioners on OA
topics and legal keywords. This approach facilitates the development of a
finely-tuned, practice-oriented categorization.

###  5.1 Participants and Measurements

To achieve consensus on OA topics and legal keywords, we recruited two groups
of paid professionals. On one hand, an internal expert panel of 11-25 patent
experts (with a female ratio ranging from 45%-33% and an average age of 45.7
years, varying due to attrition across the multi-year, multi-round Delphi
process) was responsible for decision-making on topics and legal keywords,
providing specific suggestions, feedback discussions, and modifications to OA
topics and legal keywords based on the results of the first study’s topic
modeling. These experts, selected based on criteria including at least ten
years of experience as patent attorneys, a master’s degree or higher in the IP
field, positions at deputy senior level or above, ability to provide
comprehensive opinions, and willingness to participate in the study.

On the other hand, we recruited 53-95 patent attorneys (with a female ratio of
66%-45% and an average age of 34.9 years, also subject to attrition over the
years) with at least three years of practical experience. Their primary task
was to assess the suitability of each topic and legal keyword provided by the
experts. This involved evaluating (1) whether these topics/legal keywords were
appropriate as key concepts in responding to OAs and (2) if they could be
further subdivided into lower-order concepts. This assessment was done using a
five-point Likert scale from 1 (do not agree at all) to 5 (totally agree).
Through this process, we aimed to achieve an effective consensus on patent
topics and legal keywords.

###  5.2 Convergent Delphi Procedure

The Delphi method is a structured, systematic approach for forecasting using
collective expert opinion (Brown et al., 1968). This method involves several
iterative rounds of questionnaires, ensuring an in-depth evaluation of topics
by experts, who focus on content, and a wider group, emphasizing scoring. Key
characteristics of the Delphi method include its iterative nature, allowing
for multiple rounds of questionnaires and evaluations, and its anonymous
setup, wherein experts respond without knowing other members’ identities,
promoting honest and unbiased feedback. Furthermore, the questionnaires are
specifically structured to elicit detailed information, with responses
aggregated and shared in subsequent rounds. This group-based approach
harnesses collective wisdom, rather than relying solely on individual
opinions. In our study, inspired by Hsu and Sandford (2019), we proposed and
adopted a longitudinal perspective to observe topic evolution over six years
and implemented a convergent Delphi process, detailed in Algorithm 1, to
refine the precision of the consensus further.

Algorithm 1 Covergent Delphi Process for OA Response Topics and Legal Keywords

1: $E,\ A\leftarrow$ the set of panel experts, patent attorneys

2: $T,\ C\leftarrow$ the set of topics with respective keywords $\varnothing$,
candidate topics with respective keywords $\varnothing$

3: $S_{s},\ S_{h}\leftarrow$ suitability score, higher-order score

4: $\lambda\leftarrow$ the critical threshold $4$

5: $\theta\leftarrow$ the consensus threshold $0.7$

6: converge $\leftarrow$ false

7: Construct $T$ by $E$

8: while converge $=$ false do

9: for each $t\in T$ do

10: for each $a\in A$ do

11: Assess the suitability of the construct $t$

12: Determine if $t$ is a higher order construct

13: add self-defined topic with respective keywords into $C$

14: end for

15: Calculate $s^{t}_{s}$ and $s^{t}_{h}$ over $A$

16: if $s^{t}_{h}>\lambda$ then

17: Decompose $t$ into a set $\tilde{T_{t}}$ by $E$

18: add $\tilde{T_{t}}$ into $T$

19: end if

20: end for

21: for each $c\in C$ do

22: if $E$ determine $c$ is applicable then

23: add $c$ into $T$

24: end if

25: remove $c$ from $C$

26: end for

27: $P(S_{s},\lambda)=\dfrac{|\\{s^{t}_{s}\in
S_{s}:s^{t}_{s}>\lambda\\}|}{|S_{s}|}$

28: $P(S_{h},\lambda)=\dfrac{|\\{s^{t}_{h}\in
S_{h}:s^{t}_{h}>\lambda\\}|}{|S_{h}|}$

29: if $P(S_{s},\lambda)>\theta$ and $P(S_{h},\lambda)<\theta$ and
$C=\varnothing$ then

30: converge $=$ true

31: end if

32: for each $t\in T$ do

33: if $s^{t}_{s}<\lambda$ or $s^{t}_{h}>\lambda$ then

34: remove $t$ from $T$

35: end if

36: end for

37: end while

###  5.3 Results

Figure 3: The Convergent Delphi Process Across Different Periods for OA Topics
and Legal Keywords.

Figure 3 shows the outcomes of a six-year convergent Delphi process
(2018-2023) on OA topics and corresponding legal keywords. The figure’s left
y-axis corresponds to the light blue bars, each representing the number of
topics that reached consensus after every round. The right y-axis aligns with
the orange line graph, indicating the consensus score achieved in each Delphi
process iteration. A dashed horizontal line at the 0.7 mark on the consensus
score y-axis acts as a predefined threshold. When the line plot surpasses this
threshold, it signifies a reached consensus on OA topics and legal keywords.
Notably, only 2-4 iterations per year were required to achieve consensus.
Significantly, the number of topics increased annually, reflecting the growing
demand for more detailed and accurate system requirements from both
practitioners and the expert team. To date, we have identified 132 topics,
each with corresponding legal keywords and response templates, demonstrating
the system’s evolving precision and relevance to practical applications.

##  6 Study 3: Recommendation System

Given the vast collection of over ten thousand response templates in our
database and hundreds of templates for each topic, our framework incorporates
a recommender system to further automate the selection process. This system is
designed to provide personalized response template recommendations to patent
attorneys based on their preferences and the characteristics of OAs they
receive.

###  6.1 Proposed Hybrid Recommender System

Over the past six years, our system has undergone significant iterative
enhancements, experimenting with various recommendation algorithms to identify
the most effective ones. We have developed a hybrid recommender system,
categorized as a cascade hybrid recommender system (Burke, 2002), tailored for
both PARIS and LE-PARIS frameworks.

This cascade hybrid approach employs a staged process. Initially, it utilizes
a CB recommender to identify the top related templates and topics for a given
OA. Subsequently, for each topic identified, the system applies CF on response
templates of that topic. Within each topic, the results of CF are refined by
templates identified by the CB approach, using a weighted method. This design
is specifically crafted for OA scenarios, reflecting the practical workflow of
patent attorneys who address the issues of each topic one by one when they
receive an OA. This methodology mirrors the practical phenomena encountered in
patent application responses. The specifics of the workflow of our proposed
hybrid recommender are articulated as follows:

Content-based Filtering. We denote response template database as
$\mathcal{D}=\\{(c_{i},t_{ij},o_{ij})\mid i,j\in\mathbb{N}\\}$, where each
tuple represents a response template and its related information. Here,
$c_{i}$ indicates the $i^{\mathrm{th}}$ topic, $t_{ij}$ is the
$j^{\mathrm{th}}$ response template under the $i^{\mathrm{th}}$ topic, and
$o_{ij}$ corresponds to the source OA for the given template $t_{ij}$.

Upon receiving a current OA $o_{c}$, we transform this document into an
embedding $\vec{o_{c}}$ within a predefined feature space. Similarly, each
source OA $o_{ij}$ in $\mathcal{D}$ is transformed into its corresponding
embedding $\vec{o_{ij}}$ within the same feature space, ensuring consistency
in representation across the database.

After calculating the cosine similarity between the embedding of the current
OA $\vec{o_{c}}$ and each embedding $\vec{o_{ij}}$ from the database, we
proceed to identify the top $k$ tuples from $\mathcal{D}$ whose
$\vec{o_{ij}}$s exhibit the highest cosine similarity with $\vec{o_{c}}$. We
denote this set of top $k$ tuples as $\mathcal{D}_{k}$, where
$\mathcal{D}_{k}\subset\mathcal{D}$ with $|\mathcal{D}_{k}|=k$. Accordingly,
we have the set of unique topics from the top $k$ tuples
$\mathcal{C}=\\{c_{i}\mid(c_{i},t_{ij},o_{ij})\in\mathcal{D}_{k}\\}$ and the
set of $k$ recommended response templates derived from CB
$\mathcal{T}_{k}^{\mathrm{cb}}=\\{t_{ij}\mid(c_{i},t_{ij},o_{ij})\in\mathcal{D}_{k}\\}$.

Collaborative Filtering. Define a user-template matrix $\mathbf{M}$, where
each entry $\mathbf{M}_{u\tau}$ represents the interaction between a user $u$
and a response template $\tau$. Here, $\tau$ encompasses all response
templates across the database $\mathcal{D}$. The CF process is denoted as
$\text{CF}(\cdot)$. In conventional CF methods, the top $k$ recommended
templates $\mathcal{T}_{k}^{\mathrm{cf}}$ can be derived from
$\text{CF}(\mathbf{M})$, considering the entire user-template interaction
spectrum.

As for our hybrid approach, which combines CB and CF, we consider submatrices
of $\mathbf{M}$ for each unique topic $c_{i}\in\mathcal{C}$, denoted as
$\mathbf{M}_{c_{i}}$s. Each submatrix $\mathbf{M}_{c_{i}}$ includes
interactions pertaining only to response templates under the topic $c_{i}$. By
applying the CF function $\text{CF}(\mathbf{M}_{c_{i}})$ to these submatrices,
we identify user preferences and patterns specific to each topic.
Consequently, we obtain resulting ranked response templates
$\mathcal{T}^{\mathrm{cf},c_{i}}$ for each $c_{i}$. To refine this ranking, we
consider those templates that belong to topic $c_{i}$ but are also included in
$\mathcal{T}_{k}^{\mathrm{cb}}$ and linear combination $\mathcal{L}(\cdot)$.
The refinement process can be represented as follows:

$\mathcal{T}^{\mathrm{refined},c_{i}}=\mathcal{L}(\mathcal{T}^{\mathrm{cf},c_{i}},\mathcal{T}_{k}^{\mathrm{cb}}\cap\mathcal{T}^{c_{i}})$

Following this, we extract the top $k$ templates
$\mathcal{T}_{k}^{\mathrm{refined},c_{i}}$ from
$\mathcal{T}^{\mathrm{refined},c_{i}}$ for each $c_{i}$, which are then
proposed as our final recommended response templates.

###  6.2 Experimental Setup

Table 2: Performance Metrics of Recommender Systems in 2023 Method Type | Method | Metrics (%)
---|---|---
Precision@10 | Recall@10 | nDCG@10
Collaborative Filtering (CF) | ALS | 5.16 | 2.59 | 5.83
BPR | 39.82 | 22.12 | 45.79
BiVAE | 42.17 | 22.54 | 48.67
Content-based Filtering (CB) | PARIS | 20.62 | 28.91 | 24.75
LE-PARIS | 23.64 | 32.77 | 29.00
|  |  | Median (Mean) |
Proposed Hybrid Recommender | PARIS+ALS | 14.67 (10.79) | 15.26 (9.25) | 14.26 (11.20)
PARIS+BPR | 44.82 (40.12) | 30.00 (28.33) | 50.37 (39.22)
PARIS+BiVAE | 47.98 (40.99) | 31.83 (29.05) | 53.16 (38.15)
LE-PARIS+ALS | 20.07 (11.37) | 21.12 (11.69) | 19.93 (17.64)
LE-PARIS+BPR | 46.57 (41.29) | 33.83 (29.78) | 52.45 (46.05)
LE-PARIS+BiVAE |  50.67 (45.04) |  51.05 (44.83) |  56.56 (45.67)

We explored the performance of CB, CF, and the aforementioned hybrid methods.
For the CB aspect, we adopted LM for embedding extraction, such as Sentence-
BERT (Reimers and Gurevych, 2019), as applied in the PARIS system (thus
denoted as PARIS method). With the emergence of LLM, we employed LLM-based
embedding extraction methods, like GPT-3 text-embedding-ada-002 (Greene et
al., 2022), as applied in the LE-PARIS system (thus denoted as LE-PARIS
method). Due to token limitations of these methods, we conducted syntactic and
semantic segmentation of the document when it was excessively large, and then
combined their embeddings.

In terms of CF, we used matrix factorization-based algorithms (ALS and BPR)
and autoencoder algorithms (BiVAE). Our hybrid method combines these CB and CF
techniques, resulting in six different configurations.

Our analysis is structured into three parts. Firstly, we evaluated the
performance of 11 methods using a 2023 dataset involving 151 patent attorneys
and 15,571 response templates, alongside 1,985,302 interactions. We assessed
non-hybrid systems using metrics like precision@k, recall@k, and nDCG@k. For
hybrid systems, we reported median and mean precision@k, recall@k, and nDCG@k
across all topics. Secondly, we analyzed the median precision@k and recall@k
of hybrid methods over different years, considering the iterative updates to
the PARIS and LE-PARIS systems, which could affect system performance by
reducing matrix sparsity. Finally, we compared the embedding distributions of
the PARIS and LE-PARIS methods in a t-SNE three-dimensional space and examined
the relationship between the embeddings of a received OA and its top k
embeddings, to understand the differences between these two methods. For this
analysis, we used $k=10$.

###  6.3 Results

As detailed in Table 2, outcomes for the CF, CB, and our hybrid methods are
displayed. For CF, the BPR showcased superior performance in recall@10
(22.54%), precision@10 (42.17%), and nDCG@10 (48.67%), aligning with the
previous findings (Truong et al., 2021). In the CB aspect, the LE-PARIS
method, leveraging LLM’s advanced text representation capabilities,
demonstrated higher recall@10 (32.77%), precision@10 (23.64%), and nDCG@10
(29.00%). CF’s significant impact on precision arises from its tendency to
recommend templates based on past behaviors or preferences of similar
attorneys, thus enhancing recommendation precision. However, this focus on
attorneys’ preference leads to lower coverage and recall. Conversely, CB, with
its emphasis on document similarity (i.e., OA), offers greater diversity and
hence higher recall.

Our hybrid approaches, combining the strengths of both CF and CB, outperformed
standalone methods. Notably, the LE-PARIS+BiVAE combination achieved the
highest performance, with a median precision@10 of 50.67%, median recall@10 of
51.05%, and median nDCG@10 of 56.56%.

Figure 4: Visualization Results of Our Hybrid Recommender Systems. (a) The
evolution of recommender performance over time, measured by median
Precision@10 and median Recall@10. Each line represents a combination of CB
and CF techniques used in the recommendation process. The points plotted on
each line are coordinates representing (median Recall@10, median
Precision@10), with annotations indicating the year of analysis (e.g., ’19’
for the year 2019). (b) Three dimensional embedding space visualizations,
representing the mere CB aspect of the system. The orange points mark the
current OA accessed by a user, while the ’x’ markers identify the top 10
recommended templates based on their proximity as the nearest neighbors in the
embedding space. (b-i) corresponds to the PARIS system, which adopts Sentence-
BERT embedding, while (b-ii) corresponds to the LE-PARIS system, which adopts
LLM embedding.

Subsequently, we examine the year-over-year performance of six hybrid
recommender system configurations, as illustrated in Figure 4 (a). This figure
shows each hybrid system combination of CB-CF as individual lines, with
plotted points showing their (median recall@10, median precision@10) and
respective analysis year. Notably, the configurations display varying numbers
of points due to the different inception times of the algorithms used or
proposed.

In general, these trends reveal that the LE-PARIS+BiVAE configuration
consistently surpassed others in both precision@10 and recall@10.
Additionally, there is an annual increase in both precision@10 and recall@10
across all configurations. However, the growth in precision@10 appears less
stable and more modest than that in recall@10. This pattern is likely due to
the decreasing sparsity in the user-item (i.e., attorney-template) matrix over
time, attributed to increasing user-system interactions. As the matrix becomes
denser, recall significantly improves.

Finally, we investigate the embedding distributions for both the PARIS and LE-
PARIS systems using t-SNE to project these embeddings into a three-dimensional
space, as shown in Figure 4 (b). For the PARIS system, the embeddings of
documents in the response template database appeare somewhat dispersed without
a clear pattern, suggesting a relatively weaker semantic grasp of the
documents. This is further evidenced by the scattered and distant relationship
between the embedding of a received OA and its top k embeddings.

In contrast, the LE-PARIS system displays a more clustered arrangement of
embeddings in the response template database, indicating a stronger semantic
understanding. The distances between the received OA embedding and the top k
embeddings are more concentrated and closer. This demonstrates that the LE-
PARIS system provides a better representation in our scenario, pointing to a
more effective semantic understanding of the content.

##  7 Study 4: Response Generation

Table 3: Generative Results from Different LLMs Model |  Generated Response
---|---
PatentGPT-J-6B |  The invention is not limited to the embodiments described herein. The scope of the invention is defined by the appended claims and includes other variations and modifications of the embodiments described herein.comprising: method the device, computing a with communication in being device peripheral the interface, user a with associated device peripheral a with communication in being driver device peripheral the driver, device peripheral a with communication in being system operating the system, operating an with communication in being process application the process, application an with communication in being device peripheral the device, peripheral a with communication in being interface user the interface, user a with communication in being device computing the device, computing a with communication in being system operating the system, operating an with communication in being process application the process, application an with communication in being device peripheral the device, peripheral a with communication in being interface user the interface, user a with communication in being device computing the device, [Repeats itself until the final token]
Llama 2 |  Dear Examiner, Thank you for providing the Office Action and for considering the patent application of our client. We appreciate your time and effort in reviewing the application. However, we respectfully disagree with the rejection of claims 1-5 and 7-20 under pre-AIA 35 U.S.C. 102(e) as being anticipated by Jin et al. (US 2011/0002161). We believe that the claimed invention is novel and non-obvious over the prior art, including Jin et al. Firstly, the switch structures and memory structures in the claimed invention are arranged in a specific configuration that is not disclosed in Jin et al. The switch structures are extended in the first direction (z direction) and are isolated from switch layers of adjacent switch structures in the second direction (x direction), while the memory structures are coupled with the switch structures and extend in the third direction (y direction). This configuration allows for improved performance and scalability, which is not taught by Jin et al. Secondly, the phase change memory layer in the claimed invention is isolated from adjacent memory structures in the first direction and the second direction, which is not disclosed in Jin et al. This feature provides improved thermal isolation and prevents thermal diffusion between memory cells, resulting in faster and more reliable operation. Therefore, we respectfully request that the rejection under pre-AIA 35 U.S.C. 102(e) be withdrawn and that the application be considered on its merits. We are confident that the claimed invention meets the requirements for patentability and deserves a patent.
GPT-3.5-turbo |  In response to the Office Action, we respectfully submit the following remark: We appreciate the Examiner’s evaluation of our patent application and the application of pre-AIA 35 U.S.C. 102(e) in the rejection of Claim(s) 1-5 and 7-20 based on Jin et al. (US 2011/0002161). However, we believe there may be a potential misunderstanding regarding the interpretation of the terms "first direction," "second direction," and "third direction" as used in the claims and in Jin et al. It is crucial to note that under the broadest reasonable interpretation (BRI), the terms "first direction," "second direction," and "third direction" can be interpreted differently, either as the same direction (e.g., horizontal, x direction) or as distinct directions (e.g., x, y, and z directions). We kindly request that the Examiner reconsider the interpretation of these directional terms and their applicability to the prior art. If, after reevaluation, the Examiner maintains the rejection under pre-AIA 35 U.S.C. 102(e), we respectfully request additional clarification regarding the rationale supporting the rejection to ensure a comprehensive understanding of the basis for the rejection. Thank you for your attention to this matter, and we look forward to your response.
GPT-4 |  Dear Examiner, In response to the Office Action dated [Date], which rejects claims 1-5 and 7-20 under pre-AIA 35 U.S.C. 102(e) as being anticipated by Jin et al. (US 2011/0002161), we respectfully submit the following remarks for reconsideration of the claims. Distinction of Claimed Invention from Jin et al. Structural Differences: The Office has interpreted Jin et al. as teaching a memory device with a plurality of switch structures and memory structures arranged in specified directions. However, our claimed invention distinctly involves [specific structural feature or arrangement not disclosed by Jin et al.]. This feature is critical for the functioning of our invention and is not suggested or disclosed by Jin et al. Directional Arrangement: The interpretation of the first, second, and third directions in Jin et al. is broad and lacks specificity compared to our claimed invention. Our invention explicitly defines these directions as [clearly define the directions, e.g., perpendicular, parallel, etc.], which is a configuration not disclosed or rendered obvious by Jin et al. Broadest Reasonable Interpretation (BRI) Under BRI, while the directions can be interpreted in various ways, our claim language distinctly specifies the arrangement of the components in a manner that is not anticipated by Jin. We propose to amend the claims to further clarify these directions as [specific amendments], thereby removing any ambiguity and differentiating from Jin et al. Technical Advantages Over Jin et al. The unique arrangement and features of our claimed invention provide technical advantages such as [list technical advantages], which are neither disclosed nor suggested by Jin et al. This underscores the novelty and non-obviousness of our claimed invention. In light of the above arguments, we respectfully request the reconsideration of the rejection of claims 1-5 and 7-20. We believe that our clarifications and proposed claim amendments distinctly differentiate our invention from Jin et al. and comply with the requirements of pre-AIA 35 U.S.C. 102(e). We appreciate the opportunity to respond to this Office Action and are available for further discussion or interview to resolve any remaining issues.

With the introduction of LLMs, the LE-PARIS system utilizes their distinctive
and powerful features, such as in-context learning and elicitation
capabilities. This development enables us to further automate the drafting of
responses to OAs via LLMs. This approach advances beyond merely relying on
manual input into templates or making decisions after receiving template
recommendations in the PARIS system. As outlined in Section 3, LE-PARIS
integrates multiple data sources, including response templates and patent
attorney drafts, to automatically generate responses to OAs.

###  7.1 Methodology

We evaluated four LLMs, including two closed-source models, GPT-3.5-turbo and
GPT-4 (Achiam et al., 2023), and two open-source models, Llama 2 (70B)
(Touvron et al., 2023) and PatentGPT-J-6B Lee (2023). The first three models
have been recognized as state-of-the-art in recent years. PatentGPT-J-6B, in
particular, was chosen for its pretraining on patent datasets, suggesting a
higher proficiency in patent language and related factors, making it,
arguably, more attuned to OA content attributes.

In line with Kim (2023), for domain-specific tasks, our study adopts a
qualitative examination based on our domain-specific knowledge. Considering
the lower total token count of PatentGPT-J-6B and the need for comparability,
we structured this study as a preliminary investigation into the effectiveness
of each model. This approach marks a departure from the more complex yet less
comparable procedure in Section 3. We conducted a zero-shot evaluation,
inputting a specific OA 35 USC § 102 segment into these models, along with
role instructions, and then analyzing the generated outcomes. Here is the
input text:

> Claim Rejections - 35 USC § 102
>
> In the event the determination of the status of the application as subject
> to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and
> 103) is incorrect, any correction of the statutory basis for the rejection
> will not be considered a new ground of rejection if the prior art relied
> upon, and the rationale supporting the rejection, would be the same under
> either status.
>
> The following is a quotation of the appropriate paragraphs of pre-AIA 35
> U.S.C. 102 that form the basis for the rejections under this section made in
> this Office action:
>
> A person shall be entitled to a patent unless — (e) the invention was
> described in (1) an application for patent, published under section 122(b),
> by another filed in the United States before the invention by the applicant
> for patent or (2) a patent granted on an application for patent by another
> filed in the United States before the invention by the applicant for patent,
> except that an international application filed under the treaty defined in
> section 351(a) shall have the effects for purposes of this subsection of an
> application filed in the United States only if the international application
> designated the United States and was published under Article 21(2) of such
> treaty in the English language.
>
> Claim(s) 1-5 and 7-20 is/are rejected under pre-AIA 35 U.S.C. 102(e) as
> being anticipated by Jin et al. (US 2011/0002161). Jin teaches, Figs. 1, 2 &
> 7, [0026-0028, 0046], a memory device (10) comprising:
>
> a plurality of switch structures (70) extending in a first direction (z
> direction) and comprising a switch layer (75) that is isolated from switch
> layers of adjacent switch structures (in neighbor memory cells 15) in a
> second direction (x direction); and a plurality of memory structures (20)
> coupled with the plurality of switch structures (70), a memory structure of
> the plurality of memory structures (20) comprising a phase change memory
> layer (25) between a first memory electrode layer (22) and a second memory
> electrode layer (24), the second memory electrode layer (24) extending in
> the second direction (x direction), and the phase change memory layer (25)
> and the first memory electrode layer (22) extending in a third direction (y
> direction) between a switch structure and the second memory electrode layer
> (24) of the memory structure and being isolated from adjacent memory
> structures (15) in the first direction and the second direction. Note: under
> BRI, first, second & third directions can be interpreted either same
> direction (e.g. horizontal, x direction) or different directions (as xyz
> directions).
>
> Assuming the role of a patent attorney, please craft a concise focused
> remark to the Office Action that has been provided.

On another note, for quantitative analysis, given our domain-specific nature,
we will further examine the quantitative evaluations of these models by patent
attorneys in our subsequent user study (Study 5).

###  7.2 Results

As shown in Table 3, PatentGPT-J-6B’s output often forms a continuous text
block, impacting readability. This model’s response is repetitive and lacks
depth in legal arguments and specific claim references, along with limited
technical explanation or differentiation from the prior art mentioned by
examiners.

Llama 2 improves structure with clear breaks, enhancing readability. It offers
a detailed technical rebuttal (e.g., the structural arrangements of switch and
memory structures) and highlights differences from the prior art. The model
directly states its disagreement with the rejection, emphasizing the novelty
and non-obviousness of the invention. However, it does not deeply engage with
legal arguments (e.g., BRI), focusing more on the invention’s technical
aspects. Notably, the reasoning it presents may have potential hallucination.

GPT-3.5-turbo, similar to Llama 2’s format, focuses on elucidating the
meanings of specific terms used in the invention (e.g., "first direction").
Additionally, it emphasizes adaptability in interpreting legal standards
(e.g., BRI). However, it’s important to note that the rationale provided by
this model could potentially include hallucination as well.

GPT-4 offers a comprehensive response, covering term interpretations from more
aspects (e.g., structural and directional differences). It delineates how the
invention diverges from the prior art. The model strategically suggests claim
amendments and highlights the invention’s technical advantages and novelty.
GPT-4 also employs placeholders for unknown patent details, minimizing the
risk of hallucination. This balanced approach of technical and legal
strategies positions GPT-4 as the most effective model.

##  8 Study 5: User Study

Apart from evaluating the performance of the systems PARIS and LE-PARIS using
common metrics, our objective includes assessing their practical value through
a comprehensive user study, which will be conducted in two distinct parts: (1)
Given that this work is specific to a particular domain, as mentioned in Study
4, we will focus on assessing the capabilities of the different text
generation methods employed by LE-PARIS. This assessment will be conducted by
domain experts. (2) We aim to explore whether PARIS and LE-PARIS provide
tangible aid in their defensive work. Consequently, the study is to provide a
deeper understanding of the effectiveness and usability of both systems in
real-world scenarios.

###  8.1 Participants and Procedures

Table 4: Linear Mixed Model Analysis of Attorney Performance with PARIS and
LE-PARIS

| Performance
---|---
Fixed effects | Estimate Coefficient | SE
(Intercept) | -.010 | .079
Control Variables |  |
Gender | -.035 | .079
Age | .074 | .107
Education | .294** | .079
Tenure | .642** | .109
Predictors |  |
PARIS Engagement | .086** | .012
LE-PARIS Engagement | .162** | .014
Random effects | Variance | SD
Attorney Var | .901 | .256
Time Var | .077 | .025
Attorney x Time Cov | .011 | .054

  * •

Note: * p<.05, ** p<.01. Gender coding: 1 = female, 0 = male. Age and tenure
are measured in months. Education levels: 1 = Bachelor’s, 2 = Master’s, 3 =
PhD.

In the first part of our study, we recruited 37 patent engineers, of which 27%
are female, with an average age of 34.5 years, all possessing at least three
years of practical experience. We conducted A/B testing to assess how patent
attorneys evaluate the outcomes generated by four LLMs: GPT-3.5-turbo, GPT-4,
Llama 2, and PatentGPT-J-6B. Specifically, we employed a blind testing
approach where the results produced by these four LLMs were randomly assigned
to the attorneys, who were unaware of the source of each result. Each attorney
rated the generated responses on the LE-PARIS interface using a 1-5 star
system, based on their satisfaction with the generated outcome, guided by the
instruction: Are you satisfied with the generated result?

Each patent attorney received ten results from each LLM, allowing us to record
average ratings for each LLM per attorney. This method resulted in a total of
148 observations for subsequent analysis. For this analysis, we employed
repeated measures ANOVA and post hoc testing.

In the second part of our study, we recruited 159 patent attorneys, of whom
31% are female, with an average age of 32.2 years. Using longitudinal data
from 2018 to 2023, we investigated the relationship between their engagement
with the PARIS and LE-PARIS systems and their performance in responding to
OAs. This was to explore whether using these two systems could effectively
assist in practical OA responses.

Regarding performance measurement, we specifically considered each patent
attorney’s monthly OA workload, which was adjusted for difficulty as
determined by an impartial third party responsible for task assignments. On
the other hand, the degree of system engagement was measured by the number of
times the system was fully utilized (from entry to completion of the response
draft with system assistance) and this was weighted according to the depth of
operation. This approach yielded a total of 3940 observations. Due to the
interdependence of these observations, we applied a mixed linear model (MLM)
to predict performance based on engagement with the two systems. In this
model, we controlled for factors such as gender, age, tenure, educational
level, and attorney (subject) effect, as well as the time effect.

###  8.2 Results

Figure 5: Performance Trajectories of Patent Attorneys with PARIS and LE-PARIS
Engagement Over Time. The figure presents the performance and engagement
levels of patent attorneys involved in the PARIS and LE-PARIS programs from
2018 to 2024. The x-axis represents time, and the y-axis represents
performance related to the response to OA tasks. Pink lines trace the
performance history of individual attorneys. Points superimposed on these
lines, where larger and lighter-colored bubbles indicate higher engagement.

For the first study, we examined patent attorneys’ evaluations of four
different OA generation methods: PatentGPT-J-6B, Llama 2, GPT-3.5-Turbo, and
GPT-4. The results of the repeated measures ANOVA indicated that the mean
evaluations differed significantly between the generation methods,
$F(3,108)=2052.47,p<.001,\eta_{p}^{2}=.93$. The measure of effect size,
$\eta_{p}^{2}$, suggests a large effect of generation methods on attorneys’
evaluations, with generation methods accounting for 93% of the variance in
evaluations.

Post hoc analyses were conducted using paired t-tests with corrections for
multiple comparisons using the False Discovery Rate (FDR) approach. The
comparison between PatentGPT-J-6B and Llama 2 revealed a highly significant
difference, $t(36)=-27.29,p<.001$, indicating a substantial effect when moving
from the former to the latter. Similarly, PatentGPT-J-6B compared to
GPT-3.5-Turbo showed a significant difference, $t(36)=-24.98,p<.001$, and to
GPT-4, $t(36)=-91.86,p<.001$.

However, the comparison between Llama 2 and GPT-3.5-Turbo did not yield a
significant difference, $t(36)=-0.46,p=.645$. In contrast, the transition from
Llama2 to GPT-4, $t(36)=-51.62,p<.001$, and from GPT-3.5-Turbo to GPT-4,
$t(36)=-46.19,p<.001$, were both highly significant, indicating that GPT-4
represents a marked improvement over both preceding methods. These findings
suggest that while there are incremental improvements across generations
(i.e., PatentGPT-J-6B < Llama 2 = GPT-3.5-Turbo < GPT-4), the most significant
advancements in performance are seen with the introduction of GPT-4.

In the second study, for assessing the appropriateness of MLM analysis, we
considered the evaluation of the -2 log-likelihood values between the fixed
effects model and the mixed effects model. The results indicated a more
appropriate fit for the mixed effects model. Specifically, the mixed effects
model, with a -2 log-likelihood value of 6712.29, demonstrated a better fit
compared to the fixed effects model, which had a -2 log-likelihood value of
10379.48.

As illustrated in Table 4 and depicted in Figure 5, the analysis of the
relationship between engagement with the PARIS and LE-PARIS systems and
attorneys’ performance demonstrated significant positive correlations. After
controlling for demographic variables and random effects attributable to
individual attorneys and time, it was found that engagement with the PARIS
system positively influenced performance, indicated by an estimated
coefficient of .086 ($SE=.012,p<.01$). More importantly, engagement with the
LE-PARIS system was associated with an even more pronounced positive impact on
performance, as shown by an estimated coefficient of .162 ($SE=.014,p<.01$).
These findings suggest that while both systems offer benefits, LE-PARIS might
provide more substantial support in assisting OA tasks.

##  9 Implications and Future Directions

Our study pioneers the development of comprehensive systems, PARIS and LE-
PARIS, leveraging a collaborative approach that integrates recommender systems
and LLMs to facilitate responses to OAs. We have substantiated the
effectiveness of these systems both in theory and practice. In this section,
we underscore our contributions and identify potential limitations in our
research to pave the way for future directions.

Firstly, to our knowledge, we are the first focusing on OA response
automation, a previously underexplored field. We have constructed and
categorized a historical database of OA responses. Despite employing topic
modeling techniques and relying on extensive expert consensus for iterative
database refinement to ensure quality and categorization, the construction of
an OA and response database may still be nascent. Future research is
encouraged to conduct a more detailed development in this field via advanced
LM techniques, such as tailored language models and tokenization methods to
better capture the semantic nuances of OAs and responses.

Secondly, we introduces cascade hybrid (LLM-based) recommender system tailored
for OAs (LLM as embedding extraction (Wu et al., 2023)). However, there is
room for improvement. For the CB part, future research could, similarly to
(1), capture OA and response embeddings more effectively through models
trained on OA response data, rather than using existing feature extraction
methods. For the CF aspect, future work could experiment with other mainstream
methods, like contrastive or generative self-supervised recommenders (Xie et
al., 2022), and incorporate data augmentation techniques. For LLM-based
systems, alternative approaches like LLM as recommender systems could be
explored (Wu et al., 2023).

Next, regarding the LLM-based generator component of the LE-PARIS system, our
system generates responses from multiple data sources. This process is not
about creating patent claims, which are prone to hallucinations, but rather
about arguing the technical differences between the case in hand and the prior
arts, given discerning the difference is one strength of LLMs. Nevertheless,
these LLMs are not tailor-made for OAs. Future research might involve using
open-sourced LLMs as a backbone and performing self-supervised fine-tuning
with OA response data and subsequent reinforcement learning from human
feedback for better alignment with OA contexts (Kirk et al., 2023).

Moreover, while we have conducted various user studies to understand the
practical value of both systems, we did not establish a nomological network to
elucidate the mechanisms (why), boundary conditions (how), and specific
contributing parts (ablation) of the system that particularly impact an
attorney’s performance in responding to Office Actions (OAs). Future research
could delve into refining this aspect, providing a more detailed understanding
of the underlying factors influencing system efficacy in OA response
scenarios.

Lastly, as mentioned in the introduction, our system potentially benefits
inventors, examiners, agents, and other stakeholders. However, it poses
intellectual property risks, especially in maintaining client confidentiality
when using LLMs for response drafting. The prosecution can involve unpublished
patents, known only to the applicant, examiners, and authorized patent agents,
remaining confidential to others. Using LLMs might risk breaching
confidentiality obligations. As per the American Bar Association (ABA) Rule
1.6 on the Confidentiality of Information, a fundamental principle in the
attorney-client relationship, attorneys must not disclose information related
to representing a client unless explicitly permitted by the client. Adherence
to this principle is crucial when using LLMs in prosecution to prevent
potential leaks of confidential information. Therefore, agencies should
consider using locally run, open-sourced LLMs instead of relying on external
APIs to mitigate these risks.

##  10 Conclusion

Our study advances the field of patent prosecution automation by developing
the innovative PARIS and LE-PARIS systems. These systems, designed to support
patent attorneys in crafting responses to Office Actions, integrate a range of
functionalities like OA topic classification, expert consensus, and a
comprehensive response template database. They also feature traditional and
LLM-based recommender systems, along with LLM-based response generation. This
multifaceted approach, underpinned by a diverse paradigmatic framework and
longitudinal data spanning from 2018 to 2023, not only meets experimental
metrics but also garners positive validation from practical user surveys. This
dual achievement in both theoretical robustness and practical applicability
marks a significant stride in leveraging AI to enhance the efficiency and
effectiveness of the workflow in regard to response to OAs.

## References

  * Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al., 2023.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774 .
  * Blei et al. (2003) Blei, D.M., Ng, A.Y., Jordan, M.I., 2003.  Latent dirichlet allocation.  Journal of machine Learning research 3, 993–1022.
  * Brown et al. (1968) Brown, B.B., et al., 1968.  Delphi process: a methodology used for the elicitation of opinions of experts.
  * Burke (2002) Burke, R., 2002.  Hybrid recommender systems: Survey and experiments.  User modeling and user-adapted interaction 12, 331–370.
  * Chen and Deng (2023) Chen, H., Deng, W., 2023.  Interpretable patent recommendation with knowledge graph and deep learning.  Scientific Reports 13, 2586–2586.
  * De Rassenfosse et al. (2023) De Rassenfosse, G., Jensen, P.H., Julius, T., Palangkaraya, A., Webster, E., 2023.  Is the patent system an even playing field? the effect of patent attorney firms.  The Journal of Industrial Economics .
  * Erickson (2020) Erickson, K., 2020.  Provisional patent template: Everything you need to know.  https://www.upcounsel.com/provisional-patent-template.  Accessed: 01-17-2024.
  * Falk and Train (2017) Falk, N., Train, K., 2017.  Patent valuation with forecasts of forward citations.  Journal of Business Valuation and Economic Loss Analysis 12, 101–121.
  * Gaudry (2012) Gaudry, K.S., 2012.  The lone inventor: Low success rates and common errors associated with pro-se patent applications.  PloS one 7, e33141.
  * Geetha et al. (2018) Geetha, G., Safa, M., Fancy, C., Saranya, D., 2018.  A hybrid approach using collaborative filtering and content based filtering for recommender system, in: Journal of Physics: Conference Series, IOP Publishing. p. 012101.
  * Greene et al. (2022) Greene, R., Sanders, T., Weng, L., Neelakantan, A., 2022.  New and improved embedding model.  https://openai.com/blog/new-and-improved-embedding-model.  Accessed: 2022-12-15.
  * Helmers et al. (2019) Helmers, L., Horn, F., Biegler, F., Oppermann, T., Müller, K.R., 2019.  Automating the search for a patent’s prior art with a full text similarity search.  PloS one 14, e0212103.
  * Hsu and Sandford (2019) Hsu, C.C., Sandford, B.A., 2019.  The delphi technique: making sense of consensus.  Practical assessment, research, and evaluation 12.
  * Kim (2023) Kim, J.h., 2023.  Fine-tuning the llama2 large language model using books on the diagnosis and treatment of musculoskeletal system in physical therapy.  medRxiv , 2023–11.
  * Kirk et al. (2023) Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., Raileanu, R., 2023.  Understanding the effects of rlhf on llm generalisation and diversity.  arXiv preprint arXiv:2310.06452 .
  * Koren et al. (2009) Koren, Y., Bell, R., Volinsky, C., 2009.  Matrix factorization techniques for recommender systems.  Computer 42, 30–37.
  * Krestel et al. (2021) Krestel, R., Chikkamath, R., Hewel, C., Risch, J., 2021.  A survey on deep learning for patent analysis.  World Patent Information 65, 102035.
  * Kunaver and Požrl (2017) Kunaver, M., Požrl, T., 2017.  Diversity in recommender systems–a survey.  Knowledge-based systems 123, 154–162.
  * Lee (2023) Lee, J.S., 2023.  Evaluating generative patent language models.  World Patent Information 72, 102173.
  * Lee and Hsiang (2020a) Lee, J.S., Hsiang, J., 2020a.  Patent claim generation by fine-tuning openai gpt-2.  World Patent Information 62, 101983.
  * Lee and Hsiang (2020b) Lee, J.S., Hsiang, J., 2020b.  Patent classification by fine-tuning bert language model.  World Patent Information 61, 101965.
  * Lo and Chu (2021) Lo, H.C., Chu, J.M., 2021.  Pre-trained transformer-based classification for automated patentability examination, in: 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), IEEE. pp. 1–5.
  * Lu et al. (2017) Lu, Q., Myers, A., Beliveau, S., 2017.  Uspto patent prosecution research data: Unlocking office action traits .
  * Osenga (2011) Osenga, K., 2011.  Shape of things to come: What we can learn from patent claim length.  Santa Clara Computer & High Tech. LJ 28, 617.
  * Rehurek et al. (2011) Rehurek, R., Sojka, P., et al., 2011.  Gensim—statistical semantics in python.  Retrieved from genism. org .
  * Reimers and Gurevych (2019) Reimers, N., Gurevych, I., 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084 .
  * Rendle et al. (2012) Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L., 2012.  Bpr: Bayesian personalized ranking from implicit feedback.  arXiv preprint arXiv:1205.2618 .
  * Su and Khoshgoftaar (2009) Su, X., Khoshgoftaar, T.M., 2009.  A survey of collaborative filtering techniques.  Advances in artificial intelligence 2009.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288 .
  * Truong et al. (2021) Truong, Q.T., Salah, A., Lauw, H.W., 2021.  Bilateral variational autoencoder for collaborative filtering, in: Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 292–300.
  * Tu (2021) Tu, S.S., 2021.  Fast versus slow examination: How examiners use allowances or rejections to delay or compact patent prosecution.  Intellectual Property Litigator (September/October 2021), WVU College of Law Research Paper 18.
  * USPTO (2023) USPTO, 2023.  FY 2023 Agency Financial Report.  Technical Report. United States Patent and Trademark Office.
  * Van Zeebroeck (2011) Van Zeebroeck, N., 2011.  The puzzle of patent value indicators.  Economics of innovation and new technology 20, 33–62.
  * Vayansky and Kumar (2020) Vayansky, I., Kumar, S.A., 2020.  A review of topic modeling methods.  Information Systems 94, 101582.
  * Wang et al. (2020) Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., Wang, Q., 2020.  From human-human collaboration to human-ai collaboration: Designing ai systems that can work together with people, in: Extended abstracts of the 2020 CHI conference on human factors in computing systems, pp. 1–6.
  * Wernikoff (2020) Wernikoff, D., 2020.  Create documents easily with free templates, drafted by attorneys.  https://www.legalzoom.com/templates.  Accessed: 01-17-2024.
  * Wu et al. (2021) Wu, L., He, X., Wang, X., Zhang, K., Wang, M., 2021.  A survey on neural recommendation: From collaborative filtering to content and context enriched recommendation.  arXiv preprint arXiv:2104.13030 .
  * Wu et al. (2023) Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al., 2023.  A survey on large language models for recommendation.  arXiv preprint arXiv:2305.19860 .
  * Xie et al. (2022) Xie, X., Sun, F., Liu, Z., Wu, S., Gao, J., Zhang, J., Ding, B., Cui, B., 2022.  Contrastive learning for sequential recommendation, in: 2022 IEEE 38th international conference on data engineering (ICDE), IEEE. pp. 1259–1273.
  * Zhao et al. (2023) Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al., 2023.  A survey of large language models.  arXiv preprint arXiv:2303.18223 .