Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
10.1109/ACCESS.2017.DOI

# Evaluating Generative Patent Language Models

JIEH-SHENG LEE  School of Law, National Yang Ming Chiao Tung University, No.
1001, Daxue Rd. East Dist., Hsinchu City 300093, Taiwan (e-mail:
<EMAIL_ADDRESS>
###### Abstract

This research aims to build generative language models in the patent domain
and to evaluate the models from a human-centric perspective. The evaluation
metric is to calculate the ratio of keystrokes that can be saved for a user in
an autocomplete context based on the prediction of the generative models. The
performance of models in different sizes can also be evaluated in such a
metric by measuring a number of newly granted patents. On the basis of the
metric, it is found that the largest model is not necessarily the best.
Several models are pre-trained from scratch with patent corpus and are
released. The experiments in this manuscript focus on patent claims, but the
ideas and implementation can be applied to other parts of a patent document.
Furthermore, this research is motivated to measure how close the pre-trained
language model can generate a newly granted patent claim. Or, conversely, the
task is to measure the probabilities for the model to generate each token text
given the newly granted patent claim. In addition, this manuscript raises
several legal implications on patent law for potential interdisciplinary
research in the future. In particular, can the metric based on model
prediction be a metric to measure the nonobviousness requirement in the patent
law?

###### Index Terms:

Patent, Language Model, Natural Language Generation, Deep Learning, Patent Law

=-15pt

##  I Introduction

Generative language models have shown great potential in recent years.
However, the mainstream research is to build larger models with more corpora
or modalities. Evaluation of generative language models is also a challenge.
In this manuscript, instead of pursuing a larger and general language model,
this research is motivated to build a domain-specific language model, propose
a human-centric metric, evaluate the models in different sizes, and identify
the best model. For domain-specific models, it is hypothesized that the
largest is not necessarily the best. The domain in this manuscript is the
patent domain, but the ideas and implementation can be applicable to other
domains. In this research, after being pre-trained with patent corpus from
scratch, the generative patent language model is a model capable of generating
patent text. The long-term goal in the author’s series of research is to
generate new patent text, help an inventor or a patent professional draft a
new patent application, and have it granted as a new patent by passing a
patent examiner’s review. The long-term goal will take years to achieve.

For a step closer to achieve, one task is to measure how close the pre-trained
language model can generate a newly granted patent. This is the core task to
be addressed in this manuscript. Or, conversely, the task is to measure the
probabilities for a model to generate each word (or sub-word or token) given a
newly granted patent. Since the goal is to help a user, the measurement of
probabilities is further adjusted from a human perspective. The metric in this
perspective is to calculate the ratio of keystrokes that can be saved for the
user if an autocomplete function is available to utilize the model prediction.
The performance of models in different sizes can be evaluated based on the
same metric by measuring a number of patents. New patents are granted on a
regular basis and are available for free. As a proof of concept, the metric
and model evaluation are limited to patent claims in this manuscript. The same
approach can be applied to other parts of a patent document in the future.
Since the patent domain is a legal domain, several legal implications on
patent law are raised to show the potential of interdisciplinary research. For
example, can the metric based on model prediction be a metric to measure the
nonobviousness requirement in the patent law? Or, if some patent text has a
high probability for the model to predict and, therefore, easier for a user to
access, can we say that the patent text is deemed more obvious to the model
and the user? This kind of legal question may invite legal scholars to address
them in depth in the future.

The major contributions of this manuscript include: (1) pre-trained several
language models specific to the patent domain, (2) proposed a human-centric
metric for model evaluation (how many keystrokes to be saved in an
autocomplete function), (3) found that the largest model is not necessarily
the best, (4) conducted several experiments, (5) provided the tools for
visualizing the experiment results, and (6) raised several legal implications
in the patent law for interdisciplinary research in the future. The rest of
the manuscript is organized as follows. Section II introduces the code base
that inspires this work and the author’s previous research as a background.
Section III describes the implementation of this work in detail, such as
objectives, metrics, data, model sizes, etc. Section IV provides experiments
and visualization of results. Because the metric could be relevant to the
obviousness requirement in patent law, Section V raises some legal
implications. Section VI concludes this manuscript.

##  II Related Work

This research is inspired by works such as GPT-J-6B [1] and GPT-3 [2]. The
patent language models in this manuscript are called PatentGPT-J models
because their configurations are based on GPT-J-6B’s. In addition to building
larger models, this manuscript also extends the author’s previous research on
patent text generation. Previous research includes: generating patent claims
by fine-tuning GPT-2 [3], controlling patent text generation by structural
metadata [4], and prior art search for generated patent text [5], etc. In
recent years, natural language generation (NLG) has demonstrated significant
progress in many domains. However, because of the entry barrier to legal
knowledge, very few researchers devote themselves to the generation of patent
language. This manuscript continues the research on the patent domain and aims
to scale up the generative models. Unlike GPT-3, the source code and the
GPT-J-6B model configuration [1] are available. Furthermore, in GPT-J-6B [1],
the training data include a portion of the patent text, and the size of the
model is significantly larger than the largest GPT-2 model. Therefore,
GPT-J-6B is a suitable choice for this manuscript to follow. At the time of
implementation in this research, GPT-J-6B is also the largest model open-
sourced.

According to The Pile [6], the training data for GPT-J-6B [1] is an 825 GiB
English text corpus targeted at training large-scale language models. The Pile
is constructed from 22 diverse and high-quality datasets. Among them, the
“USPTO Backgrounds” is a dataset of the background sections extracted from
patents granted by the United States Patent and Trademark Office (USPTO). The
raw data are available in the bulk archive of the USPTO website [7]. In [6],
the authors explain that a typical patent background lays out the general
context of the invention and gives an overview of the technical field. They
included USPTO Backgrounds because it contains a large volume of technical
writing aimed at a non-technical audience. The raw size of the dataset is
22.90 GiB. Furthermore, according to [1] and [8], the GPT-J-6B model was
trained on 400B tokens from The Pile dataset and the training took roughly 5
weeks with TPU v3-256.

##  III Implementation

###  III-A Objectives

The majority of the source code in this research is forked from Mesh
Transformer JAX [1]. In the code base [1], the GPT-J-6B model is released as
the largest pre-trained model open to the public at the time. The code base
also provides a guide for fine-tuning[9] the GPT-J-6B model. The first
objective of this research is to build the same largest model dedicated to the
patent domain, i.e., the PatentGPT-J-6B model. Instead of fine-tuning, the
author pre-trained the PatentGPT-J-6B model from scratch with US patents. A
corpus like patents is a specialized domain. The reason why pre-training from
scratch is that it is critical for the model to generate nothing but patent
text. Fine-tuning a model pre-trained from other corpora may generate non-
patent text. Non-patent text makes little or no sense to patent professionals.
With respect to the general requirements in training data, the patent corpus
is also suitable for pre-training because the corpus is sizable and is
incremental on a regular basis.

The second objective is to measure the performance of PatentGPT-J models from
a human-centric perspective. Specifically, the metric is to measure how many
keystrokes the model can save for a user by providing model predictions in an
autocomplete function. The metric is used to evaluate various PatentGPT-J
models in different sizes and to challenge the assumption that larger models
are always better. The third and auxiliary objective is to explore the
capability of neural models by testing whether the patent-specific model may
perform a general NLP task in few-shot learning. The content of the general
task is not within the patent domain. If the test result is positive, what
size is required for the model to demonstrate the capability of few-shot
learning? Is the largest model the best?

###  III-B Metric: Autocomplete Effectiveness

In this manuscript, the metric for evaluating model performance is the
Autocomplete Effectiveness (AE) ratio. The AE ratio is about how many
keystrokes can be saved by using an autocomplete function. The AE ratio is
defined below. A higher AE ratio is better because more keystrokes are saved.

$AE=\frac{\text{\footnotesize{(keystrokes w/o autocomplete) - (keystrokes w/
autocomplete)}}}{\text{\footnotesize{keystrokes w/o autocomplete}}}$ (1)

The AE ratio works at the token level. The details of implementing the AE
ratio are provided below. Given a sequence of tokens _$t_{1}$_ , _$t_{2}$_ ,
…, _$t_{k}$_ , the numerator of the ratio is the accumulated number of
keystrokes that can be saved by an autocomplete function at each token
position. For each token _$t_{i}$_ , its rank in model prediction determines
whether the autocomplete function will be triggered. In programmer’s popular
editors for code completion, it is common to have an autocomplete function.
For example, a user can press a special key, such as the “tab” key, to
complete the next function name, parameter, or the remaining of the statement
in the source code. In this manuscript, the autocomplete function works
similarly, except that the target to complete is the next token in the patent
text. Without involving the actual implementation of the autocomplete
function, the calculation of the AE ratio can be explained later in
pseudocode. The calculation is implemented based on the predictions and ranks
of model inference in an emulated fashion. In the following experiments, no
actual autocomplete function of an editor is required. In the past, to
demonstrate how fluent and coherent the generated patent text can be, an
autocomplete prototype was implemented to complete an entire patent claim as a
proof of concept in [3] based on GPT-2 [10]. Integration of the previous
prototype and the new AE ratio is planned in the future.

In an emulated fashion, the calculation of the total number of keystrokes is
based on how a user may compose patent text with minimal number of keystrokes
in an editor. For example, given some patent text, if the editor has no
autocomplete function, the minimal number of keystrokes for a user to compose
the patent text is the total number of characters of the entire patent text
(ignoring special function keys such as “shift”, “CapsLock”, “tab”, etc.). If
the editor provides an autocomplete function, the total number of keystrokes
is likely to decrease. The user can press the conventional “tab” key to
autocomplete the next token and type less. The performance of the model can be
evaluated by the degree to which the user can type less. A model helping the
user with fewer keystrokes is therefore a better model.

New patents are granted on a frequent basis. These new patents are free and
suitable for evaluating the AE ratio in this research. In the emulated
calculation, the cut-off of ranks in model prediction is set as 10. Given a
sequence of tokens _$t_{1}$_ , _$t_{2}$_ , …, and _$t_{k}$_ , the _rank_ of
the token _$t_{k}$_ is its position in the model’s predictions based on tokens
_$t_{1}$_ , _$t_{2}$_ , …, and _$t_{k-1}$_. The actual _$t_{k}$_ may or may
not locate within the _top-10_ tokens predicted by the model. In this
manuscript, the hypothetical autocomplete function works by showing the
_top-10_ tokens predicted by the model, based on tokens _$t_{1}$_ , _$t_{2}$_
, …, _$t_{k-1}$_. If the _top-1_ token is exactly the token _$t_{k}$_ , it
means that a user can press the “tab” key to complete the next token. If the
_top-2_ token is the token _$t_{k}$_ , the user can press “downarrow”
($\downarrow$) key and “tab” key to complete. If the token _$t_{k}$_ is not
within the _top-10_ , it means that the user would have to type manually
without the autocomplete function. It is assumed that the length of ten is
effective for a glance and that a longer list can be unproductive. In brief,
depending on whether the token _$t_{k}$_ is within the _top-10_ tokens
predicted by the model, the keystrokes may come from the autocomplete function
or manual typing. In such settings, the number of keystrokes is accumulated by
counting the “tab”, the “downarrow” ($\downarrow$) key, and the number of
characters in the user’s manual typing.

The pseudocode to calculate the minimal number of keystrokes with autocomplete
is provided in Algorithm 1. As described, an emulated user can confirm the
_top-1_ token by pressing the “tab” key to complete. The “tab” counts as one
keystroke. If the token _$t_{k}$_ is the token with rank as two in model
prediction, the autocomplete function will require pressing the “downarrow”
key and the “tab” key to select the target token. Two keystrokes will be
counted accordingly. If the token _$t_{k}$_ is the token with rank as three in
model prediction, using the autocomplete function will require three
keystrokes (two “downarrow” keys and one “tab” key). As a general rule, the
number of keystrokes is equal to the rank matched shown in the _top-10_ list
predicted by the model. By emulating how the sequence of tokens _$t_{1}$_ ,
_$t_{2}$_ , …, _$t_{k}$_ may be composed with the hypothetical autocomplete
function, it is feasible to count the total number of keystrokes to complete
the entire text of the token sequence.

Algorithm 1 Calculating Autocomplete Keystrokes

1:$patent\\_text$ $\leftarrow$ read one new patent claim

2:$patent\\_tokens$ $\leftarrow$ tokenizer.encode($patent\\_text$)

3:$keystroke$ $\leftarrow$ 0

4:for $i=1$ to len($patent\\_tokens$)-1 do

5: $prompt$ $\leftarrow$ $patent\\_tokens[:i]$

6: $next\\_token$ $\leftarrow$ $patent\\_tokens[i]$

7: $top$ $10$ $tokens$ $\leftarrow$ model.predict($prompt$)

8: if $next\\_token$ == $top$ $1$ $token$ then

9: $keystroke$ += 1 $\triangleright$ autocomplete by “tab” key

10: else if $next\\_token$ == $rank$ $n$ $token$ then $\triangleright$ $n$ =
$2$ to $10$

11: $next\\_text$ $\leftarrow$ tokenizer.decode($next\\_token$)

12: $keystroke$ += min($n$, len($next\\_text$)) $\triangleright$ auto or
manual

13: else if $next\\_token$ is out of $top$ $10$ then

14: $keystroke$ += len($next\\_text$) $\triangleright$ manual typing

15: end if

16:end for

A caveat is that if the text length of the matching token in the _top-10_ list
is shorter than the number of keystrokes by pressing the “tab” and the
“downarrow”, it will be more efficient for a user to type the token text
manually. Therefore, in this scenario, the number of keystrokes should be the
text length of the match token because of fewer keystrokes. In a nutshell, if
the token _$t_{k}$_ matches the _top-1_ token in model prediction, the
keystroke counts as one. If the token _$t_{k}$_ matches a token in model
prediction (_rank = 2 $\sim$10_), the keystroke(s) will count as _n_ or the
length of the token text, which is smaller. If the rank of the token _$t_{k}$_
is beyond 10 and not shown in the autocomplete function, a user will have to
type the token text manually. The number of keystrokes therefore equals to the
text length of the token _$t_{k}$_.

It is noted that MRR (Mean Reciprocal Rank) is a common metric for measuring
model performance. For example, in [11] and [12], the authors use the metric
to measure code completion and describe: MRR assesses the whole top _N_
recommended completions and takes into account the first position in which the
target is matched. For a single query, the reciprocal rank is $\frac{1}{rank}$
where _rank_ is the position of the highest-ranked answer. If no correct
answer was returned in the query, then the reciprocal rank is 0. Compared to
the AE ratio in this manuscript, the reciprocal rank serves a similar purpose
in measuring autocomplete effectiveness. However, a non-zero MRR value does
not precisely reflect the actual human effort in terms of keystrokes. The
number of keystrokes is also lost if the MRR value is zero. These are the
reasons why this research takes a different metric from a more human-centric
perspective. Furthermore, an MRR metric aggregates the calculations of
reciprocal ranks in average. Without any other visualization, the distribution
of ranks for each answer/token is lost in the mean value of the MRR. In this
research a visualization is provided to inspect the distributions of ranks in
this manuscript. The distribution of ranks might be useful for understanding
few-shot or zero-shot learning in the future.

###  III-C Data

A granted patent has several sections in the patent document. This manuscript
is motivated to have a broader coverage and complement the USPTO Backgrounds
dataset in The Pile. The dataset in this research is named the USPTO TACD
dataset to include titles, abstracts, claims, and descriptions. Both
independent and dependent claims are included. The raw data ranges from 1976
to 2021 (1976$\sim$2020 for training and 2021 for validation). For a benchmark
purpose, the USPTO TACD dataset does not include the background section.
Interested researchers may combine the USPTO TACD dataset and the USPTO
Backgrounds dataset, if necessary. The script to collect the USPTO Backgrounds
dataset is provided in the repository [13], which leverages the code in [14]
(forked from [15]). The author in this manuscript leveraged these repositories
to build the USPTO TACD dataset. The downloaded and shuffled patent text for
building the USPTO TACD dataset are compressed and available upon request.

As mentioned in The Pile [6], the background section of a patent document is
aimed at a non-technical audience. This should be the reason why the USPTO
Backgrounds dataset was built and included in The Pile. In contrast, the USPTO
TACD dataset in this manuscript aims at a more technical audience. Especially
the patent claims aim at patent professionals. From a legal perspective,
patent claims are the most important part of a patent document because they
define the scope of the legal protection conferred by a patent. Among patent
claims, a unique feature is claim dependency: A patent claim may depend on
another patent claim. A patent claim without any dependency is called an
independent claim. A patent claim that depends on another patent claim is
called a dependent claim. Compared to other corpus in plain text, the claim
dependency poses a unique challenge and opportunity in capturing its
structure. The approach proposed in this research to expand the text of patent
claims will be explained in III-E. The approach makes the USPTO TACD dataset
significantly larger than the downloaded raw text and the USPTO Backgrounds
dataset.

Another factor enlarging the USPTO TACD dataset is related to bidirectional
text generation. The patent text is a source of inventive ideas. It is
hypothesized that a generative patent language model can generate new patent
text for inventor brainstorming. In the use case of brainstorming, it is not
necessary that an inventor starts from the beginning of patent text. The
inventor may start from the middle or from anywhere. Therefore, it is
preferable that the generative language model can generate patent text
bidirectionally. To reserve this possibility for future implementation, the
patent text is duplicated in reverse order as part of the dataset.

After adding data augmentation with claim dependency and reserve order, the
total size of the USPTO TACD dataset in text is about 713 GiB. The size is
significantly larger than the 22.90 GiB of the USPTO Backgrounds dataset. In
comparison, the size of The Pile is 825 GiB. In terms of tokens, the total
number of the USPTO TACD dataset is about 147B, compared to the 400B tokens in
The Pile. The tokens required for encoding one GiB of raw text are fewer for
PatentGPT-J models, because the tokenizer is pre-trained from scratch with
patent text, too. By using the pre-trained tokenizer, the efficiency for
encoding the USPTO TACD dataset in tokens is higher. In this research, pre-
training a PatentGPT-J model consumes about 11B tokens for training 350,000
steps. The 11B tokens are about 7.48% of the USPTO TACD dataset (147B). It is
noted that the vocabulary for the pre-trained tokenizer contains some special
tags defining the unique structure and meta data in patent documents. The
practice of using domain-specific tags, such as <|start_of_claim|>, <|dep|>,
<|end_of_claim|>, etc., was implemented in the author’s previous research [4]
and reused.

###  III-D Model sizes & Training Loss

The pre-training from scratch is done following the fine-tuning guide [9] and
without using any saved checkpoint. Based on the code base [1], the
configuration of the GPT-J-6B model includes 4,096 dimensions (_d_model_) and
28 layers (_layers_). The model size of the GPT-J-6B is 6B. In this research,
by changing the number of dimensions, layers, or both, several PatentGPT-J
models are pre-trained and configured in different sizes. The model sizes in
this research are 6B, 1.6B, 456M, 279M, 191M, 128M, and 115M, as shown in
Table I. It is noted that the original GPT-J-6B was pre-trained with TPU
v3-256 (_tpu_size = 256_) and the training steps are 383,500. In the fine-
tuning guide [9], the training step is configured as 350,000. PatentGPT-J
models follow the fine-tuning guide and train 350,000 steps. In terms of
computing resource, this research utilizes TPU v3-8 which is sufficient for
this stage of research. TPU v3-8 is more accessible and cost less.
Consequently, _tpu_size_ is set as 8 in the configuration.

TABLE I: Model Comparison model | size | layer | d_model | tpu_size | steps
---|---|---|---|---|---
PatentGPT-J | 115M | 1 | 1024 | 8 | 350,000
PatentGPT-J | 128M | 2 | 1024 | 8 | 350,000
PatentGPT-J | 191M | 7 | 1024 | 8 | 350,000
PatentGPT-J | 279M | 14 | 1024 | 8 | 350,000
PatentGPT-J | 456M | 28 | 1024 | 8 | 350,000
PatentGPT-J | 1.6B | 28 | 2048 | 8 | 350,000
PatentGPT-J | 6B | 28 | 4096 | 8 | 350,000
GPT-J | 6B | 28 | 4096 | 256 | 383,500

Fig. 1 shows the training loss of PatentGPT-J models. In terms of training
loss, both the 6B model (_d_model=4096_) and the 1.6B model (_d_model=2048 and
layer=28_) have lower values ranging from 1.56 to 1.53 [16] around the end of
training. Compared to GPT-J-6B, its training loss ranges from 1.59 to 1.39
around the end according to [17]. In this research, it is found that the model
with the lowest loss is not necessarily the best for the AE ratio. As shown in
section IV, the 456M model (_d_model=1024 and layer=28_ , the third curve from
the bottom in Fig. 1) has the highest AE ratio in the experiment in IV-A. The
1.6B model has the highest AE ratio in the experiment in IV-B. The 6B model is
not the best in both experiments. One probable reason is that the AE ratio
takes into account the length of the token text. This length of text is not
included in the calculation of the training loss. Therefore, the training loss
in cross-entropy does not align with the metric squarely. For future
researchers, the pre-trained PatentGPT-J models in 279M, 456M, and 1.6B,
including checkpoints and vocabulary files, are released at [18]. The sample
code at [18] also shows how to use a PatentGPT-J model for inference. The
sample code is for the sentiment analysis in section IV-D.

Figure 1: Training loss curve

###  III-E Claim Dependency

This section explains how to expand the claim text based on the claim
dependency. For patent professionals in daily practice, it is common for an
independent claim to have several dependent claims that describe different
technical elements or limitations in the claimed invention. The several
dependent claims do not, in general, depend on each other. For example, _claim
2_ , _claim 3_ , …, and _claim n_ are several dependent claims and depend only
on _claim 1_ , and there is no claim dependency among _claim 2_ , _claim 3_ ,
…, and _claim n_. In such a scenario, a patent language model should learn
only the dependency between _claim 1_ and _claim 2_ , _claim 1_ and _claim 3_
, …, and _claim 1_ and _claim n_. However, in the raw text downloaded from the
USPTO, the claims are described in sequence such as _claim 1_ , _claim 2_ ,
_claim 3_ , …, and _claim n_ consecutively. The _claim x (x=2 $\sim$n)_ is
implicitly dependent on all of it previous claim(s) including _claim 1_ to
_claim x-1_ because the language model is an autoregressive model. Such an
implicit claim sequence in the raw text should not be treated as a claim
dependency in the dataset.

The objective of a language model is to predict the next token based on all
preceding tokens. Therefore, without removing the implicit sequence and
incorrect dependency, after feeding the raw patent text, predicting _claim n_
will mean predicting the claim based on _claim 1_ , _claim 2_ , _claim 3_ ,…
and _n-1_ altogether. However, as described, the claim dependency between
_claim n_ and _claim 1_ is remote. _Claim n_ should depend on _claim 1_ only
and nothing in between. _Claim 2_ , _claim 3_ , …, and _claim n-1_ should not
involve when training _claim n_ and _claim 1_. In the training dataset, the
dependency in the pair (_claim n_ and _claim 1_) should be only one record.
The other pairs, such as _cliam x (x=2 $\sim$n)_ and _claim 1_ should be
separate training records too. To correctly reflect the dependency of claims,
this research takes an expansive approach to mitigate the above issues. The
approach is to prepare a pair of claims as a training record according to the
dependency between the claims. The pair is prepared by concatenating these two
claims with a special token _$ <$—dep—$>$_ to denote the dependency. For
example, if _claim n2_ depends on _claim n1_ , the training data will be
prepared as “ _claim n1 $<$—dep—$>$claim n2_.” The paired structure reflects
the dependency of the two claims. In this way, “ _claim 1 $<$—dep—$>$claim 2_”
is a training record, and “ _claim 1 $<$—dep—$>$claim 3_” is another record,
etc. The duplication of the _claim 1_ for all pairs “ _claim 1
$<$—dep—$>$claim x (x=2$\sim$n)_” is the reason why the USPTO TACD dataset in
this research is large.

It should be noted that the expansion of claims in this manuscript is limited
to paired claims only. In reality, sometimes the situation could be more
complicated than the general scenario of paired claims. A dependent claim may
depend on another dependent claim or depend on multiple claims (independent or
dependent). For example, _claim n3_ may depend on _claim n2_ , and _claim n2_
may depend on _claim n1_. In this situation, the claim dependency is like a
chain of dependency. The USPTO TACD dataset does not formulate this chain as
one single training record. It is hypothesized that the paired claims in this
manuscript can make the model learn the dependency in the chain discretely in
pairs. This hypothesis is to be validated in the future. Another situation not
addressed in this research is multiple-dependent claims. For example, _claim
n6_ depends on _claim n4_ and also on _claim n5_. In this scenario, _claim n6_
is a multiple-depedent claim. Conceptually, an advanced data structure, such
as tree or graph, is required to capture the complicated claim dependency.
Both the sequential plain text and the paired structure described are
deficient in representing the correct dependency among multiple-dependent
claims. This challenge is reserved for future research.

##  IV Experiments & Visualization

The first experiment in section IV-A and the second experiment in section IV-B
evaluate the PatentGPT-J models based on the AE ratio defined in Equation 1. A
model performs better if its AE ratio is higher. The third experiment in
section IV-C is to test the models with non-patent text. The purpose is to
observe how much the AE ratio will drop. The fourth experiment in section IV-D
is sentiment analysis. During this research, it was found by accident that
larger PatentGPT-J models can perform sentiment analysis reasonably well in
few-shot learning. The content of the sentiment analysis is outside the patent
domain. PatentGPT-J models are trained specifically for the patent domain
only. Therefore, a patent language model capable of doing sentiment analysis
is counterintuitive. To visualize the results of the experiment, statistics
and figures are provided in section IV-E and an inspection tool is
demonstrated in section IV-F.

###  IV-A Experiment 1

The first experiment covers 500 independent claims randomly selected from
patents in 2022 (range: 01/01$\sim$04/12). These patents are new and are not
in the USTPO TACD dataset. The performance of different model sizes is shown
in Table II. In the table, the first column is the size of the model. The
second column is the “AE ratio,” and the 456M model is the best (56.8%). It is
noted that all the models perform relatively well, except for the smallest
model. Even the 128M model has an AE ratio of more than 50%. Also, the largest
model is not better, although its size is about 13 times larger than the best
model. This finding implies that, in a specific domain and with a specific
metric, a modest range of model sizes may suffice. Continue pursuing larger
models might be unnecessary.

TABLE II: PatentGPT-J Performance Evaluation #1 Evaluation by Total Number of
Keystrokes
---
size |  AE ratio ($\uparrow$) |  total w/ |  top 10 ($\downarrow$) |  out of top 10 ($\downarrow$) |  top 1 ($\uparrow$) |  total w/o
|  |  autocomplete ($\downarrow$) |  (auto/manual) |  (manual) |  (auto) |  autocomplete
6B |  56.2% |  281,789 |  167,246 |  114,543 |  72,457 |  643,646
1.6B |  56.7% |  278,948 |  168,635 |  110,313 |  73,295 |  643,646
456M |  56.8% |  277,800 |  168,112 |  109,688 |  73,600 |  643,646
279M |  56.2% |  281,619 |  169,082 |  112,537 |  72,498 |  643,646
191M |  55.6% |  285,843 |  169,478 |  116,365 |  71,676 |  643,646
128M |  50.1% |  320,907 |  172,604 |  148,303 |  64,284 |  643,646
115M |  36.0% |  411,884 |  177,304 |  234,580 |  44,660 |  643,646
* “keys w/ autocomplete” = keystrokes of “top 10” + keystrokes of “out of top 10” |
** AE (autocomplete effectiveness) ratio = (“key w/o autocomplete” - “key /w autocomplete”) / “key w/o autocomplete” |

The third column of the table is the total number of keystrokes needed to
compose the 500 independent claims with an autocomplete function. The number
is the minimal number after taking advantage of the predictions of the model.
The lower the number of keystrokes, the better. The 456M model has the lowest
number in this experiment. The last column is the total number of keystrokes
without using any autocomplete function. The number is the number of
characters that a user has to manually type for all token text. This number is
the same for all rows in the table because manual efforts are the same. As a
whole, the AE ratio is calculated on the basis of the third column and the
last column, as shown in Equation 1. The fourth column is the number of
keystrokes based on the top 10 predictions of the model. By definition, _“top
n”_ includes tokens with ranks ranging from _1_ to _n_ , and _“out of top n”_
includes tokens with ranks ranging from _n+1_ to _the size of vocabulary_
(50,400 in both GPT-J-6B and PatentGPT-J). In summary, the “top 10” include
all tokens with ranks from 1 to 10. The top 1 is equivalent to rank 1. The
details of mixing autocomplete and manual typing in the fourth column is
described in section III-B and omitted here. The fifth column shows the number
of keystrokes if a user manually types when the rank of the target token is
greater than 10. The relation of the third, fourth, and fifth columns is
described below:

total keystrokes with autocomplete = (2) (keystrokes of “top 10”) +
(keystrokes of “out of top 10”)

For both “top 10” and “out of top 10, the fewer the keystrokes, the better.
The sixth column (top 1) is a supplemental column showing the top-1 counts in
the fourth column (top 10). The higher the counts of the top 1, the better. In
this experiment, the 456M model performs best. This means that by using the
456M model, a user can save more keystrokes than using other models.

Figure 2: Inspecting AE Ratio and Keystroke Distribution of Patent 11,248,512
(claim 1).

In addition to the performance evaluation in Table II, a visualization tool is
developed to inspect the distribution of the ranks on a token basis given a
patent claim. All inspection results in this experiment are archived and
released through an interactive web page at [19]. The following is one of the
results. Fig. 2 shows claim 1 of patent 11,248,512 with color highlights and
statistics. Highlights are colored in three categories: (1) “rank 1” in light
green, (2) “rank 2$\sim$10” in dark green, and (3) “out of top 10” in red. In
this style, it is convenient to recognize which tokens can be completed simply
by pressing a “tab” key (rank 1), or pressing a “tab” key and “downarrow” key
(rank 2$\sim$10) or typing manually if the keystrokes are fewer, or typing all
the token text manually (out of top 10). It is also convenient to compare
different models in such a visualization. The model selected in this example
is the 456M model. The AE ratio is 55.3%. It is noted that such visualization
might be a tool for saliency analysis. The visualization is like a saliency
map. Salient tokens in the patent context might mean something that is not
obvious to a generalized patent language model. Nonobviousness is a legal
requirement for patentability in patent law. Nonobviousness is critical not
only in patent prosecution but also in patent litigation. Therefore, such a
saliency analysis may have practical value and merit legal research in the
future. The legal implications with respect to patent law will be briefly
described in V.

###  IV-B Experiment 2

The second experiment covers another 500 patent claims, including independent
and dependent claims. These patents are randomly selected from the patents in
the first week of 2022 and outside the USTPO TACD dataset. The purpose of this
experiment is to verify the findings in the first experiment with different
test data. The performance of different models is shown in Table III. The
table structure is the same as the previous one. In this experiment, the 1.6B
model is the best model in terms of the AE ratio. The 456B model is the second
best by a narrow margin. The largest model (6B) is not better even though its
size is 3.75 times larger than the 1.6B model. Therefore, it is validated
again that continuing to pursue larger models might be unnecessary in a
specific domain that has a specific metric such as the AE ratio. A reasonable
range of model sizes can be sufficient. The rest of Table III is self-
explanatory and the details are omitted for brevity. Fig. 3 shows the same
visualization to inspect the distribution of the ranks on a token basis given
a patent claim. The patent claim to analyze is claim 5 of patent 11,212,959.
The color codes and their purposes are the same and are omitted here. After
comparing different models, the best model for this patent claim is the 1.6B
model, as shown below. The AE ratio is 65.9%.

TABLE III: PatentGPT-J Performance Evaluation #2 Evaluation by Total Number of
Keystrokes
---
size |  AE ratio ($\uparrow$) |  total w/ |  top 10 ($\downarrow$) |  out of top 10 ($\downarrow$) |  top 1 ($\uparrow$) |  total w/o
|  |  autocomplete ($\downarrow$) |  (auto/manual) |  (manual) |  (auto) |  autocomplete
6B |  54.9% |  238,845 |  150,101 |  88,744 |  73,282 |  529,900
1.6B |  55.3% |  237,015 |  150,423 |  86,592 |  73,708 |  529,900
456M |  55.2% |  237,650 |  151,140 |  86,510 |  73,687 |  529,900
279M |  54.4% |  241,708 |  151,275 |  90,433 |  72,687 |  529,900
191M |  53.6% |  245,713 |  151,375 |  94,338 |  72,274 |  529,900
128M |  48.5% |  272,636 |  153,575 |  119,061 |  65,492 |  529,900
115M |  35.8% |  340,012 |  157,193 |  182,819 |  48,702 |  529,900
* “keys w/ autocomplete” = keystrokes of “top 10” + keystrokes of “out of top 10” |
** AE (autocomplete effectiveness) ratio = (“key w/o autocomplete” - “key /w autocomplete”) / “key w/o autocomplete” |
Figure 3: Inspecting AE Ratio and Keystroke Distribution of Patent 11,212,959
(claim 5).

It should be noted that, depending on the model size, producing the inspection
results for all 500 patent claims in this experiment took about 14 hours to 3
days per model. It took a long time for two reasons. First, the model
inference required for inspection will start from the beginning for each token
every time. For example, calculating the rank of token _$t_{k}$_ requires the
model inference running through tokens _$t_{1}$_ , _$t_{2}$_ , …, _$t_{k-1}$_.
Calculating the rank of token _$t_{k+1}$_ requires the model inference running
through tokens _$t_{1}$_ , _$t_{2}$_ , …, _$t_{k}$_. The sample code for model
inference provided in [1] does not have a cache mechanism for calculating the
rank of token _$t_{k+1}$_ based on token _$t_{k}$_ without starting over from
token _$t_{1}$_. In the author’s impression, probably no sample code of other
popular generative language models provides such a cache function. How to
improve the efficiency of model inference is a practical topic for future
research. The second reason behind the long processing time is that, for
another visualization in IV-F, patent text generation is executed and prepared
as another task in model inference so that reviewing the generated text _post
hoc_ can be done in real time. Both efficiency issues could be mitigated by
better algorithms in the future.

###  IV-C Experiment 3

This experiment calculates the AE ratio of PatentGPT-J models by using non-
patent text. The AE ratio should drop because the models were pre-trained with
patent corpus only. The purpose of this experiment is to know how much the AE
ratio will drop. In this experiment, the non-patent text is the sample
“unicorn” text generated by the original GPT-2 model and published by OpenAI
[20], as shown below.

⬇

In a shocking finding, scientist discovered a herd of unicorns living in a
remote, previously unexplored valley, in the Andes Mountains. Even more
surprising to the researchers was the fact that the unicorns spoke perfect
English. The scientist named the population, after their distinctive horn,
Ovid’s Unicorn. These four-horned, silver-white unicorns were previously
unknown to science. Now, after almost two centuries, the mystery of what
sparked this odd phenomenon is finally solved.

The performance of different models for non-patent text is shown in Table IV.
The table structure is the same as the previous ones. In this experiment, the
6B model is the best model in terms of the AE ratio. However, its performance
drops significantly to 18.2%. This ratio shows that an autocomplete function
based on PatentGPT-J models is less helpful for composing non-patent text. It
is reasonable to see that a domain-specific model fails to predict well for
out-of-domain content. The rest of Table IV is self-explanatory and is
omitted. Fig. 4 shows a similar visualization for inspecting the distribution
of the ranks on a token basis given the non-patent text. The model is the 6B
model in this figure. It is noted that more tokens are displayed in red (out
of top 10) and few are displayed in light green (top 1). Obviously the non-
patent text is harder for PatentGPT-J models to predict. Even the largest
PatentGPT-J model has a low AE ratio. For all model inspections in this
experiment, a web demo is available at [21].

TABLE IV: PatentGPT-J Performance Evaluation #3 Evaluation by Total Number of
Keystrokes
---
size |  AE ratio ($\uparrow$) |  total w/ |  top 10 ($\downarrow$) |  out of top 10 ($\downarrow$) |  top 1 ($\uparrow$) |  total w/o
|  |  autocomplete ($\downarrow$) |  (auto/manual) |  (manual) |  (auto) |  autocomplete
6B |  18.2% |  401 |  130 |  271 |  37 |  490
1.6B |  16.7% |  408 |  125 |  283 |  35 |  490
456M |  16.7% |  408 |  100 |  308 |  29 |  490
279M |  14.1% |  421 |  97 |  324 |  31 |  490
191M |  13.9% |  422 |  105 |  317 |  29 |  490
128M |  10.8% |  437 |  90 |  347 |  25 |  490
115M |  8.8% |  447 |  86 |  361 |  17 |  490
* “keys w/ autocomplete” = keystrokes of “top 10” + keystrokes of “out of top 10” |
** AE (autocomplete effectiveness) ratio = (“key w/o autocomplete” - “key /w autocomplete”) / “key w/o autocomplete” |
Figure 4: Inspecting AE Ratio and Keystroke Distribution of the unicorn text.

###  IV-D Experiment 4

Apart from testing the AE ratio, this experiment tests the capability of
PatentGPT-J models in few-shot learning. According to [22], models like GPT-J,
GPT-Neo, and GPT-3 are so big that they can easily adapt to many contexts
without being re-trained. Giving only a few examples to the model can help it
dramatically increase its accuracy. The webpage at [22] provides one example
of sentiment analysis with GPT-J. Using the example, this experiment tests
whether the PatentGPT-J models can correctly predict sentiment in few-shot
learning. The non-patent prompt for few-shot learning is as follows:

⬇

prompt = """Message: Support has been terrible for 2 weeks...

Sentiment: Negative

###

Message: I love your API, it is simple and so fast!

Sentiment: Positive

###

Message: It is really bad! How could it be possible?

Sentiment: Negative

###

Message: The API is great. It is really good!

Sentiment: Positive

###

Message: GPT-J has been released 2 months ago.

Sentiment: Neutral

###

Message: Your team has been amazing, thanks!

Sentiment:"""

Table V shows the results of the experiment. For each model, the test is run
1,000 times with the same prompt and parameters (_top_p=0.9_ and
_temperature=0.75_). Although the smallest model (115M) cannot produce
anything sensible, surprisingly, the best model (1.6B) predicts 731 times
correctly. It is also surprising that a model as small as the 191M model is
already capable of predicting the correct answer 526 times. In terms of the
format for possible answers (Positive, Negative, or Neutral), the 191M model
can predict the answers in correct format 871 times (=526+85+260). The results
of the experiment and the sample code are released as “Unreasonable
effectiveness in Few-Shot Learning” at [18]. Whether the PatentGPT-J models
can solve other tasks in few-shot learning is a research topic in the future.
Another research topic is whether the capability of few-shot learning is more
related to model size, rather than the training corpus. According to [23],
pre-trained large language models (LLMs) are decent zero-shot reasoners. The
range of model sizes that will demonstrate similar zero-shot capabilities is a
further topic down the line.

TABLE V: Sentiment Analysis in Few-Shot Learning size | positive | negative | neutral | others
---|---|---|---|---
6B | 636 | 263 | 86 | 15
1.6B | 731 | 167 | 67 | 35
456M | 535 | 152 | 37 | 276
279M | 586 | 285 | 6 | 123
191M | 526 | 85 | 260 | 129
128M | 358 | 268 | 24 | 350
115M | 0 | 0 | 0 | 1,000

###  IV-E Statistics

In experiment IV-A, the best model is the 456M model. In this section, based
on the model, two charts are provided to show the distribution of token ranks
at each token position. Figure 5 shows the distribution of ranks for the first
50 tokens of the 500 patent claims in the experiment. In this figure, the
percentage of the “top 1” counts increases gradually along the token position.
The first three tokens could be ignored because a patent claim always begins
with “1. A” or “1. An” in terms of formality. To observe the finding with more
tokens, Figure 6 shows the distribution of ranks with all token sequences of
the 500 patent claims in the same experiment. A similar finding is that the
percentage of the “top 1” counts continues to dominate along the token
position. Based on these findings, it is hypothesized that a PatentGPT-J model
may save more keystrokes with an autocomplete function if the patent text to
draft is longer. This hypothesis requires more data to validate at scale in
the future.

Figure 5: Statistics of First 50 Token Positions (PatentGPT-J-456M) Figure 6:
Statistics of All Token Positions (PatentGPT-J-456M)

###  IV-F Interactive Inspection

The interactive tool in this section provides a user interface (UI) for
inspecting the details of the top 10 predictions at each token position. All
evaluations in experiment IV-B and their visualization are released as an
interactive web page at [24]. The UI is interactive in real time because the
details of all evaluation results were generated and stored in advance when
calculating the AE ratio. In Algorithm 1 (line: 10), the model predicts the
top 10 tokens based on the prompt. When implemented, what is stored are not
only the predicted top 10 tokens but also the related details at that point of
time. The stored details make it possible for the UI to show the results
interactively without triggering the time-consuming model inference again. For
example, Fig. 7 shows the details at token position 30. Explaining from the
top of the UI, the first part “context” is a text area showing the entire
patent text to calculate the AE ratio. The second part is a slider to adjust
the token position. A token position _n_ means the _nth_ token in the
“context.” Based on the token position, the third part of the UI shows all the
tokens before that position. The text area “prompt” is the input for the model
to generate the next token. In this example, the next actual token in the
“context” is the “duct” and the token is the top 1 choice in the prediction of
the model. Therefore, the rank (pick sequence) is 1. The probability of
picking the top 1 choice is 0.22, as shown below the “prompt” part. However,
whether the model will pick the top 1 choice is a random process. In fact, in
this example, the model picks the fifth token (“conduit”) during text
generation. In the UI, the next part shows the top 10 tokens and their text
and probabilities. The last part of the UI shows the generated text at that
token position, as shown in the text area “gen_text.” Depending on the
parameters for sampling, the model may generate patent text with different
degrees of randomness. The model continues to generate text until it reaches
the end of the claim. In this example, the model generates “conduit; a cooling
system operatively connected to the exhaust port of the power source; and …”
(carriage return omitted). For saving time, the length of text generation was
limited to 128 tokens in model inference. Although the patent claim generated
is not finished in this example, the generated text is coherent, fluent, and
sufficient for brainstorming. If not told, a patent professional may not
recognize the text as machine-generated immediately.

In the future, a practical use case for inventors based on such a tool might
be exploring inventive ideas in the generated patent text. Since the model is
specific to the patent domain, the model can generate domain-specific text
that may contain ideas different from the prompt. Different ideas may augment
inventors to invent better and faster. This idea of augmentation was proposed
as the “Augmented Inventing” idea in the author’s previous research [3]. The
difference and progress here is that, in this manuscript, the patent text is
generated at scale and at each token position. All text is generated and
stored in one batch. The UI in this section is a tool for an inventor to slide
through each position quickly and brainstorm with all generated patent text.
If needed, multiple samples of patent text can be generated with the same or
different parameters in sampling. How to enhance the tool after conducting
field tests and collecting users’ feedback is a crucial step to move forward.

Figure 7: Inspection: (1) next top 10 tokens, (2) generated text at position
_n_

##  V Legal Implications

Section 103 of the US Patent Law defines “A patent for a claimed invention may
not be obtained, …, if the _differences_ between the claimed invention and the
_prior art_ are such that the claimed invention as a whole would have been
_obvious_ before the effective filing date of the claimed invention to _a
person having ordinary skill in the art_ to which the claimed invention
pertains. Patentability shall not be negated by _the manner in which the
invention was made_ (emphasis added).” PatentGPT-J models may trigger several
legal questions from the perspective of Section 103. First, can a PatentGPT-J
model be deemed as or approximate the person having ordinary skill in the art
(PHOSITA)? A PHOSITA is a hypothetical person in patent law. The reason why a
neural model may serve the role of PHOSITA is that: The training of a neural
model is about generalizing among training data. If the training data are past
patents (prior arts), the model is a generalized model based on prior arts. If
such a generalized model approximates the ordinary skill in the prior arts,
the model may be deemed as the PHOSITA. In this line of thinking, a
fundamental question to be debated in the future will be: Can the
“generalization” in the technical sense be equivalent to the “ordinary” in the
legal sense?

Second, can we use a PatentGPT-J model to measure the nonobviousness
requirement in the patent law? A PatentGPT-J model is a generative model.
Assuming that the model is a PHOSITA, what the model can generate with high
probabilities might be deemed obvious to the model. If high probabilities can
approximate obviousness, a generative model such as PatentGPT-J may act as a
PHOSITA to measure nonobviousness with low probabilities. In this line of
thinking, a fundamental question to be debated will be: Can the “high or low
probabilities” in the technical sense be equivalent to the “obviousness or
nonobviousness” in the legal sense? Conceptually, it should be reasonable to
say that if one thing is harder (of lower probability or predictability) for
an inventor to predict, it should be something less obvious to the inventor.

In KSR, 111KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398 (2007) the Supreme
Court particularly emphasized “the need for caution in granting a patent based
on the combination of elements found in the prior art,” and importantly, the
Supreme Court reaffirmed principles based on its precedent that “[t]he
combination of familiar elements according to known methods is likely to be
obvious when it does no more than predictable results.” In this manuscript, if
the tokens of a language model can be considered the elements of the patent
language, can we say that the combination of high-probability tokens is the
combination of familiar elements? Can we say that the combination is likely to
be obvious when the language model yields no more than predictable results
(tokens)? In this line of thinking, a fundamental question to be debated will
be: Can the “tokens” in the technical sense be equivalent to the “elements” in
the legal sense?

Third, can an inventor combine the patent text generated from a PatentGPT
model and make the entire patent application eligible for patent protection?
According to Section 103 of the patent law, patentability shall not be negated
by _the manner in which the invention was made_. Although the statute was
codified to overturn the flash-of-genuine doctrine, can we say that the manner
of using a patent language model to conceive new ideas or combine generated
patent text will not negate any patentability? Like other questions in this
section, these challenging questions are unexplored and need more
interdisciplinary researchers to explore in the future.

##  VI Conclusion

This manuscript introduces the concept of evaluating model performance from a
human-centric perspective based on saved keystrokes in an autocomplete
context. The evaluation metric is applied to patent language models and the
best models are released after experiments. Although the models are pre-
trained from scratch with patent corpus and dedicated to the patent domain,
the concept in the metric and model evaluation is domain-agnostic and can
apply to a broader scope. For a specific domain, it is found that the largest
generative patent language model is not necessarily the best for the human-
centric metric. From a user’s perspective, several tools are also developed
for visualizing the experiment results and understanding the models deeper.
The visualization in saliency may particularly help inventors and patent
professionals identify the legal nonobviousness in patent text. Furthermore,
the overall implementation in this manuscript can serve as a proof of concept
for interdisciplinary research between computer science and patent law in the
future.

##  VII Acknowledgments

The research reported in this manuscript has been funded by the Ministry of
Science and Technology (MOST) in Taiwan (Project ID: 111-2222-E-A49-005). In
addition, the author would like to thank TensorFlow Research Cloud (TRC)
greatly for providing significant computational resources. Without the
resources, it would not be easy to build various patent language models
efficiently in this manuscript and cover a range of tasks (pre-training, fine-
tuning, inferencing, calculating the Autocomplete Effective ratio in this
research, etc.).

## References

  * [1] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [3] Jieh-Sheng Lee and Jieh Hsiang.  Patent claim generation by fine-tuning openai gpt-2.  World Patent Information, 62:101983, 2020.
  * [4] Jieh-Sheng Lee.  Controlling patent text generation by structural metadata.  In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM ’20, page 3241–3244, New York, NY, USA, 2020. Association for Computing Machinery.
  * [5] Jieh-Sheng Lee and Jieh Hsiang.  Prior art search and reranking for generated patent text.  In Proceedings of the 2nd Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech) 2021, co-located with the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), PatentSemTech 2021, pages 18–24, 2021.
  * [6] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2020.
  * [7] The USPTO.  Bulk data storage system.  https://bulkdata.uspto.gov/.
  * [8] Aran Komatsuzaki.  GPT-J-6B: 6B JAX-Based Transformer.  https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/, June 2021\.
  * [9] Ben Wang, Stella Biderman, and Javier de la Rosa.  How to Fine-Tune GPT-J.  https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md, December 2021.
  * [10] Alec Radrof, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2018.
  * [11] Maliheh Izadi, Roberta Gismondi, and Georgios Gousios.  Codefill: Multi-token code completion by jointly learning from structure and naming sequences, 2022.
  * [12] Gareth Ari Aye, Seohyun Kim, and Hongyu Li.  Learning autocompletion from real-world datasets.  In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 131–139, 2021\.
  * [13] EleutherAI.  pile-uspto.  https://github.com/EleutherAI/pile-uspto, 2020.
  * [14] cfoster0.  uspto-patent-data-parser (forked from tamerkhraisha).  https://github.com/cfoster0/uspto_patent_data_parser, 2020.
  * [15] TamerKhraisha.  uspto-patent-data-parser.  https://github.com/TamerKhraisha/uspto-patent-data-parser, 2020\.
  * [16] Jieh-Sheng Lee.  Training loss of PatentGPT-J-models.  https://wandb.ai/jason-lee/patent-gpt-j-d-4096/reports/PatentGPT-J-models--VmlldzoyMDgwNjI5?accessToken=u30ezzk5mxvfjdgqsyhhajfgm1mnoi9plj4x832jafpu4zl82zr5fc4fk3bn9zyl, May 2022.
  * [17] Ben Wang.  Training loss of GPT-J-6B.  https://wandb.ai/eleutherai/mesh-transformer-jax/reports/GPT-J-Pretraining--VmlldzoxMjMzOTgy, 2021\.
  * [18] Jieh-Sheng Lee.  PatentGPT-J repository.  https://github.com/jiehsheng/PatentGPT-J, 2022.
  * [19] Jieh-Sheng Lee.  (Demo 1) Inspecting AE ratio and keystroke distribution.  https://huggingface.co/spaces/patent/demo1, May 2022.
  * [20] OpenAI.  Better Language Models and Their Implications.  https://openai.com/blog/better-language-models, February 2019.
  * [21] Jieh-Sheng Lee.  (Demo 3) Inspecting AE Ratio and Keystroke Distribution of the unicorn text.  https://huggingface.co/spaces/patent/demo3, May 2022.
  * [22] NLP Cloud.  How to use gpt-3, gpt-j and gpt-neo, with few-shot learning.  https://nlpcloud.io/effectively-using-gpt-j-gpt-neo-gpt-3-alternatives-few-shot-learning.html, 2022\.
  * [23] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners, 2022.
  * [24] Jieh-Sheng Lee.  (Demo 2) Inspection: (1) next top 10 tokens, (2) generated text at position _n_.  https://huggingface.co/spaces/patent/demo2, May 2022.

|  JIEH-SHENG LEE received the M.S. and Ph.D. degrees from the Department of
Computer Science and Information Engineering, National Taiwan University,
Taiwan, and the LL.M. degree from the George Washington University Law School,
USA. He is currently an Assistant Professor with the School of Law, National
Yang Ming Chiao Tung University (NYCU), Taiwan. He was a recipient of the
National Collegiate Programming Contest Championship in Taiwan, and his team
represented Taiwan to participate in the World Final of the ACM International
Collegiate Programming Contest. He was also a recipient of the Top Prize of
Acer Dragon Thesis Award twice. Before joining the NYCU School of Law, he was
experienced in both IT (engineer & manager) and IP (in-house patent council)
in the semiconductor industry. In the IP field, he passed the USPTO patent bar
exam and is licensed in New York. In the IT field, he was certified as MCSE
(Microsoft), OCP (Oracle), SCJP & SCJD (Sun). His current research interests
include Deep Learning for patents, LegalTech, Patent Law, and AI & Law.
---|---