# ConSequence: Synthesizing Logically Constrained Sequences for Electronic
Health Record Generation

Brandon Theodorou1, 2, Shrusti Jain1, Cao Xiao3, Jimeng Sun1, 2

###### Abstract

Generative models can produce synthetic patient records for analytical tasks
when real data is unavailable or limited. However, current methods struggle
with adhering to domain-specific knowledge and removing invalid data. We
present ConSequence, an effective approach to integrating domain knowledge
into sequential generative neural network outputs. Our rule-based formulation
includes temporal aggregation and antecedent evaluation modules, ensured by an
efficient matrix multiplication formulation, to satisfy hard and soft logical
constraints across time steps. Existing constraint methods often fail to
guarantee constraint satisfaction, lack the ability to handle temporal
constraints, and hinder the learning and computational efficiency of the
model. In contrast, our approach efficiently handles all types of constraints
with guaranteed logical coherence. We demonstrate ConSequence’s effectiveness
in generating electronic health records, outperforming competitors in
achieving complete temporal and spatial constraint satisfaction without
compromising runtime performance or generative quality. Specifically,
ConSequence successfully prevents all rule violations while improving the
model quality in reducing its test perplexity by 5% and incurring less than a
13% slowdown in generation speed compared to an unconstrained model.

## Introduction

Sequential data generation applications are used in various fields, such as
healthcare (Choi et al. 2017; Biswal et al. 2021; Zhang et al. 2021;
Theodorou, Xiao, and Sun 2023), finance (Assefa et al. 2020; Dogariu et al.
2022), natural language processing (Gatt and Krahmer 2018; Dong et al. 2022;
Reiter and Dale 1997), and computer vision (Tulyakov et al. 2018; Yang,
Srivastava, and Mandt 2022; Ho et al. 2022). In these applications, generative
models demonstrate the capability to produce synthetic data closely resembling
real-world datasets. Notably, language models have showcased remarkable
achievements in domains like text generation (Chowdhery et al. 2022; Thoppilan
et al. 2022; Brown et al. 2020) and health record generation (Theodorou, Xiao,
and Sun 2023), primarily attributed to their adeptness in accurate next-token
forecasting. Beyond mimicking real data, generative models often require
adhering to specific rules and temporal dependencies unique to an application
domain. For instance, in medicine, generative models should follow logical
constraints such as indications (the reason for using a specific treatment for
a disease) and contraindications (the reason that makes a particular treatment
inadvisable for that disease) to create realistic sequential data.

To support domain-specific applications, generated samples must not only
approximate the true underlying data distribution but also encapsulate the
relationships and dependencies encoded as rules derived from external
knowledge. Failing to meet these constraints can hinder efficacy and deter
adoption. Various current approaches attempt to ensure logical consistency in
neural network outputs. These methods encompass strategies such as integrating
rule constraints into the loss function (Xu et al. 2018; Fischer et al. 2019),
employing post-processing modules designed to rectify model output violations
(Manhaeve et al. 2018; Hoernle et al. 2022; Ahmed et al. 2022), or directly
adding model components that ensure the final weights and outputs align with
domain knowledge (Towell and Shavlik 1994; Avila Garcez and Zaverucha 1999;
Giunchiglia and Lukasiewicz 2021). Despite the existing effort in constraint
enforcement, sequential data generation still faces several challenges in
aligning with real-world knowledge.

  * •

Inadequate treatment of temporal logical constraints. Existing models struggle
with temporal constraints, which can be static or evolve across multiple time
steps. These constraints are key to enhancing accuracy and reliability in
sequential generation models. Yet, no existing models effectively manage these
constraints as the information conditioning them may not be accessible at each
time step.

  * •

Efficiency and scalability. Sequential tasks typically encompass a multitude
of time steps and entail a substantial output dimensionality, underscoring the
need for a streamlined approach to facilitate large-scale sequence generation.
Current methods can be slow and computationally demanding, impairing real-
world generation speed (Bond-Taylor et al. 2021). Thus, an effective solution
should not only be asymptotically efficient but also maintain high generation
speeds for practical use.

  * •

Difficulty in achieving full logical consistency. Realism in sequential data
generation is paramount, but logical inconsistencies can compromise it. Hence,
encoding domain knowledge without error is essential. For full
trustworthiness, methods must uphold all logical constraints; even one
violation can erode user confidence. This is especially vital in critical
areas like healthcare.

In this paper, we present a method called ConSequence for addressing the
challenges of sequential constrained knowledge infusion. ConSequence can
handle entailment formatted rules with both soft and hard logical constraints,
covering static, temporal, and combined antecedents.

  * •

Temporal antecedent compilation via attentive history aggregation. We employ
an attention-based temporal aggregator for effectively consolidating
historical data at each time step, supporting both absolute and relative
aggregation while accommodating sequence variations.

  * •

Deterministic constraint execution via rule neuron. We introduce a graphical
module that verifies each antecedent component, providing absolute constraint
satisfaction.

  * •

Parallel constraint application via efficient GPU implementation. We represent
the mentioned components as weight matrices. These are then seamlessly
incorporated as a constraint head in neural network models, processing many
rules and records in parallel, resulting in minimal to no GPU slowdown and
maximizing overall efficiency.

To assess its effectiveness, we undertake a series of experiments on a
benchmark task employing the cutting-edge health record generation model, HALO
(Theodorou, Xiao, and Sun 2023), across two real-world datasets. We benchmark
our method against prevailing constraint enforcement approaches, evaluating
them on: (1) rule violation count, (2) overall model quality, and (3)
generative efficiency.

Our results demonstrate that ConSequence outperforms all other methods over
all three criteria. While most baselines result in over 25% of their generated
records for one dataset being invalid, ConSequence successfully prevents any
rule violations. Furthermore, ConSequence improves the model quality and
incurs less than a 13% slowdown in generation speed compared to an
unconstrained model. In comparison, all of the baselines that are not loss-
based exhibit slowdowns exceeding 37%, with the majority performing even more
poorly. This highlights the effectiveness of ConSequence for generating high-
quality health records while satisfying domain-specific constraints.

## Related Work

Various strategies have been proposed to apply logical constraints to neural
network models, especially in multi-label predictions (Dash et al. 2022;
Giunchiglia, Stoian, and Lukasiewicz 2022). One method is to include logical
constraints in the loss function during training, as regularization or penalty
terms (Xu et al. 2018; Fischer et al. 2019). Another approach involves
adjusting the model’s weights to satisfy the constraints (Towell and Shavlik
1994; Avila Garcez and Zaverucha 1999; Ahmed et al. 2022). Alternatively, some
techniques focus on mapping the model’s outputs to meet the constraints
(Manhaeve et al. 2018; Hoernle et al. 2022; Giunchiglia and Lukasiewicz 2021).

Despite the range of approaches, the ultimate goal is consistent: to ensure
the model’s outputs align with logical constraints, enhancing interpretability
and trustworthiness. While progress has been made in applying constraint
methods to various applications, adapting these techniques to our specific
task faces challenges. For instance, loss-based approaches (Xu et al. 2018;
Fischer et al. 2019) reduce constraint violations but do not guarantee logical
consistency, potentially limiting practical use for our needs. Additionally,
certain methods ensuring consistency (Giunchiglia and Lukasiewicz 2021;
Hoernle et al. 2022) can be slow, which might not suit large-scale sequential
generation settings.

Moreover, our task involves integrating temporal rules that may change over
time, without their logical precursors necessarily present at each step.
Although some methods have applied constraints to sequential text and molecule
data (Hokamp and Liu 2017; Liu et al. 2020), none have specifically addressed
the challenge of satisfying general temporal logical constraints efficiently.
This highlights the need for innovative approaches capable of handling complex
temporal constraints in a computationally efficient manner.

## Problem Formulation

We present the problem formulation for the task of longitudinal patient record
generation. We first define patient data.

######  Definition 1 (Sequential Patient Health Record Data).

A patient record is a time-sequenced series of visits, denoted as
$\mathcal{P}=\mathcal{V}^{(1)},\mathcal{V}^{(2)},\cdots,\mathcal{V}^{(T)}$.
Each visit, $\mathcal{V}^{(t)}$, consists of a variable number of unique
medical codes $c^{(t)}_{i}$ from a set $\mathcal{C}$. These codes,
$(c^{(t)}_{1},c^{(t)}_{2},\cdots c^{(t)}_{k})$, encode medical information
such as diagnoses, procedures, and medications.

To prepare for machine learning models, we transform $\mathcal{P}$ into a
matrix representation
$\mathbf{P}=[\mathbf{v}_{1},\mathbf{v}_{2},\cdots,\mathbf{v}_{T}]$ for a
patient with $T$ visits. Each visit
$\mathbf{v}_{t}\in\mathbb{R}^{|\mathcal{C}|}$ is a multi-hot binary vector,
with $c_{t}^{i}\in\\{0,1\\}$ representing the presence of the $i$-th code in
the $t$-th visit of $\mathcal{P}$. Static features such as gender, ethnicity,
and birth year are encoded in the first visit $\mathcal{V}^{(1)}$.

In synthetic patient generation, our aim is to create synthetic patient
sequences $\mathbf{P}^{\prime}$ from scratch, given a real patient dataset
$\mathcal{D}$ to train on, that mimic real records $\mathbf{P}\in\mathcal{D}$
and offer equivalent downstream utility. We incorporate real-world knowledge
through logical constraints $\mathcal{R}$ to guide generation. These
constraints, defined as a series of logical entailment statements over codes
and visits, may be grounded in a single time step or span multiple time steps.
We encode them in a format called Conjunctive Implicative Form.

######  Definition 2 (Conjunctive Implicative Form).

A statement is in Conjunctive Implicative Form (CIF) if it is of the form
$a_{1}\land a_{2}\land\ldots\land a_{k}\implies a_{k+1}$ for $k\geq 0$, where
$a_{i}$ is a literal such that $a_{i}=x_{j}$ or $a_{i}=\lnot x_{j}$ for a
variable $x_{j}$.

Encoding Rules. Our set of variables consists of generated codes across all
timesteps, that is, all $c_{t}^{i}$. We let $a_{t}^{i}$ denote either of
$c_{t}^{i}$ and $\lnot c_{t}^{i}$. Given this, our rules must be in one of the
following formats.

  1. 1.

$True\Rightarrow a_{t}^{j}$, a rule with an empty antecedent, though this does
not generally appear in practice.

  2. 2.

$a_{t}^{i}\Rightarrow a_{t}^{j}$, represents a hierarchical relationship: a
specified truth value of $c_{t}^{i}$ necessitates one of $c_{t}^{j}$. For
example, if a patient is taking insulin, they must have diabetes.

  3. 3.

$a_{t}^{i}\land a_{t}^{j}\land\cdots\land a_{t}^{k}\Rightarrow a_{t}^{l}$,
which extends the former to relate the truth value of a boolean combination of
terms to that of a singular term. For example, if a patient has heart disease
and takes a statin, they must have high cholesterol.

  4. 4.

$a_{t-1}^{i}\Rightarrow a_{t}^{j}$, a temporal rule relating the prior
timestep and current time. For example, lifelong diseases like diabetes will
always continue to appear once they begin.

  5. 5.

$a_{I}^{i}\Rightarrow a_{t}^{j}$, a boolean combination of terms across a set
of specified previous timesteps, $I$, which relate to the current time. For
example, a patient without a past history of pregnancy can’t have
complications during childbirth.

  6. 6.

$a_{I}^{i}\land\cdots\land a_{I}^{j}\land\cdots\land a_{t}^{k}\land\cdots\land
a_{t}^{l}\Rightarrow a_{t}^{m}$, relating a boolean combination of terms
across both past and current timesteps to a singular boolean variable
$c_{t}^{m}$. For example, a patient with past diabetes and current numbness in
their feet must have diabetic peripheral neuropathy.

Each rule $r$ consists of a temporal component
$I^{(r)}=\\{t_{1},t_{2},\cdots,t_{|I|}\\}$ which determines the indices of the
past time steps referred to, the variables in the past and present which
condition the output, and the output literal111We assume rules are acyclic but
make no additional assumptions, and this approach can be generalized to cyclic
rules with an initial transformation and elimination step, not discussed
here.. If a rule has a non-empty temporal component, we classify it as
temporal, and we otherwise consider it to be static. We use entailment
statements with conjunctive antecedents to represent rules, offering
conciseness, efficient processing, and aligning with logical constraints in
real-world settings. In contrast, other models (Xu et al. 2018; Hoernle et al.
2022) use logical normal forms combining variables with AND, OR, or entailment
connectors, which can be cumbersome for large outputs, without additional
representational benefits. Next we show that our entailment-based format
equals the representational power of alternative encoding schemes.

######  Theorem 1.

Given a boolean expression $C_{1}\land\dots\land C_{k}$ in conjunctive normal
form, there exists an equivalent set of statements in conjunctive implicative
form.

###### Proof.

An expression in conjunctive normal form is the conjunction of one or more
clauses $C_{1}\land\dots\land C_{k}$, where each clause $C_{i}$ is the
disjunction of one or more literals $a_{i,1}\lor\dots\lor a_{i,m}$. Given an
expression in conjunctive normal form, we proceed as follows. First, we
convert each clause $C_{i}=a_{i,1}\lor\dots\lor a_{i,m}$ to to the equivalent
statement $\lnot(a_{i,1}\lor\dots\lor a_{i,m-1})\implies a_{i,m}$. Applying De
Morgan’s Laws, we can further convert this to the statement in conjunctive
implicative form $\lnot a_{i,1}\land\dots\land\lnot a_{i,m-1}\implies
a_{i,m}$. That is, the converted expression is now the conjunction of
conjunctive implicative statements. Within our format, the conjunction of
statements is implicitly represented through their mutual presence within a
set, so $f$ is logically equivalent to the set of statements in conjunctive
implicative form $\\{(\lnot a_{1,1}\land\dots\land\lnot a_{1,m-1}\implies
a_{1,m}),\dots,(\lnot a_{k,1}\land\dots\land\lnot a_{k,p-1}\implies
a_{k,p})\\}$ formed through this conversion. ∎

##  ConSequence Method

We introduce our ConSequence method (seen in Figure 1) for generating
logically constrained sequential data. ConSequence is designed to directly
mirror the logical entailment process. It involves two modules: temporal
aggregation and antecedent evaluation, followed by a matrix multiplication-
based approach for efficient parallel constraint application optimized for
modern GPU architectures.

Figure 1: ConSequence refines EHR generation models by incorporating rule-
based prior knowledge. The rule evaluation process aggregates past history and
feeds it with the current visit to the rule neuron as the logical antecedent.
If the neuron’s activation threshold is met, it fires and overrides the prior
model’s output to ensure logical consistency in the generated EHRs

### A. Temporal Aggregation

To support temporal rules, we first conduct temporal sequence aggregation.
Specifically, given the sequence of all visits
$\mathbf{P}=[\mathbf{v}_{1},\mathbf{v}_{2},\dots,\mathbf{v}_{T}]$, where
$\mathbf{v}_{t}$ is the visit at time $t$, we consider a temporal rule
$r\in\mathcal{R}$ that contains a temporal component $I^{(r)}$. $I^{(r)}$
refers to the set of indices of past time steps whose visits are referenced
within $r$, and may be both absolutely (e.g. the second time step) and
relatively (e.g. the previous time step) numbered, where we represent relative
numberings as negative indices. We define $f_{I^{(r)}}$ as a function that
maps a time step $t$ to a set of referenced past time steps using $I^{(r)}$,
converting relatively numbered visits to absolutely numbered ones and handling
the inability of time steps to reference themselves and future visits as a
part of their past history. For example, if $I^{(r)}=\\{-1,1,4\\}$, then we
have
$f_{I^{(r)}}(1)=\\{\\},f_{I^{(r)}}(2)=\\{1\\},f_{I^{(r)}}(3)=\\{1,2\\},f_{I^{(r)}}(10)=\\{1,4,9\\}$.
Next, we aggregate the relevant past visits according to $f_{I^{(r)}}(t)$. We
define binary mask vector $\mathbf{m}_{t}^{(r)}$ which varies across both
rules and time steps:

$m_{t,i}^{(r)}=\begin{cases}1&\text{if }i\in f_{I^{(r)}}(t)\\\
0&\text{otherwise}\end{cases}\quad\text{where
}\mathbf{m}_{t}^{(r)}\in\\{0,1\\}^{T}.$ (1)

We then construct the aggregated history representation
$\mathbf{h}_{t}^{(r)}\in\mathbb{R}^{|\mathcal{C}|}$ for time step $t$ and rule
$r$ by:

$\mathbf{h}_{t}^{(r)}={\bigcup}_{i=1}^{t-1}m_{t,i}^{(r)}\mathbf{v}_{i}$ (2)

Here, $\bigcup$ denotes the boolean OR operation, and $\mathbf{h}_{t}^{(r)}$
is a static vector that aggregates the relevant codes within past visits
according to the temporal component $I^{(r)}$. This allows us to aggregate
past sequences into fixed representations for each time step and rule, which
are treated as standard variables for the entailment calculation in the next
module.

### B. Antecedent Evaluation

Once we have $\mathbf{h}_{t}^{(r)}$, we turn the problem into a standard
entailment constraint problem. We solve this problem using a recurrent neuron
model to evaluate the antecedent of a logical entailment and set the
consequent. Each logical rule is constructed as a neuron, inspired by
activation thresholds in biological and machine learning processes (Lin 2017),
that only activates when all antecedent literals are satisfied, feeding back
into the input to set the consequent.

#### Construction of the Rule Neuron

Let the variables that the rule operates over be represented by the binary
vector
$\mathbf{x}_{t}^{(r)}=\text{concat}(\mathbf{h}_{t}^{(r)},\mathbf{v}_{t})$
which consists of both the aggregated historical variables and current visit
representation. The antecedent of one of our logical entailment rules is
defined as a conjunction of literals, each of the form $x_{t,i}^{(r)}$ or
$\neg x_{t,i}^{(r)}$, referring to $i$-th element of $\mathbf{x}_{t}^{(r)}$.
Let $L=\\{x_{t,p_{1}}^{(r)},\cdots,x_{t,p_{p}}^{(r)},\neg
x_{t,n_{1}}^{(r)},\cdots,\neg x_{t,n_{n}}^{(r)}\\}$ be the set of such
literals in the antecedent which refer collectively to a subset of
$\mathbf{x}_{t}^{(r)}$. The rule neuron accepts inputs from the variables in
$\mathbf{x}_{t}^{(r)}$ and is constructed as follows: we denote the weight of
the connection between the neuron and the $i$-th variable in
$\mathbf{x}_{t}^{(r)}$ as $w_{i}^{(r)}$ and set it to $1$ if the variable is
in $L$ and $-1$ if its negation is in $L$. For all other variables, the weight
is set to 0. This construction is based on the logical interpretation of the
antecedent, where the presence of a variable in the antecedent implies its
positive contribution to the truth value, and the presence of its negation
implies its negative contribution. More explicitly, we set the neuron’s
weights by:

$w_{i}^{(r)}=\begin{cases}1&\text{if }x_{t,i}\in L\\\ -1&\text{if }\neg
x_{t,i}\in L\\\ 0&\text{otherwise}\end{cases}\\\ $ (3)

For instance, consider a rule stipulating that patients with past history of
gastric ulcer and current symptom of melena, without past colorectal cancer,
current hematemesis, or current esophageal varices must have gastrointestinal
bleeding. The weights corresponding to the variables for past gastric ulcer
and current melena in $\mathbf{x}_{t}^{(r)}$ would be 1 while the variables
for past colorectal cancer, current hematesis, and current esophageal varices
to -1. All others would be 0.

#### Activation Process

Given the constructed neuron, we use it as follows. We first set the
activation threshold $\theta^{(r)}$ to the number of positive, non-negated
variables (of the form $x_{t,i}$ rather than $\neg x_{t,i}$) in $L$. Given
$\mathbf{x}_{t}^{(r)}$, the activation process begins. The weighted sum of the
inputs is calculated by Eq. (4):

$s_{t}^{(r)}=\sum_{i=1}^{2\cdot|C|}w_{i}^{(r)}\cdot x_{t,i}$ (4)

If the $s_{t}^{(r)}$ is greater than or equal to $\theta^{(r)}$, the neuron
fires and sets the output variable $o^{(r)}$ in $\mathbf{v}_{t}$ to the output
value, $\alpha^{(r)}$. Otherwise, it remains inactive. It’s important to note
that, based on the construction, $\theta^{(r)}$ is reached only if all of the
positive variables and none of the negations are found in the input. For
instance, using the previously mentioned example rule, $\theta^{(r)}$ would be
set to 2. $s_{t}^{(r)}$ would then equal 2 only if the variables for past
gastric ulcer and current melena are both 1 in $\mathbf{x}_{t}^{(r)}$, as they
are the sole potential positive contributions during the activation sum.
Furthermore, the variables for past colorectal cancer, current hematesis, and
current esophageal varices must all be 0 to avoid any reduction below 2 due to
their corresponding weight of -1.

#### Handling Soft Constraints

We also want to support settings where $r$ is a soft constraint with
$\alpha^{(r)}$ between 0 and 1. This flexibility is essential when the
relation between variables is probabilistic. For example, most cases of COPD
involve a past history of smoking, but exceptions do exist. So, we might
include a rule stating that a patient without a past history of smoking has
just a 1% chance of presently showing COPD and set $\alpha^{(r)}=0.01$
accordingly. To address such cases, we add a final step before setting the
output variable $o^{(r)}$ in $\mathbf{v}_{t}$. During training, we set the
variable to the correct probability, but the output variables must equal 0 or
1 during generation. To achieve this, we sample from a Bernoulli distribution
with probability $p=\alpha^{(r)}$ before setting the value, resulting in a
binary output while satisfying soft constraints by aligning that output with
the proper underlying probabilities. Note that we can also apply this sampling
process to hard constraints because sampling from $p=0$ and $p=1$ always yield
0 and 1, respectively. Therefore, we can integrate this sampling process into
our core activation process without special handling for different constraints
types.

#### Rule Evaluation

The rule evaluation process involves encoding the temporal history of each
binary variable and feeding them, along with the current variables, to the
rule neuron. The rule neuron evaluates the antecedent by computing the
weighted sum of its inputs, comparing it to the activation threshold, and
activating the neuron if the threshold is reached. If the neuron fires, it
overrides any modeling by the preceding architecture and sets the output
variable to the mandated output value, sampling from that value if during
generation as opposed to training, as in Eq. (5):

$v_{t,o^{(r)}}=\begin{cases}\text{Bernoulli}(\alpha^{(r)})&\text{if
}s_{t}^{(r)}\geq\theta^{(r)}\\\ v_{t,o^{(r)}}&\text{otherwise}\end{cases}\\\ $
(5)

This end-to-end process can be seen in Figure 1.

### C. Parallel Constraint Application

In the previous sections, we have presented ConSequence in the context of a
single rule and, in most cases, a single time step. During generation, we will
only be concerned with the most recent visit at any given step, but during
training it will be valuable to constrain all time steps simultaneously.
Furthermore, applying multiple rules simultaneously is valuable for improving
efficiency in any setting. So, we present a matrix multiplication formulation
of ConSequence that leverages modern GPU computing architectures to
simultaneously and efficiently apply groups of rules across time steps and
across batches of patient records.

#### Rule Grouping and Combination

We begin by organizing rules into distinct groups, denoted as
$G=\\{g_{1},\cdots,g_{n}\\}$ where each group
$g_{i}=\\{r_{1},\cdots,r_{|g_{i}|}\\}$ contains a set of rules that can be
processed simultaneously. While the temporal aggregation module prevents the
combination of rules with different temporal components, we note that most
rules use one of a few common components such as the first visit, the previous
visit, or all past history. With this in mind, we categorize rules according
to their temporal components.

We can then combine rules in each category of temporal components into rule
groups if they have different output variables and their antecedent variables
do not include output variables from previous rules in the same category. To
do this, we iterate through the rules of each category, adding them to the
current group and starting a new group each time we encounter a rule whose
input or output variables overlaps with the output of a preceding rule in the
current group.

#### Temporal Aggregation

To perform temporal aggregation across a group of rules, $g$, with a shared
temporal component, we use a technique inspired by masked self-attention. It
involves constructing a temporal mask matrix
$\mathbf{M}^{(g)}\in\mathbb{R}^{T\times T}$, where each row
$\mathbf{m}_{t}^{(g)}\in\mathbb{R}^{T}$ is a binary vector that determines
which other time steps the $t$-th time step can attend to. Specifically, each
$m_{t,i}^{(g)}$ in $\mathbf{m}_{t}^{(g)}$ is set by Eq. (6):

$m_{t,i}^{(g)}=\begin{cases}1&\text{if }i\in f_{I^{(g)}}(t)\\\
0&\text{otherwise}\end{cases}\quad\text{where
}\mathbf{m}_{t}^{(g)}\in\\{0,1\\}^{T}.$ (6)

By employing boolean matrix multiplication, we compute
$\mathbf{H}^{(g)}\in\mathbb{R}^{T\times|\mathcal{C}|}$ using the patient
record $\mathbf{P}$ by $\mathbf{H}^{(g)}=\mathbf{M}^{(g)}\mathbf{P}$, where
each row is the aggregated temporal history vector $\mathbf{h}^{(r)}_{t}$
introduced previously (for any $r\in g$).

#### Rule Execution and Output

This aggregated temporal history matrix can then be concatenated along the
code dimension to $\mathbf{P}$ to form
$\mathbf{X}^{(g)}=\left[\mathbf{H}^{(g)},\mathbf{P}\right]$. We then construct
matrix $\mathbf{W}^{(g)}\in\mathbb{R}^{2\cdot|C|\times|C|}$ to represent the
rule neuron weights. For each rule in the group, we set the values in the
column corresponding to the output code based on the construction of
$\mathbf{w}^{(r)}$ detailed earlier. Note that since there is no overlap in
output variables within a group, each column is constructed without
interference between different rules.

Utilizing matrix multiplication, we compute the antecedent sum output
$\mathbf{S}^{(g)}=\mathbf{X}^{(g)}\mathbf{W}^{(g)}$, resulting in a matrix
$\mathbf{S}^{(g)}\in\mathbb{R}^{T\times|C|}$ where each value whose row
corresponds to time step $t$ and whose column corresponds to the output
variable in rule $r$ is equal to the sum $s_{t}^{(r)}$ from earlier. This
value is a count of how many positive literals in the corresponding antecedent
were satisfied at that time step minus how many negatives were failed (0 if
there was no rule corresponding to that variable). When this count is equal to
the number of positive literals, $\theta$, we set the output code to the
output value, $\alpha$. As we have multiple $\theta$ and $\alpha$ values per
group, we combine the constants into vectors
$\boldsymbol{\theta}\in\mathbb{R}^{|C|}$ (setting values corresponding to
output codes which are not used in the rule group to -1 so that the equality
is never satisfied) and $\boldsymbol{\alpha}\in\mathbb{R}^{|C|}$ respectively
which may perform comparison and instantiations element-wise in parallel. To
support both hard and soft constraints simultaneously, we can also first
sample from the Bernoulli distribution defined by $p=\boldsymbol{\alpha}$ and
set the output code to the resulting binary values. As a single step equation,
this process can be summarized:

$\mathbf{P}[\mathbf{X}^{(g)}\mathbf{W}^{(g)}==\boldsymbol{\theta}]=\text{Bernoulli}(\boldsymbol{\alpha})$
(7)

where the group of rules are applied simultaneously over all time steps and
all records in a batch, effectively enabling parallel enforcement of rules.

| Outpatient | Inpatient
---|---|---
# Records | 1,006,321 | 46,520
Mean Visits/Record | 35.40 | 1.27
Mean Codes /Visit | 1.69 | 13.61
# Phecodes | 1,817 | 1,610
Demographic | 7 | 6
Table 1: Dataset Statistics | Violations on Outpatient Dataset | Violations on Inpatient Dataset
---|---|---
| Static | Temporal | % Valid | Static | Temporal | % Valid
Vanilla HALO | 1265.4 (18.1) | 17.5 (2.9) | 87.2% (0.002) | 4320.6 (22.2) | 2046.2 (35.6) | 65.4% (0.002)
Post Processing | 0.0 | 0.0 | 100% | 0.0 | 0.0 | 100.0%
Semantic Loss | 955.5 (9.2) | 18.7 (3.6) | 90.3% (0.001) | 3827.0 (23.3) | 1817.2 (28.8) | 65.3% (0.002)
CCN | 1013.6 (15.0) | 17.7 (2.1) | 91.0% (0.001) | 3070.24 (18.3) | 2149.76 (32.5) | 70.7% (0.002)
MultiPlexNet | 0.0 | 2714.2 (3.6) | 89.7% (0.001) | 0.0 | 3304.8 (20.1) | 72.7% (0.002)
SPL | 0.0 | 41 | 99.9% | 0.0 | 11.5 (1.4) | 99.9% (0.0001)
ConSequence | 0.0 | 0.0 | 100% | 0.0 | 0.0 | 100%
Table 2: Rule Violation Counts

### D. Runtime Analysis

ConSequence can process, evaluate, and adhere to rules in $O(T^{2}|C|)$
running time. We include a proof in our supplement, but the outline involves
breaking the process down into its smaller components, analyzing the time to
aggregate the history, generate the rule neuron, perform the entailment
calculation, and set the output variables.

### E. Training and Generation Process

We conclude our presentation of ConSequence by outlining how it works with the
underlying generative architecture. ConSequence is designed to integrate
seamlessly into any underlying sequentially generating architecture. During
both training and generation, the process involves feeding the patient record
input into the underlying model to predict the probabilities of the variables
in the next visit before updating those probabilities via the ConSequence
module.

During training, the true binary labels for the predicted visits are used as
inputs to ConSequence. The antecedent evaluation process updates the model’s
predicted probabilities based on logical constraints. The enhanced model’s
predictions, now incorporating constraint-driven logic, can then be compared
to the ground truth labels, allowing for the calculation of a loss value that
guides the model’s training process without having to worry about learning
prior

During generation, the underlying model predicts the probabilities for the
next visit based on the current record, and a full, binary visit is sampled
from those probabilities. The predicted visit is then passed through
ConSequence, which enforces the logical constraints. This ensures that the
generated sequence adheres to the desired constraints while still being
synthesized by the underlying architecture.

So, ConSequence enhances the underlying generative architecture by
incorporating logical constraints seamlessly into both the training and
generation processes. This enables the end-to-end training of a stronger model
and the generation of logically constrained sequential data that maintains
clinical coherence while meeting specific requirements.

Figure 2: Training and Generation Speeds on Outpatient (left) and Inpatient
(right) datasets

## Experiments

### Experimental Setting

We evaluate ConSequence against state-of-the-art constraint enforcement
approaches by applying each to the HALO (Theodorou, Xiao, and Sun 2023)
architecture (as one exemplar for these model agnostic techniques) for
synthetic EHR generation. We perform a series of experiments on a pair of
underlying real-world EHR datasets. Below are details concerning our data,
baselines, and experimental design with remaining information found in our
supplement.

#### Datasets

We used an outpatient EHR dataset from real-world US claims data and an
inpatient EHR dataset from the public MIMIC-III dataset. Both datasets
represent patients as visits. We encoded gender and age categories into the
patient matrix representation $\mathbf{P}$ using binary variables and added a
“label visit” with only this demographic information at the beginning of each
set of visits. This enables us to condition subsequent medical codes by the
patient’s gender or age. Each dataset contains visits with ICD-9 diagnosis
codes that we aggregated to higher level codes based on their respective
ontologies. For instance, we combined diagnosis codes based on initial letter
and first two digits (e.g., A24, A24.1, and A24.25 are aggregated to A24). We
also limited each patient to a max of 100 visits, which removed very few
visits in practice. Table 1 provides the statistics of the final datasets.
Additional details are available in the supplement.

#### Rules

Although we expect experts to provide most rules in practice, we generated
them in a data-driven approach here.

We identify static rules as: 1) For each demographic category, we permit at
most one variable to be set (or none if unknown). Thus, autoregressive rules
were added to prevent setting subsequent variables if a previous one was
already set. 2) We identify pairs of codes that exclusively co-occur. For each
pair, we establish a rule mandating the presence of the first code whenever
the second one is present.

We identify temporal rules as follows: 1) We identify prevalent codes that
never appear within a demographic category. For each demographic category, we
establish a rule disallowing these codes if the corresponding demographic
variable is set in the first visit. 2) We detect common codes that only occur
following a specific code. In each instance, we impose a rule preventing the
former code unless the latter has previously occurred. 3) We find frequent
codes that consistently appear in subsequent visits once they’ve occurred. For
each case, we formulate a rule necessitating the code in the current visit if
it was present in the previous one.

Our final rule sets consist of 71 rules (14 static, 57 temporal) for the
outpatient dataset and 95 (11 static, 84 temporal) for the inpatient dataset.
More details about cutoffs, counts by type, and some examples can be found in
the appendix.

#### Baselines

We compare our method with several baseline approaches for handling logical
constraints.

  * •

Post Processing is a naive approach removing any generated samples violating
the constraints and generates more samples to replace them. It guarantees
consistency with the constraints but is slower due to repeated generation and
rule-checking, and may incur distribution shifts.

  * •

Semantic Loss (Xu et al. 2018) is a regularization method based on the
probability of breaking a constraint. It can handle temporal rules but does
not guarantee compliance.

  * •

CCN (Giunchiglia and Lukasiewicz 2021) is a rule-based method that uses
maximum and minimum operations to set the output based on the values of the
inputs in the antecedent. It guarantees compliance but cannot handle temporal
rules or any where the output variable is set to 0.

  * •

MultiPlexNet (Hoernle et al. 2022) introduces a categorical latent variable to
select between satisfying assignments of constraints. It guarantees the
satisfaction of static rules but does not support temporal constraints.

  * •

Semantic Probability Layer (SPL) (Ahmed et al. 2022) constructs probabilistic
circuits to handle a wide variety of constraints. It can handle any static but
no temporal rules222We modified both the SPL and corresponding HALO model for
partial compatibility with our setting but couldn’t achieve full capabilities.
Further details are available in the supplement..

#### Implementation

We train HALO models on both datasets and apply each constraint enforcement
method in turn. We generate a synthetic dataset of 10,000 patients from
scratch with each method and without any constraints for comparison. These
generations are performed on identical hardware (a single Tesla P100 GPU), and
the generation process is repeated 25 times for each method (except for SPL on
the outpatient dataset as it takes too long) to validate our results.

### Results on the Evaluation of Logical Adherence

Our first evaluation measures each method’s effectiveness in preventing rule
violations. We count rule violations within the synthetic datasets, stratified
by rule type, and calculate the percentage of fully valid generated patients.

Table 2 presents these statistics, highlighting that ConSequence is the only
non-post-processing method guaranteeing full prevention of rule violations.
Although CCN, MultiPlexNet, and SPL adhere to some static rules and Semantic
Loss regularizes all constraints, no baseline methods fully satisfy temporal
rules, leading to numerous invalid generations. In contrast, ConSequence
manages all rule types, preventing invalid generations more efficiently than
the naive Post Processing approach.

### Results on the Evaluation of Data Enrichment

This section assesses whether knowledge infusion impacts learning and
generation by evaluating the quality of the model and its synthetic data. We
first calculate the model’s perplexity on a held-out test set, which signifies
the log probability of the test set, normalized by the number of codes in the
dataset. Lower perplexity values indicate superior performance. Table 3
displays the perplexities for each method on each dataset. We find that
ConSequence enhances the model’s performance on both datasets by eliminating
error sources from predicted rule violations, allowing for absolute non-
violation predictions, and facilitating the model to learn dataset patterns
during training beyond the rules. ConSequence exhibits the most significant
improvement, while CCN and Semantic Loss moderately enhance learning. In
contrast, MultiPlexNet and SPL baselines hinder the modeling of the underlying
data distribution by introducing more complex loss formulations during
training.

| Outpatient | Inpatient
---|---|---
Vanilla HALO | 24.019 | 13.443
Semantic Loss | 23.995 | 13.432
CCN | 23.976 | 12.985
MultiPlexNet | 119.560 | 24.260
SPL | 34.454 | 22.794
ConSequence | 23.924 | 12.712
Table 3: Modeling Perplexity Performance

We also evaluated the realism of the generated dataset by comparing the
probabilities of codes and their combinations in each synthetic dataset to
their original training datasets, and we provide those results in our
supplement.

### Results on Method Efficiency

The final evaluation checks if ConSequence maintains robust real-world
efficiency. Figure 2 presents the training and generation speeds for each
model. Our results highlight that ConSequence provides commendable real-world
efficiency. It outpaces other constraint methods during training, while the
baselines slow down due to memory or algorithmic inefficiencies. During
generation, ConSequence surpasses all baselines barring Semantic Loss, which
matches Vanilla HALO model during inference but can’t guarantee constraint
adherence. ConSequence experiences less than a 13% slowdown in generation time
on both datasets, showing that it meets the real-world efficiency criterion.

## Conclusion

We introduce ConSequence, an effective approach that enforces logical
constraints in sequential generative models for the first time. ConSequence
seamlessly integrates rule-based domain knowledge into generative neural
network outputs through temporal aggregation and antecedent evaluation
modules. It efficiently enforces these constraints using a matrix
multiplication formulation of these modules, ensuring the satisfaction of both
hard and soft logical constraints. Through extensive experimentation in the
domain of electronic health record generation, we demonstrate that ConSequence
outperforms comparable models in both efficiency and effectiveness, incurring
a minimal slowdown compared to an unconstrained model while eliminating all
rule violations and enhancing overall generative quality.

## Ethics Statement

We view the possible broader impact of ConSequence through the lens of
enabling better and more realistic sequential synthetic data. This includes
facilitating better video generation, more effective synthetic financial data,
and improved healthcare data, These increases in generative quality alone can
have transformative effects on machine learning research in related domains,
improve the creative development of media, and permit freer data sharing.
However, even beyond overall improvements in data quality, we see the
elimination of logical inconsistencies as especially effective for increasing
adoption of these generative models and the data they produce. One of the
typical checks for utilizing such systems, particularly in the healthcare
domain, is a manual review of the created data. Logical inconsistencies are
particularly visible and therefore among the main barriers in the adoption
process. Therefore, we believe that ConSequence’s ability to prevent such
inconsistencies can facilitate not just better generative models but allow
them to disseminate more widely and exponentially propagate the positive
impact of synthetic data.

## References

  * Ahmed et al. (2022) Ahmed, K.; Teso, S.; Chang, K.-W.; Van den Broeck, G.; and Vergari, A. 2022.  Semantic probabilistic layers for neuro-symbolic learning.  _Advances in Neural Information Processing Systems_ , 35: 29944–29959.
  * Assefa et al. (2020) Assefa, S. A.; Dervovic, D.; Mahfouz, M.; Tillman, R. E.; Reddy, P.; and Veloso, M. 2020.  Generating synthetic data in finance: opportunities, challenges and pitfalls.  In _Proceedings of the First ACM International Conference on AI in Finance_ , 1–8.
  * Avila Garcez and Zaverucha (1999) Avila Garcez, A. S.; and Zaverucha, G. 1999.  The connectionist inductive learning and logic programming system.  _Applied Intelligence_ , 11: 59–77.
  * Biswal et al. (2021) Biswal, S.; Ghosh, S.; Duke, J.; Malin, B.; Stewart, W.; Xiao, C.; and Sun, J. 2021\.  EVA: Generating longitudinal electronic health records using conditional variational autoencoders.  In _Machine Learning for Healthcare Conference_ , 260–282. PMLR.
  * Bond-Taylor et al. (2021) Bond-Taylor, S.; Leach, A.; Long, Y.; and Willcocks, C. G. 2021.  Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Brown et al. (2020) Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020.  Language Models are Few-Shot Learners.  _CoRR_ , abs/2005.14165.
  * Choi et al. (2017) Choi, E.; Biswal, S.; Malin, B.; Duke, J.; Stewart, W. F.; and Sun, J. 2017.  Generating multi-label discrete patient records using generative adversarial networks.  In _Machine learning for healthcare conference_ , 286–305. PMLR.
  * Chowdhery et al. (2022) Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Dash et al. (2022) Dash, T.; Chitlangia, S.; Ahuja, A.; and Srinivasan, A. 2022.  A review of some techniques for inclusion of domain-knowledge into deep neural networks.  _Scientific Reports_ , 12(1): 1040.
  * Dogariu et al. (2022) Dogariu, M.; Ştefan, L.-D.; Boteanu, B. A.; Lamba, C.; Kim, B.; and Ionescu, B. 2022.  Generation of realistic synthetic financial time-series.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 18(4): 1–27.
  * Dong et al. (2022) Dong, C.; Li, Y.; Gong, H.; Chen, M.; Li, J.; Shen, Y.; and Yang, M. 2022.  A survey of natural language generation.  _ACM Computing Surveys_ , 55(8): 1–38.
  * Fischer et al. (2019) Fischer, M.; Balunovic, M.; Drachsler-Cohen, D.; Gehr, T.; Zhang, C.; and Vechev, M. 2019.  DL2: training and querying neural networks with logic.  In _International Conference on Machine Learning_ , 1931–1941. PMLR.
  * Gatt and Krahmer (2018) Gatt, A.; and Krahmer, E. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _Journal of Artificial Intelligence Research_ , 61: 65–170.
  * Giunchiglia and Lukasiewicz (2021) Giunchiglia, E.; and Lukasiewicz, T. 2021.  Multi-label classification neural networks with hard logical constraints.  _Journal of Artificial Intelligence Research_ , 72: 759–818.
  * Giunchiglia, Stoian, and Lukasiewicz (2022) Giunchiglia, E.; Stoian, M. C.; and Lukasiewicz, T. 2022.  Deep learning with logical constraints.  _arXiv preprint arXiv:2205.00523_.
  * Ho et al. (2022) Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022\.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_.
  * Hoernle et al. (2022) Hoernle, N.; Karampatsis, R. M.; Belle, V.; and Gal, K. 2022.  Multiplexnet: Towards fully satisfied logical constraints in neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 5700–5709.
  * Hokamp and Liu (2017) Hokamp, C.; and Liu, Q. 2017.  Lexically constrained decoding for sequence generation using grid beam search.  _arXiv preprint arXiv:1704.07138_.
  * Lin (2017) Lin, J.-W. 2017.  Artificial neural network related to biological neuron network: a review.  _Advanced Studies in Medical Sciences_ , 5(1): 55–62.
  * Liu et al. (2020) Liu, X.; Liu, Q.; Song, S.; and Peng, J. 2020.  A chance-constrained generative framework for sequence optimization.  In _International Conference on Machine Learning_ , 6271–6281. PMLR.
  * Manhaeve et al. (2018) Manhaeve, R.; Dumancic, S.; Kimmig, A.; Demeester, T.; and De Raedt, L. 2018.  Deepproblog: Neural probabilistic logic programming.  _advances in neural information processing systems_ , 31.
  * Reiter and Dale (1997) Reiter, E.; and Dale, R. 1997.  Building applied natural language generation systems.  _Natural Language Engineering_ , 3(1): 57–87.
  * Theodorou, Xiao, and Sun (2023) Theodorou, B.; Xiao, C.; and Sun, J. 2023.  Synthesize high-dimensional longitudinal electronic health records via hierarchical autoregressive language model.  _Nature communications_ , 14(1): 5305.
  * Thoppilan et al. (2022) Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.-T.; Jin, A.; Bos, T.; Baker, L.; Du, Y.; et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * Towell and Shavlik (1994) Towell, G. G.; and Shavlik, J. W. 1994.  Knowledge-based artificial neural networks.  _Artificial intelligence_ , 70(1-2): 119–165.
  * Tulyakov et al. (2018) Tulyakov, S.; Liu, M.-Y.; Yang, X.; and Kautz, J. 2018.  Mocogan: Decomposing motion and content for video generation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1526–1535.
  * Xu et al. (2018) Xu, J.; Zhang, Z.; Friedman, T.; Liang, Y.; and Broeck, G. 2018.  A semantic loss function for deep learning with symbolic knowledge.  In _International conference on machine learning_ , 5502–5511. PMLR.
  * Yang, Srivastava, and Mandt (2022) Yang, R.; Srivastava, P.; and Mandt, S. 2022.  Diffusion probabilistic modeling for video generation.  _arXiv preprint arXiv:2203.09481_.
  * Zhang et al. (2021) Zhang, Z.; Yan, C.; Lasko, T. A.; Sun, J.; and Malin, B. A. 2021.  SynTEG: a framework for temporal structured electronic health data simulation.  _Journal of the American Medical Informatics Association_ , 28(3): 596–604.


[a]Andrzej Czarnecki

# Negative energy effects in processes involving bound particles: finite
nuclear size effects

###### Abstract

Probability of finding negative energy states in a hydrogen atom is decreased
when the nucleus is extended. Reduced electron Compton wavelength is the
characteristic size of the region around the nucleus where negative energy
contributions to the wave function are sizeable.

##  1 Introduction

The hydrogen atom consists most of the time of a nucleus and a single
electron. There is, however, a small probability that additional electron-
positron pairs appear. Positrons are represented by negative-energy solutions
of the Dirac equation. The probability of finding negative-energy components
of the wave function was first determined by Bethe [1]. In the ground state of
a hydrogen-like ion with the atomic number $Z$ that probability is

$P_{-}\left(Z\right)=\frac{8\alpha_{Z}^{5}}{15\pi},$ (1)

where $\alpha_{Z}=Z\alpha$ and $\alpha\simeq 1/137$ is the fine structure
constant. Eq. (1) applies in the case of a point-like nucleus. Here we
demonstrate that in the case of an extended charge distribution in the
nucleus, this probability $P_{-}$ decreases. This is interpreted as follows.
Negative energy states appear mainly in the region of a strong potential, in
analogy with the Klein paradox [2, 3, 4]. When the nucleus is extended, the
region of the strong potential is removed.

In the present work we focus on hydrogen-like ions containing spin-1/2
electrons. Ions with bound spin-0 particles have recently been considered in
[5], in the limit of a point-like nucleus. For an excellent review of a
variety of bound-state phenomena, see for example Ref. [6].

##  2 Models of an extended nucleus

###  2.1 Spherical shell

Instead of the whole charge $Ze$ concentrated in a point, we consider a
uniform surface charge distribution on a spherical shell of radius $R$. The
reason why we choose this particular model is that it is easy to treat
analytically. Besides, the details of the charge distribution are not
important for us. All we want to determine is how the probability of finding
negative energy states decreases in the absence of the region of a very strong
field. Inside a spherical shell there is no field at all and we can change the
field outside by changing the radius $R$ of the shell.

We use such units that $\hbar=c=\epsilon_{0}=1$. Then the electrostatic
potential energy of an electron interacting with the shell is ([7], Chapter
15)

$V_{\text{shell}}\left(r\right)=\begin{cases}-\frac{\alpha_{Z}}{R}&r<R,\\\
-\frac{\alpha_{Z}}{r}&r>R.\end{cases}$ (2)

This function has a Fourier transform that is easy to remember: it is similar
to the Fourier transform of the Coulomb potential,
$V_{\text{C}}\left(k\right)=-4\pi\alpha_{Z}/k^{2}$, but it is modulated by the
sinc function,

$V_{\text{shell}}\left(k\right)=V_{\text{C}}\left(k\right)\cdot\frac{\sin\kappa}{\kappa},\quad\kappa=kR.$
(3)

In the limit $R\to 0$ we reproduce the Coulomb potential of a pointlike
nucleus.

###  2.2 Uniformly charged ball

Although we will be using the spherical shell model, here we introduce a
uniformly charged sphere as an alternative charge distribution,

$\rho\left(r\right)=\begin{cases}\text{constant}&r<R,\\\ 0&r>R.\end{cases}$
(4)

Our goal is to demonstrate that the Fourier spectrum of the potential is
similar in both cases, strengthening the argument that the details of the
charge distribution are not decisive for our conclusions. Electron’s potential
energy resulting from the interaction with the density in Eq. (4) is

$V_{\text{ball}}\left(r\right)=\begin{cases}-\frac{\alpha_{Z}}{R}\left(\frac{5}{4}-\frac{r^{4}}{4R^{4}}\right)&r<R,\\\
-\frac{\alpha_{Z}}{r}&r>R.\end{cases}$ (5)

Its Fourier transform is

$V_{\text{ball}}\left(k\right)=5V_{\text{C}}\left(k\right)\cdot\frac{3\left(\kappa^{2}-2\right)\sin\kappa-\kappa\left(\kappa^{2}-6\right)\cos\kappa}{\kappa^{5}},\quad\kappa=kR.$
(6)

Fourier transforms of both the shell and the solid ball potentials, normalized
to the Coulomb potential, are shown in Figure 1.

Figure 1: Fourier transforms of electron’s potential energy due to its
interaction with two charge distributions: a solid ball (red, dashed) and a
hollow shell (blue, solid), normalized to the Coulomb potential of a point-
like distribution. For a given radius of the charge distribution, the two
Fourier transforms are similar. In both cases, the smearing of the charge
decreases the magnitude of high-momentum (large $kR$) Fourier components.

##  3 Negative energy states in the case of an extended nucleus

In order to find the probability $P_{-}\left(Z\right)$ of finding negative
energy states in the potential of Eq. (3), we need the negative energy
component of the wave function $\phi_{-}\left(\bm{k}\right)$. Once that is
determined, the needed probability is

$P_{-}\left(Z\right)=\int\frac{\mbox{d}^{3}k}{\left(2\pi\right)^{3}}\left|\phi_{-}\left(\bm{k}\right)\right|^{2}.$
(7)

That wave function component satisfies an integral equation, derived from the
Fourier transform of the Dirac equation [8]. That equation is greatly
simplified in the approximation where $\phi_{-}\left(\bm{k}\right)$ is small
and can be neglected under the integral [1, 5]. In the case of the hollow
shell potential in Eq. (3) one finds

$\phi_{-}\left(\bm{k}\right)=-\frac{4\pi\alpha_{Z}}{\left(m+E\right)k^{2}}\frac{\sin
kR}{kR}\sqrt{\frac{E-m}{2E}}\psi\left(0\right)$ (8)

where $E=\sqrt{m^{2}+k^{2}}$, $m$ is the electron mass, and the wave function
at the origin is approximately $\psi\left(0\right)\simeq 1/\sqrt{\pi a^{3}}$,
where $a=1/\left(m\alpha_{Z}\right)$ is the Bohr radius. The negative energy
probability can now be calculated,

$P_{-}\left(Z\right)=\frac{4\alpha_{Z}^{5}m^{3}}{\pi
R^{2}}\int_{0}^{\infty}\mbox{d}k\frac{\sin^{2}kR}{E\left(E+m\right)^{4}\left(E-m\right)}.$
(9)

Introduce a new variable $\epsilon$, $E=\epsilon m$,
$k\mbox{d}k=m^{2}\epsilon\mbox{d}\epsilon$, and denote $mR$ by $\mu$,

$P_{-}\left(Z\right)=\frac{4\alpha_{Z}^{5}}{\pi\mu^{2}}J,\qquad
J=\int_{1}^{\infty}\mbox{d}\epsilon\frac{\sin^{2}\left(\sqrt{\epsilon^{2}-1}\mu\right)}{\left(\epsilon+1\right)^{9/2}\left(\epsilon-1\right)^{3/2}},\qquad\mu=mR.$
(10)

When the radius $R$ of the charge distribution is large in comparison with the
reduced Compton wavelength of the electron $1/m$, $\mu$ is large, the sine
function in the numerator oscillates rapidly, and its square could in some
integrands be replaced by $1/2$. In our present case, this does not work
because in the region $\epsilon-1\ll 1$ the square root factor in the argument
of the sine compensates the large value of $\mu$. Thus we first add and
subtract a simpler function that has a similar behavior for $\epsilon$ near 1,

$J=\int_{1}^{\infty}\mbox{d}\epsilon\left\\{\frac{\sin^{2}\left(\sqrt{\epsilon^{2}-1}\mu\right)}{\left(\epsilon+1\right)^{9/2}\left(\epsilon-1\right)^{3/2}}-\frac{\sin^{2}\left[\sqrt{2}\sqrt{\epsilon-1}\mu\right]}{2^{9/2}\left(\epsilon-1\right)^{3/2}}+\frac{\sin^{2}\left[\sqrt{2}\sqrt{\epsilon-1}\mu\right]}{2^{9/2}\left(\epsilon-1\right)^{3/2}}\right\\}.$
(11)

In the first two terms we replace the sine-squared factors by 1/2; this
eliminates the dependence of these terms on $\mu$, they become subleading for
large $\mu$ and can be neglected. The last term, evaluated analytically, gives

$P_{-}\left(Z\right)\xrightarrow{\mu\gg 1}\frac{\alpha_{Z}^{5}}{4\mu}.$ (12)

We see that the probability of negative energy contributions decreases with
the radius of the charge distribution, confirming the intuition that these
contributions arise in the region of a strong potential.

Subleading terms in Eq. (11) introduce a small negative term. Including them
significantly increases the range of $\mu$ over which the large-$\mu$
asymptotics agrees with the exact $J$,

$P_{-}\left(Z\right)\xrightarrow{\mu\gg
1}\alpha_{Z}^{5}\left(\frac{1}{4\mu}-\frac{16}{35\pi\mu^{2}}\right).$ (13)

For completeness, we determine the behavior of $J$ for small $\mu$. The sine
function can be replaced by the first two terms of its Taylor expansion. We
retain only the square of the leading term and its product with the subleading
term and find

$P_{-}\left(Z\right)\xrightarrow{\mu\ll
1}\frac{4\alpha_{Z}^{5}}{45\pi}\left(6-5\mu^{2}\right).$ (14)

When $\mu\to 0$, this reproduces Bethe’s result [1]. Both the large and small
$R$ asymptotics of $P_{-}\left(Z\right)$ are plotted in Figure 2.

Figure 2: Probability of finding negative energy states in an hydrogen-like
ion with an extended nucleus: a shell of radius $R$. The solid blue curve
shows the numerical integral as in Eq. (10). For $R$ small in comparison with
the electron’s reduced Compton wavelength (red, dashed line defined by Eq.
(14)), the result approaches Bethe’s value of Eq. (1), with $8/15\simeq 0.53$.
For large $R$ (greed, dotted line defined by Eq. (12)), negative energy states
are suppressed because the potential is weak.

##  4 Conclusions

Finite size of the nucleus removes the region where the potential is very
strong. This decreases the probability of finding negative energy components
in the electron wave function. For large radii of the charge distribution the
decrease is linear in the ratio of the the electron’s reduced Compton
wavelength to the nuclear radius.

## Acknowledgement

I thank David Broadhurst for suggesting the point of view explored in this
work. This research was supported by Natural Sciences and Engineering Research
Canada (NSERC).

## References

  * [1] H. A. Bethe, _Bemerkungen über die Wasserstoff-Eigenfunktionen in der Diracschen Theorie_ , Zeitschrift für Naturforschung A 3, 470 – 477 (1948).
  * [2] O. Klein, _Die Reflexion von Elektronen an einem Potentialsprung nach der relativistischen Dynamik von Dirac_ , Zeitschrift für Physik 53, 157–165 (1929).
  * [3] B. R. Holstein, _Klein’s paradox_ , Am. J. Phys. 66, 507–512 (1998).
  * [4] A. Hansen and F. Ravndal, _Klein’s Paradox and Its Resolution_ , Physica Scripta 23, 1036–1042 (1981).
  * [5] A. Czarnecki, _Negative Energy States in Pionic Hydrogen_ , Acta Phys. Polon. Supp. 15, 1 (2022), 2202.03538.
  * [6] P. Hoyer, _Journey to the Bound States_ , SpringerBriefs in Physics, Springer (2021), 2101.06721.
  * [7] S. Chandrasekhar, _Newton’s Principia for the Common Reader_ , Clarendon Press (2003).
  * [8] H. Feshbach and F. Villars, _Elementary relativistic wave mechanics of spin 0 and spin 1/2 particles_ , Rev. Mod. Phys. 30, 24–45 (1958).