# GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level
Relation Extraction

Andrei C. Coman
Idiap Research Institute, EPFL
<EMAIL_ADDRESS>
&Christos Theodoropoulos
KU Leuven
<EMAIL_ADDRESS>Marie-Francine Moens
KU Leuven
<EMAIL_ADDRESS>&James Henderson
Idiap Research Institute
<EMAIL_ADDRESS>
###### Abstract

Document-level relation extraction aims to identify relationships between
entities within a document. Current methods rely on text-based encoders and
employ various hand-coded pooling heuristics to aggregate information from
entity mentions and associated contexts. In this paper, we replace these rigid
pooling functions with explicit graph relations by leveraging the intrinsic
graph processing capabilities of the Transformer model. We propose a joint
text-graph Transformer model, and a graph-assisted declarative pooling
(GADePo) specification of the input which provides explicit and high-level
instructions for information aggregation. This allows the pooling process to
be guided by domain-specific knowledge or desired outcomes but still learned
by the Transformer, leading to more flexible and customizable pooling
strategies. We extensively evaluate our method across diverse datasets and
models, and show that our approach yields promising results that are
comparable to those achieved by the hand-coded pooling functions.

##  1 Introduction

Document-level relation extraction, a critical task in natural language
processing, involves identifying and categorizing meaningful relationships
between entities within a document. This task is foundational to many
applications, including knowledge base population and completion Banko et al.
(2007), information retrieval Schütze et al. (2008), question answering Chen
et al. (2017), sentiment analysis Pang and Lee (2008), and more.

Standard methods that approach this challenge generally employ pretrained
text-based encoders Devlin et al. (2019); Beltagy et al. (2019); Zhuang et al.
(2021); Cui et al. (2021). These encoders are responsible for capturing the
nuances of information contained in the entity mentions and their contextual
surroundings. Previous successful methods often then use hand-coded pooling
heuristics, such as $logsumexp$ Jia et al. (2019) or directly exploit the
attention weights, to aggregate the information learned by the encoder Zhou et
al. (2021); Xiao et al. (2022); Tan et al. (2022a); Ma et al. (2023). But
hand-coded pooling functions lead to limited flexibility and customization.

In this paper, we challenge the reliance on rigid aggregation functions and
propose a novel approach, leveraging the power of Transformer models Vaswani
et al. (2017), to allow for a more flexible and adaptable information
aggregation using Transformer’s self-attention function. We introduce a joint
text-graph Transformer model, which capitalizes on the intrinsic graph
processing capabilities of the Transformer model Mohammadshahi and Henderson
(2020, 2021) by adding graph relations to new tokens which perform the desired
aggregation. This graph-assisted declarative pooling (GADePo) method uses
these graph relations to provide an explicit high-level specification for the
information aggregation process. Inputting these relations as features to the
self-attention function Mohammadshahi and Henderson (2020) enables the pooling
to be steered by domain-specific knowledge or desired outcomes but still
learned by the Transformer, opening up the possibility for a more tailored and
customized relation extraction process. Our method is versatile as it
facilitates multimodal learning, where the integration of unstructured (text)
and structured (graph) data is essential. By representing the cross-modal
interdependencies via a structure, our method could extend its applicability
beyond the specific task discussed in this work, such as information
extraction Theodoropoulos et al. (2021), question answering Feng et al.
(2022), and knowledge base population and completion Ji et al. (2020).

We extensively evaluate our method across diverse datasets and models commonly
employed in document-level relation extraction tasks, and show that our
approach yields promising results that are comparable to those achieved by the
hand-coded pooling functions. In summary, the main contributions of this paper
are:

  * •

We propose a novel method for using pretrained Transformer models that moves
beyond the constraints of hand-coded aggregation functions, offering enhanced
flexibility and adaptability.

  * •

We introduce a novel joint text-graph Transformer model, bringing together the
capabilities of graph processing and text-based encoders.

  * •

We extensively evaluate our proposed approach across various datasets and
models, providing a robust assessment of its performance, and show that it
yields promising results that are comparable to those achieved by the hand-
coded pooling functions.

##  2 Related Work

Figure 1: The comparison between ATLOP (left) and GADePo (right), illustrating
a document containing two entities, each with two mentions. In ATLOP, the
textual input annotated with mention boundaries $*\text{m}*$ is encoded via a
Transformer model. The output of this model is then used to aggregate the
mentions and the surrounding context according to the Entity Embedding (EE)
Equation 2 and Localized Context Embedding (LCE) Equation 4, respectively. EE
computes $\bm{h}_{e}$ by aggregating mentions relative to a single entity,
while LCE computes $\bm{c}^{(s,o)}$ by using attention weights to identify
context elements that are relevant to a pair of subject and object entities.
In GADePo, the textual input is extended to include the graph special tokens
<ent> (i.e., entity) and <pent> (i.e., pair entity), and explicit directional
graph relations of types $\texttt{<ent>}\longleftrightarrow*\text{m}*$ and
$\texttt{<pent>}\longleftrightarrow*\text{m}*$ are used in place of the EE and
LCE equations, respectively. A joint text-graph Transformer model is then used
to encode the graph-assisted declarative pooling specification of the input.

In recent studies, the scope of relation extraction has been expanded to
encompass not only individual sentences but entire documents. This extension,
known as document-level relation extraction, presents a more realistic and
challenging scenario compared to the sentence-level counterpart as it seeks to
extract relations both within sentences and across multiple sentences Yao et
al. (2019).
Transformer-based Vaswani et al. (2017) models have shown great potential in
addressing this task. Wang et al. (2019) and Tang et al. (2020) show that the
BiLSTM-based Hochreiter and Schmidhuber (1997) baselines lack the capacity to
model complex interactions between multiple entities. Wang et al. (2019)
propose a more robust approach, which consists of using the BERT Devlin et al.
(2019) model and a two-step prediction process, i.e, first identifying if a
link between two entities exists, followed by predicting the specific relation
type. CorefBERT Ye et al. (2020) improves over BERT by leveraging a novel task
specifically designed to enhance its coreferential reasoning ability. GAIN
Zeng et al. (2020) leverages BERT as a text encoder and GCNs Kipf and Welling
(2017) to process two types of graphs, one at mention-level and the another
one at entity level. GAIN has been shown to be very effective in inter-
sentence and inferential scenarios. Mohammadshahi and Henderson (2020, 2021)
show how to leverage the intrinsic graph processing capabilities of the
Transformer model and incorporate structural dependencies between input
elements. Xu et al. (2021) leverage this idea in the scenario of document-
level relation extraction, and consider the structure of entities. Their
proposed SSAN model employs a transformation module that creates attentive
biases from this structure to regulate the attention flow during the encoding
phase. DocuNet Zhang et al. (2021) presents a new approach to document-level
relation extraction. It uses an entity-level relation matrix to capture both
local and global information among relational triples, contrasting with prior
models that considered entities independently. The system also incorporates a
U-shaped segmentation module and an encoder module to capture global
interdependencies and contextual information of entities respectively. SAIS
Xiao et al. (2022) explicitly models key information sources such as relevant
contexts and entity types. It improves extraction quality and
interpretability, while also boosting performance through evidence-based data
augmentation and ensemble inference. KD-DocRE Tan et al. (2022a) proposes a
semi-supervised document relation extraction framework with three key
components. Firstly, an axial attention module enhances performance in
handling two-hop relations by capturing the interdependence of entity pairs.
Secondly, an adaptive focal loss solution addresses the class imbalance issue
in document-level relation extraction tasks. Lastly, the framework employs
knowledge distillation to improve robustness and overall effectiveness by
bridging the gap between human-annotated and distantly supervised data. DREEAM
Ma et al. (2023) is a method designed to enhance document-level relation
extraction by addressing memory efficiency and annotation limitations in
evidence retrieval. It employs evidence as a supervisory signal to guide
attention and introduces a self-training strategy to learn evidence retrieval
without requiring evidence annotations.
SAIS Xiao et al. (2022), KD-DocRE Tan et al. (2022a), and DREEAM Ma et al.
(2023) draw direct inspiration from ATLOP Zhou et al. (2021) in developing
their respective methods. ATLOP introduces two innovative techniques, adaptive
thresholding, and localized context pooling, to address challenges in multi-
label and multi-entity problems. Adaptive thresholding employs a learnable
entities-dependent threshold, replacing the global threshold used in previous
approaches for multi-label classification Peng et al. (2017); Christopoulou et
al. (2019); Nan et al. (2020); Wang et al. (2020). Localized context pooling
leverages attention from pretrained language models to identify and extract
relevant context crucial for determining the relation between entities.

We recognize the significant influence of ATLOP, and adopt it as the
foundation for comparisons with our proposed method. However, we challenge the
conventional reliance on rigid aggregation functions presented in ATLOP and
propose a novel approach that enables more flexible and adaptable information
aggregation.

##  3 Preliminaries

###  3.1 Problem Formulation

The document-level relation extraction task involves analyzing a document $D$
that contains a set of entities
$\mathcal{E}_{D}{=}\\{e_{i}\\}_{i=1}^{|\mathcal{E}_{D}|}$. The main objective
is to determine the presence or absence of various relation types between all
entity pairs $(e_{s},e_{o})_{s,o\in\mathcal{E}_{D},s\neq o}$, where the
subject and object entities are denoted as $e_{s}$ and $e_{o}$, respectively.
A key aspect to consider is that an entity can appear multiple times in the
document, resulting in a cluster of multiple mentions
$\mathcal{M}_{e}{=}\\{m_{i}\\}_{i=1}^{|\mathcal{M}_{e}|}$ for each entity $e$.
The set of relations is defined as $\mathcal{R}\cup{\emptyset}$, where
$\emptyset$ represents the absence of a relation, often referred to as "no-
relation". The task consists of a multi-label classification problem where
there can be multiple relations between entities ${e_{s}}$ and ${e_{o}}$.

###  3.2 Previous Method: ATLOP

As stated in Section 2, work such as SAIS Xiao et al. (2022), KD-DocRE Tan et
al. (2022a), and DREEAM Ma et al. (2023) draw direct inspiration from ATLOP
Zhou et al. (2021). We therefore use ATLOP as the basis for comparisons with
our proposed method. The left portion of Figure 1 illustrates the components
of the ATLOP model, which we present in detail in this subsection.

#### Text Encoding

A special token $*$ is added at the start and end of every mention. Tokens
$\mathcal{T}_{D}{=}\\{t_{i}\\}_{i=1}^{|\mathcal{T}_{D}|}$ are encoded via a
Pretrained Language Model (PLM) as follows:

$\bm{H},\bm{A}=PLM(\mathcal{T}_{D}),$ (1)

where $\bm{H}\in\mathbb{R}^{|\mathcal{T}_{D}|\times d}$ and
$\bm{A}\in\mathbb{R}^{|\mathcal{T}_{D}|\times|\mathcal{T}_{D}|}$ represent the
token embeddings and the average attention weights of all attention heads from
the last layer, respectively.

#### Entity Embedding (EE)

For each individual entity $e$ with mentions
$\mathcal{M}_{e}{=}\\{m_{i}\\}_{i=1}^{|\mathcal{M}_{e}|}$, an entity embedding
$\bm{h}_{e}\in\mathbb{R}^{d}$ is computed as follows:

$\bm{h}_{e}=log\sum_{i=1}^{|\mathcal{M}_{e}|}exp(\bm{H}_{m_{i}}),$ (2)

where $\bm{H}_{m_{i}}\in\mathbb{R}^{d}$ is the embedding of the special token
$*$ at the starting position of mention $m_{i}$. The $logsumexp$ pooling
function has been empirically shown in Jia et al. (2019) to be more effective
than $max$ pooling, as it facilitates accumulating weak signals from
individual mentions, thanks to its smoother characteristics.

#### Localized Context Embedding (LCE)

ATLOP introduces the concept of localized context embeddings to accommodate
the variations in relevant mentions and context for different entity pairs
$(e_{s},e_{o})$. Since the attention mechanism in pretrained language models
captures the importance of each token within the context, it can be utilized
to determine the context relevant for both entities. The importance of each
token can be computed from the cross-token dependencies matrix $\bm{A}$
obtained in Equation 1. When evaluating entity $e_{s}$, the importance of
individual tokens is determined by examining the cross-token dependencies
across all mentions associated with $e_{s}$, denoted as $\mathcal{M}_{e_{s}}$.
Initially, ATLOP collects and averages the attention
$\bm{A}_{m_{i}}\in\mathbb{R}^{|\mathcal{T}_{D}|}$ at the special token $*$
preceding each mention $m_{i}\in\mathcal{M}_{e_{s}}$. This process results in
$\bm{a}_{s}\in\mathbb{R}^{|\mathcal{T}_{D}|}$, which represents the importance
of each token concerning entity $e_{s}$ (and $\bm{a}_{o}$ for $e_{o}$).
Subsequently, the importance of each token for a given entity pair
$(e_{s},e_{o})$, denoted as $\bm{q}^{(s,o)}\in\mathbb{R}^{|\mathcal{T}_{D}|}$,
is computed using $\bm{a}_{s}$ and $\bm{a}_{o}$ as follows:

$\bm{q}^{(s,o)}=\frac{\bm{a}_{s}\circ\bm{a}_{o}}{\bm{a}_{s}^{\top}\bm{a}_{o}},$
(3)

where $\circ$ represents the Hadamard product. Consequently, $\bm{q}^{(s,o)}$
represents a distribution that indicates the importance of each token for
$(e_{s},e_{o})$. Finally, the localized context embedding is computed as
follows:

$\bm{c}^{(s,o)}=\bm{H}^{\top}\bm{q}^{(s,o)},$ (4)

where $\bm{c}^{(s,o)}\in\mathbb{R}^{d}$ corresponds to a weighted average over
all token embeddings.

#### Relation Classification

To predict the relation between the subject entity $e_{s}$ and object entity
$e_{o}$, ATLOP first generates context-aware subject and object
representations as follows:

$\bm{z}_{s}=tanh(\bm{W}_{s}[\bm{h}_{e_{s}};\bm{c}^{(s,o)}]+\bm{b}_{s})$ (5)
$\bm{z}_{o}=tanh(\bm{W}_{o}[\bm{h}_{e_{o}};\bm{c}^{(s,o)}]+\bm{b}_{o}),$ (6)

where $\bm{z}_{s},\bm{z}_{o}\in\mathbb{R}^{d}$, $[\cdot;\cdot]$ represents the
concatenation of two vectors, and $\bm{W}_{s},\bm{W}_{o}\in\mathbb{R}^{d\times
2d}$ together with $\bm{b}_{s},\bm{b}_{o}\in\mathbb{R}^{d}$ are trainable
parameters. Then, the entity pair representation is computed as follows:

$\bm{x}^{(s,o)}=\bm{z}_{s}\otimes\bm{z}_{o},$ (7)

where $\bm{x}^{(s,o)}\in\mathbb{R}^{d^{2}}$ and $\otimes$ stands for the
Kronecker product. Finally, relation scores are computed as:

$\bm{y}^{(s,o)}=\bm{W}_{r}\bm{x}^{(s,o)}+\bm{b}_{r},$ (8)

where $\bm{y}^{(s,o)}\in\mathbb{R}^{|\mathcal{R}|}$, with
$\bm{W}_{r}\in\mathbb{R}^{|\mathcal{R}|\times d^{2}}$ and
$\bm{b}_{r}\in\mathbb{R}^{|\mathcal{R}|}$ representing learnable parameters.
The probability of relation $r\in\mathcal{R}$ between the subject and object
entities is computed as follows:

$P(r|s,o)=\sigma(\bm{y}^{(s,o)}),$ (9)

where $\sigma$ is the sigmoid function. To reduce the number of parameters in
the classifier, a grouped function is used, which splits the embedding
dimensions into $k$ equal-sized groups and applies the function within the
groups as follows:

$\bm{z}_{s}=[\bm{z}_{s}^{1};\ldots;\bm{z}_{s}^{k}]$ (10)
$\bm{z}_{o}=[\bm{z}_{o}^{1};\ldots;\bm{z}_{o}^{k}]$ (11)
$\bm{x}^{(s,o)}=[\bm{x}^{(s,o)^{1}};\ldots;\bm{x}^{(s,o)^{k}}]$ (12)
$\bm{y}^{(s,o)}=\sum_{i=1}^{k}\bm{W}_{r}^{i}\bm{x}^{(s,o)^{i}}+\bm{b}_{r},$
(13)

where $\bm{z}_{s}^{i},\bm{z}_{o}^{i}\in\mathbb{R}^{d/k}$,
$\bm{x}^{(s,o)^{i}}\in\mathbb{R}^{d^{2}/k}$, and
$\bm{W}_{r}^{i}\in\mathbb{R}^{|\mathcal{R}|\times d^{2}/k}$. This way, the
number of parameters can be reduced from $d^{2}$ to $d^{2}/k$.

#### Loss Function

ATLOP introduces the adaptive thresholding loss concept. This approach
involves training a model to learn a hypothetical threshold class $T\\!H$,
which dynamically adjusts for each relation class $r\in\mathcal{R}$. During
training, for each entity pair $(e_{s},e_{o})$, the loss enforces the model to
generate scores above $T\\!H$ for positive relation classes $\mathcal{R}_{P}$
and scores below $T\\!H$ for negative relation classes $\mathcal{R}_{N}$. The
loss is computed as follows:

$\displaystyle\mathcal{L}=$ $\displaystyle-\sum_{s\neq
o}\sum_{r\in\mathcal{R}_{P}}\frac{exp(y_{r}^{(s,o)})}{\sum_{r^{{}^{\prime}}\in\mathcal{R}_{P}\cup\\{T\\!H\\}}exp(y_{r^{\prime}}^{(s,o)})}$
(14)
$\displaystyle-\frac{exp(y_{T\\!H}^{(s,o)})}{\sum_{r^{{}^{\prime}}\in\mathcal{R}_{N}\cup\\{T\\!H\\}}exp(y_{r^{\prime}}^{(s,o)})}$

##  4 Proposed Method: GADePo

Our proposed graph-assisted declarative pooling (GADePo) method replaces the
hand-coded aggregation functions EE and LCE with a declarative graph
specification. By using the intrinsic graph processing capabilities of the
Transformer model, the specified graph serves as an explicit high-level
directive for the information aggregation process of the Transformer. By
inputting the graph relations to the Transformer’s self-attention layers,
GADePo enables the aggregation to be steered by domain-specific knowledge or
desired outcomes, while still allowing it to be learned by the Transformer,
opening up the possibility for a more tailored and customized yet data-driven
relation extraction.

Our method facilitates multimodal learning by representing the cross-modal
interdependencies via a structure, making it a versatile approach that can be
applied to various tasks and domains beyond the one addressed in this work.

As described in Subsection 3.2, ATLOP leverages two hand-coded pooling
functions: the Entity Embedding (EE) and the Localized Context Embedding
(LCE). Both functions try to capture distinct aspects of the document and are
applied to the output of the Pretrained Language Model (PLM).

Our GADePo model is illustrated in the right portion of Figure 1. We address
both EE and LCE with the introduction of two special tokens, <ent> (i.e.,
entity) and <pent> (i.e., pair entity), and two explicit graph relations of
types $\texttt{<ent>}\longleftrightarrow*$ and
$\texttt{<pent>}\longleftrightarrow*$ in both directions, where $*$ represents
the special token at the starting position of a specific mention. The set of
relations is specified as $c_{ij}\in\mathcal{C}$ which each identify the
relation label from $i$ to $j$. Each of these relation labels is associated
with an embedding vector of dimension $d$, as are the special token inputs
<ent> and <pent>. These two special tokens are added to the PLM’s vocabulary,
while relations are input to the self-attention functions at every layer.
These new embeddings represent learnable parameters that are trained during
the PLM fine-tuning on the downstream tasks. As reported in Table 4, GADePo
adds a negligible number of extra parameters, accounting solely for the
inclusion of the special tokens and the graph directional relations.

#### Special Token <ent>

To tackle the EE pooling function, we add to the input tokens
$\mathcal{T}_{D}$ as many <ent> special tokens as entities in the document.
This way each entity $e$ has a corresponding entity token <ent> in the input.
We connect each <ent> token with its corresponding cluster of mentions
$\mathcal{M}_{e}{=}\\{m_{i}\\}_{i=1}^{|\mathcal{M}_{e}|}$, and vice-versa. The
two graph relations we use are thus $\texttt{<ent>}\longrightarrow*$ and
$*\longrightarrow\texttt{<ent>}$, where $*$ represents the special token at
the starting position of mention $m_{i}$. The <ent> token can thus be seen as
having a direct correspondence to the computed $\bm{h}_{e}$ in Equation 2.

#### Special Token <pent>

ATLOP performs information filtering by calculating via Equation 4 a localized
context embedding (LCE) $\bm{c}^{(s,o)}$ that is dependent on the cross-token
dependencies matrix $\bm{A}$ output by the PLM. The intuition behind is that
the dependencies between different tokens are encoded as attention weights. We
propose a straightforward adjustment of the input graph used for the EE
pooling to effectively model and capture these dependencies. To address the
LCE pooling function we add to the input tokens $\mathcal{T}_{D}$ as many
<pent> special tokens as the number of all possible pairs of entities. The
special token <pent> thus refers to a pair of entities $(e_{s},e_{o})$. We
connect each <pent> token with each mention in the two clusters of mentions
$\mathcal{M}_{e_{s}}{=}\\{m_{i}\\}_{i=1}^{|\mathcal{M}_{e_{s}}|}$ and
$\mathcal{M}_{e_{o}}{=}\\{m_{i}\\}_{i=1}^{|\mathcal{M}_{e_{o}}|}$ and vice-
versa. Since the attention weights used in LCE are computed from these mention
embeddings, we expect that they are sufficient for the Transformer to learn to
find the relevant contexts. The two graph relations we use are thus
$\texttt{<pent>}\longrightarrow*$ and $*\longrightarrow\texttt{<pent>}$. The
<pent> token can be seen as having a direct correspondence to the computed
$\bm{c}^{(s,o)}$ in Equation 4.

All equations relative to the relation classification reported in Subsection
3.2 remain valid as we merely substitute the hand-coded computations of
$\bm{h}_{e}$ and $\bm{c}^{(s,o)}$ with the embedding of <ent> and <pent>,
respectively.

#### Text-Graph Encoding

The previous work of Mohammadshahi and Henderson (2020, 2021) shows how to
leverage the intrinsic graph processing capabilities of the Transformer model
by incorporating graph relations between input tokens as relation embeddings
input to the self-attention function. We use the same idea but modify their
attention weight computation to:

$e_{ij}=\frac{\bm{x}_{i}\bm{W}_{Q}~{}\operatorname*{diag}(\operatorname*{LN}(\bm{c}_{ij}\bm{W}_{C}))~{}(\bm{x}_{j}\bm{W}_{K})^{\top}}{\sqrt{d}},$
(15)

where $\bm{x}_{i},\bm{x}_{j}\in\mathbb{R}^{d}$ represent the token embeddings
on which the attention weight $e_{ij}$ must be computed.
$\bm{W}_{Q},\bm{W}_{K}\in\mathbb{R}^{d\times d}$ represent the query and key
matrices, respectively. $\bm{c}_{ij}\in\\{0,1\\}^{|\mathcal{C}|}$ represents a
one-hot encoded vector of graph relation $c\in\mathcal{C}$ between the $i$-th
and $j$-th input elements. $\bm{W}_{C}\in\mathbb{R}^{|\mathcal{C}|\times d}$
represents the relations embedding matrix. Finally, $\operatorname*{LN}$
stands for the $LayerNorm$ operation and $\operatorname*{diag}$ returns a
diagonal matrix. Compared to the standard attention function, where
$e_{ij}=\bm{x}_{i}\bm{W}_{Q}(\bm{x}_{j}\bm{W}_{K})^{\top}/\sqrt{d}$, the
relation embedding determines a weighting of the different dimensions.

##  5 Experiments

| DocRED
---|---
Model | Dev | Test
| Ign $F_{1}$ | $F_{1}$ | Ign $F_{1}$ | $F_{1}$
ATLOP Zhou et al. (2021) | $61.32\pm 0.14$ | $63.18\pm 0.19$ | $61.39$ | $63.40$
ATLOP (ours) | $61.41\pm 0.20$ | $63.38\pm 0.19$ | $61.61$ | $63.79$
GADePo (ours) | $61.48\pm 0.35$ | $63.51\pm 0.36$ | $61.29$ | $63.54$
ATLOP w/o LCE Zhou et al. (2021) | $-$ | $-$ | $-$ | $-$
ATLOP w/o LCE (ours) | $59.62\pm 0.17$ | $61.56\pm 0.17$ | $59.63$ | $61.74$
GADePo w/o <pent> (ours) | $59.20\pm 0.29$ | $61.32\pm 0.27$ | $59.74$ | $62.00$
| Re-DocRED
Model | Dev | Test
| Ign $F_{1}$ | $F_{1}$ | Ign $F_{1}$ | $F_{1}$
ATLOP Tan et al. (2022b) | $76.79$ | $77.46$ | $76.82$ | $77.56$
ATLOP (ours) | $77.77\pm 0.22$ | $78.48\pm 0.22$ | $78.42$ | $79.10$
GADePo (ours) | $77.28\pm 0.17$ | $78.01\pm 0.21$ | $77.75$ | $78.51$
ATLOP w/o LCE Tan et al. (2022b) | $-$ | $-$ | $-$ | $-$
ATLOP w/o LCE (ours) | $75.37\pm 0.25$ | $76.12\pm 0.27$ | $75.46$ | $76.18$
GADePo w/o <pent> (ours) | $75.51\pm 0.34$ | $76.36\pm 0.31$ | $74.93$ | $75.73$
Table 1: Results in percentage for the development and test sets of DocRED and
Re-DocRED. We report the mean and standard deviation of Ign $F_{1}$ and
$F_{1}$ on the development set, calculated from five training runs with
distinct random seeds. We provide the official test score achieved by the best
checkpoint on the development set. Ign $F_{1}$ refers to the $F_{1}$ score
that excludes relational facts shared between the training and
development/test sets.

(a) DocRED

(b) Re-DocRED

(c) HacRED

Figure 2: Performance evaluation of ATLOP and GADePo on the development set under varying data availability conditions on DocRED, Re-DocRED, and HacRED. The $x$-axis represents the percentage of the dataset used for training, while the $y$-axis displays the $F_{1}$ score in percentage. Each point on the graph represents the mean value, while error bars indicate the standard deviation derived from five distinct training runs with separate random seeds. | HacRED
---|---
Model | Dev | Test
| $P$ | $R$ | $F_{1}$ | $P$ | $R$ | $F_{1}$
ATLOP Cheng et al. (2021) | $-$ | $-$ | $-$ | $77.89$ | $76.55$ | $77.21$
ATLOP (ours) | $77.34\pm 0.41$ | $77.13\pm 0.51$ | $77.23\pm 0.37$ | $77.40$ | $77.04$ | $77.22$
GADePo (ours) | $75.96\pm 0.49$ | $79.88\pm 0.87$ | $77.99\pm 0.46$ | $76.40$ | $80.14$ | $78.23$
ATLOP Cheng et al. (2021) | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
ATLOP w/o LCE (ours) | $77.34\pm 1.19$ | $76.65\pm 0.73$ | $76.16\pm 0.48$ | $76.43$ | $75.11$ | $75.76$
GADePo w/o <pent> (ours) | $72.72\pm 0.98$ | $79.86\pm 1.17$ | $75.33\pm 1.04$ | $72.91$ | $80.77$ | $76.64$
Table 2: Results in percentage for the development and test sets of HacRED. We
report the mean and standard deviation of Precision ($P$), Recall ($R$) and
$F_{1}$ on the development set, calculated from five training runs with
distinct random seeds. We provide the official test score achieved by the best
checkpoint on the development set.

###  5.1 Datasets and Models

We extensively evaluate our GADePo model across diverse datasets and models.
We report the statistics of the datasets used in this work in Table 3.

#### DocRED

Yao et al. (2019) is built from Wikipedia and Wikidata. It contains human-
annotated documents with named entities, coreference data, and intra- and
inter-sentence relations, supported by evidence. DocRED involves analyzing
multiple sentences to identify entities and establish their relationships,
integrating the whole document’s information. In our experiments, all models
are based on RoBERTalarge Zhuang et al. (2021).

#### Re-DocRED

Tan et al. (2022b) point out that DocRED’s annotations are often incomplete,
leading to a high number of false negatives. After investigating these
inconsistencies, overlooked relation triples were reintegrated through a re-
annotation of more than 4000 documents in the original DocRED. In our
experiments, all models are based on RoBERTalarge Zhuang et al. (2021).

#### HacRED

Cheng et al. (2021) is a large-scale, high-quality Chinese document-level
relation extraction dataset. Its construction involved a comprehensive
analysis of the performance gap between popular datasets and practical
applications, leading to a case-oriented construction framework focusing on
challenging practical cases. Given that Cheng et al. (2021) do not offer
details on the model used in their study, our experiments are based on the
Chinese BERTbase with whole word masking Cui et al. (2021)
model111https://huggingface.co/hfl/chinese-bert-wwm-ext.

###  5.2 Results and Discussion

The results of our experiments on DocRED, Re-DocRED, and HacRED datasets are
reported in Table 1, Figure 2, and Table 2.

We evaluate all three datasets using the $F_{1}$ metric. For DocRED and Re-
DocRED Ign $F_{1}$ (or Ignored $F_{1}$) is also reported, and refers to the
$F_{1}$ score that excludes relational facts that are shared between the
training and development/test sets. This is done to avoid potential biases in
the evaluation metrics due to overlap in content between the sets, which might
not reflect the model’s ability to generalize to truly unseen data. For
HacRED, we comply with the format introduced by Cheng et al. (2021) and report
also the Precision ($P$) and Recall ($R$) metrics.

For all datasets we report the mean and standard deviation on the development
set, calculated from five training runs with distinct random seeds. We provide
the official test score achieved by the best checkpoint on the development
set. Training details and hyperparameters are reported in Subsection A.3.

For all datasets and models, we provide our implementation of the ATLOP
baseline that achieves or surpasses previously reported results and compare
the proposed GADePo model against it.

#### DocRED Results

In Table 1, GADePo achieves the highest development Ign $F_{1}$ and $F_{1}$
scores, showing significant improvements over the other models, while on the
test set it falls shortly behind ATLOP. Interestingly, the performance drops
for all models when the LCE or <pent> features are removed from ATLOP and
GADePo, respectively, highlighting the importance of these components in the
models’ effectiveness. Notably, when we compare ATLOP without LCE and GADePo
without <pent>, the results are inverted, with ATLOP achieving the highest
development scores, and GADePo performing better on the test set.

#### Re-DocRED Results

In Table 1, our implementation of ATLOP outperforms the other models in both
development and test sets. It performs better than the original ATLOP
implementation by Tan et al. (2022b) as well, suggesting that a better
hyperparameter search can lead to better performance. The GADePo model also
shows robust results, although slightly trailing behind our ATLOP variant.
Again, removing the LCE for ATLOP and <pent> for GADePo, led to a decline in
performance across all models.

#### HacRED Results

In Table 2, the ATLOP model, as presented by Cheng et al. (2021), demonstrates
a strong performance with an $F_{1}$ score of $77.21\%$ on the test set. Our
version of the ATLOP model closely matches the previously reported result. The
proposed GADePo model shows a slightly improved performance with an $F_{1}$
score of $78.23\%$ on the test set, which was higher than both the original
and our own versions of the ATLOP model. This improvement seems to be
primarily driven by a substantial increase in Recall ($R$), indicating that
the GADePo model is more effective at identifying relevant instances. As
already reported for the DocRED and Re-DocRED datasets, the drops in
performance when the LCE and <pent> components are removed from ATLOP and
GADePo, respectively, highlight the significant contributions of these
features.

#### Data Ablation

The performance evaluation depicted in Figure 2 compares ATLOP and GADePo on
the development set, considering different levels of data availability across
all datasets. ATLOP appears to have relatively consistent variance, while
GADePo exhibits higher variance in the initial data sizes for the DocRED and
Re-DocRED datasets while being significantly more stable on the HacRED
dataset. Across all datasets, GADePo generally catches up with or outperforms
ATLOP as data size increases. The substantial improvement observed in HacRED
could potentially be attributed to the fact that it contains more than 4000
and 5000 documents compared to the DocRED and Re-DocRED datasets,
respectively. This could suggest that GADePo has a higher potential for
optimization or is better at exploiting larger datasets.

#### General Remarks

Both ATLOP and GADePo have been trained in the same hyperparameters regime as
reported in A.3. However, the ATLOP model is exclusively focused on the
relation extraction task. It is worth noting that besides learning to perform
well on the relation extraction, GADePo must also learn how exploit the graph
relations to the special tokens <ent> and <pent> and pool information through
them. Remarkably, GADePo manages to effectively learn the extra tasks and
compete with the ATLOP counterpart.

##  6 Conclusion

In this study, we proposed an alternative approach to document-level relation
extraction, challenging the conventional reliance on rigid pooling functions
in information aggregation. Our method leverages the power of Transformer
models by incorporating explicit graph relations as instructions for
information aggregation. By combining graph processing with text-based
encoding, we introduced the graph-assisted declarative pooling (GADePo)
specification, which allows for more flexible and customizable pooling
strategies.

We conducted extensive evaluations using diverse datasets and models commonly
employed in document-level relation extraction tasks. The results of our
experiments demonstrated that our approach achieves promising performance that
is comparable to that of hand-coded pooling functions. This suggests that our
method can serve as a viable alternative for relation extraction, providing a
more adaptable and tailored approach.

Our study opens up several directions for future research, such as exploring
different graph representations and their impact on pooling strategies.
Moreover, given that via the GADePo specification a Transformer model can
effectively encode both unstructured text and structured graph information, it
holds great potential for various applications beyond relation extraction. One
such application is entity linking and disambiguation, where the model can
utilize external knowledge bases to link entities mentioned in the text to
their corresponding entities in the knowledge base. Additionally, tasks
involving graph and node-level predictions in conjunction with text could
benefit from the GADePo’s multimodal learning capabilities. For instance,
question answering systems that operate over knowledge bases could employ the
joint text-graph Transformer model to better understand complex queries and
retrieve relevant information. Further research is needed to fully uncover and
leverage the potential of GADePo.

## Limitations

The proposed GADePo model, like the Transformer, is constrained by its
quadratic computational complexity relative to input length. The inclusion of
the <ent> and <pent> special tokens further exacerbates this issue. As a
consequence, the method may encounter significant challenges when processing
inputs of larger sizes.

## References

  * Banko et al. (2007) Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007.  Open information extraction from the web.  In _CACM_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Cheng et al. (2021) Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021.  HacRED: A large-scale relation extraction dataset toward hard cases in practical applications.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2819–2831, Online. Association for Computational Linguistics.
  * Christopoulou et al. (2019) Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019.  Connecting the dots: Document-level neural relation extraction with edge-oriented graphs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4925–4936, Hong Kong, China. Association for Computational Linguistics.
  * Cui et al. (2021) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021.  Pre-training with whole word masking for chinese bert.  _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , 29:3504–3514.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Falcon and The PyTorch Lightning team (2019) William Falcon and The PyTorch Lightning team. 2019.  PyTorch Lightning.
  * Feng et al. (2022) Yue Feng, Zhen Han, Mingming Sun, and Ping Li. 2022.  Multi-hop open-domain question answering over structured and unstructured knowledge.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 151–156, Seattle, United States. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9:1735–1780.
  * Ji et al. (2020) Shaoxiong Ji, Shirui Pan, E. Cambria, Pekka Marttinen, and Philip S. Yu. 2020.  A survey on knowledge graphs: Representation, acquisition, and applications.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33:494–514.
  * Jia et al. (2019) Robin Jia, Cliff Wong, and Hoifung Poon. 2019.  Document-level n-ary relation extraction with multiscale representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_.
  * Liu et al. (2020) Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020.  On the variance of the adaptive learning rate and beyond.  In _International Conference on Learning Representations_.
  * Ma et al. (2023) Youmi Ma, An Wang, and Naoaki Okazaki. 2023.  DREEAM: Guiding attention with evidence for improving document-level relation extraction.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1971–1983, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Mohammadshahi and Henderson (2020) Alireza Mohammadshahi and James Henderson. 2020.  Graph-to-graph transformer for transition-based dependency parsing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3278–3289, Online. Association for Computational Linguistics.
  * Mohammadshahi and Henderson (2021) Alireza Mohammadshahi and James Henderson. 2021.  Recursive non-autoregressive graph-to-graph transformer for dependency parsing with iterative refinement.  _Transactions of the Association for Computational Linguistics_ , 9:120–138.
  * Nan et al. (2020) Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020.  Reasoning with latent structure refinement for document-level relation extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1546–1557, Online. Association for Computational Linguistics.
  * Pang and Lee (2008) Bo Pang and Lillian Lee. 2008.  Opinion mining and sentiment analysis.  _Found. Trends Inf. Retr._ , 2:1–135.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017\.  Cross-sentence n-ary relation extraction with graph LSTMs.  _Transactions of the Association for Computational Linguistics_ , 5:101–115.
  * Schütze et al. (2008) Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_ , volume 39.  Cambridge University Press Cambridge.
  * Tan et al. (2022a) Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a.  Document-level relation extraction with adaptive focal loss and knowledge distillation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1672–1681, Dublin, Ireland. Association for Computational Linguistics.
  * Tan et al. (2022b) Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b.  Revisiting DocRED - addressing the false negative problem in relation extraction.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8472–8487, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Tang et al. (2020) Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020.  Hin: Hierarchical inference network for document-level relation extraction.  In _Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Singapore, May 11–14, 2020, Proceedings, Part I 24_ , pages 197–209. Springer.
  * Theodoropoulos et al. (2021) Christos Theodoropoulos, James Henderson, Andrei Catalin Coman, and Marie-Francine Moens. 2021.  Imposing relation structure in language-model embeddings using contrastive learning.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 337–348, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_.
  * Wang et al. (2020) Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020.  Global-to-local neural networks for document-level relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3711–3721, Online. Association for Computational Linguistics.
  * Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Yang Wang. 2019.  Fine-tune bert for docred with two-step process.  _ArXiv_ , abs/1909.11898.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xiao et al. (2022) Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang, and Jiawei Han. 2022.  SAIS: Supervising and augmenting intermediate steps for document-level relation extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2395–2409, Seattle, United States. Association for Computational Linguistics.
  * Xu et al. (2021) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021.  Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14149–14157.
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  DocRED: A large-scale document-level relation extraction dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 764–777, Florence, Italy. Association for Computational Linguistics.
  * Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020.  Coreferential Reasoning Learning for Language Representation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7170–7186, Online. Association for Computational Linguistics.
  * Zeng et al. (2020) Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020.  Double graph based reasoning for document-level relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1630–1640, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021.  Document-level relation extraction as semantic segmentation.  In _International Joint Conference on Artificial Intelligence_.
  * Zhou et al. (2021) Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021.  Document-level relation extraction with adaptive thresholding and localized context pooling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zhuang et al. (2021) Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021.  A robustly optimized BERT pre-training approach with post-training.  In _Proceedings of the 20th Chinese National Conference on Computational Linguistics_ , pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China.

##  Appendix A Appendix

###  A.1 Datasets Statistics

Table 3 shows the statistics of the DocRED, Re-DocRED, and HacRED datasets.

Dataset | Samples | Facts | Avg. Ents | Relations
---|---|---|---|---
DocRED | 5,053 | 56,354 | 26.2 | 96
Re-DocRED | 4,053 | 120,664 | 19.4 | 96
HacRED | 9,231 | 65,225 | 10.8 | 26
Table 3: Statistics of the DocRED Yao et al. (2019), Re-DocRED Tan et al.
(2022b), and HacRED Cheng et al. (2021) datasets.

###  A.2 GADePo’s Extra Parameters

GADePo introduces few extra parameters to the PLM. The amount of parameters is
reported in Table 4.

Parameter | Model
---|---
| RoBERTalarge | BERTbase
<ent> | 1024 | 768
<pent> | 1024 | 768
$\texttt{<ent>}\longrightarrow*$ | 24 $\times$ 1024 | 12 $\times$ 768
$*\longrightarrow\texttt{<ent>}$ | 24 $\times$ 1024 | 12 $\times$ 768
$\texttt{<pent>}\longrightarrow*$ | 24 $\times$ 1024 | 12 $\times$ 768
$*\longrightarrow\texttt{<pent>}$ | 24 $\times$ 1024 | 12 $\times$ 768
Total | 100,352 | 38,400
Table 4: GADePo’s extra parameters count.

###  A.3 Training Details

In all our experiments we perform early stopping on the development set based
on the Ign $F_{1}+F_{1}$ score for DocRED and Re-DocRED, and $F_{1}$ score for
HacRED. The five different seeds we use are $\\{73,21,37,7,3\\}$

We use RAdam Liu et al. (2020) as our optimizer. On the RoBERTalarge based
models we train for 8 epochs and set the learning rates to $3e^{-5}$ and
$1e^{-4}$ for the PLM parameters and the new additional parameters,
respectively. On the BERTbase based models we train for 10 epochs and set the
learning rates to $1e^{-5}$ and $1e^{-4}$ for the PLM parameters and the new
additional parameters, respectively. We use a cosine learning rate decay
throughout the training process.

In all our experiments the batch size is set to 4 for ATLOP and 2 for GADePo,
with gradient accumulation set to 1 and 2, for ATLOP and GADePo, respectively.
We clip the gradients to a max norm of 1.0. All models are trained with mixed
precision.

We run our experiments on two types of GPUs, namely the NVIDIA V100 32GB for
the RoBERTalarge based models and NVIDIA RTX 3090 24GB for the BERTbase based
models, respectively.

We use PyTorch Paszke et al. (2019), Lightning Falcon and The PyTorch
Lightning team (2019), and Hugging Face’s Transformers Wolf et al. (2020)
libraries to develop our models.


# Dual variational methods for an indefinite nonlinear Helmholtz equation

Rainer Mandel, Dominic Scheider, Tolga Yeşil

###### Abstract

We prove new existence results for a Nonlinear Helmholtz equation with sign-
changing nonlinearity of the form

$-\Delta u-k^{2}u=Q(x)|u|^{p-2}u,\quad u\in W^{2,p}(\mathbb{R}^{N})$

with $k>0,$ $N\geq 3$,
$p\in\left[\left.\frac{2(N+1)}{N-1},\frac{2N}{N-2}\right)\right.$ and $Q\in
L^{\infty}(\mathbb{R}^{N})$. Due to the sign-changes of $Q$, our solutions
have infinite Morse-Index in the corresponding dual variational formulation.

##  1 Introduction

In the present article, we consider nonlinear Helmholtz equations of the form

(1.1) $-\Delta u-k^{2}u=Q(x)|u|^{p-2}u\qquad\text{on }\mathbb{R}^{N}$

for $p\in\left[\left.\frac{2(N+1)}{N-1},\frac{2N}{N-2}\right)\right.$ and
$k>0$ with a weight function $Q\in L^{\infty}(\mathbb{R}^{N})$ that may change
sign. To allow for the latter is nontrivial given that one of the main tools
for proving the existence of solutions is the dual variational method that, in
its classical form, relies on the nonnegativity of the weight function. In the
context of Nonlinear Helmholtz equations it was first implemented in a paper
by Evéquoz and Weth [4]. To highlight the role of the nonnegativity of $Q$ we
briefly recapitulate the approach. To simplify formulas and notation we set
the wavenumber $k=1$. The analysis of the general case then results from a
rescaling.

Instead of (1.1) one considers a reformulation as the integral equation

(1.2) $u=\textbf{R}(Q|u|^{p-2}u)\qquad u\in L^{p}(\mathbb{R}^{N}),$

where R is the real part of a resolvent type operator ${\mathcal{R}}$, i.e., a
right inverse of the Helmholtz operator on $\mathbb{R}^{N}$. For
$f\in\mathcal{S}(\mathbb{R}^{N})$ the operator ${\mathcal{R}}$ is given by
${\mathcal{R}}(f)=\Phi\ast f$ where

$\Phi(x):=\frac{\text{i}}{4}(2\pi|x|)^{\frac{2-N}{2}}H^{(1)}_{\frac{N-2}{2}}(|x|),\quad
x\in\mathbb{R}^{N}\setminus\\{0\\}$

is the fundamental solution of the Helmholtz equation associated with
Sommerfeld’s outgoing radiation condition

(1.3)
$\left|\nabla\Phi(x)-\text{i}\Phi(x)\frac{x}{|x|}\right|=o(|x|^{\frac{1-N}{2}}),\quad\text{
as }|x|\to\infty.$

Here, $H^{(1)}_{\frac{N-2}{2}}$ denotes the Hankel function of the first kind
and order $\frac{N-2}{2}$. So the operator R from (1.2) is given by
$\textbf{R}(f)=\Psi\ast f$ where $\Psi:=\text{Re}(\Phi)$ is given by

(1.4)
$\Psi(x)=-\frac{1}{4}(2\pi|x|)^{\frac{2-N}{2}}Y_{\frac{2-N}{2}}(|x|),\quad
x\in\mathbb{R}^{N}\setminus\\{0\\}$

It is known [7, Theorem 2.3] that $\mathcal{R}$ extends as a continuous linear
map from $L^{p^{\prime}}(\mathbb{R}^{N})\to L^{p}(\mathbb{R}^{N})$ precisely
for $p\in\left[\frac{2(N+1)}{N-1},\frac{2N}{N-2}\right]$. One then introduces
the dual variable $\tilde{u}:=Q^{1/p^{\prime}}|u|^{p-2}u$ and observes that
solutions of (1.2) are precisely the critical points of the (dual) energy
functional $I:L^{p^{\prime}}(\mathbb{R}^{N})\to L^{p}(\mathbb{R}^{N})$ given
by

$I(\tilde{u}):=\frac{1}{p^{\prime}}\left\|\tilde{u}\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\int\limits_{\mathbb{R}^{N}}\tilde{u}{\mathcal{K}}\tilde{u}\leavevmode\nobreak\
dx.$

Here, ${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to
L^{p}(\mathbb{R}^{N}),\tilde{u}\mapsto
Q^{\frac{1}{p}}\textbf{R}(Q^{\frac{1}{p}}\tilde{u})$ is a symmetric operator
in the sense of

(1.5) $\int\limits_{\mathbb{R}^{N}}f\,{\mathcal{K}}g\leavevmode\nobreak\
dx=\int\limits_{\mathbb{R}^{N}}g\,{\mathcal{K}}f\leavevmode\nobreak\
dx\qquad\text{for all }f,g\in L^{p^{\prime}}(\mathbb{R}^{N}),$

see [4, Lemma 4.1]. Under the additional assumption that $Q$ vanishes at
infinity, one obtains that $I$ is an odd functional of class
${\mathcal{C}}^{1}$ that has the Mountain Pass Geometry and satisfies the
Palais-Smale Condition. So the existence of an unbounded sequence of solutions
to (1.2) follows from the Symmetric Mountain Pass Theorem. Inverting the
transformation $u\mapsto\tilde{u}$ one thus obtains an unbounded sequence of
solutions to the nonlinear Helmholtz equation (1.1). This is the strategy
proposed by Evéquoz and Weth [4] for the focusing nonlinear Helmholtz equation
$Q\geq 0$. We refer to [9] for analogous results in the defocusing case $Q\leq
0$, where the dual variational approach can be implemented for the dual
variable $\tilde{u}:=|Q|^{1/p^{\prime}}|u|^{p-2}u$. In view of these two
results it is natural to ask for a dual variational approach work in the
intermediate case of sign-changing $Q$. In this paper, we provide a solution
for this problem.

To treat sign-changing coefficients $Q\in L^{\infty}(\mathbb{R}^{N})$ we have
to come up with a new idea to make the dual variation approach work. We write
$Q=Q_{+}-Q_{-}$ where $Q_{\pm}:=|Q|\mathds{1}_{A_{\pm}}$ and

(1.6) $A_{+}:=\\{Q>0\\},\qquad A_{-}:=\\{Q\leq 0\\}.$

In fact we will consider $Q_{\lambda}:=\lambda Q_{+}-Q_{-}$ for $\lambda>0$ in
the following. Our main idea is to introduce a tuple of dual variables
$(\varphi,\psi)\in L^{p^{\prime}}(A_{+})\times L^{p^{\prime}}(A_{-})$
associated with $(u|_{A_{+}},u|_{A_{-}})$ and to derive a coupled system of
nonlinear integral equations the solutions of which are precisely the critical
points of an associated strongly indefinite dual energy functional. We will
see that the indefiniteness comes from the presence of $Q_{-}$ and thus
vanishes in the case of a nonnegative function $Q\geq 0$. In particular, the
critical points of this dual energy functional will have infinite Morse index,
which clearly distinguishes these solutions from the dual bound and ground
states obtained in [4] in the case $Q\geq 0$. We will explain the dual
variational framework in detail in Section 2. Our conditions for the existence
of critical points involve the linear operator
${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to
L^{p}(\mathbb{R}^{N}),f\mapsto|Q|^{\frac{1}{p}}\textbf{R}(|Q|^{\frac{1}{p}}f)$
as well as the numbers

(1.7)
$\alpha:=\max\limits_{\left\|\varphi\right\|_{p^{\prime}}=1}\int\limits_{\mathbb{R}^{N}}\varphi{\mathcal{K}}\varphi\leavevmode\nobreak\
dx,\qquad\beta:=\max\limits_{\left\|\varphi\right\|_{p^{\prime}}=\left\|\psi\right\|_{p^{\prime}}=1}\int\limits_{\mathbb{R}^{N}}\varphi{\mathcal{K}}\psi\leavevmode\nobreak\
dx.$

Since we will assume ${\mathcal{K}}$ to be compact, both values are indeed
attained. Moreover we have $\beta\geq 0$ and [4, Lemma 4.2(ii)] gives
$\alpha>0$ once we assume that $A^{+}$ has positive measure, i.e.,
$Q^{+}\not\equiv 0$. Our main result reads as follows.

######  Theorem 1.1.

Let $p\in[\frac{2(N+1)}{N-1},\frac{2N}{N-2})$ and $Q\in
L^{\infty}(\mathbb{R}^{N})$, $Q^{+}\not\equiv 0$. Moreover assume that

(1.8) ${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to
L^{p}(\mathbb{R}^{N})\text{ is compact and
}\int\limits_{\mathbb{R}^{N}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq
0\text{ for all }\psi\in L^{p^{\prime}}(A_{-}).$

Then for almost all $\lambda>\lambda_{0}:=(2\beta\alpha^{-1})^{p}$ there is a
nontrivial strong solution $u\in
W^{2,q}(\mathbb{R}^{N})\cap{\mathcal{C}}^{1,\gamma}(\mathbb{R}^{N})$ for all
$q\in\left(\frac{2N}{N-1},\infty\right)$ and $\gamma\in(0,1)$ of

(1.9) $-\Delta u-u=Q_{\lambda}(x)|u|^{p-2}u\qquad\text{on }\mathbb{R}^{N}.$

The proof relies on a combination of a saddle-point reduction and the abstract
monotonicity trick by Jeanjean-Toland [6], which provides bounded Palais-Smale
sequences (only) for almost all $\lambda>\lambda_{0}$. It would clearly be
desirable to extend our result to all $\lambda>\lambda_{0}$, but related a
priori bounds seem to be out of reach. Notice also that [9, Theorem 1.4]
suggests the existence of nontrivial solutions also for small $\lambda>0$,
possibly assuming the set $\\{Q>0\\}$ to be small enough and following a
different variational approach. Let us point out that $\lambda_{0}$ is small
provided that the number $\beta$, which is the same as the operator norm
$\|\mathds{1}_{A_{+}}{\mathcal{K}}(\mathds{1}_{A_{-}})\|_{p^{\prime}\to p}$,
is small compared to $\alpha>0$. In the case $p>\frac{2(N+1)}{N-1}$ this can
be achieved by considering coefficient functions $Q$ such that
$\operatorname*{dist}(A_{-},A_{+})$ is large enough, see [5, Lemma 2.6].

In the following Corollary, we show that the abstract conditions (1.8) hold
for a reasonable class of sign-changing functions $Q\in
L^{\infty}(\mathbb{R}^{N})$. If for instance $Q$ vanishes at infinity, then
[4, Lemma 4.1(ii)] applied to $|Q|$ implies that
${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to L^{p}(\mathbb{R}^{N})$ is
compact. It is less immediate to verify the non-negativity assumption on the
bilinear form
$f\mapsto\int\limits_{\mathbb{R}^{N}}f{\mathcal{K}}f\leavevmode\nobreak\ dx$
on $L^{p^{\prime}}(A_{-})$. From [2, Corollary 5.4] we infer that this
condition holds for measurable sets $A_{-}$ with small enough diameter. To be
more precise, if $y_{\frac{N-2}{2}}$ denotes the first (positive) zero of the
Bessel function $Y_{\frac{N-2}{2}}$, then the condition
$\operatorname*{diam}(A_{-})\leq y_{\frac{N-2}{2}}$ is sufficient. We thus
conclude as follows.

######  Corollary 1.2.

Assume $p\in[\frac{2(N+1)}{N-1},\frac{2N}{N-2})$ and $Q\in
L^{\infty}(\mathbb{R}^{N}),Q^{+}\not\equiv 0$ . Moreover assume

$\lim_{R\to\infty}\operatorname{esssup}_{|x|\geq
R}|Q|=0\qquad\text{and}\qquad\operatorname*{diam}{(A_{-})}\leq
y_{\frac{N-2}{2}}.$

Then for almost all $\lambda>\lambda_{0}:=(2\beta\alpha^{-1})^{p}$ there is a
nontrivial strong solution $u\in
W^{2,q}(\mathbb{R}^{N})\cap{\mathcal{C}}^{1,\gamma}(\mathbb{R}^{N})$ for all
$q\in\left(\frac{2N}{N-1},\infty\right)$ and $\gamma\in(0,1)$ of (1.9).

Let us remark that for the general case $k>0$ the condition on the diameter of
$A_{-}$ changes to $\operatorname*{diam}(A_{-})\leq k^{-1}y_{\frac{N-2}{2}}$.
To put this condition into perspective, note that for $N=3$ we have
$Y_{\frac{1}{2}}(t)=-\sqrt{\frac{2}{\pi t}}\cos{t}$, thus $y_{1/2}=\pi/2$ and
$y_{\frac{N-2}{2}}>y_{1/2}$ for $N>3$ (see [1, Section 9.5]). The regularity
results in Theorem 1.1 and Corollary 1.2 are direct consequences of [4, Lemma
4.3] and of the iteration procedure from Step 3 in the proof of [8, Theorem
1]. Notice that this theorem provides solutions to far more general Nonlinear
Helmholtz equations than (1.1) regardless of whether sign-changes occur or
not, but the constructed solutions are small. This result relies on a fixed
point approach. Let us also mention [3] where nontrivial solutions of
Nonlinear Helmholtz equations are constructed for rather general and possibly
sign-changing nonlinearities vanishing identically outside some compact subset
of $\mathbb{R}^{N}$. Our method is entirely different from any of these
approaches.

This paper is organzied as follows: In Section 2 we introduce our basic tools
and develop the dual variational framework by reformulating the indefinite
Nonlinear Helmholtz equation as a coupled system of integral equations. Then
we prove that nontrivial critical points of the associated energy functional
$J_{\lambda}$ are indeed nontrivial solutions $u\in L^{p}(\mathbb{R}^{N})$ of
the integral equation $u=\textbf{R}(Q_{\lambda}|u|^{p-2}u)$. This motivates
the search for critical points of $J_{\lambda}$. In Section 3 we perform the
saddle-point reduction of $(\varphi,\psi)\mapsto J_{\lambda}(\varphi,\psi)$
leading to a reduced function $\tilde{J}_{\lambda}$ that depends on $\varphi$
only. In Section 4 we establish the existence of bounded Palais-Smale
sequences for these reduced functionals for almost all $\lambda>\lambda_{0}$.
As mentioned above, this step entirely relies on the monotonicity trick by
Jeanjean and Toland [6]. Finally, we combine all the auxiliary results to
prove Theorem 1.1 and Corollary 1.2 in Section 5.

Let us close this introduction by fixing some notation: Throughout the paper
we denote by $B_{r}(x)$ the open ball in $\mathbb{R}^{N}$ with radius $r>0$
and center at $x\in\mathbb{R}^{N}$. Moreover, we set $B_{r}=B_{r}(0)$ for any
$r>0$. For $1\leq s\leq\infty$, we abbreviate the standard norm on
$L^{s}(\mathbb{R}^{N})$ by $\left\|\cdot\right\|_{s}$. The Schwartz-class of
rapidly decreasing functions on $\mathbb{R}^{N}$ is denoted by
$\mathcal{S}(\mathbb{R}^{N})$. For any $p\in(1,\infty)$ we always denote by
$p^{\prime}:=\frac{p}{p-1}$ the Hölder conjugate of $p$. The indicator
function of a measurable set $B\subset\mathbb{R}^{N}$ is $\mathds{1}_{B}$. By
$\operatorname*{diam}{(B)}$ we always denote the diametere of a set. We will
always use the symbols $\varphi,\psi$ to denote $L^{p^{\prime}}(A_{+})-$ and
$L^{p^{\prime}}(A_{-})$-functions that are continued trivially to the whole of
$\mathbb{R}^{N}$.

Acknowledgements
The authors would like to thank Tobias Weth for helpful suggestions and
stimulating discussions. The first two authors are funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) \- Project-ID
258734477 - SFB 1173.

##  2 Dual variational formulation

In this section we will formulate a variational framework to the equation
(1.9). We recall from the introduction that solutions of our problem are
obtained as solutions of the integral equation

(2.1) $u=\textbf{R}(Q_{\lambda}|u|^{p-2}u),\qquad\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ u\in
L^{p}(\mathbb{R}^{N}).$

where $\textbf{R}(f)=\Psi\ast f$ for the function $\Psi$ introduced in (1.4)
and

(2.2) $\left\|\textbf{R}(f)\right\|_{L^{p}(\mathbb{R}^{N})}\leq
C\left\|f\right\|_{L^{p^{\prime}}(\mathbb{R}^{N})}$

for all $p\in\left[\frac{2(N+1)}{N-1},\frac{2N}{N-2}\right]$ and some constant
$C>0$.

To obtain the dual variational formualation of (2.1) we introduce
$v:=\mathds{1}_{A_{+}}u$ and $w:=\mathds{1}_{A_{-}}u$. Then (2.1) is
equivalent to the system

$\displaystyle\begin{aligned}
v&=\lambda\mathds{1}_{A_{+}}\textbf{R}\left[Q_{+}|v|^{p-2}v\right]-\mathds{1}_{A_{+}}\textbf{R}\left[Q_{-}|w|^{p-2}w\right],\\\
w&=\lambda\mathds{1}_{A_{-}}\textbf{R}\left[Q_{+}|v|^{p-2}v\right]-\mathds{1}_{A_{-}}\textbf{R}\left[Q_{-}|w|^{p-2}w\right].\end{aligned}$

Setting

$\varphi:=\lambda Q_{+}^{1/p^{\prime}}|v|^{p-2}v\in
L^{p^{\prime}}(A_{+}),\qquad\psi:=Q_{-}^{1/p^{\prime}}|w|^{p-2}w\in
L^{p^{\prime}}(A_{-})$

we deduce

$\displaystyle\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi$
$\displaystyle=Q_{+}^{1/p}v$ $\displaystyle=\lambda
Q_{+}^{1/p}\textbf{R}\left[Q_{+}|v|^{p-2}v\right]-Q_{+}^{1/p}\textbf{R}\left[Q_{-}|w|^{p-2}w\right]$
$\displaystyle=Q_{+}^{1/p}\textbf{R}\left[Q_{+}^{1/p}\varphi\right]-Q_{+}^{1/p}\textbf{R}\left[Q_{-}^{1/p}\psi\right]$
$\displaystyle=Q_{+}^{1/p}\textbf{R}\left[|Q|^{1/p}(\varphi-\psi)\right].$

Similarly

$\displaystyle|\psi|^{p^{\prime}-2}\psi$
$\displaystyle=Q_{-}^{1/p}\textbf{R}\left[|Q|^{1/p}(\varphi-\psi)\right].$

In terms of the Birman-Schwinger operator
${\mathcal{K}}:f\mapsto|Q|^{\frac{1}{p}}\textbf{R}\big{(}|Q|^{\frac{1}{p}}f\big{)}$
introduced above this can be reformulated as

$\displaystyle\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi$
$\displaystyle=\mathds{1}_{A_{+}}{\mathcal{K}}(\varphi-\psi),$
$\displaystyle|\psi|^{p^{\prime}-2}\psi$
$\displaystyle=\mathds{1}_{A_{-}}{\mathcal{K}}(\varphi-\psi)$

and therefore carries a variational structure through the (dual) energy
functional $J_{\lambda}$ on $L^{p^{\prime}}(A_{+})\times
L^{p^{\prime}}(A_{-})$ given by

(2.3)
$J_{\lambda}(\varphi,\psi):=\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\left\|\varphi\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{p^{\prime}}\left\|\psi\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\int\limits_{\mathbb{R}^{N}}(\varphi-\psi){\mathcal{K}}(\varphi-\psi)\leavevmode\nobreak\
dx.$

This functional is of class ${\mathcal{C}}^{1}$ with

$\displaystyle\partial_{1}J_{\lambda}(\varphi,\psi)[h_{1}]$
$\displaystyle=\int\limits_{\mathbb{R}^{N}}\left(\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi-{\mathcal{K}}(\varphi-\psi)\right)h_{1}\leavevmode\nobreak\
dx,$ $\displaystyle h_{1}\in L^{p^{\prime}}(A_{+})$
$\displaystyle\partial_{2}J_{\lambda}(\varphi,\psi)[h_{2}]$
$\displaystyle=\int\limits_{\mathbb{R}^{N}}\left(-|\psi|^{p^{\prime}-2}\psi-{\mathcal{K}}(\varphi-\psi)\right)h_{2}\leavevmode\nobreak\
dx,$ $\displaystyle h_{2}\in L^{p^{\prime}}(A_{-}).$

Here $\partial_{1},\partial_{2}$ standard for partial derivatives with respect
to $\varphi$ and $\psi$. For this reason we will look for critical points of
$J_{\lambda}$. These solve the integral equation (2.1). Thus by the regularity
results [4, Lemma 4.3] and [8, p.13] these are indeed strong solutions to our
original problem (1.9).

######  Proposition 2.1.

Let $(\varphi,\psi)\in L^{p^{\prime}}(A_{+})\times
L^{p^{\prime}}(A_{-})\setminus\\{(0,0)\\}$ be a critical point of
$J_{\lambda}$ where $\lambda>0$. Then

$u:=\mathrm{{\bf R}}\left(|Q|^{\frac{1}{p}}(\varphi-\psi)\right)\in
L^{p}(\mathbb{R}^{N})$

is a nontrivial solution of (2.1).

###### Proof.

Let $(\varphi,\psi)\in L^{p^{\prime}}(A_{+})\times
L^{p^{\prime}}(A_{-})\setminus\\{(0,0)\\}$ be a critical point of
$J_{\lambda}$. Thus we have

$\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi=\mathds{1}_{A_{+}}{\mathcal{K}}(\varphi-\psi),\qquad|\psi|^{p^{\prime}-2}\psi=\mathds{1}_{A_{-}}{\mathcal{K}}(\varphi-\psi)$

as well as

$\displaystyle Q_{\lambda}|u|^{p-2}u$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q||u|^{p-2}u$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q|^{\frac{1}{p}}\cdot||Q|^{\frac{1}{p}}u|^{p-2}\cdot|Q|^{\frac{1}{p}}u$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q|^{\frac{1}{p}}\cdot||Q|^{\frac{1}{p}}\textbf{R}\left[|Q|^{1/p}(\varphi-\psi)\right]|^{p-2}\cdot|Q|^{\frac{1}{p}}\textbf{R}\left[|Q|^{1/p}(\varphi-\psi)\right]$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q|^{\frac{1}{p}}\cdot|{\mathcal{K}}\left[\varphi-\psi\right]|^{p-2}\cdot{\mathcal{K}}\left[\varphi-\psi\right]$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q|^{\frac{1}{p}}\cdot\left|\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi+|\psi|^{p^{\prime}-2}\psi\right|^{p-2}\cdot\left(\lambda^{1-p^{\prime}}|\varphi|^{p^{\prime}-2}\varphi+|\psi|^{p^{\prime}-2}\psi\right)$
$\displaystyle=(\lambda\mathds{1}_{A_{+}}-\mathds{1}_{A_{-}})|Q|^{\frac{1}{p}}\cdot\left(\lambda^{(1-p^{\prime})(p-1)}\varphi+\psi\right)$
$\displaystyle=|Q|^{\frac{1}{p}}\cdot\left(\varphi-\psi\right).$

Applying R then gives
$\textbf{R}\left(Q_{\lambda}|u|^{p-2}u\right)=\textbf{R}\left(|Q|^{\frac{1}{p}}(\varphi-\psi)\right)=u.$
Hence $u$ solves (2.1). ∎

So we conclude that it remains to find nontrivial critical points of the
functionals $J_{\lambda}$ for as many $\lambda>0$ as possible. This will be
achieved with the Mountain Pass Theorem for families of
${\mathcal{C}}^{1}$-functionals by Jeanjean and Toland [6].

##  3 Saddle-point reduction

In this section we perform the saddle-point reduction of $J_{\lambda}$ with
respect to the $\psi$-variable. To this end, we prove that for any fixed
$\varphi\in L^{p^{\prime}}(A_{+})$ the functional $\psi\mapsto
J_{\lambda}(\varphi,\psi)$ attains its maximum at some uniquely defined
function in $L^{p^{\prime}}(A_{-})$ that we will call $Z(\varphi)$ in the
following. We shall see that the positivity assumption
$\int\limits_{\mathbb{R}^{N}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq
0$ for all $\psi\in L^{p^{\prime}}(A_{-})$ ensures that the functional
$\psi\mapsto J_{\lambda}(\varphi,\psi)$ is strictly concave so that the global
maximization with respect to $\psi$ is the only reasonable approach to perform
a saddle point reduction. We introduce the reduced functional
$\tilde{J}_{\lambda}:L^{p^{\prime}}(A_{+})\to\mathbb{R}$ via

(3.1) $\tilde{J}_{\lambda}(\varphi):=\sup\limits_{\psi\in
L^{p^{\prime}}(A_{-})}J_{\lambda}(\varphi,\psi).$

######  Proposition 3.1.

Assume that ${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to
L^{p}(\mathbb{R}^{N})$ is compact and that
$\int\limits_{A_{-}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq 0$ for
all $\psi\in L^{p^{\prime}}(A_{-})$. Then for every $\varphi\in
L^{p^{\prime}}(A_{+})$ there exists a unique $Z(\varphi)\in
L^{p^{\prime}}(A_{-})$ such that for all $\lambda>0$ we have

$\tilde{J}_{\lambda}(\varphi)=J_{\lambda}(\varphi,Z(\varphi)).$

Moreover:

  1. (i)

For any $\varphi\in L^{p^{\prime}}(A_{+})$ the corresponding maximizer
$Z(\varphi)$ satisfies

(3.2)
$\left\|Z(\varphi)\right\|_{p^{\prime}}\leq\left(p^{\prime}\beta\left\|\varphi\right\|_{p^{\prime}}\right)^{\frac{1}{p^{\prime}-1}}$

where $\beta$ is defined in (1.7).

  2. (ii)

The map $Z:L^{p^{\prime}}(A_{+})\to L^{p^{\prime}}(A_{-})$ is continuous.

  3. (iii)

The reduced functional
$\tilde{J}_{\lambda}:L^{p^{\prime}}(A_{+})\to\mathbb{R}$ is of class
${\mathcal{C}}^{1}$ with derivative

$\tilde{J}_{\lambda}^{\prime}[h]=\partial_{1}J_{\lambda}(\varphi,Z(\varphi))[h].$

###### Proof.

We first establish the existence of a maximizer. So fix $\varphi\in
L^{p^{\prime}}(A_{+})$ and consider a maximizing sequence
$(\psi_{n})_{n}\subset L^{p^{\prime}}(A_{-})$. Using
$J_{\lambda}(\varphi,0)\leq\sup\limits_{\psi\in
L^{p^{\prime}}(A_{-})}J_{\lambda}(\varphi,\psi)=J_{\lambda}(\varphi,\psi_{n})+o(1)$
as $n\to\infty$ we obtain

$o(1)\leq-\frac{1}{p^{\prime}}\|\psi_{n}\|_{p^{\prime}}^{p^{\prime}}+\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\psi_{n}-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi_{n}\mathcal{K}\psi_{n}\leq-\frac{1}{p^{\prime}}\|\psi_{n}\|_{p^{\prime}}^{p^{\prime}}+\beta\|\varphi\|_{p^{\prime}}\|\psi_{n}\|_{p^{\prime}}\quad(n\to\infty).$

Here we used the nonnegativity assumption on ${\mathcal{K}}$ as well as (2.2).
Hence,

(3.3)
$\|\psi_{n}\|_{p^{\prime}}\leq(p^{\prime}\beta\|\varphi\|)^{\frac{1}{p^{\prime}-1}}+o(1)\qquad(n\to\infty),$

so $(\psi_{n})_{n}$ is bounded. Passing to a subsequence we find
$\psi^{\ast}\in L^{p^{\prime}}(A_{-})$ such that
$\psi_{n}\rightharpoonup\psi^{\ast}$ in $L^{p^{\prime}}(A_{-})$ as
$n\to\infty.$ Using the compactness of ${\mathcal{K}}$ and the weak lower
semicontinuity of the norm we find

$\displaystyle\sup_{\psi}J_{\lambda}(\varphi,\psi)$
$\displaystyle\quad=\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\|\varphi\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{p^{\prime}}\|\psi_{n}\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{2}\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\varphi\leavevmode\nobreak\
dx+\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\psi_{n}\leavevmode\nobreak\
dx-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi_{n}\mathcal{K}\psi_{n}\leavevmode\nobreak\
dx+o(1)$
$\displaystyle\quad=\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\|\varphi\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{p^{\prime}}\|\psi_{n}\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{2}\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\varphi\leavevmode\nobreak\
dx+\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\psi^{\ast}\leavevmode\nobreak\
dx-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi^{\ast}\mathcal{K}\psi^{\ast}\leavevmode\nobreak\
dx+o(1)$
$\displaystyle\quad\leq\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\|\varphi\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{p^{\prime}}\|\psi^{\ast}\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{2}\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\varphi\leavevmode\nobreak\
dx+\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\psi^{\ast}\leavevmode\nobreak\
dx-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi^{\ast}\mathcal{K}\psi^{\ast}\leavevmode\nobreak\
dx+o(1)$ $\displaystyle\quad=J_{\lambda}(\varphi,\psi^{\ast})+o(1).$

Hence the supremum is attained at $\psi^{\ast}$. Since equality must hold in
the above estimate we conclude
$\left\|\psi_{n}\right\|_{p^{\prime}}\to\left\|\psi^{\ast}\right\|_{p^{\prime}}$,
whence $\psi_{n}\to\psi^{\ast}$ in $L^{p^{\prime}}(A_{-})$ as $n\to\infty$.
This shows the existence of a maximizer satisfying the estimate stated in (i).
So (i) is proved once we have established the uniqueness of the maximizer.

To this end assume that $\psi^{\ast},\psi^{\dagger}\in L^{p^{\prime}}(A_{-})$
are maximizers. Then we have

$\displaystyle 0$
$\displaystyle\leq\frac{1}{2}J_{\lambda}(\varphi,\psi^{\ast})+\frac{1}{2}J_{\lambda}(\varphi,\psi^{\dagger})-J_{\lambda}\left(\varphi,\frac{1}{2}(\psi^{\ast}+\psi^{\dagger})\right)$
$\displaystyle=\frac{1}{p^{\prime}}\left(\left\|\frac{\psi^{\ast}+\psi^{\dagger}}{2}\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\ast}\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\dagger}\|^{p^{\prime}}_{p^{\prime}}\right)$
$\displaystyle\quad\quad+\frac{1}{2}\left(\int_{\mathbb{R}^{N}}\frac{\psi^{\ast}+\psi^{\dagger}}{2}\mathcal{K}\left[\frac{\psi^{\ast}+\psi^{\dagger}}{2}\right]\>\mathrm{d}x-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi^{\ast}\mathcal{K}\psi^{\ast}\>\mathrm{d}x-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi^{\dagger}\mathcal{K}\psi^{\dagger}\>\mathrm{d}x\right)$
$\displaystyle=\frac{1}{p^{\prime}}\left(\left\|\frac{\psi^{\ast}+\psi^{\dagger}}{2}\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\ast}\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\dagger}\|^{p^{\prime}}_{p^{\prime}}\right)-\frac{1}{8}\int_{\mathbb{R}^{N}}(\psi^{\ast}-\psi^{\dagger})\mathcal{K}[\psi^{\ast}-\psi^{\dagger}]\>\mathrm{d}x$
$\displaystyle\leq\frac{1}{p^{\prime}}\left(\left\|\frac{\psi^{\ast}+\psi^{\dagger}}{2}\right\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\ast}\|^{p^{\prime}}_{p^{\prime}}-\frac{1}{2}\|\psi^{\dagger}\|^{p^{\prime}}_{p^{\prime}}\right)$
$\displaystyle\leq 0,$

where we have used the non-negativity condition in the second last step and
the convexity of $z\mapsto|z|^{p^{\prime}}$ in the last step. So we have
equality in each estimate and conclude $\psi^{\ast}=\psi^{\dagger}$. Note that
the maximizer does not depend on $\lambda$ since the only $\lambda$-dependent
term in $J_{\lambda}(\varphi,\psi)$ is
$\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\|\varphi\|_{p^{\prime}}^{p^{\prime}}$,
which is independent of $\psi$.

We now prove (ii), i.e., the continuity of the map $Z$: Assume
$\varphi_{n}\to\varphi$ in $L^{p^{\prime}}(A_{+})$ and let
$(\psi_{n})_{n}:=(Z(\varphi_{n}))_{n}\subset L^{p^{\prime}}(A_{-})$ be the
associated maximizers. By (3.3), the sequence $(\psi_{n})_{n}$ is bounded and
after passing to a subsequence we may assume $\psi_{n}\rightharpoonup\psi_{0}$
in $L^{p^{\prime}}(A_{-})$ as $n\to\infty$. Arguing as above we deduce

(3.4)
$\limsup_{n\to\infty}\tilde{J}_{\lambda}(\varphi_{n})=\limsup_{n\to\infty}J_{\lambda}(\varphi_{n},\psi_{n})=\limsup_{n\to\infty}J_{\lambda}(\varphi,\psi_{n})\leq
J_{\lambda}(\varphi,\psi_{0})\leq\tilde{J}_{\lambda}(\varphi).$

using weak lower semicontinuity and
$\liminf\limits_{n\to\infty}\left\|\psi_{n}\right\|_{p^{\prime}}\geq\left\|\psi_{0}\right\|_{p^{\prime}}$.
On the other hand, with the special choice $\psi=Z(\varphi)$ we obtain

(3.5) $\liminf_{n\to\infty}\tilde{J}_{\lambda}(\varphi_{n})\geq
J_{\lambda}(\varphi,\psi)=J_{\lambda}(\varphi,Z(\varphi))=\tilde{J}_{\lambda}(\varphi).$

Combining both estimates gives
$\left\|\psi_{n}\right\|_{p^{\prime}}\to\left\|\psi_{0}\right\|_{p^{\prime}}$
as well as $\tilde{J}_{\lambda}(\varphi_{n})\to\tilde{J}_{\lambda}(\varphi)$
as $n\to\infty$. Thus we have equality in (3.4), (3.5). Since maximizers are
unique, we obtain $\psi_{0}=Z(\varphi)$ and in particular
$Z(\varphi_{n})=\psi_{n}\to\psi_{0}=Z(\varphi_{0})$ in $L^{p^{\prime}}(A_{-})$
as $n\to\infty$.

We are left to prove (iii). Let $h\in L^{p^{\prime}}(A_{+})$ be arbitrary. We
can estimate the difference quotients as follows:

$\displaystyle\liminf_{\tau\to 0}\frac{\tilde{J}_{\lambda}(\varphi+\tau
h)-\tilde{J}_{\lambda}(\varphi)}{\tau}$ $\displaystyle\geq\liminf_{\tau\to
0}\frac{J_{\lambda}(\varphi+\tau
h,Z(\varphi))-J_{\lambda}(\varphi,Z(\varphi))}{\tau}$
$\displaystyle=\liminf_{\tau\to
0}\int_{0}^{1}\partial_{1}J_{\lambda}(\varphi+\tau\sigma
h,Z(\varphi))[h]\,\mathrm{d}\sigma$
$\displaystyle=\partial_{1}J_{\lambda}(\varphi,Z(\varphi))[h],$
$\displaystyle\limsup_{\tau\to 0}\frac{\tilde{J}_{\lambda}(\varphi+\tau
h)-\tilde{J}_{\lambda}(\varphi)}{\tau}$ $\displaystyle\leq\limsup_{\tau\to
0}\frac{J_{\lambda}(\varphi+\tau h,Z(\varphi+\tau
h))-J_{\lambda}(\tilde{\varphi},Z(\varphi+\tau h))}{\tau}$
$\displaystyle=\limsup_{\tau\to
0}\int_{0}^{1}\partial_{1}J_{\lambda}(\varphi^{*}+\tau\sigma h,Z(\varphi+\tau
h))[h]\,\mathrm{d}\sigma$
$\displaystyle=\partial_{1}J_{\lambda}(\varphi,Z(\varphi))[h].$

Here we used that $Z$ is continuous and that $\partial_{1}J_{\lambda}$ is
continuous, see [10, Proposition 9] for a similar computation. We conclude
that $\tilde{J}_{\lambda}$ is Gâteaux-differentiable with continuous
derivative $\varphi\mapsto\partial_{1}J_{\lambda}(\varphi,Z(\varphi))[\cdot]$,
see Proposition 3.1 (iii). Hence, the reduced functional $\tilde{J}_{\lambda}$
is continuously (Fréchet-)differentiable with

$\displaystyle\tilde{J}_{\lambda}^{\prime}(\varphi)[h]=\partial_{1}J_{\lambda}(\varphi,Z(\varphi))[h]\qquad\forall
h\in L^{p^{\prime}}(A_{+})$

as claimed. ∎

Notice that the condition
$\int_{\mathbb{R}^{N}}\psi\mathcal{K}\psi\>\mathrm{d}x\geq 0$ is also
necessary for the existence of a global maximizer of $\psi\mapsto
J_{\lambda}(\varphi,\psi)$ because otherwise this functional is unbounded from
above.

##  4 Palais-Smale sequences for the reduced functional

In view of the results of the previous sections, we obtain a solution to our
problem by proving the existence of a nontrivial critical point of the reduced
functional $\tilde{J}_{\lambda}:L^{p^{\prime}}(A_{+})\to\mathbb{R}$ introduced
in (3.1). This will be done via Mountain-pass techniques for monotone families
of functionals originating from the work of Jeanjean and Toland [6].

######  Definition 4.1.

Let $X$ be a Banach space, $M\subset\mathbb{R}$ a compact interval. Then the
family $(I_{\nu})_{\nu\in M}$ of ${\mathcal{C}}^{1}$-functionals on $X$ is
said to have the Mountain Pass Geometry if there exist $v_{1},v_{2}\in X$ such
that for all $\nu\in M$ it holds

$c_{\nu}:=\inf_{\gamma\in\Gamma}\sup_{t\in[0,1]}I_{\nu}(\gamma(t))>\max\\{I_{\nu}(v_{1}),I_{\nu}(v_{2})\\},$

where $\Gamma:=\\{\gamma\in C([0,1],X):\gamma(0)=v_{1},\gamma(1)=v_{2}\\}$.

######  Theorem 4.2.

(Jeanjean, Toland) [6, Theorem 2.1] Assume that $X$ is a Banach space,
$M\subset\mathbb{R}$ a compact interval and $(I_{\nu})_{\nu\in M}$ a family of
${\mathcal{C}}^{1}$-functionals on $X$ having the Mountain Pass Geometry.
Assume further that $(I_{\nu})_{\nu\in M}$ has the following property:

(H) $\displaystyle\begin{aligned} &\text{For every sequence
}(\nu_{n},\varphi_{n})\in M\times X\text{ with }\nu_{n}\nearrow\nu_{*}\in
M\text{ and with }\\\ &\quad-I_{\nu_{*}}(\varphi_{n}),\quad
I_{\nu_{n}}(\varphi_{n}),\quad\frac{I_{\nu_{n}}(\varphi_{n})-I_{\nu_{*}}(\varphi_{n})}{\nu_{*}-\nu_{n}}\quad\text{
bounded from above, }\\\ &\text{the sequence }(\varphi_{n})\text{ is bounded
itself, and
}\limsup_{n\to\infty}(I_{\nu_{*}}(\varphi_{n})-I_{\nu_{n}}(\varphi_{n}))\leq
0.\end{aligned}$

Then for almost all $\nu\in M$ there is a bounded Palais-Smale sequence (BPS)
for $I_{\nu}$ at the level $c_{\nu}$.

We shall apply this result to $X=L^{p^{\prime}}(A_{+})$ and the family of
${\mathcal{C}}^{1}$-functionals
$I_{\lambda}:=\tilde{J}_{\lambda}:X\to\mathbb{R}$. We first verify the
Mountain Pass Geometry for parameters $\lambda\in(\lambda_{0},\infty)$ where
$\lambda_{0}=(2\beta\alpha^{-1})^{p}$. Let us recall that $\alpha,\beta$ were
defined as

$\alpha=\max\limits_{\left\|\varphi\right\|_{p^{\prime}}=1}\int\limits_{\mathbb{R}^{N}}\varphi{\mathcal{K}}\varphi\leavevmode\nobreak\
dx>0,\qquad\beta=\max\limits_{\left\|\varphi\right\|_{p^{\prime}}=\left\|\psi\right\|_{p^{\prime}}=1}\int\limits_{\mathbb{R}^{N}}\varphi{\mathcal{K}}\psi\leavevmode\nobreak\
dx\geq 0.$

######  Proposition 4.3.

Let ${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to L^{p}(\mathbb{R}^{N})$ be
compact and assume
$\int\limits_{\mathbb{R}^{N}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq
0$ for all $\psi\in L^{p^{\prime}}(A_{-})$. Then, for any given compact
subinterval $M\subset(\lambda_{0},\infty)$, the family of functionals
$(\tilde{J}_{\lambda})_{\lambda\in M}$ has the Mountain Pass Geometry
according to Definition 4.1.

###### Proof.

For $\lambda\in M$ we define
$r_{\lambda}:=(\lambda^{p^{\prime}-1}\alpha)^{1/(p^{\prime}-2)}$. Then we have

$\displaystyle\inf_{\|\varphi\|_{p^{\prime}}=r_{\lambda}}\tilde{J}_{\lambda}(\varphi)$
$\displaystyle=\inf_{\|\varphi\|_{p^{\prime}}=r_{\lambda}}\sup_{\psi}J_{\lambda}(\varphi,\psi)\geq\inf_{\|\varphi\|_{p^{\prime}}=r_{\lambda}}J_{\lambda}(\varphi,0)$
$\displaystyle=\inf_{\|\varphi\|_{p^{\prime}}=r_{\lambda}}\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}\|\varphi\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{2}\int_{\mathbb{R}^{N}}\varphi\mathcal{K}\varphi\>\mathrm{d}x$
$\displaystyle=\frac{\lambda^{1-p^{\prime}}}{p^{\prime}}r_{\lambda}^{p^{\prime}}-\frac{\alpha}{2}r_{\lambda}^{2}=\alpha\left(\frac{1}{p^{\prime}}-\frac{1}{2}\right)(\lambda^{p^{\prime}-1}\alpha)^{\frac{2}{p^{\prime}-2}}$
$\displaystyle>0.$

On the other hand, we have

$\displaystyle\tilde{J}_{\lambda}(0)=\sup_{\psi}J_{\lambda}(0,\psi)=\sup_{\psi}\left[-\frac{1}{p^{\prime}}\|\psi\|_{p^{\prime}}^{p^{\prime}}-\frac{1}{2}\int_{\mathbb{R}^{N}}\psi\mathcal{K}\psi\>\mathrm{d}x\right]=0.$

According to Definition 4.1 it therefore remains to find some $\varphi^{*}\in
L^{p^{\prime}}(A_{+})$ with $\|\varphi^{*}\|_{p^{\prime}}\geq r_{\lambda_{0}}$
such that $\tilde{J}_{\lambda_{0}}(\varphi^{*})\leq 0$ holds. Notice that in
this case we actually have
$\tilde{J}_{\lambda}(\varphi^{*})<\tilde{J}_{\lambda_{0}}(\varphi^{*})\leq 0$
for all $\lambda\in M\subset(\lambda_{0},\infty)$. To achieve this we estimate
$\tilde{J}_{\lambda_{0}}$ from above as follows

$\displaystyle\tilde{J}_{\lambda_{0}}(\varphi)$
$\displaystyle=J_{\lambda_{0}}(\varphi,0)-\frac{1}{p^{\prime}}\|Z(\varphi)\|_{p^{\prime}}^{p^{\prime}}+\int_{\mathbb{R}^{N}}\varphi{\mathcal{K}}(Z(\varphi))-\frac{1}{2}\int_{\mathbb{R}^{N}}Z(\varphi)\mathcal{K}(Z(\varphi))$
$\displaystyle\leq
J_{\lambda_{0}}(\varphi,0)-\frac{1}{p^{\prime}}\|Z(\varphi)\|_{p^{\prime}}^{p^{\prime}}+\beta\|\varphi\|_{p^{\prime}}\|Z(\varphi)\|_{p^{\prime}}$
$\displaystyle\leq
J_{\lambda_{0}}(\varphi,0)+\frac{\beta^{p}}{p}\|\varphi\|_{p^{\prime}}^{p}$

where we have used that $\max\limits_{c\geq
0}\left(-\frac{c^{p^{\prime}}}{p^{\prime}}+\beta\left\|\varphi\right\|^{p^{\prime}}_{p^{\prime}}c\right)=\frac{\beta^{p}}{p}\|\varphi\|_{p^{\prime}}^{p}.$
We choose $\varphi^{*}=r_{0}\varphi_{0}$ where the function $\varphi_{0}\in
L^{p^{\prime}}(A_{+}),\|\varphi_{0}\|_{p^{\prime}}=1$ attains the maximum
$\alpha=\int_{\mathbb{R}^{N}}\varphi_{0}\mathcal{K}\varphi_{0}\>\mathrm{d}x>0$.
Then the choice $R:=(\frac{1}{2}\alpha\beta^{-p})^{1/(p-2)}$ yields after some
computations (recall $\lambda_{0}=(2\beta\alpha^{-1})^{p}$)

$\|\varphi^{*}\|_{p^{\prime}}=R=\left(\frac{1}{2}\alpha\beta^{-p}\right)^{\frac{1}{p-2}}\geq(\lambda_{0}^{p^{\prime}-1}\alpha)^{\frac{1}{p^{\prime}-2}}>(\lambda^{p^{\prime}-1}\alpha)^{\frac{1}{p^{\prime}-2}}=r_{\lambda}\qquad\text{for
all }\lambda\in M.$

Using again the explicit formulas for $R,\lambda_{0}$ we find

$\displaystyle\tilde{J}_{\lambda_{0}}(R\varphi_{0})$ $\displaystyle\leq
J_{\lambda_{0}}(R\varphi_{0})+\frac{\beta^{p}R^{p}}{p}$
$\displaystyle=\frac{\lambda_{0}^{1-p^{\prime}}}{p^{\prime}}R^{p^{\prime}}-\frac{\alpha}{2}R^{2}+\frac{\beta^{p}}{p}R^{p}$
$\displaystyle=\frac{R^{p^{\prime}}}{p^{\prime}}\cdot\left(\lambda_{0}^{1-p^{\prime}}-\frac{p\alpha}{2(p-1)}R^{2-p^{\prime}}+\frac{\beta^{p}}{p-1}R^{p-p^{\prime}}\right)$
$\displaystyle=\frac{R^{p^{\prime}}}{p^{\prime}}\cdot\left(\lambda_{0}^{1-p^{\prime}}-\frac{1}{2}\alpha
R^{2-p^{\prime}}\right)$
$\displaystyle=\frac{R^{p^{\prime}}}{p^{\prime}}\cdot\left((2\beta\alpha^{-1})^{-\frac{p}{p-1}}-\frac{1}{2}\alpha\cdot\left(\frac{1}{2}\alpha\beta^{-p}\right)^{\frac{1}{p-1}}\right)$
$\displaystyle=0$

and thus the claim holds with $v_{1}=0$ and $v_{2}=\varphi^{*}=R\varphi_{0}$.
∎

Having established the Mountain Pass Geometry of our functionals we now verify
the condition (H) in order to use Theorem 4.2

######  Proposition 4.4.

For any compact subinterval $M\subset(\lambda_{0},\infty)$ the family of
${\mathcal{C}}^{1}-$functionals $(\tilde{J}_{\lambda})_{\lambda\in M}$
satisfies the condition (H).

###### Proof.

Consider a sequence $(\varphi_{n},\lambda_{n})\in L^{p^{\prime}}(A_{+})\times
M$ with $\lambda_{n}\nearrow\lambda_{*}$ and

$\displaystyle-\tilde{J}_{\lambda_{*}}(\varphi_{n})\leq
C,\quad\tilde{J}_{\lambda_{n}}(\varphi_{n})\leq
C,\quad\frac{\tilde{J}_{\lambda_{n}}(\varphi_{n})-\tilde{J}_{\lambda_{*}}(\varphi_{n})}{\nu_{*}-\nu_{n}}\leq
C$

for all $n\in\mathbb{N}$. Then we have

$C\geq\frac{\tilde{J}_{\lambda_{n}}(\varphi_{n})-\tilde{J}_{\lambda^{*}}(\varphi_{n})}{\lambda_{*}-\lambda_{n}}=\frac{\lambda_{n}^{1-p^{\prime}}-\lambda_{*}^{1-p^{\prime}}}{\lambda_{*}-\lambda_{n}}\|\varphi_{n}\|_{p^{\prime}}^{p^{\prime}}=\left((p^{\prime}-1)\lambda_{*}^{-p^{\prime}}+o(1)\right)\|\varphi_{n}\|_{p^{\prime}}^{p^{\prime}}\qquad(n\to\infty)$

So we conclude that $(\varphi_{n})$ is bounded. Furthermore,
$\lambda_{n}\to\lambda^{*}>0$ gives

$\limsup_{n\to\infty}\,(I_{\lambda_{*}}(\varphi_{n})-I_{\lambda_{n}}(\varphi_{n}))=\limsup_{n\to\infty}\,(\lambda_{n}^{1-p^{\prime}}-\lambda_{*}^{1-p^{\prime}})\|\varphi_{n}\|^{p^{\prime}}_{p^{\prime}}=0,$

which is all we had to show. ∎

We thus conclude that Theorem 4.2 applies in our context and yields BPS
sequences for $\tilde{J}_{\lambda}$ at the corresponding Mountain pass levels
$c_{\lambda}$ for almost all $\lambda\in(\lambda_{0},\infty)$. From the
existence of BPS sequences we deduce rather easily the existence of critical
points at the corresponding Mountain Pass level.

######  Proposition 4.5.

Let ${\mathcal{K}}:L^{p^{\prime}}(\mathbb{R}^{N})\to L^{p}(\mathbb{R}^{N})$ be
compact and assume
$\int\limits_{\mathbb{R}^{N}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq
0$ for all $\psi\in L^{p^{\prime}}(A_{-})$. Then for all
$\lambda\in(\lambda_{0},\infty)$ every BPS sequence of $\tilde{J}_{\lambda}$
at its Mountain Pass level $c_{\lambda}$ converges to a critical point of
$\tilde{J}_{\lambda}$ at the level $c_{\lambda}$.

###### Proof.

Let $(\varphi_{j})_{j}$ in $L^{p^{\prime}}(A_{+})$ be a BPS sequence for
$\tilde{J}_{\lambda}$, i.e., $\tilde{J}_{\lambda}(\varphi_{j})\to c>0$ and
$\tilde{J}_{\lambda}^{\prime}(\varphi_{j})\to 0$. We may thus assume w.l.o.g.
$\varphi_{j}\rightharpoonup\varphi^{*}$. Moreover, Proposition 3.1 (i) implies
the boundedness of $(\psi_{j})_{j}:=(Z(\varphi_{j}))_{j}$ and hence w.l.o.g.
also weak convergence. For all $h\in L^{p^{\prime}}(A_{+})$ we then have, in
view of the formula for $\tilde{J}_{\lambda}^{\prime}$ from Proposition 3.1
(iii),

$\displaystyle\left|\int_{\mathbb{R}^{N}}|\varphi_{j}|^{p^{\prime}-2}\varphi_{j}h-|\varphi_{k}|^{p^{\prime}-2}\varphi_{k}h\>\mathrm{d}x\right|$
$\displaystyle\quad=\left|\tilde{J}_{\lambda}^{\prime}(\varphi_{j})h-\tilde{J}^{\prime}(\varphi_{k})h+\int_{\mathbb{R}^{N}}h\mathcal{K}[\varphi_{j}-\varphi_{k}]\>\mathrm{d}x-\int_{\mathbb{R}^{N}}h\mathcal{K}[\psi_{j}-\psi_{k}]\>\mathrm{d}x\right|$
$\displaystyle\quad\leq\|h\|_{p^{\prime}}\cdot\left[\|\tilde{J}_{\lambda}^{\prime}(\varphi_{j})\|+\|\tilde{J}_{\lambda}^{\prime}(\varphi_{k})\|+\|\mathcal{K}[\varphi_{j}-\varphi_{k}]\|_{p}+\|\mathcal{K}[\psi_{j}-\psi_{k}]\|_{p}\right]$
$\displaystyle\quad=\|h\|_{p^{\prime}}\cdot o(1)\qquad(j,k\to\infty).$

We infer that $(|\varphi_{j}|^{p^{\prime}-2}\varphi_{j})_{j}$ converges
strongly in $L^{p}(A_{+})$. By uniqueness of weak limits, we infer
$|\varphi_{j}|^{p^{\prime}-2}\varphi_{j}\to|\varphi^{*}|^{p^{\prime}-2}\varphi^{*}$
strongly in $L^{p}(A_{+})$ and hence in particular
$\|\varphi_{j}\|_{p^{\prime}}\to\|\varphi^{*}\|_{p^{\prime}}$. This finally
implies $\varphi_{j}\to\varphi^{*}$ strongly in $L^{p^{\prime}}(A_{+})$. A
standard computation finally shows
$\tilde{J}_{\lambda}(\varphi^{*})=c_{\lambda}$ as well as
$\tilde{J}_{\lambda}^{\prime}(\varphi^{*})=0$. ∎

##  5 Proof of Theorem 1.1 and Corollary 1.2

We finally combine all auxiliary results to prove Theorem 1.1.

###### Proof of Theorem 1.1.

From Proposition 2.1 and Proposition 3.1 we infer that for almost all
$\lambda\in(\lambda_{0},\infty)$ a nontrivial solution $u\in
L^{p}(\mathbb{R}^{N})$ of the nonlinear Helmholtz equation (1.9) is found once
we have proved the existence of nontrivial critical points of the reduced
functional $\tilde{J}_{\lambda}$ for almost all $\lambda\in M$ where $M$ is an
arbitrary compact subinterval of $(\lambda_{0},\infty)$. From Proposition 4.3
we infer that the family $(\tilde{J}_{\lambda})_{\lambda\in M}$ has the
Mountain Pass Geometry. Moreover, by Proposition 4.4, condition (H) holds. So
Theorem 4.2 yields for almost all $\lambda\in M$ a BPS sequence for
$\tilde{J}_{\lambda}$ at the corresponding Mountain Pass level. By Proposition
4.5 each of these BPS sequences converges to a critical point
$\varphi_{\lambda}$ of $\tilde{J}_{\lambda}$ at the Mountain Pass level. Since
this critical point is necessarily nontrivial, we have thus obtained the
desired claim for $L^{p}(\mathbb{R}^{N})$-solutions of (1.9). From [4, Lemma
4.3] we infer that each of these solutions belongs to
$W^{2,q}(\mathbb{R}^{N})\cap{\mathcal{C}}^{1,\alpha}(\mathbb{R}^{N})$ for all
$p\leq q<\infty$ and $\alpha\in(0,1)$. Arguing as in Step 3 and Step 4 [8,
p.13] one even obtains that these solutions belong to
$W^{2,q}(\mathbb{R}^{N})$ for all $q\in(\frac{2N}{N-1},p)$. In particular,
these solutions are strong solutions of (1.9), which finishes the proof. ∎

###### Proof of Collorary 1.2.

We will prove that $\delta:=\operatorname*{diam}{(A_{-})}\leq
y_{\frac{N-2}{2}}$ implies
$\int\limits_{\mathbb{R}^{N}}\psi{\mathcal{K}}\psi\leavevmode\nobreak\ dx\geq
0$ for all $\psi\in L^{p^{\prime}}(A_{-})$. Note that due to (2.2),
$\mathcal{K}=|Q|^{1/p}\mathcal{R}(|Q|^{1/p}\cdot)$ and $Q\in
L^{\infty}(\mathbb{R}^{N})$ it suffices to prove

(5.1) $\int\limits_{\mathbb{R}^{N}}\psi\textbf{R}\psi\leavevmode\nobreak\
dx\geq 0,\qquad\text{ for all }\psi\in\mathcal{S}(A_{-}).$

Using that $x,y\in A_{-}$ implies $x-y\in B_{\delta}$ and Corollary 5.4 in [2]
we infer

$\int\limits_{\mathbb{R}^{N}}\psi\textbf{R}\psi\leavevmode\nobreak\
dx=\int\limits_{\mathbb{R}^{N}}\psi[\mathds{1}_{B_{\delta}}\Psi\ast\psi](x)\leavevmode\nobreak\
dx\geq 0,$

which proves the nonnegativity. Moreover, $\operatorname{esssup}_{|x|\geq
R}|Q(x)|\to 0$ implies the compactness of $\mathcal{K}$ as was shown in Lemma
4.2 in [4]. So the assumptions of Theorem 1.1 are satisfied and the result
follows. ∎

## References

  * [1] M. Abramowitz and I. A. Stegun.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables.  Dover, New York, ninth dover printing, tenth gpo printing edition, 1964\.
  * [2] H. Chen, G. Evéquoz, and T. Weth.  Complex solutions and stationary scattering for the nonlinear Helmholtz equation.  arXiv:1911.09557, 2019.
  * [3] G. Evéquoz and T. Weth.  Real solutions to the nonlinear Helmholtz equation with local nonlinearity.  Arch. Ration. Mech. Anal., 211(2):359–388, 2014.
  * [4] G. Evéquoz and T. Weth.  Dual variational methods and nonvanishing for the nonlinear Helmholtz equation.  Advances in Mathematics, 280:690–728, 2015.
  * [5] G. Evéquoz and T. Yeşil.  Dual ground state solutions for the critical nonlinear Helmholtz equation.  Proceedings of the Royal Society of Edinburgh Section A: Mathematics, 150(3):1155–1186, 2020.
  * [6] L. Jeanjean and J. F. Toland.  Bounded Palais-Smale mountain-pass sequences.  C. R. Acad. Sci. Paris Sér. I Math., 327(1):23–28, 1998.
  * [7] C. E. Kenig, A. Ruiz, C. D. Sogge, et al.  Uniform Sobolev inequalities and unique continuation for second order constant coefficient differential operators.  Duke Mathematical Journal, 55(2):329–347, 1987.
  * [8] R. Mandel.  Uncountably many solutions for nonlinear Helmholtz and curl-curl equations.  Adv. Nonlinear Stud., 19(3):569–593, 2019.
  * [9] R. Mandel, E. Montefusco, and B. Pellacci.  Oscillating solutions for nonlinear Helmholtz equations.  Z. Angew. Math. Phys., 68(6):Paper No. 121, 19, 2017.
  * [10] A. Szulkin and T. Weth.  The method of Nehari manifold.  In Handbook of nonconvex analysis and applications, pages 597–632. Int. Press, Somerville, MA, 2010.