scale=0.9, color=black, opacity=0.6, angle=0, position=current page.north,
vshift=-1.5cm, hshift=0cm, contents=  Accepted at 2024 IEEE International
Conference on Robotics and Automation (ICRA 2024): © 2024 IEEE. Personal use
of this material is permitted. Permission from IEEE must be obtained for all
other uses, in any current or future media, including reprinting/republishing
this material for advertising or promotional purposes, creating new collective
works, for resale or redistribution to servers or lists, or reuse of any
copyrighted component of this work in other works.

# Are you a robot? Detecting Autonomous Vehicles from Behavior Analysis

Fabio Maresca∗, Filippo Grazioli∗, Antonio AlbaneseΨ,
Vincenzo Sciancalepore∗, Gianpiero Negri†, Xavier Costa-Perez‡ ∗F. Maresca, F.
Grazioli and V. Siancalepore are with NEC Laboratories Europe GmbH
<EMAIL_ADDRESS>Albanese is with Flyhound Co.
<EMAIL_ADDRESS>Costa-Perez is with i2CAT Foundation, NEC
Laboratories Europe GmbH and ICREA<EMAIL_ADDRESS>Negri is with
Amazon Global Robotics - EU Innovation Lab<EMAIL_ADDRESS>work was
partially supported by EU FP for Research and Innovation Horizon 2020 under
Grant Agreements No. 956670 (5GSmartFact) and No. 101139130 (6G-DISAC).

###### Abstract

The tremendous hype around autonomous driving is eagerly calling for emerging
and novel technologies to support advanced mobility use cases. As car
manufactures keep developing SAE level 3+ systems to improve _the safety and
comfort_ of passengers, traffic authorities need to establish new procedures
to manage the transition from human-driven to fully-autonomous vehicles while
providing a feedback-loop mechanism to fine-tune envisioned autonomous
systems. Thus, a way to automatically profile autonomous vehicles and
differentiate those from human-driven ones is a _must_.

In this paper, we present a fully-fledged framework that monitors active
vehicles using camera images and state information in order to determine
whether vehicles are autonomous, without requiring any active notification
from the vehicles themselves. Essentially, it builds on the cooperation among
vehicles, which share their data acquired on the road feeding a machine
learning model to identify autonomous cars. We extensively tested our solution
and created the NexusStreet dataset, by means of the CARLA simulator,
employing an autonomous driving control agent and a steering wheel maneuvered
by licensed drivers. Experiments show it is possible to discriminate the two
behaviors by analyzing video clips with an accuracy of $\sim\\!80\%$, which
improves up to $\sim\\!93\%$ when the target’s state information is available.
Lastly, we deliberately degraded the state to observe how the framework
performs under non-ideal data collection conditions.

##  I Introduction

With the advent of emerging communication technologies, the automotive sector
has exhibited relevant industrial profits towards the so-called revolution of
_connected and autonomous cars_. This digital transformation enables advanced
services drawing attention to two crucial societal aspects, such as safety of
road users and sustainability of involved infrastructure, thereby delaying the
transition to connected roads with fully-autonomous cars as part of the SAE
level 3+: this implies vehicles implementing an environmental perception
module [1]. Within this transition phase, it appears clear a need for the
coexistence of human drivers and autonomously-driven vehicles: computers will
take over the maneuvering control of the cars to take specific actions (e.g.,
highway driving or parking), while giving it back upon critical situations
[2].

Figure 1: Envisioned vehicular communication scenario. Vehicles report state
information and collectively identify autonomous cars.

Such mixed traffic conditions [3] will involve a number of severe safety
issues that must be tackled [4]: autonomously-driven vehicles will slowly
build the _risk perception_ [5] exacerbating safeguarding concerns; it is
crucial to spot autonomous cars with (potential) inconsistent behaviors [6].
We propose an overarching framework to automatically detect whether a vehicle
is autonomous or human-driven based on gathered behavioral features and to
provide feedbacks for improving autonomous systems: it benchmarks established
machine learning models trained on temporal series of images and additional
numeric features as the state information of the target vehicle to classify.
To tackle this problem, we built and exploited a dataset that collects human
and autonomous driving scenes, in the form of videos, i.e., sequences of RGB
images. These are recorded by the EGO vehicle (EV) following a target vehicle
(TV), whose state, collected alongside, is characterized by speed,
acceleration, rotation, distance from the EV and distance from lane center. We
have built the dataset using the CARLA simulator [7] and the Baidu Apollo
autonomous driving agent to control the target in the autonomous settings [8,
9], whereas the human driving behavior has been acquired by manually steering
the TV with a Logitech G29 steering wheel. The dataset is dubbed as
NexusStreets111NexusStreets is inspired by the Nexus series of replicants in
Blade Runner, who were designed to be nearly indistinguishable from humans.
The name emphasizes the idea of a mix of human-driven and autonomous vehicles
on the same streets.  and is publicly released on Zenodo at
https://zenodo.org/records/7682484.

###  I-A Related Work

A useful taxonomic characterization of behavior prediction methods is based on
the type of input features employed by the models [10]. In [11, 12], the
authors propose machine learning models that predict vehicle behavior from the
track history of the TVs states. The accuracy of such methods is limited by
the performance of the EV’s perception module. Note that they do not consider
vehicle-vehicle, nor vehicle-environment interaction patterns. In [13] the
authors propose a deep learning framework to automatically learn regular
mobile traffic patterns along roads thereby detecting non-recurring events and
classify them by severity level to assist the emergency operations and support
advanced services provided by the network operator.

When other surrounding vehicles’ track histories are considered, patterns are
extracted from the interactions between them and the target vehicles [14, 15].
Analogously to the works presented above, these methods’ performance is
limited by the accuracy of the EV’s perception that provides the track
histories. However, they can learn from a richer input representations and
hence capture more complex dynamics.

Figure 2: Illustration of the Autonomous Vehicle Detection Methodology

To account for vehicle-environment interactions, a further class of methods
inputs simplified bird’s eyes views of the scene [16, 17]. Finally, [18, 19]
propose to directly predict vehicle behavior from the raw sensor data to
overcome the information loss injected by the perception stack.

To the best of our knowledge, no existing work investigates the problem of
spotting autonomous vehicles from observations providing feedbacks on
inconsistent behaviors.

###  I-B Contributions

The paper contributions can be summarized as follows: $i$) we generate a novel
dataset combining autonomous and human driving scenes, $ii$) we present
benchmark results of various machine learning models on the driving scene
classification task, $iii$) we model the problem as a supervised
classification of multivariate time series describing the temporal evolution
of the Target Vehicle (TV) state, including a second multivariate time series
representing 2D bounding box detection of the TV in the pixel coordinates
space over time, $iv$) we carry out an exhaustive simulation campaign to
validate our framework, $v$) we make the NexusStreets dataset publicly
available to the scientific community. Fig. 2 provides a schematic
illustration of the full pipeline.

Our experiments show that it is possible to discriminate with high accuracy
($>0.95$ auROC) whether a TV is autonomous or human-driven alone from the
observation of its driving behavior in the form of state and 2D object
detection.

##  II Dataset Generation

###  II-A Simulation Environment and Considered Scenarios

Autonomous and manual driving scenes have been collected using a nearly-
realistic simulator, namely CARLA [7] used in driving-related research. It
offers access to the vehicles’ sensors and formally describes the roads
according to OpenDRIVE specifications [20]. It also provides interfaces with
external tools and software, e.g., ROS [21, 22], and autonomous driving agents
to control vehicles. Additionally, it allows the integration of external
steering wheels to directly drive the vehicles: human behaviors have been
reproduced with a Logitech G29 steering wheel, which supports realistic force
feedback and first-person driving experience.

Five default maps are available offering a variety of streets, from narrow to
larger ones, up to a $5$-lanes highway. Each map is characterized by different
curves, traffic signs and lights dispositions. For each city, we considered
different levels of traffic, spanning from no traffic to heavy traffic
conditions, where a hundred non-player characters (NPCs) spawn randomly in the
simulated environment. Every scene is repeated with $10$ different weather
conditions allowing to acquire video sequences in a richer spectrum of light
and visibility conditions222The weather conditions do not affect the physics
of the simulator, nor the actors and NPCs’ behaviors. Different weather
conditions have been considered to make the detection task (section II-D)
closer to the reality..

Figure 3: Schematic illustration of the carla-apollo bridge architecture

###  II-B Description of the Main Actors

We assume in our dataset two main actors:

  * •

Target vehicle (TV), the leading vehicle for which we acquire state
information and whose behavior we aim to classify;

  * •

EGO vehicle (EV), the vehicle following and monitoring the TV. It records the
scenarios with a monocular RBG front-camera and, ideally, it retrieves and
computes data by means of additional sensors (radar, LiDAR, etc.) and/or
external sources, e.g., other communicating vehicles and objects in smart
cities.

Figure 4: $T$ and $E$ are the two reference systems for TV and EV,
respectively. Their origins lay in their center of mass. For simplicity, we
only depict the $x$\- and $y$-axis, however the $z$-axis is also considered in
the simulations. The TV state is constituted by its position vector $\bm{d}$,
speed $\dot{\bm{d}}$ and acceleration $\ddot{\bm{d}}$ w.r.t. the $E$ frame, as
well as the distance from the lane center $l$ and its yaw angle $\omega$
calculated in $T$. Moreover, we use the reference system $F$ to represent the
2D bounding box coordinates. $F(0,0)$ corresponds to the upper left pixel of
each 960x540 JPEG image acquired by the EV camera.  Figure 5: Distribution of
each TV state feature, highlighting $25$th quartile, median, $75$th quartile,
and values falling outside such two boundaries.

The dataset properly combines autonomous driving scenes wherein the TV is
implemented using the Baidu Apollo [8, 9] autonomous driving agent, and human-
driven scenes wherein the TV is controlled by skilled people directly driving
with a steering wheel in a first-person perspective [23].

The EV follows the TV via a PID controller that mimics the TV’s path by
automatically adjusting its speed basing on the distance from the TV.

###  II-C Sampling Scenes and Target States

The dataset consists of 520 scenes, i.e., 260 pairs of scenes where the TV is
controlled by the autonomous agent and the human driver in identical
scenarios. Two $60$s runs for every (city, traffic, weather) combination
result in nearly $9$ hours of total driving time. For each timestamp, the
$960\times 540$ RGB image captured by the EV’s front camera is acquired,
together with the TV state provided by the simulator.

Each video is truncated at $60$s to ensure uniform duration. Additionally, we
down-sample the scenes and the acquired states to the minimum sampling rate,
guaranteeing a correspondence between the images and the state at all times.
This results in $\sim\\!2$fps and all scenes present $120$ timestamps.

At each sampling timestamp, the EV captures the TV state
$\bm{x^{S}}=\big{[}\|\bm{d}\|_{2},\|\dot{\bm{d}}\|_{2},\|\ddot{\bm{d}}\|_{2},l,\omega\big{]}^{T}$,
where $\bm{d}\in\mathbb{R}^{3}$ is the position vector of TV in EV-centered
reference system $E$, $\|\cdot\|_{2}$ denotes the $L^{2}$-norm of the vector,
$l\in\mathbb{R}$ is the distance of the TV from the lane center w.r.t. the TV-
centered reference system $T$ and $\omega\in\mathbb{R}$ is the TV’s yaw angle.
We denote the time derivative with the dot-notation, i.e., $\dot{\bm{d}}$ is
the speed vector and $\ddot{\bm{d}}$ is the acceleration. Fig. 4 provides a
depiction of the $E$ and $T$ reference systems.

While the TV state $\bm{x^{S}}$ is provided by the simulator, in real-life
scenarios it could be estimated from the EV’s perception stack, or also shared
by other surrounding vehicles and objects by means of Vehicle-to-Everything
(V2X) communications. Interestingly, multiple followers can monitor and
profile the same target vehicle while exchanging acquired information. Figs. 5
and 6 depict the distributions of the features that constitute the TV state.

Figure 6: Trends of the state features over time. The maximum value $15,600$s
on the x-axis corresponds to the concatenation of $260$ scenes lasting $60$s
each.

###  II-D 2D Target Vehicle Detection

In addition to the TV state, we provide 2D object detections in the form of
bounding box coordinates w.r.t the reference system $F$ at each timestamp (see
Fig. 4). We operate 2D object detection using a pre-trained YOLOv7 [24] model.
This convolutional neural network (CNN) outputs bounding boxes and class
probabilities for each object detected in the image. For each acquired frame,
we consider objects labeled as “cars” with a confidence $\geq 0.9$ and ignore
other classes. We store the detection as vectors
$\bm{x^{D}}=\big{[}x,y,h,w\big{]}$, where $(x,y)$ are the bounding box center
coordinates, $h$ its height and $w$ its width. For each frame, multiple
vehicles are detected. It is therefore necessary to identify the bounding box
of the TV. To achieve this goal, for each weather and light condition, we
first collect a set of $2$ images (as template) depicting the TV rear and
side. Then, we embed the template images using a ResNet-50v2 [25] model
trained on ImageNet [26]. Embedding vectors are obtained from the activation
of the final pre-softmax fully-connected layer. Afterwards, for each frame, we
embed all detection crops and then compute their average Euclidean distance
from the TV template. The detection crop whose embedding presents the smallest
average Euclidean distance is designated as the true TV
detection333Analogously, to a real-world perception module, this approach is
not exempted from noise. Nevertheless, the combination of YOLO and
ResNet-50v2-based template matching is able to detect the TV most of the
times. Mistakes are considered outliers given their small numbers..

##  III Experiments and Discussion

Hereafter, we show experimental results for the vehicle behavior
classification task, which is modeled as a supervised learning problem. We
then present empirical results for the future state prediction process, which
is modeled as an autoregressive problem.

###  III-A Vehicle Behavior Classification

We first present a formalization of the problem, then we compare different
machine learning models performances for the behavior classification task.
Specifically, this is based on the history of the target vehicle (TV) state,
2D detection in the pixel coordinates space and a combination of both.
Finally, we show how information loss in the TV state leads to a performance
degradation at the classifier.

Problem formulation. Formally, for a given scene $i$, let us define the whole
history of TV state vectors as the following:

$\bm{X}^{S}_{i}=\Big{[}\bm{x}^{S}_{i,t_{1}},\dots,\bm{x}^{S}_{i,t_{T_{i}}}\Big{]}^{T}\in\mathbb{R}^{T_{i}\times
5}\>\>,$ (1)

where $\bm{x}^{S}_{i,t}$ is the state vector at timestamp $t$ and $T_{i}$
represent the number of available timestamps. Analogously, let us define the
whole history of 2D detections for a given scene as the following

$\bm{X}^{D}_{i}=\Big{[}\bm{x}^{D}_{i,t_{1}},\dots,\bm{x}^{D}_{i,t_{T_{i}}}\Big{]}^{T}\in\mathbb{R}^{T_{i}\times
4}\>\>.$ (2)

To each scene, we assign a binary label $y_{i}\in\\{0,1\\}$, whose value
represents whether the TV is classified as autonomous or human-driven.

Given a total number of scenes $N$, we can now define two datasets, i.e., the
state information and the 2D detection information in the pixel coordinates
space, accordingly:

$\displaystyle\mathcal{X}^{S}=\Big{\\{}\Big{(}\bm{X}^{S}_{i},y_{i}\Big{)}\Big{\\}}^{N}_{i=1};\qquad\mathcal{X}^{D}=\Big{\\{}\Big{(}\bm{X}^{D}_{i},y_{i}\Big{)}\Big{\\}}^{N}_{i=1}.$
(3)

Analogously, considering the concatenation of the state vectors with the 2D
detections, we can define a joint dataset as follows:

$\mathcal{X}^{S+D}=\Big{\\{}\Big{(}\Big{[}\bm{X}^{S}_{i},\bm{X}^{D}_{i}\Big{]},y_{i}\Big{)}\Big{\\}}^{N}_{i=1}.$
(4)

For the three settings considered, i.e., state-only (S), detection-only (D)
and both (S+D), our goal is to estimate the true probability distribution
$p(y|\bm{X})$ by a parametric approximation $q_{\bm{\theta}}(y|\bm{X})$, such
that

$\displaystyle q_{\bm{\theta}}(y|\bm{X})\approx p(y|\bm{X}),$ (5)

where $\bm{\theta}$ is the set of parameters. Hereafter, we describe the two
machine learning models employed in our experiments for the estimation of
$q_{\bm{\theta}}(y|\bm{X})$.

Machine Learning Models. We benchmark two different machine learning models,
namely a Random Forest (RF) [27] and a deep neural network based on Long
Short-Term Memory (LSTM) cells [28].

Figure 7: Vehicle behavior classification results. These have been obtained by
applying RF and LSTM models to the datasets $\mathcal{X}^{S}$ and
$\mathcal{X}^{D}$ separately. Reported values are test mean scores over $5$
repeated experiments. Confidence intervals are standard deviations.

For both models, we perform hyper-parameter optimization via a $5$-fold
stratified cross-validated grid search. Given a training/test split, only the
training set is used for the hyper-parameter optimization in order to avoid
information leak. Each hyper-parameter configuration obtains $5$ validation
scores. The configuration that achieves the maximum average area under the
receiver operator characteristic curve (auROC) is selected. A new model with
the optimal configuration is trained on the training set and tested on the
left-out test set.

For the RF, we optimize the number of trees over $\\{100k\>|\>1\leq k\leq
10\\}$, the minimum number of samples required to be at a leaf node over
$\\{k\>|\>1\leq k\leq 5\\}$, the function to measure the quality of a split
over $\\{\text{Gini impurity},\text{Shannon entropy}\\}$ and the number of
features to consider when looking for the best split over
$\\{\sqrt{n},\log_{2}n\\}$, where $n$ is the number of input features. Since
the RF inputs vectors, we unroll the input matrices $\bm{X}$ to $1$D vectors.

For the LSTM-based model, we optimize the number of stacked LSTM cells over
$\\{1,4,8\\}$, the drop-out rate over $\\{0.1,0.3\\}$ and the output dimension
of the LSTM cells over $\\{32,64,128\\}$. The LSTM-based network is trained
for a maximum of $1000$ epochs by optimizing a binary cross-entropy loss,
employing the Adam optimizer and early stopping with $5$ epochs patience to
monitor the validation auROC. The batch size was set to $256$ and the learning
rate to $10^{-4}$. ReLU activations have been employed in the model and a
final sigmoid function was used for the output.

Performance Evaluation. With the goal of providing an unbiased estimate of the
models classification performance, we perform $5$ independent random
training/validation/test splits. For a given split, $70\%$ of the scenes
constitute the training set, 10% the validation set and $20\%$ the test set.
Test scores are then averaged over the $5$ repeated experiments.

Sliding Windows Sampling. The samples contained in the datasets
$\mathcal{X}^{S}$, $\mathcal{X}^{D}$ and $\mathcal{X}^{S+D}$ are constituted
by the full history of states and 2D detection associated to the scenes, which
span $60$s and present $120$ timestamps each. Given that our aim is to make
predictions based on shorter time intervals, we sample time windows of shorter
duration from each scene. First, we define $5$ values for time window
duration: $5$s, $10$s, $15$s, $20$s and $25$s. Then, for each window size, we
apply a sliding window approach and extract sub-scenes from the whole dataset
of the defined time duration. We train the machine learning models on the
obtained sub-scenes. This allows to deeply observe how the scene duration can
affect the models performance.

Vehicle Behavior Classification Results. Fig. 7 shows results achieved by the
RF and the LSTM-based model on the behavior classification task. We report as
relevant KPIs the following: auROC, area under the precision-recall curve
(auPR), accuracy, recall, precision and F1 score. We train and test the models
on both the $\mathcal{X}^{S}$ and $\mathcal{X}^{D}$ datasets and show results
for all considered time window duration values.

Figure 8: Effect of state information loss across various drop rates. The
degradation has been simulated by obfuscating state information from the
$X^{S+D}$ dataset with various drop rates. $r=0$ corresponds to the best case
scenario where all the data is intact. Test scores are averaged over $5$
repeated experiments.

We observe that, as expected, the states allow for better generalization
performance than the 2D detection in the pixel coordinates space. Overall, the
RF outperforms the LSTM-based model. For the RF, we observe that test scores
improve as the time window duration increases. Conversely, the LSTM-based
model achieves its peak performance on $30$ timestamps for state-based
predictions, and $40$ timestamps for detection-based predictions. Both models
maintain a similar performance gap between $\mathcal{X}^{S}$ and
$\mathcal{X}^{D}$ across all time window duration values.

Figure 9: Effect of different history and future horizon lengths. Test results
for future state autoregression are averaged over $5$ repeated experiments.
Error bars represent standard deviation. RMSE: root mean squared error. Time
duration is quantified in terms of timestamp number. $1$ timestamp is
approximately $0.5$s.

Effect of Target State Degradation. For this experiment campaign, we degrade
the state information from the $\mathcal{X}^{S+D}$ dataset to test the system
robustness against sensor failures and/or disruptions in communication, such
as delays or disconnections. Please note that 2D detection information is not
perturbed.

Given a drop rate $r\in(0,1)$, we randomly obfuscate a portion of rows in
$\bm{X}^{S}$ defined by $r$, while leaving $\bm{X}^{D}$ unaffected. In
particular, the state vectors corresponding to the selected rows are set to
NaN at test time. Subsequently, for the affected rows, the models consider the
latest known state for the classification. Fig. 8 highlights the performances
of both models in response to different drop rate values
$\\{0.2,0.4,0.6,0.8\\}$. The results are compared to the ideal scenario, i.e.,
$r=0.0$, which represents the upper limit. We observe that RF test scores
results in a $\sim 3\%$ of performance drop. Analogously, the LSTM-based model
faces a $\sim 5\%$ of performance degradation.

All in all, the car detection task is successfully executed demonstrating the
feasibility of our proposed approach while keeping the overall solution
complexity affordable. The output of this task can be fed into the autonomous
control system pursuing a settings fine-tuning while minimizing the difference
between a skilled-human driven behaviors and consistent autonomously driving
operations.

###  III-B Future State Autoregression

In the following, we present experimental results for the future state
autoregression task. The term “auto” in autoregressive indicates that a
variable is regressed against itself [29]. We first present a formalization of
the problem. Then, we show the effect of different past (i.e., history) and
future horizon lengths on the prediction error. Finally, we compare the
performance achieved by the model on scenes wherein the target vehicle (TV) is
controlled by the autonomous agent against scenes in which the vehicle is
driven by a human.

Problem Formulation. Given a history of $H$ past timestamps and a future
horizon defined by $F$ timestamps, our goal is to learn an autoregressive
parametric function $f_{\bm{\theta}}:\mathbb{R}^{H\times
5}\xrightarrow{}\mathbb{R}^{F\times 5}$ such that

$\hat{\bm{X}}_{H:F}^{S}=f_{\bm{\theta}}\big{(}\bm{X}_{1:H}^{S}\big{)},$ (6)

where $\bm{\theta}$ are learnable parameters and
$\bm{X}_{A:B}^{S}=\big{[}\bm{x}^{S}_{i,t_{A}},\dots,\bm{x}^{S}_{i,t_{B}}\big{]}^{T}\in\mathbb{R}^{(B-A)\times
5}$ represents the state vectors of a scene from timestamp A to B.

We train the model by optimizing a mean squared error loss, as follows:

$\min_{\bm{\theta}}L\big{(}\bm{\theta}\big{)}=\frac{1}{N}\sum_{i=1}^{N}\Big{(}f_{\bm{\theta}}\big{(}\bm{X}_{i,1:H}^{S}\big{)}-\bm{X}_{i,H:F}^{S}\Big{)}^{2}\>\>.$
(7)

Implementation Details. Specifically, we implement $f_{\bm{\theta}}$ as a
multi-layer perceptron (MLP) with 3 layers. We unroll the $\bm{X}_{1:H}^{S}$
and $\bm{X}_{H:F}^{S}$ matrices and treat them as 1D vectors. The dimension of
the $2$ hidden layers is set to $32$ and $16$ accordingly, while the dimension
of the input and output is set to $5H$ and $5F$, respectively. We employ
sigmoid activations and train the network with the Adam optimizer with an
initial learning rate of $10^{-4}$. We set a maximum of $10,000$ epochs and
adopt early stopping with $10$ patience epochs monitoring a validation score.

Performance Evaluation and Data Preparation. Analogously to the experiments
described above, we perform $5$ independent training/validation/test splits,
so that $70\%$ of the scenes is employed at training time, $10\%$ for
validation and $20\%$ for test. After employing the sliding window sampling as
described in the previous section, we extract sub-scenes from the full scenes
set. For the sampling, we set the window size to $20$ timestamps, i.e. $\sim
10$s. We train a model for each combination of history length
$H\in\\{1,2,4,6,8,10,12\\}$ and future horizon length $F\in\\{1,2,4,6,8\\}$.

Figure 10: Future state autoregression error for autonomous agent and human
driver. Error bars represent the standard deviation over $5$ repeated
experiments with different training/test splits.

Effects of History and Future Horizon Lengths. We depict in Fig. 9 the
corresponding effect of multiple history and future horizon lengths on the
prediction error of the TV future state. We observe that, as the number of
timestamps that constitute the future horizon length increases, the prediction
error tends to increase. Interestingly, while the input history length
increases, the system experiences a little performance improvement, due to
more available information to rely on.

Comparison of Autonomous and Human Driving. Finally, in Fig. 10 we compare the
prediction error achieved by the same autoregressive model for both autonomous
and human driving scenes. Results are averaged over all considered future
horizon lengths. The results show that the TV future state is significantly
easier to predict when a human is driving compared to the case when the Baidu
Apollo agent is controlling the vehicle (autonomous). This is in line with our
subjective assessment of the scenes in which the TV is controlled by the
autonomous agent. It appears, in fact, that the agent often performs abrupt
braking maneuvers and accelerations, which make its overall state harder to
predict compared to the smoother driving style of a human driver, as well as
less safe due to potential hazards caused by the sudden change of speed and/or
direction.

##  IV Risk-based Training Process

An additional result from the comparison described in the previous section is
that the lower predictability of the autonomous behavior can result in an
increased exposure to the safety risks, as it leads to abrupt braking and/or
sudden acceleration or deceleration. This is usually considered among the
major causes of hazards for the bystanders and the EV passengers [30], as an
exhaustive hazards analysis and risk assessment conducted according to ISO
26262 [31] could highlight. The higher predictability of human driving could
result from the inherent human capability to perceive the risk in diverse
scenarios [32], which suggests the idea to perform an iterative training
process of the autonomous driving control algorithm using as objective
function the difference between autonomous and human driver prediction errors
or similar metrics based on risk estimation, to finally minimize unpredictable
behavior effects with subsequent safety risks. This iterative training process
is out of the scope of this paper and will be the subject of future works.
Future works will also seek to integrate data sourced from real vehicles,
validating its suitability for real world applications.

##  V Conclusions

In this paper, we have presented a novel autonomous vehicle detection solution
capable of identifying whether vehicles are driven by humans or computers. We
generated and released publicly a custom dataset for this purpose,
NexusStreet. It combines both autonomous-driven scenes, obtained on the CARLA
simulator by means of Baidu Apollo control agent, and human-driven ones,
acquired by using a steering wheel maneuvered by licensed (and skilled) human
drivers. After training various machine learning models, we show that it is
possible to automatically detect autonomous vehicles by evaluating video
segments. Accuracy and auROC improves to $+90\%$ when combined with histories
of the state vector, i.e., position, speed, acceleration and orientation,
locally estimated by all installed sensors. Moreover, information as state
vector and local classification can be shared among road users through
Vehicle-to-everything (V2X)-based infrastructure, further increasing the
overall autonomous vehicles detection accuracy to fine-tune the car autonomous
control system.

## References

  * [1] Fábio Duarte and Carlo Ratti.  The impact of autonomous vehicles on cities: A review.  Journal of Urban Technology, 25(4):3–18, 2018.
  * [2] Navreet Virdi, Hanna Grzybowska, S Travis Waller, and Vinayak Dixit.  A safety assessment of mixed fleets with connected and autonomous vehicles using the surrogate safety assessment module.  Accident Analysis & Prevention, 131:95–111, 2019.
  * [3] Parth Bhavsar, Plaban Das, Matthew Paugh, Kakan Dey, and Mashrur Chowdhury.  Risk analysis of autonomous vehicles in mixed traffic streams.  Transportation Research Record, 2625(1):51–61, 2017.
  * [4] Jinyun Zhou, Rui Wang, Xu Liu, Yifei Jiang, Shu Jiang, Jiaming Tao, Jinghao Miao, and Shiyu Song.  Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasterization.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1450–1457, 2021.
  * [5] Liang Peng, Boqi Li, Wenhao Yu, Kai Yang, Wenbo Shao, and Hong Wang.  Sotif entropy: Online sotif risk quantification and mitigation for autonomous driving.  IEEE Transactions on Intelligent Transportation Systems, 2023.
  * [6] Manzoor Hussain, Nazakat Ali, and Jang-Eui Hong.  Deepguard: a framework for safeguarding autonomous driving systems from inconsistent behaviour.  Automated Software Engineering, 29(1):1, 2022.
  * [7] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017.
  * [8] Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang, Yuanqing Lin, and Ruigang Yang.  The apolloscape dataset for autonomous driving.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition workshops, pages 954–960, 2018.
  * [9] Baidu apollo team (2017), apollo: Open source autonomous driving.  https://github.com/ApolloAuto/apollo. Accessed: 2023-02-20.
  * [10] Sajjad Mozaffari, Omar Y Al-Jarrah, Mehrdad Dianati, Paul Jennings, and Alexandros Mouzakitis.  Deep learning-based vehicle behavior prediction for autonomous driving applications: A review.  IEEE Transactions on Intelligent Transportation Systems, 23(1):33–47, 2020.
  * [11] Alex Zyner, Stewart Worrall, and Eduardo Nebot.  Naturalistic driver intention and path prediction using recurrent neural networks.  IEEE Transactions on Intelligent Transportation Systems, 21(4):1584–1594, 2019.
  * [12] Long Xin, Pin Wang, Ching-Yao Chan, Jianyu Chen, Shengbo Eben Li, and Bo Cheng.  Intention-aware long horizon trajectory prediction of surrounding vehicles using dual lstm networks.  In 21st International Conference on Intelligent Transportation Systems (ITSC), pages 1441–1446. IEEE, 2018.
  * [13] Armin Okic, Lanfranco Zanzi, Vincenzo Sciancalepore, Alessandro Redondi, and Xavier Costa-Pérez.  $\pi$-road: a learn-as-you-go framework for on-demand emergency slices in v2x scenarios.  In IEEE INFOCOM - IEEE Conference on Computer Communications, pages 1–10, 2021.
  * [14] Frederik Diehl, Thomas Brunner, Michael Truong Le, and Alois Knoll.  Graph neural networks for modelling traffic participant interaction.  In IEEE Intelligent Vehicles Symposium (IV), pages 695–701. IEEE, 2019.
  * [15] Yuexin Ma, Xinge Zhu, Sibo Zhang, Ruigang Yang, Wenping Wang, and Dinesh Manocha.  Trafficpredict: Trajectory prediction for heterogeneous traffic-agents.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6120–6127, 2019.
  * [16] Donghan Lee, Youngwook Paul Kwon, Sara McMains, and J Karl Hedrick.  Convolution neural network-based lane change intention prediction of surrounding vehicles for acc.  In IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1–6. IEEE, 2017.
  * [17] Marcel Schreiber, Stefan Hoermann, and Klaus Dietmayer.  Long-term occupancy grid prediction using recurrent neural networks.  In International Conference on Robotics and Automation (ICRA), pages 9299–9305. IEEE, 2019.
  * [18] Wenjie Luo, Bin Yang, and Raquel Urtasun.  Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3569–3577, 2018.
  * [19] Sergio Casas, Wenjie Luo, and Raquel Urtasun.  Intentnet: Learning to predict intention from raw sensor data.  In Conference on Robot Learning, pages 947–956. PMLR, 2018.
  * [20] Marius Dupuis, Martin Strobl, and Hans Grezlikowski.  Opendrive 2010 and beyond–status and future of the de facto standard for the description of road networks.  In Proc. of the Driving Simulation Conference Europe, pages 231–242, 2010.
  * [21] Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, Andrew Y Ng, et al.  Ros: an open-source robot operating system.  In ICRA Workshop on Open Source Software, volume 3, page 5. Kobe, Japan, 2009.
  * [22] Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall.  Robot operating system 2: Design, architecture, and uses in the wild.  Science Robotics, 7(66):eabm6074, 2022.
  * [23] Malik Haris and Adam Glowacz.  Navigating an automated driving vehicle via the early fusion of multi-modality.  Sensors, 22(4):1425, 2022.
  * [24] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7464–7475, 2023.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009.
  * [27] Tin Kam Ho.  Random decision forests.  In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 1, pages 278–282. IEEE, 1995.
  * [28] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [29] Rob J Hyndman and George Athanasopoulos.  Forecasting: principles and practice.  OTexts, 2018.
  * [30] Torben Stolte, Gerrit Bagschik, Andreas Reschka, and Markus Maurer.  Hazard analysis and risk assessment for an automated unmanned protective vehicle.  In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 1848–1855, 2017.
  * [31] International organization for standardization, ISO 26262-1:2018 - Road vehicles — Functional safety — Part 1: Vocabulary, 2018.
  * [32] Sarvesh Kolekar, Joost de Winter, and David Abbink.  Human-like driving behaviour emerges from a risk-based driver model.  Nature Communications, 11(1), September 2020.


# Liouville type theorem of integral equation with anisotropic structure

Yating Niu

###### Abstract

In this paper, we classify all positive solutions for the following integral
equation:

$u(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(u(y))dy,$ (0.1)

where $b>1$ is a constant. Here $K_{b}(x,y)$ is the Green function of the
following homogeneous Neumann boundary problem

$\left\\{\begin{aligned} -\text{div}(x^{b}_{n}\nabla u)&=f\quad\text{in}\
\mathbb{R}^{n}_{+}\\\ \frac{\partial u}{\partial x_{n}}&=0\quad\text{on}\
\partial\mathbb{R}^{n}_{+}.\end{aligned}\right.$ (0.2)

By using the method of moving planes in integral form, we derive the symmetry
of positive solutions. We also establish the equivalence between the integral
equation and its corresponding partial differential equation. Similarly, the
results can be generalized to the integral system.

##  1 Introduction

In this paper, we study the positive solutions $u(x)$ of the following type of
integral equation

$u(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(u(y))dy$ (1.1)

and integral system

$\left\\{\begin{aligned}
u(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(v(y))dy\quad
x\in\mathbb{R}^{n}_{+}\\\
v(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}g(u(y))dy\quad
x\in\mathbb{R}^{n}_{+}.\end{aligned}\right.$ (1.2)

on upper half space $\mathbb{R}^{n}_{+}$, where $n\geq 3$.

In recent years, there has been great interest in using the method of moving
planes to classify the solutions of equation. It is a very powerful tool to
study the symmetry of solution. The method of moving planes for PDEs was
invented by the Alexandroff in the early 1950’s. Later, it was further
developed by Serrin [22] and Gidas, Ni and Nirenberg [9], [10]. In the paper
of Chen, Li [2], they studied the following partial differential equation

$\Delta u+u^{p}=0\quad x\in\mathbb{R}^{n}.$ (1.3)

They proved that for $p=\frac{n+2}{n-2}$ all the positive $C^{2}$ solutions of
(1.3) are radially symmetric about some point.

It is a natural question to ask whether similar results hold for the following
system

$\left\\{\begin{aligned} -\Delta u&=v^{p}\quad x\in\mathbb{R}^{n}\\\ -\Delta
v&=u^{q}\quad x\in\mathbb{R}^{n}.\end{aligned}\right.$ (1.4)

In 1993, Mitidieri [20] has considered system (1.4) the nonexistence of radial
positive solutions, where $\frac{1}{p+1}+\frac{1}{q+1}>\frac{n-2}{n}$ and
$p>1$, $q>1$. Later, Mitidieri in [21] also extended this results to more
general system. In [7], they proved that for $p=q=\frac{n+2}{n-2}$ the
positive solutions of (1.4) are radially symmetric with respect to some point.
Guo and Liu extended to more general elliptic system in [11]. For other
results, we refer to [6, 12, 14, 15, 23].

For the integral equation, we can use the method of moving planes in integral
form to study the properties of the solution. The integral equation (1.1) and
integral system (1.2) is closely related to [25]. In this paper, Yu studied
the positive solutions for the following integral equation

$u(x)=\int_{\mathbb{R}^{n}}\frac{1}{|x-y|^{n-\alpha}}f(u(y))dy\quad
x\in\mathbb{R}^{n},$ (1.5)

where $n\geq 2$, $0<\alpha<n$. If $f(u)=u^{\frac{n+\alpha}{n-\alpha}}$,
integral equation (1.5) arises as an Euler-Lagrange equation for a functional
under a constraint in the context of Hardy-Littlewood-Sobolev inequalities.
Lieb [17] posed the classification of all the critical points of the
functional - the solutions of the integral equation as an open problem. Chen,
Li and Ou [5] solved the problem by using the method of moving planes in an
integral form. They proved that all the positive regular solutions are
radially symmetric and monotone decreasing about some point.

For the integral system, Yu also established the Liouville type result for the
following integral system

$\left\\{\begin{aligned}
u(x)&=\int_{\mathbb{R}^{n}}\frac{1}{|x-y|^{n-\alpha}}f(u(y),v(y))dy\quad
x\in\mathbb{R}^{n}\\\
v(x)&=\int_{\mathbb{R}^{n}}\frac{1}{|x-y|^{n-\alpha}}g(u(y),v(y))dy\quad
x\in\mathbb{R}^{n}\end{aligned}\right.$ (1.6)

in [25], where $n\geq 2$, $0<\alpha<n$. If $f=v^{q}$ and $g=u^{p}$, Chen, Li
and Ou [4] proved that the positive solutions of (1.6) are radially symmetric
for $\frac{1}{q+1}+\frac{1}{p+1}=\frac{n-\alpha}{n}$. Later, more general
integral equations and systems have also been studied in papers [3, 11, 12,
16, 18, 19, 21].

The above results are all about the whole space $\mathbb{R}^{n}$. For the
upper half space

$\mathbb{R}^{n}_{+}=\\{x=(x_{1},x_{2},\cdots,x_{n})\in\mathbb{R}^{n}\ |\
x_{n}>0\\},$

Fang and Chen [8] considered the integral equation
$u(x)=\int_{\mathbb{R}^{n}_{+}}G^{+}_{\infty}(x,y)u^{p}dy$, where
$G^{+}_{\infty}(x,y)$ is the Green’s function in $\mathbb{R}^{n}_{+}$ with the
following Dirichlet problem

$\left\\{\begin{aligned} (-\Delta)^{m}u&=u^{p}\qquad\qquad\qquad\quad\
\text{in}\ \mathbb{R}^{n}_{+}\\\ u=\frac{\partial u}{\partial
x_{n}}&=\cdots=\frac{\partial^{m-1}u}{\partial x^{m-1}_{n}}=0\quad\text{on}\
\partial\mathbb{R}^{n}_{+}.\end{aligned}\right.$ (1.7)

They proved that the Dirichlet problem (1.7) is equivalent to the integral
equation. Later, Tang and Dou [24] studied the system of integral equations on
upper half space. In 2015, Chen, Fang and Yang [1] considered the Dirichlet
problem involving the fractional Laplacian on upper half space.

In this paper, we study the upper half space results for problem (1.1) and
(1.2). Our result is the following

######  Theorem 1.1.

Let $u(x)\in C(\overline{\mathbb{R}^{n}_{+}})$ be a positive solution of
(1.1), where $n\geq 3$ and let $f:[0,+\infty)\rightarrow\mathbb{R}$ is a
continuous function with the properties
$(i)\ f(t)$ is non-decreasing in $[0,+\infty)$;
$(ii)g(t)=\frac{f(t)}{t^{\frac{n+b+2}{n+b-2}}}$ is non-increasing in
$(0,+\infty)$.
Then either $u\equiv 0$ or there exists $x_{0}\in\partial\mathbb{R}^{n}_{+}$
such that
$u(x)=u_{a,x_{0}}(x)=\left(\frac{ca}{a^{2}+|x-x_{0}|^{2}}\right)^{\frac{n+b-2}{2}}$
and $g(t)\equiv\bar{c}$, where $c=\sqrt{\frac{(n+b)(n+b-2)}{\bar{c}}}$ and
$a\geq 0$.

######  Remark 1.1.

In Theorem 1.1, we can get $f(t)\geq 0$ from conditions $(i)$ and $(ii)$.

######  Theorem 1.2.

Let $(u,v)\in C(\overline{\mathbb{R}^{n}_{+}})\times
C(\overline{\mathbb{R}^{n}_{+}})$ be a positive solution of problem (1.2).
Suppose that $f$, $g$: $[0,+\infty)\rightarrow\mathbb{R}$ are continuous and
satisfy
(i) $f(t)$, $g(t)$ are non-decreasing in $[0,+\infty)$;
(ii) $h(t)=\frac{f(t)}{t^{\frac{n+b+2}{n+b-2}}}$,
$k(t)=\frac{g(t)}{t^{\frac{n+b+2}{n+b-2}}}$ are non-increasing in
$(0,+\infty)$.
Then either $(u,v)\equiv(0,0)$ or $u$, $v$ has the form
$u(x)=\left(\frac{ca}{a^{2}+|x-x_{0}|^{2}}\right)^{\frac{n+b-2}{2}}$ and
$v(x)=\left(\frac{\tilde{c}a}{a^{2}+|x-x_{0}|^{2}}\right)^{\frac{n+b-2}{2}}$,
for some $x_{0}\in\partial\mathbb{R}^{n}_{+}$ and $a\geq 0$.

######  Remark 1.2.

In Theorem 1.2, we can get $f(t)\geq 0$ and $g(t)\geq 0$ from conditions $(i)$
and $(ii)$.

The main method in this paper is the moving plane method in integral form. We
also use Hardy-Littlewood-Sobolev inequalities and Kelvin’s transform to prove
the symmetry of positive solutions with respect to the $x_{1},\cdots,x_{n-1}$
directions. And $x_{n}$ is the anisotropic direction.

This paper is organized as follow. In Section 2, we use the method of moving
planes to get the symmetry of the solutions. In Section 3, we prove the
equivalence of the integral equation with differential equation and the
nonexistence of an ordinary differential equation. We also give the proof of
Theorem 1.1. In Section 4, we obtain the symmetry of solutions of integral
equation system and and we prove Theorem 1.2.

##  2  Symmetry of solutions

Let us study the positive solutions to the integral equation

$u(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(u(y))dy\quad
x\in\mathbb{R}^{n}_{+},$

where

$\begin{split}&K_{b}(x,y)=D_{b}\int_{0}^{1}(|x-y|^{2}(1-\tau)+|x-y^{*}|^{2}\tau)^{-\frac{n+b-2}{2}}[\tau(1-\tau)]^{\frac{b}{2}-1}d\tau,\\\
&D_{b}=2^{b-2}\pi^{-\frac{n}{2}}\frac{\Gamma(\frac{n+b-2}{2})}{\Gamma(\frac{b}{2})},\quad
y^{*}=(y_{1},\cdots,y_{n-1},-y_{n}).\end{split}$ (2.1)

Moreover, for $b>0,n\geq 3$, the following estimates for $K_{b}(x,y)$ holds

$|\partial_{x}^{\gamma}K_{b}(x,y)|\leq
C(\gamma,b)|x-y^{*}|^{-b}|x-y|^{2-n-|\gamma|},$ (2.2)

where $\gamma\in\mathbb{N}^{n}$ (see Proposition 1-2 of [13]). Since we don’t
know the behaviors of $u$ at infinity, we introduce the Kelvin’s transform of
$u$ as
$v(x)=\frac{1}{|x-x_{0}|^{n+b-2}}u\left(\frac{x-x_{0}}{|x-x_{0}|^{2}}\right)$.

$\forall x_{0}\in\partial\mathbb{R}^{n}_{+}$, we define $w(x):=u(x-x_{0})$.
Since

$\displaystyle w(x)$ $\displaystyle=u(x-x_{0})$
$\displaystyle=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y+x_{0})y^{b}_{n}f(u(y))dy$
$\displaystyle=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y^{b}_{n}f(w(y))dy,$

so we take $x_{0}=0$. We consider
$v(x)=\frac{1}{|x|^{n+b-2}}u\left(\frac{x}{|x|^{2}}\right)$. Then a direct
calculation shows that $v(x)$ solves

$v(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(v(y)|y|^{n+b-2})\frac{1}{|y|^{n+b+2}}dy\quad
x\in\mathbb{R}^{n}_{+}.$

We define $\tilde{\tau}=\frac{n+b+2}{n+b-2}$. By the definition of
$g(t)=\frac{f(t)}{t^{\tilde{\tau}}}\displaystyle$, we deduce that $v(x)$
satisfies

$v(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}g(v(y)|y|^{n+b-2})v^{\tilde{\tau}}dy.$

Since $u$ is continuous in $\overline{\mathbb{R}^{n}_{+}}$, we conclude that
$v$ is continuous and positive in
$\overline{\mathbb{R}^{n}_{+}}\setminus\\{0\\}$ with possible singularity at
the origin. Moreover, $v$ decays at infinity as $u(0)|x|^{2-n-b}$. By the
asymptotic behavior of $v$ at $\infty$, we get

$v(x)\in L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))\cap
L^{\infty}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))$

for all $r>0$. Now we use the moving plane method to prove our result.

For a given real number $\lambda$, define

$\Sigma_{\lambda}=\\{x=(x_{1},\cdots,x_{n})\in\mathbb{R}^{n}_{+}\ |\
x_{1}\geq\lambda\\},\quad T_{\lambda}=\\{x\in\mathbb{R}^{n}_{+}\ |\
x_{1}=\lambda\\}$

and let $x^{\lambda}=(2\lambda-x_{1},x_{2},\cdots,x_{n})$ and
$u_{\lambda}(x)=u(x^{\lambda})$.

######  Lemma 2.1.

$\displaystyle v(x)-v(x^{\lambda})=$
$\displaystyle\int_{\Sigma_{\lambda}}(K_{b}(x,y)-K_{b}(x^{\lambda},y))y^{b}_{n}\times$
$\displaystyle[g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}}-g(v(y^{\lambda})|y^{\lambda}|^{n+b-2})v(y^{\lambda})^{\tilde{\tau}}]dy.$

_Proof._ By the definition of $K_{b}(x,y)$, we know
$K_{b}(x,y)=K_{b}(x^{\lambda},y^{\lambda})$,
$K_{b}(x^{\lambda},y)=K_{b}(x,y^{\lambda})$. A direct calculation implies

$\displaystyle
v(x)=\int_{\Sigma_{\lambda}}K_{b}(x,y)y^{b}_{n}g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}}dy+\int_{\Sigma_{\lambda}}K_{b}(x,y^{\lambda})y^{b}_{n}g(v(y^{\lambda})|y^{\lambda}|^{n+b-2})v(y^{\lambda})^{\tilde{\tau}}dy,$

and

$\displaystyle
v(x^{\lambda})=\int_{\Sigma_{\lambda}}K_{b}(x^{\lambda},y)y^{b}_{n}g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}}dy+\int_{\Sigma_{\lambda}}K_{b}(x^{\lambda},y^{\lambda})y^{b}_{n}g(v(y^{\lambda})|y^{\lambda}|^{n+b-2})v(y^{\lambda})^{\tilde{\tau}}dy.$

We get the desired result. $\Box$

######  Lemma 2.2.

Under the assumptions of Theorem 1.1, there exists $\lambda_{0}>0$ such that
for all $\lambda\geq\lambda_{0}$, we have $v_{\lambda}(x)\geq v(x)$ for all
$x\in\Sigma_{\lambda}$.

_Proof._ $\forall\lambda>0$, it is easy to see that $|y|>|y^{\lambda}|$,
$\forall y\in\Sigma_{\lambda}$. Let

$\Sigma^{-}_{\lambda}=\\{y\in\Sigma_{\lambda}\ |\ v(y)>v_{\lambda}(y)\\},$

then $\forall y\in\Sigma^{-}_{\lambda}$, we have

$v(y)|y|^{n+b-2}>v_{\lambda}(y)|y^{\lambda}|^{n+b-2}.$

Since $g(t)$ is non-increasing (see Theorem 1.1(ii)), we get

$g(v(y)|y|^{n+b-2})\leq g(v_{\lambda}(y)|y^{\lambda}|^{n+b-2}).$

This implies

$\displaystyle
g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}}-g(v_{\lambda}(y)|y^{\lambda}|^{n+b-2})v_{\lambda}(y)^{\tilde{\tau}}\leq
g(v(y)|y|^{n+b-2})(v(y)^{\tilde{\tau}}-v_{\lambda}(y)^{\tilde{\tau}}).$ (2.3)

As for $y\in\Sigma_{\lambda}\setminus\Sigma^{-}_{\lambda}$, using the
assumptions of Theorem 1.1 we have

$\displaystyle g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}}$
$\displaystyle=\frac{f(v(y)|y|^{n+b-2})}{|y|^{n+b+2}}$
$\displaystyle\leq\frac{f(v_{\lambda}(y)|y|^{n+b-2})}{|y|^{n+b+2}}$
$\displaystyle=\frac{f(v_{\lambda}(y)|y|^{n+b-2})}{[|y|^{n+b-2}v_{\lambda}(y)]^{\tilde{\tau}}}v_{\lambda}(y)^{\tilde{\tau}}$
$\displaystyle=g(v_{\lambda}(y)|y|^{n+b-2})v_{\lambda}(y)^{\tilde{\tau}}$
$\displaystyle\leq
g(v_{\lambda}(y)|y^{\lambda}|^{n+b-2})v_{\lambda}(y)^{\tilde{\tau}}.$ (2.4)

Then by Lemma 2.1 and the mean value theorem, we know that there exists a
$\xi$ between $y$ and $y^{\lambda}$ that makes

$v(y)^{\tilde{\tau}}-v_{\lambda}(y)^{\tilde{\tau}}=\tilde{\tau}v(\xi)^{\tilde{\tau}-1}(v(y)-v_{\lambda}(y))$
(2.5)

true. Combining Lemma 2.1, (2.3) and (2.4), we have

$\displaystyle v(x)-v_{\lambda}(x)$
$\displaystyle\leq\int_{\Sigma^{-}_{\lambda}}(K_{b}(x,y)-K_{b}(x^{\lambda},y))y^{b}_{n}g(v(y)|y|^{n+b-2})(v(y)^{\tilde{\tau}}-v_{\lambda}(y)^{\tilde{\tau}})dy$
$\displaystyle\leq\tilde{\tau}\int_{\Sigma^{-}_{\lambda}}(K_{b}(x,y)-K_{b}(x^{\lambda},y))y^{b}_{n}g(v(y)|y|^{n+b-2})v(y)^{\tilde{\tau}-1}(v(y)-v_{\lambda}(y))dy.$

In getting the above inequality, we also used $\forall
x,y\in\Sigma_{\lambda}$, $K_{b}(x,y)\geq K_{b}(x^{\lambda},y)$ and $\forall
y\in\Sigma^{-}_{\lambda}$, $v_{\lambda}(y)\leq v(\xi)\leq v(y)$. Since

$|y|^{n+b-2}v(y)=u\left(\frac{y}{|y|^{2}}\right)\geq\min_{B_{\frac{1}{\lambda}}}u(x)\geq
C_{\lambda}>0,\quad\forall\lambda>0,\ y\in\Sigma_{\lambda},$

and $f$ is continuous, we conclude that $g(|y|^{n+b-2}v(y))$ is bounded for
$y\in\Sigma_{\lambda}$. Hence we can deduce from the above inequality that

$v(x)-v_{\lambda}(x)\leq
C\int_{\Sigma^{-}_{\lambda}}K_{b}(x,y)y^{b}_{n}v(y)^{\tilde{\tau}-1}(v(y)-v_{\lambda}(y))dy,\quad\forall\lambda>0,\
x\in\Sigma_{\lambda}.$

By (2.2) and since $\frac{y^{b}_{n}}{|x-y*|^{b}}\leq 1$, then

$v(x)-v_{\lambda}(x)\leq
C\int_{\Sigma^{-}_{\lambda}}\frac{1}{|x-y|^{n-2}}v(y)^{\tilde{\tau}-1}(v(y)-v_{\lambda}(y))dy.$

By the Hardy-Littlewood-Sobolev inequality, it follows that for any
$q>\frac{n}{n-2}$,

$\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}\leq
C\|v^{\tilde{\tau}-1}(v-v_{\lambda})\|_{L^{\frac{qn}{n+2q}}(\Sigma^{-}_{\lambda})}.$

By using the generalized H$\ddot{o}$lder inequality, we obtain

$\displaystyle\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}$
$\displaystyle\leq
C\|v^{\tilde{\tau}-1}\|_{L^{\frac{n}{2}}(\Sigma^{-}_{\lambda})}\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}$
$\displaystyle=C\left(\int_{\Sigma^{-}_{\lambda}}v^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}.$

Due to $v\in L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))$, we
can choose $\lambda_{0}$ sufficiently large, such that for
$\lambda\geq\lambda_{0}$, we have

$C\left(\int_{\Sigma_{\lambda}}v^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\frac{1}{2}.$

Then we conclude

$\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}=0$

for all $\lambda\geq\lambda_{0}$. Thus, $\Sigma^{-}_{\lambda}$ must be measure
$0$. Since $v$ is continuous, we deduce that $\Sigma^{-}_{\lambda}$ is empty.
$\Box$

We define $\lambda_{1}=\inf\\{\lambda\ |\ v(x)\leq v_{\mu}(x),\
\forall\mu\geq\lambda,\ x\in\Sigma_{\mu}\\}$.

######  Lemma 2.3.

If $\lambda_{1}>0$, then $v(x)\equiv v_{\lambda_{1}}(x)$ for all
$x\in\Sigma_{\lambda_{1}}$.

_Proof._ Suppose that conclusion does not hold. We have $v(x)\leq
v_{\lambda_{1}}(x)$, but $v(x)\not\equiv v_{\lambda_{1}}(x)$ in
$\Sigma_{\lambda_{1}}$. We will prove that the plane can be moved further to
the left. We infer from the proof of Lemma 2.2 that

$\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}\leq
C\left(\int_{\Sigma^{-}_{\lambda}}v^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}.$
(2.6)

If one can show that for $\varepsilon$ sufficiently small so that
$\forall\lambda\in(\lambda_{1}-\varepsilon,\lambda_{1}]$, there holds

$C\left(\int_{\Sigma^{-}_{\lambda}}v^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\frac{1}{2},$
(2.7)

then by (2.6), we have $\|v-v_{\lambda}\|_{L^{q}(\Sigma^{-}_{\lambda})}=0$,
and therefore $\Sigma^{-}_{\lambda}$ must be measure zero. Since $v$ is
continuous, we deduce that $\Sigma^{-}_{\lambda}$ is empty. This contradicts
the definition of $\lambda_{1}$.

Now we verify inequality (2.7). Since $v\in
L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))$, for any small
$\eta>0$ we can choose $R$ sufficiently large so that

$C\left(\int_{\mathbb{R}^{n}_{+}\backslash
B_{R}}v^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\eta.$ (2.8)

We fix this $R$ and then show that the measure of $\Sigma^{-}_{\lambda}\cap
B_{R}$ is sufficiently small for $\lambda$ close to $\lambda_{1}$. Since the
measure of $\Sigma^{-}_{\lambda_{1}}$ is zero, by Lemma 2.1 and (2.4) we
deduce $v(x)<v_{\lambda_{1}}(x)$ in the interior of $\Sigma_{\lambda_{1}}$.

For any $\gamma>0$, let

$E_{\gamma}=\\{x\in\Sigma_{\lambda_{1}}\cap B_{R}\ |\
v_{\lambda_{1}}(x)-v(x)>\gamma\\},\qquad F_{\gamma}=(\Sigma_{\lambda_{1}}\cap
B_{R})\backslash E_{\gamma}.$

It is obviously that

$\lim_{\gamma\rightarrow 0}\mu(F_{\gamma})=0.$

For $\lambda<\lambda_{1}$, let

$D_{\lambda}=(\Sigma_{\lambda}\backslash\Sigma_{\lambda_{1}})\cap B_{R}.$

Apparently, the measure of $D_{\lambda}$ is small for $\lambda$ to close to
$\lambda_{1}$. Then it is easy to see that

$(\Sigma^{-}_{\lambda}\cap B_{R})\subseteq(\Sigma^{-}_{\lambda}\cap
E_{\gamma})\cup F_{\gamma}\cup D_{\lambda}.$ (2.9)

In fact, $\forall x\in\Sigma_{\lambda}^{-}\cap E_{\gamma}$, we have

$v(x)-v_{\lambda}(x)=v(x)-v_{\lambda_{1}}(x)+v_{\lambda_{1}}(x)-v_{\lambda}(x)>0.$

Hence

$v_{\lambda_{1}}(x)-v_{\lambda}(x)>v_{\lambda_{1}}(x)-v(x)>\gamma.$

It follows that

$(\Sigma_{\lambda}^{-}\cap E_{\gamma})\subseteq G_{\gamma}\equiv\\{x\in B_{R}\
|\ v_{\lambda_{1}}(x)-v_{\lambda}(x)>\gamma\\}.$ (2.10)

By the well-known Chebyshev inequality, we have

$\displaystyle\mu(G_{\gamma})$
$\displaystyle\leq\frac{1}{\gamma^{p+1}}\int_{G_{\gamma}}|v_{\lambda_{1}}(x)-v_{\lambda}(x)|^{p+1}dx$
$\displaystyle\leq\frac{1}{\gamma^{p+1}}\int_{B_{R}}|v_{\lambda_{1}}(x)-v_{\lambda}(x)|^{p+1}dx,$

where $p>0$. For each fixed $\gamma$, as $\lambda$ is close to $\lambda_{1}$,
the right hand side of the above inequality can be made as small as we wish.
Therefore by (2.9) and (2.10), the measure of $\Sigma^{-}_{\lambda}\cap B_{R}$
can also be made sufficiently small. Combining this with (2.8), we obtain
(2.7). $\Box$

##  3  Proof for Theorem 1.1

We define

$\mathcal{L}_{b}(u)=-\text{div}(y_{n}^{b}\nabla u).$

Take a cutoff function $\varphi_{R}(y)\in C^{\infty}_{c}(B_{R})$ such that
$0\leq\varphi_{R}\leq 1$ in $B_{R}$, $\varphi_{R}=1$ in $B_{\frac{R}{2}}$.
Setting

$u_{R}(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y^{b}_{n}f(u(y))\varphi_{R}(y)dy,$

we clearly have $f(u(y))\varphi_{R}(y)\in
C(\overline{\mathbb{R}^{n}_{+}})\cap\mathcal{E}^{\prime}$, where
$\mathcal{E}^{\prime}$ is the dual space of
$C^{\infty}(\overline{\mathbb{R}^{n}_{+}})$. By the result of [13], we have
$\mathcal{L}_{b}K_{b}(x,y)=\delta(x-y)$ and

$\left\\{\begin{aligned}
\mathcal{L}_{b}(u_{R})&=y^{b}_{n}f(u(y))\varphi_{R}(y)\quad\text{in}\quad\mathbb{R}^{n}_{+}\\\
\frac{\partial u_{R}}{\partial x_{n}}&=0\quad\qquad\qquad\qquad\
\text{on}\quad\partial\mathbb{R}^{n}_{+}\end{aligned}\right.$ (3.1)

(see Proposition 1-1 of [13]). By letting $R\rightarrow\infty$, we then
conclude that

$\left\\{\begin{aligned} -(y^{b}_{n}\Delta
u+by^{b-1}_{n}u_{n})&=y^{b}_{n}f(u(y))\quad\text{in}\quad\mathbb{R}^{n}_{+}\\\
\frac{\partial u}{\partial x_{n}}&=0\quad\qquad\quad\ \
\text{on}\quad\partial\mathbb{R}^{n}_{+}.\end{aligned}\right.$ (3.2)

Therefore, the integral equation (1.1) satisfies the following partial
differential equation:

$\left\\{\begin{aligned} \Delta
u+\frac{b}{y_{n}}u_{n}+f(u(y))&=0\quad\text{in}\quad\mathbb{R}^{n}_{+}\\\
\frac{\partial u}{\partial
x_{n}}&=0\quad\text{on}\quad\partial\mathbb{R}^{n}_{+}.\end{aligned}\right.$
(3.3)

Then we will establish the equivalence between the integral equation (1.1) and
the partial differential equation (3.3). We need the asymptotic behavior
$u(x)\sim\frac{1}{|x|^{n+b-2}}$ as $x\rightarrow\infty$; that is, there exist
two constant $R$, $C$ such that

$u(x)=\frac{C}{|x|^{n+b-2}},\quad|x|>R.$

######  Lemma 3.1.

Let $f(t)=\bar{c}t^{\frac{n+b+2}{n+b-2}}$ in Theorem 1.1 and
$u(x)\sim\frac{1}{|x|^{n+b-2}}$ as $x\rightarrow\infty$. Then the positive
solution $u$ of $\eqref{s3}$ is $C^{2}(\overline{\mathbb{R}^{n}_{+}})$.

Proof.
Step 1: _We prove that $u\in C^{\alpha}(\overline{\mathbb{R}^{n}_{+}})$. _

Since $u\in C^{0}(\overline{\mathbb{R}^{n}_{+}})$, one know $u\in
L^{\infty}(\overline{B_{R}\cap\mathbb{R}^{n}_{+}})$. Since
$u(x)\sim\frac{1}{|x|^{n+b-2}}$ as $x\rightarrow\infty$, we obtain $u\in
L^{p}(\mathbb{R}^{n}_{+})$, $\forall p>\frac{n}{n+b-2}$. By $\eqref{G2}$ we
have

$|Du(x)|\leq
C\int_{\mathbb{R}^{n}_{+}}\frac{1}{|x-y|^{n-1}}u(y)^{\tilde{\tau}}dy.$

We apply the Hardy-Littlewood-Sobolev inequality to get

$\|Du\|_{L^{p}(\mathbb{R}^{n}_{+})}\leq
C\|u^{\tilde{\tau}}\|_{L^{\frac{np}{n+p}}(\mathbb{R}^{n}_{+})}$

for any $p>\frac{n}{n-1}$. Since $\tilde{\tau}\frac{np}{n+p}>\frac{n}{n+b-2}$,
we obtain $Du\in L^{p}(\mathbb{R}^{n}_{+})$. This implies the desired result
for all $p>n$.

Step 2: _We prove that $u\in C^{2}(\overline{\mathbb{R}^{n}_{+}})$._

We write $W_{R}=\\{x=(x^{\prime},x_{n})\ |\ 0<x_{n}<R,\ |x^{\prime}|<R\\}$.
For an arbitrarily fixed $\bar{x}\in\overline{\mathbb{R}^{n}_{+}}$, there is
positive constant $R$, such that $\bar{x}\in W_{R}$ and
$\text{dist}(\bar{x},\tilde{\partial}W_{R})>1$, where
$\tilde{\partial}W_{R}=\partial W_{R}\backslash\\{x\ |\ x_{n}=0,\
|x^{\prime}|<R\\}$. We have $B_{1}(\bar{x})\cap\mathbb{R}^{n}_{+}\subset
W_{R}$. Take a cutoff function $\varphi(y)\in C^{\infty}_{c}(B_{1}(\bar{x}))$
such that $0\leq\varphi\leq 1$ in $B_{1}(\bar{x})$, $\varphi=1$ in
$B_{\frac{1}{2}}(\bar{x})$. We define

$u_{1}(x)=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y^{b}_{n}f(u(y))\varphi(y)dy,\quad
x\in B_{\frac{1}{4}}(\bar{x}).$

By [13], we have $u_{1}\in C^{2,\alpha}(\overline{\mathbb{R}^{n}_{+}})$.

Set

$\displaystyle u_{2}(x)$
$\displaystyle=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y^{b}_{n}f(u(y))(1-\varphi(y))dy$
$\displaystyle=\int_{\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x})}K_{b}(x,y)y^{b}_{n}f(u(y))(1-\varphi(y))dy,\quad x\in
B_{\frac{1}{4}}(\bar{x}).$

Then by the H$\ddot{o}$lder inequality and $f(t)=\bar{c}t^{\tilde{\tau}}$, we
conclude

$\displaystyle|D^{2}u_{2}(x)|$
$\displaystyle\leq\left\|\frac{1}{|x-y|^{n}}\right\|_{L^{q^{\ast}}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}\|u^{\tilde{\tau}}\|_{L^{p^{\ast}}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}$ $\displaystyle\leq
C\|u\|^{\tilde{\tau}}_{L^{p}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}$

for $p>\max\left\\{\frac{n+b+2}{n+b-2},n\right\\}$, where
$p^{\ast}=\frac{p}{\tilde{\tau}}$ and
$\frac{1}{p^{\ast}}+\frac{1}{q^{\ast}}=1$. Similarly,

$\displaystyle|D^{3}u_{2}(x)|$
$\displaystyle\leq\left\|\frac{1}{|x-y|^{n+1}}\right\|_{L^{q^{\ast}}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}\|u^{\tilde{\tau}}\|_{L^{p^{\ast}}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}$ $\displaystyle\leq
C\|u\|^{\tilde{\tau}}_{L^{p}(\mathbb{R}^{n}_{+}\setminus
B_{\frac{1}{2}}(\bar{x}))}.$

This implies the desired result. $\Box$

Since $u\in C^{2}(\overline{\mathbb{R}^{n}_{+}})$, we may assume that (see
[14])

$\bar{u}(x^{\prime},x_{n},x_{n+1})=u\left(x^{\prime},\sqrt{x^{2}_{n}+x^{2}_{n+1}}\right).$
(3.4)

It follows that,

$\left\\{\begin{aligned}
\Delta_{n+1}\bar{u}+\frac{b-1}{x_{n+1}}\bar{u}_{n+1}+\bar{c}\bar{u}(x)^{\tilde{\tau}}&=0\quad\text{in}\quad\mathbb{R}^{n+1}_{+}\\\
\frac{\partial\bar{u}}{\partial
x_{n+1}}&=0\quad\text{on}\quad\partial\mathbb{R}^{n+1}_{+}.\end{aligned}\right.$
(3.5)

$\bar{u}$ is a classical solution to (3.5).

######  Lemma 3.2.

Let $\bar{u}(x)$ be the positive solution of $\eqref{sec2}$, and the
asymptotic behavior of $\bar{u}$ at infinity is
$\bar{u}(x)\sim\frac{1}{|x|^{n+b-2}}$. Then $\bar{u}(x)$ satisfies the
corresponding integral equation

$\bar{u}(x)=\bar{c}\int_{\mathbb{R}^{n+1}_{+}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{u}(y)^{\tilde{\tau}}dy.$
(3.6)

_Proof._ Consider $R$ large enough and $|x|>R$, and set
$A_{1}:=\left\\{y\in\mathbb{R}^{n+1}_{+}\ |\ |y-x|\leq\frac{|x|}{2}\right\\}$,
and $A_{2}:=\left\\{y\in\mathbb{R}^{n+1}_{+}\ |\
|y-x|\geq\frac{|x|}{2}\right\\}$. Assume that
$\zeta(x):=\bar{c}\int_{\mathbb{R}^{n+1}_{+}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{u}(y)^{\tilde{\tau}}dy$.
By the asymptotic behavior of $\bar{u}$ at infinity, one can easily verify
that

$\left|\bar{c}\int_{A_{1}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{u}(y)^{\tilde{\tau}}dy\right|\leq
C\frac{1}{|x|^{n+b}}$

and

$\displaystyle\left|\bar{c}\int_{A_{2}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{u}(y)^{\tilde{\tau}}dy\right|$
$\displaystyle\leq$ $\displaystyle\ C\int_{A_{2}\cap
B_{\sqrt{|x|}}}\frac{y^{b-1}_{n+1}}{|x-y^{*}|^{b-1}}\frac{1}{|x-y|^{n-1}}\left(\frac{1}{1+|y|}\right)^{n+b+2}dy+C\int_{A_{2}\backslash
B_{\sqrt{|x|}}}\frac{1}{|x-y|^{n-1}}\frac{1}{|y|^{n+b+2}}dy$
$\displaystyle\leq$ $\displaystyle\
\frac{C}{|x|^{n-1+\frac{b-1}{2}}}\int_{A_{2}\cap
B_{\sqrt{|x|}}}\left(\frac{1}{1+|y|}\right)^{n+b+2}dy+\frac{C}{|x|^{n-1}}\int_{A_{2}\backslash
B_{\sqrt{|x|}}}\frac{1}{|y|^{n+b+2}}dy$ $\displaystyle\leq$ $\displaystyle\
\frac{C}{|x|^{n+\frac{b-3}{2}}}+\frac{C}{|x|^{n-1+\frac{b+1}{2}}}$
$\displaystyle\leq$ $\displaystyle\ \frac{C}{|x|^{n+\frac{b-3}{2}}},$

where $|x|>R$ and $B_{R}=\\{x\in\mathbb{R}^{n+1}\ |\ |x|<R\\}$. In getting the
above inequality, we used $\forall|y|<\sqrt{|x|}$,
$u(y)\sim\left(\frac{1}{1+|y|}\right)^{n+b-2}$ and

$|x-y^{*}|\geq||x|-|y^{*}||\geq|x|-\sqrt{|x|}\geq\frac{|x|}{2}.$

Then we have $|\zeta(x)|\leq C\frac{1}{|x|^{n+\frac{b-3}{2}}}$. Similarly, we
also obtain $|\nabla\zeta(x)|\leq C\frac{1}{|x|^{n+\frac{b-1}{2}}}$. We know
that the asymptotic behavior $\bar{u}(x)\sim\frac{1}{|x|^{n+b-2}}$ and
$\nabla\bar{u}(x)\sim\frac{1}{|x|^{n+b-1}}$ at $\infty$. Let
$w=\bar{u}-\zeta$, then $w$ satisfies the following equation

$\Delta
w+\frac{b-1}{x_{n+1}}w_{n+1}=0\quad\text{in}\quad\mathbb{R}^{n+1}_{+}.$

Multiplying this identity by $wx_{n+1}$ and integrating by parts, we get

$\displaystyle 0$
$\displaystyle=\lim_{R\rightarrow\infty}\int_{B^{+}_{R}(0)}(\Delta
w+\frac{b-1}{x_{n+1}}w_{n+1})wx_{n+1}dx$
$\displaystyle=-\lim_{R\rightarrow\infty}\int_{B^{+}_{R}(0)}x_{n+1}|\nabla
w|^{2}dx+\lim_{R\rightarrow\infty}\int_{B^{+}_{R}(0)}(b-2)ww_{n+1}dx.$

Since the asymptotic properties of $w$ and $\nabla w$, the boundary integral
term is zero. We deduce that

$\displaystyle\lim_{R\rightarrow\infty}\int_{B^{+}_{R}(0)}x_{n+1}|\nabla
w|^{2}dx$ $\displaystyle=\frac{b-2}{2}\lim_{R\rightarrow\infty}\int_{\partial
B^{+}_{R}(0)}w^{2}\nu^{n+1}dS$ $\displaystyle\leq
C\lim_{R\rightarrow\infty}\int_{\partial
B^{+}_{R}(0)}\frac{1}{|x|^{2n+b-3}}dS$ $\displaystyle=0,$

where $\nu=(\nu^{1},\nu^{2},\cdots,\nu^{n+1})$ is the outward pointing unit
normal vector field. Then we have $\nabla w=0$ and $w=0$. $\Box$

######  Proposition 3.1.

Let $f(t)$ satisfy the conditions of Theorem 1.1. The problem

$\left\\{\begin{aligned}
t^{b}u^{\prime\prime}(t)+bt^{b-1}u^{\prime}(t)+t^{b}f(u(t))&=0\quad
t\in[0,+\infty)\\\ u^{\prime}(0)&=0\end{aligned}\right.$ (3.7)

has no positive solutions of class $C^{2}$.

In order to prove Proposition 3.1 we make use of the following results. And
the idea of the proof from Theorem 3.2 of [20].

######  Lemma 3.3.

Let $u(t)\in C^{2}$ be a positive solution of $(\ref{pr1})$, then for every
$t>0$ we have

$tu^{\prime}(t)+(b-1)u(t)\geq 0.$ (3.8)

Proof. Since $(\ref{pr1})$ we have

$tu^{\prime\prime}+bu^{\prime}+tf(u)=0,\quad t>0,$

hence

$tu^{\prime\prime}+bu^{\prime}=(tu^{\prime})^{\prime}+(b-1)u^{\prime}\leq 0.$

The function $M(t):=tu^{\prime}(t)+(b-1)u(t)$ is non-increasing. Now we
proceed by contradiction. If there exists $t_{1}>0$ such that

$M(t_{1})=t_{1}u^{\prime}(t_{1})+(b-1)u(t_{1})<0,$

then the monotonicity of $M(t)$ and the positivity of $u(t)$ imply that

$u^{\prime}(t)\leq u^{\prime}(t)+\frac{b-1}{t}u(t)\leq\frac{M(t_{1})}{t}\
\text{for}\ t\geq t_{1}.$

Integrating the inequality $u^{\prime}(t)\leq\frac{M(t_{1})}{t}$ on
$(t_{1},t)$, we obtain

$-u(t_{1})\leq u(t)-u(t_{1})\leq M(t_{1})\ln\left(\frac{t}{t_{1}}\right).$

Letting $t\rightarrow\infty$ and recalling that $M(t_{1})<0$, we get a
contradiction. $\Box$

The next lemma contains the necessary a priori estimate for our study.

######  Lemma 3.4.

Let $u\in C^{2}$ be a positive solution of $(\ref{pr1})$, then

$u(t)\leq Ct^{-\frac{n+b-2}{2}},$ (3.9) $|u^{\prime}(t)|\leq
Ct^{-\frac{n+b}{2}},$ (3.10)

where $C$ is a positive constant.

Proof. We claim $f(0)=0$. Suppose $f(0)>0$. Since $f(t)$ is non-decreasing, we
have

$(t^{b}u^{\prime}(t))^{\prime}=-t^{b}f(u(t))\leq-f(0)t^{b}.$ (3.11)

Integrating $(\ref{l4})$ on $(0,t)$ and using the fact that $u^{\prime}(0)=0$,
we obtain

$t^{b}u^{\prime}(t)\leq-\frac{f(0)}{b+1}t^{b+1}.$ (3.12)

Integrating $(\ref{l5})$, we get

$u(t)\leq-\frac{f(0)}{2(b+1)}t^{2}+u(0).$

This contradicts $u(t)>0$ by letting $t\rightarrow\infty$.

We have $f(0)=0$. Since $f(t)$ is non-decreasing, by Remark 1.1 there are
three cases for $f$. For the case $f\equiv 0$, we obtain $u\equiv 0$ by
applying $(\ref{s3})$. If $f(t)=0\ \text{in}\ [0,t_{0}]$ for some $t_{0}>0$,
then we have $g(t)=0\ \text{in}\ [0,t_{0}]$ and $g(t)>0\ \text{in}\
(t_{0},+\infty)$. This contradicts the non-increasing property of $g$. Then we
consider that $f(t)>0$ for every $t>0$.

Since

$(t^{b}u^{\prime}(t))^{\prime}=-t^{b}f(u(t))<0,\quad\forall t>0,$

$t^{b}u^{\prime}(t)$ is strictly monotonically decreasing. Noticing that
$u^{\prime}(0)=0$, we see that $u^{\prime}(t)\leq 0$, $\forall t\geq 0$.

Since $u^{\prime}\leq 0$ and $u>0$, we get $u$ is bounded. Assume that
$u(t)\leq M$ in $[0,+\infty)$. Since $g(t)$ is non-increasing, we have

$\frac{f(t)}{t^{\tilde{\tau}}}=g(t)\geq
g(M)=\frac{f(M)}{M^{\tilde{\tau}}}\quad\forall t\in(0,M].$

We can prove $f(u)\geq Cu^{\tilde{\tau}}$.

Therefore, we have

$-(t^{b}u^{\prime}(t))^{\prime}=t^{b}f(u(t))\geq Ct^{b}u(t)^{\tilde{\tau}}.$
(3.13)

Integrating $(\ref{l6})$ on $(0,t)$, by using $u^{\prime}(t)\leq 0$, we have

$-t^{b}u^{\prime}\geq Ct^{b+1}u(t)^{\tilde{\tau}}.$

Integrating $-u^{\prime}\geq Ctu(t)^{\tilde{\tau}}$ over $(0,t)$, we get
$u(t)^{1-\tilde{\tau}}\geq Ct^{2}+u(0)^{1-\tilde{\tau}}$. Therefore, we obtain

$u(t)\leq Ct^{-\frac{2}{\tilde{\tau}-1}}=Ct^{-\frac{n+b-2}{2}}.$

In order to prove $(\ref{l8})$ it is sufficient to $(\ref{l3})$ and
$(\ref{l7})$ . $\Box$

Proof of Proposition 3.1

We will proceed by contradiction argument. We define
$F(t)=\int^{t}_{0}f(s)ds$. Multiplying $(\ref{pr1})$ by $u$ and integrating by
parts on $(0,t)$, we obtain

$t^{b}u(t)u^{\prime}(t)-\int^{t}_{0}s^{b}(u^{\prime}(s))^{2}ds+\int^{t}_{0}s^{b}f(u(s))u(s)ds=0.$
(3.14)

On the other hand, multiplying $(\ref{pr1})$ by $tu^{\prime}(t)$ and
integrating by parts on $(0,t)$, we get

$\frac{1}{2}t^{b+1}(u^{\prime}(t))^{2}+\frac{b-1}{2}\int^{t}_{0}s^{b}(u^{\prime}(s))^{2}ds+t^{b+1}F(u(t))-(b+1)\int^{t}_{0}s^{b}F(u(s))ds=0.$
(3.15)

Using $(\ref{l7})$ and $(\ref{l8})$, we conclude that

$\lim_{t\rightarrow\infty}t^{b}u(t)u^{\prime}(t)=\lim_{t\rightarrow\infty}t^{b+1}(u^{\prime}(t))^{2}=0$

and

$\int^{\infty}_{0}s^{b}(u^{\prime}(s))^{2}ds<\infty.$

Hence by $(\ref{l9})$, we have

$\int^{\infty}_{0}s^{b}(u^{\prime}(s))^{2}ds=\int^{\infty}_{0}s^{b}f(u(s))u(s)ds<\infty.$

We claim that there is a sequence $t_{k}\rightarrow\infty$ such that
$t^{b+1}_{k}F(u(t_{k}))\rightarrow 0$. Suppose not, $t^{b+1}F(u(t))\geq
C_{0}>0$, $\forall t>1$, for some positive constant $C_{0}$, we have

$\frac{C_{0}}{t}\leq t^{b}F(u(t))=t^{b}\int^{u(t)}_{0}f(s)ds\leq
t^{b}f(u(t))u(t).$ (3.16)

Integrating the inequality $(\ref{l11})$ on $(1,+\infty)$, we obtain

$\int^{\infty}_{1}\frac{C_{0}}{t}dt\leq\int^{\infty}_{1}t^{b}f(u(t))u(t)dt.$

The left-hand side of the inequality is unbounded, but the right-hand side is
bounded. This is a contradiction. The claim is proved. Since
$g(t)=\frac{f(t)}{t^{\tilde{\tau}}}$ is non-increasing, we have

$F(u(t))=\int^{u(t)}_{0}\frac{f(s)}{s^{\tilde{\tau}}}s^{\tilde{\tau}}ds\geq\frac{1}{\tilde{\tau}+1}f(u(t))u(t).$
(3.17)

Combining $(\ref{l9})$, $(\ref{l10})$ and $(\ref{l12})$, and taking $t=t_{k}$,
we get

$\displaystyle\frac{1}{2}t^{b+1}_{k}(u^{\prime}(t_{k}))^{2}+\frac{b-1}{2}\left[t^{b}_{k}u(t_{k})u^{\prime}(t_{k})+\int^{t_{k}}_{0}s^{b}f(u(s))u(s)ds\right]$
$\displaystyle+t^{b+1}_{k}F(u(t_{k}))-\frac{(b+1)}{\tilde{\tau}+1}\int^{t_{k}}_{0}s^{b}f(u(s))u(s)ds\geq
0.$

Letting $t_{k}\rightarrow\infty$, we have

$\left(\frac{b-1}{2}-\frac{(b+1)}{\tilde{\tau}+1}\right)\int^{\infty}_{0}s^{b}f(u(s))u(s)ds\geq
0.$

This contradicts the fact that

$\left(\frac{b-1}{2}-\frac{(b+1)}{\tilde{\tau}+1}\right)=\frac{1}{2}\left(\frac{2(b+1)}{n+b}-2\right)<0.$

$\Box$
Proof of Theorem 1.1

By Lemma 2.1, Lemma 2.2 and Lemma 2.3, we choose the $x_{1}$ direction and
prove that $v$ is symmetric in the $x_{1}$ direction. If $\lambda_{1}>0$, then
$v$ is symmetric in the direction of $x_{1}$. If $\lambda_{1}=0$, then we
conclude by continuity that $v(x)\leq v_{0}(x)$ for all $x\in\Sigma_{0}$. We
can also start the moving plane from $-\infty$ and find a corresponding
$\lambda^{\prime}_{1}$. If $\lambda^{\prime}_{1}=0$, then we get $v_{0}(x)\leq
v(x)$ for $x\in\Sigma_{0}$. So $v(x)$ is symmetric with respect to $T_{0}$. If
$\lambda^{\prime}_{1}<0$, an analogue to Lemma 2.3 shows that $v$ is symmetric
with respect to $T_{\lambda^{\prime}_{1}}$. For the $x_{2},x_{3}\cdots
x_{n-1}$, we can carry out the procedure as the above. There are two cases for
solutions.

Case 1 If $\lambda_{1}>0$ or $\lambda^{\prime}_{1}<0$ in some direction for
some $x_{0}\in\partial\mathbb{R}^{n}_{+}$, we have $v=v_{\lambda_{1}}$ or
$v=v_{\lambda^{\prime}_{1}}$. Since $g$ is non-increasing and by Lemma 2.1, we
get $g(t)=\bar{c}$. This implies $f(t)=\bar{c}t^{\frac{n+b+2}{n+b-2}}$. For
$\bar{c}=0$, one has $u\equiv 0$. In the following, we always assume
$\bar{c}>0$. Since $v$ is regular at the origin, we have the asymptotic
behavior of $u$ at infinity is $u(x)\sim\frac{1}{|x|^{n+b-2}}$.

According to Lemma 3.1, we obtain $u\in C^{2}(\overline{\mathbb{R}^{n}_{+}})$.
We know $u$ satisfies (3.3). After transformation (3.4), $\bar{u}$ satisfies
$n+1$ dimensional equation (3.5). Thanks to Lemma 3.2, we obtain the
equivalence between the integral equation (3.6) and its corresponding
differential equation (3.5). And we know that $\bar{u}$ satisfies $n+1$
dimensional integral equation (3.6). Using the above moving plane in integral
form for $\bar{u}$, we obtain $\bar{u}$ is radially symmetric in the direction
of $x_{1},\cdots x_{n-1},x_{n}$. We define
$(x^{\prime},x_{n})=(x_{1},\cdots,x_{n-1},x_{n})$. There exists
$p=(p_{1},\cdots,p_{n})\in\mathbb{R}^{n}$ such that

$u(x^{\prime},|x_{n}|)=\bar{u}(x^{\prime},x_{n},0)=\bar{u}(\bar{x}^{\prime},\bar{x}_{n},0)=u(\bar{x}^{\prime},|\bar{x}_{n}|)$

if $\sum^{n}_{i=1}|x_{i}-p_{i}|^{2}=\sum^{n}_{i=1}|\bar{x}_{i}-p_{i}|^{2}$.
$u$ is radially symmetric about $p$. In fact, $p_{n}$ must be zero. Otherwise,
it follows that

$\bar{u}(x^{\prime},2p_{n}-x_{n},0)=\bar{u}(x^{\prime},x_{n},0)=u(x^{\prime},|x_{n}|)=\bar{u}(x^{\prime},-x_{n},0)$

It shows that for the fixed $x^{\prime}$, $\bar{u}$ is periodic with respect
to $x_{n}$ with period $2p_{n}$. Similarly to the proof of Lemma 2.2, we
obtain that $\bar{u}$ is monotonic with respect to $p$. Thanks to the decay
estimate of $\bar{u}$ we directly get $\bar{u}\equiv 0$ and $u\equiv 0$. We
get a contradiction. Thus, we have $p_{n}=0$ and $u$ is radially symmetric
about $p\in\partial\mathbb{R}^{n}_{+}$. By a result of [5], we deduce that
$u(x)=u_{a,p}(x)=\left(\frac{ca}{a^{2}+|x-p|^{2}}\right)^{\frac{n+b-2}{2}}$,
where $c=\sqrt{\frac{(n+b)(n+b-2)}{\bar{c}}}$ and $a\geq 0$.

Case 2 Now we suppose that $\lambda_{1}=\lambda^{\prime}_{1}=0$ for
$x_{1},x_{2}\cdots x_{n-1}$ directions and for all
$x_{0}\in\partial\mathbb{R}^{n}_{+}$, then $v$ and hence $u$ are radially
symmetric in the $x_{1},x_{2}\cdots x_{n-1}$ directions. We define
$S_{C}:=\\{x\in\mathbb{R}^{n}_{+}\ |\ x_{n}=C\\}$. For any given $p$, $q\in
S_{C}$, there exists $x_{0}\in\partial\mathbb{R}^{n}_{+}$ such that
$d(p,x_{0})=d(q,x_{0})$. Since $u$ is radially symmetric, we have $u(p)=u(q)$.
According to the arbitrariness of $p$ and $q$, the solution $u$ depends only
on $x_{n}$. Set $\tilde{u}(x_{n}):=u(x)$, and we have:

$\left\\{\begin{aligned}
x^{b}_{n}\tilde{u}^{\prime\prime}+bx^{b-1}_{n}\tilde{u}^{\prime}+x^{b}_{n}f(\tilde{u})&=0\\\
\tilde{u}^{\prime}(0)&=0.\end{aligned}\right.$ (3.18)

According to Proposition 3.1, we have $\tilde{u}\equiv 0$. Hence, $u(x)\equiv
0$. $\Box$

##  4  Integral system

The idea of proving Theorem 1.2 is similar to the proof of Theorem 1.1. We
consider the Kelvin’s transform $w$, $z$ of $u$, $v$ defined by

$w(x)=\frac{1}{|x|^{n+b-2}}u\left(\frac{x}{|x|^{2}}\right),\
z(x)=\frac{1}{|x|^{n+b-2}}v\left(\frac{x}{|x|^{2}}\right).$

The definition of $\Sigma_{\lambda}$, $T_{\lambda}$, $x^{\lambda}$ and
$u_{\lambda}(x)$ in the moving plane method is the same as above.

By a simple calculation, we have

$\left\\{\begin{aligned}
w(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}f(z(y)|y|^{n+b-2})\frac{1}{|y|^{n+b+2}}dy\quad
x\in\mathbb{R}^{n}_{+},\\\
z(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}g(w(y)|y|^{n+b-2})\frac{1}{|y|^{n+b+2}}dy\quad
x\in\mathbb{R}^{n}_{+}.\end{aligned}\right.$ (4.1)

By the definition of $h(t)=\frac{f(t)}{t^{\tilde{\tau}}}$,
$k(t)=\frac{g(t)}{t^{\tilde{\tau}}}$ we deduce that

$\left\\{\begin{aligned}
w(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}h(z(y)|y|^{n+b-2})z(y)^{\tilde{\tau}}dy,\\\
z(x)&=\int_{\mathbb{R}^{n}_{+}}K_{b}(x,y)y_{n}^{b}k(w(y)|y|^{n+b-2})w(y)^{\tilde{\tau}}dy.\end{aligned}\right.$
(4.2)

Since that $w$ and $z$ are continuous and positive in
$\overline{\mathbb{R}^{n}_{+}}$ with possible singularity at the origin. They
decay at infinity as $u(0)|x|^{2-n-b}$ and $v(0)|x|^{2-n-b}$. We obtain $w$,
$z\in L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))\cap
L^{\infty}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))$ for any $r>0$.

######  Lemma 4.1.

$\displaystyle w(x)-w_{\lambda}(x)=$
$\displaystyle\int_{\Sigma_{\lambda}}(K_{b}(x,y)-K_{b}(x^{\lambda},y))y_{n}^{b}\times$
$\displaystyle[h(z(y)|y|^{n+b-2})z(y)^{\tilde{\tau}}-h(z(y^{\lambda})|y^{\lambda}|^{n+b-2})z(y^{\lambda})^{\tilde{\tau}}]dy,$
$\displaystyle z(x)-z_{\lambda}(x)=$
$\displaystyle\int_{\Sigma_{\lambda}}(K_{b}(x,y)-K_{b}(x^{\lambda},y))y_{n}^{b}\times$
$\displaystyle[k(w(y)|y|^{n+b-2})w(y)^{\tilde{\tau}}-k(w(y^{\lambda})|y^{\lambda}|^{n+b-2})w(y^{\lambda})^{\tilde{\tau}}]dy.$

_Proof._ The proof is similar to that of Lemma 2.1 and is omitted. $\Box$

######  Lemma 4.2.

Under the conditions of Theorem 1.2, there exists $\lambda_{0}>0$ such that
for all $\lambda\geq\lambda_{0}$, we have $w_{\lambda}(x)\geq w(x)$ and
$z_{\lambda}(x)\geq z(x)$ for all $x\in\Sigma_{\lambda}$.

_Proof._ The proof is similar to Lemma 2.2. We denote by
$\Sigma^{w}_{\lambda}=\\{y\in\Sigma_{\lambda}\ |\ w(y)>w_{\lambda}(y)\\}$ and
$\Sigma^{z}_{\lambda}=\\{y\in\Sigma_{\lambda}\ |\ z(y)>z_{\lambda}(y)\\}$. By
the Hardy-Littlewood-Sobolev inequality, we get

$\displaystyle\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})}$
$\displaystyle\leq
C\left(\int_{\Sigma_{\lambda}^{z}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})},$
$\displaystyle\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})}$
$\displaystyle\leq
C\left(\int_{\Sigma_{\lambda}^{w}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})},$

where $q>\frac{n}{n-2}$. The above two inequalities imply

$\displaystyle\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})}$
$\displaystyle\leq
C\left(\int_{\Sigma_{\lambda}^{w}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\left(\int_{\Sigma_{\lambda}^{z}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})},$
(4.3)

and

$\displaystyle\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})}$
$\displaystyle\leq
C\left(\int_{\Sigma_{\lambda}^{w}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\left(\int_{\Sigma_{\lambda}^{z}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})}.$
(4.4)

Since $w(x)$, $z(x)\in L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus
B_{r}(0))$ for any $r>0$, we choose $\lambda_{0}$ large enough such that

$C\left(\int_{\Sigma_{\lambda}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\left(\int_{\Sigma_{\lambda}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\frac{1}{2}$

for all $\lambda\geq\lambda_{0}$. Then we have

$\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})}=\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})}=0$

for all $\lambda\geq\lambda_{0}$. So we get the desired result. $\Box$

We define $\lambda_{1}=\inf\\{\lambda\ |\ w(x)\leq w_{\mu}(x)\ \text{and}\
z(x)\leq z_{\mu}(x),\ \forall\mu\geq\lambda,\ x\in\Sigma_{\mu}\\}$.

######  Lemma 4.3.

If $\lambda_{1}>0$, then $w(x)\equiv w_{\lambda_{1}}(x)$ and $z(x)\equiv
z_{\lambda_{1}}(x)$ for all $x\in\Sigma_{\lambda_{1}}$.

_Proof._ Suppose that $z(x)\not\equiv z_{\lambda_{1}}(x)$, then we can infer
from Lemma 4.1 that $w<w_{\lambda_{1}}$ in the interior of
$\Sigma_{\lambda_{1}}$ and this further implies $z<z_{\lambda_{1}}$ in the
same area. If one can show that for $\varepsilon$ sufficiently small so that
$\forall\lambda\in(\lambda_{1}-\varepsilon,\lambda_{1}]$, there holds

$C\left(\int_{\Sigma_{\lambda}^{w}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\left(\int_{\Sigma_{\lambda}^{z}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\frac{1}{2},$
(4.5)

then by (4.3) and (4.4), we have

$\|w-w_{\lambda}\|_{L^{q}(\Sigma^{w}_{\lambda})}=\|z-z_{\lambda}\|_{L^{q}(\Sigma^{z}_{\lambda})}=0,$

and therefore $\Sigma^{w}_{\lambda}$ and $\Sigma^{z}_{\lambda}$ must be
measure zero. Since $w$ and $z$ are continuous, we deduce that
$\Sigma^{w}_{\lambda}$ and $\Sigma^{z}_{\lambda}$ are empty. This contradicts
the definition of $\lambda_{1}$.

Now we verify inequality (4.5). Since $w$, $z\in
L^{\frac{2n}{n+b-2}}(\mathbb{R}^{n}_{+}\setminus B_{r}(0))$, for any small
$\eta>0$, we can choose $R$ sufficiently large so that

$C\left(\int_{\mathbb{R}^{n}_{+}\backslash
B_{R}}w^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\eta,\qquad
C\left(\int_{\mathbb{R}^{n}_{+}\backslash
B_{R}}z^{\frac{2n}{n+b-2}}(y)dy\right)^{\frac{2}{n}}\leq\eta.$

We fix this $R$ and then show that the measure of $\Sigma^{w}_{\lambda}\cap
B_{R}$ and $\Sigma^{z}_{\lambda}\cap B_{R}$ are sufficiently small for
$\lambda$ close to $\lambda_{1}$. The rest part of the proof is similar to
that of Lemma 2.3 and is omitted. $\Box$
Proof of Theorem 1.2

By Lemma 4.1, Lemma 4.2 and Lemma 4.3, we choose the $x_{1}$ direction and
prove that $w$, $z$ are symmetric in the $x_{1}$ direction. If
$\lambda_{1}>0$, then $w$, $z$ are symmetric in the direction of $x_{1}$. If
$\lambda_{1}=0$, then we conclude by continuity that $w(x)\leq w_{0}(x)$,
$z(x)\leq z_{0}(x)$ for all $x\in\Sigma_{0}$. We can also start the moving
plane from $-\infty$ and find a corresponding $\lambda^{\prime}_{1}$. If
$\lambda^{\prime}_{1}=0$, then we get $w_{0}(x)\leq w(x)$, $z_{0}(x)\leq z(x)$
for $x\in\Sigma_{0}$. So $w(x)$ and $z(x)$ are symmetric with respect to
$T_{0}$. If $\lambda^{\prime}_{1}<0$, an analogue to Lemma 4.3 shows that $w$
and $z$ are symmetric with respect to $T_{\lambda^{\prime}_{1}}$. For the
$x_{2},x_{3}\cdots x_{n-1}$, we can carry out the procedure as the above.
There are two cases for solutions.

Case 1 If $\lambda_{1}>0$ or $\lambda^{\prime}_{1}<0$ in some direction for
some $x_{0}\in\partial\mathbb{R}^{n}_{+}$, we have $w=w_{\lambda_{1}}$,
$z=z_{\lambda_{1}}$ or $w=w_{\lambda^{\prime}_{1}}$,
$z=z_{\lambda^{\prime}_{1}}$. Since $h$ and $k$ are non-increasing and by the
Lemma 4.1, then we get $h(t)=c_{1}$ and $k(t)=c_{2}$, where $c_{1}$ and
$c_{2}$ are positive constants. We have $f(t)=c_{1}t^{\frac{n+b+2}{n+b-2}}$
and $g(t)=c_{2}t^{\frac{n+b+2}{n+b-2}}$. Since $w$ and $z$ are regular at the
origin, we have the asymptotic behavior of $u$ and $v$ at infinity are
$u\sim\frac{1}{|x|^{n+b-2}}$, $v\sim\frac{1}{|x|^{n+b-2}}$.

Similarly, we can prove $u$, $v\in C^{2}(\overline{\mathbb{R}^{n}_{+}})$.
Thus, we take the transformation

$\bar{u}(x^{\prime},x_{n},x_{n+1})=u\left(x^{\prime},\sqrt{x^{2}_{n}+x^{2}_{n+1}}\right)\quad\text{and}\quad\bar{v}(x^{\prime},x_{n},x_{n+1})=v\left(x^{\prime},\sqrt{x^{2}_{n}+x^{2}_{n+1}}\right).$

It is easy to verify that $\bar{u}$ and $\bar{v}$ satisfy equation

$\left\\{\begin{aligned}
\Delta_{n+1}\bar{u}+\frac{b-1}{x_{n+1}}\bar{u}_{n+1}+c_{1}\bar{v}(x)^{\tilde{\tau}}&=0\quad\text{in}\quad\mathbb{R}^{n+1}_{+}\\\
\frac{\partial\bar{u}}{\partial
x_{n+1}}&=0\quad\text{on}\quad\partial\mathbb{R}^{n+1}_{+}.\end{aligned}\right.$
(4.6) $\left\\{\begin{aligned}
\Delta_{n+1}\bar{v}+\frac{b-1}{x_{n+1}}\bar{v}_{n+1}+c_{2}\bar{u}(x)^{\tilde{\tau}}&=0\quad\text{in}\quad\mathbb{R}^{n+1}_{+}\\\
\frac{\partial\bar{v}}{\partial
x_{n+1}}&=0\quad\text{on}\quad\partial\mathbb{R}^{n+1}_{+}.\end{aligned}\right.$
(4.7)

Exactly as the proof of Lemma 3.2, using the asymptotic behavior of $\bar{u}$
and $\bar{v}$ at infinity, one can easily deduce that

$\bar{u}(x)=c_{1}\int_{\mathbb{R}^{n+1}_{+}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{v}(y)^{\tilde{\tau}}dy,$
(4.8)
$\bar{v}(x)=c_{2}\int_{\mathbb{R}^{n+1}_{+}}K_{b-1}(x,y)y^{b-1}_{n+1}\bar{u}(y)^{\tilde{\tau}}dy.$
(4.9)

We establish the equivalence between the integral equation (4.8), (4.9) and
its corresponding differential equation (4.6), (4.7). Using the above moving
plane method for the $n+1$ dimensional integral equations (4.8) and (4.9), we
obtain $\bar{u}$ and $\bar{v}$ are radially symmetric in the direction of
$x_{1},\cdots x_{n-1},x_{n}$. There exists $p\in\partial\mathbb{R}^{n}_{+}$
such that

$u(x^{\prime},|x_{n}|)=\bar{u}(x^{\prime},x_{n},0)=\bar{u}(\bar{x}^{\prime},\bar{x}_{n},0)=u(\bar{x}^{\prime},|\bar{x}_{n}|)$

and

$v(x^{\prime},|x_{n}|)=\bar{v}(x^{\prime},x_{n},0)=\bar{v}(\bar{x}^{\prime},\bar{x}_{n},0)=v(\bar{x}^{\prime},|\bar{x}_{n}|).$

if $\sum^{n}_{i=1}|x_{i}-p_{i}|^{2}=\sum^{n}_{i=1}|\bar{x}_{i}-p_{i}|^{2}$.
Thus, $u$ and $v$ are radially symmetric about $p$. By the results of [4] and
[5] we obtain $u(x)=\left(\frac{ca}{a^{2}+|x-p|^{2}}\right)^{\frac{n+b-2}{2}}$
and $v(x)=\left(\frac{\tilde{c}a}{a^{2}+|x-p|^{2}}\right)^{\frac{n+b-2}{2}}$
where $p\in\partial\mathbb{R}^{n}_{+}$.

Case 2 (The idea of the proof from Theorem 3.2 of [20]) Now we suppose that
$\lambda_{1}=\lambda^{\prime}_{1}=0$ for $x_{1},x_{2}\cdots x_{n-1}$
directions and for all $x_{0}\in\partial\mathbb{R}^{n}_{+}$, then $w$, $z$ and
hence $u$, $v$ are radially symmetric in the $x_{1},x_{2}\cdots x_{n-1}$
directions. We define $S_{C}:=\\{x\in\mathbb{R}^{n}_{+}\ |\ x_{n}=C\\}$.
Similarly, we obtain the solutions $u$ and $v$ depend only on $x_{n}$. Set
$\tilde{u}(x_{n}):=u(x)$, $\tilde{v}(x_{n}):=v(x)$ and we have:

$\left\\{\begin{aligned}
x^{b}_{n}\tilde{u}^{\prime\prime}+bx^{b-1}_{n}\tilde{u}^{\prime}+x^{b}_{n}f(\tilde{v})&=0\\\
\tilde{u}^{\prime}(0)&=0,\end{aligned}\right.$ (4.10) $\left\\{\begin{aligned}
x^{b}_{n}\tilde{v}^{\prime\prime}+bx^{b-1}_{n}\tilde{v}^{\prime}+x^{b}_{n}g(\tilde{u})&=0\\\
\tilde{v}^{\prime}(0)&=0.\end{aligned}\right.$ (4.11)

Exactly as the proof of Lemma 3.3, using $f(t)\geq 0$ and $g(t)\geq 0$ we
deduce that

$t\tilde{u}^{\prime}(t)+(b-1)\tilde{u}(t)\geq 0,$ (4.12)
$t\tilde{v}^{\prime}(t)+(b-1)\tilde{v}(t)\geq 0.$ (4.13)

Similarly to the proof of Lemma 3.4, we obtain

$(b-1)\tilde{u}(t)\geq-t\tilde{u}^{\prime}(t)\geq
Ct^{2}\tilde{v}^{\tilde{\tau}}(t),$
$(b-1)\tilde{v}(t)\geq-t\tilde{v}^{\prime}(t)\geq
Ct^{2}\tilde{u}^{\tilde{\tau}}(t).$

Solving these inequalities, we get for all $t>0$,

$\tilde{u}(t)\leq Ct^{-\frac{n+b-2}{2}},\qquad\tilde{v}(t)\leq
Ct^{-\frac{n+b-2}{2}}.$ (4.14)

Since (4.12) and (4.13), we have

$|\tilde{u}^{\prime}(t)|\leq
Ct^{-\frac{n+b}{2}},\qquad|\tilde{v}^{\prime}(t)|\leq Ct^{-\frac{n+b}{2}}.$
(4.15)

Multiplying (4.10) by $\tilde{v}$ and (4.11) by $\tilde{u}$ and integrating by
parts on $(0,t)$, we get

$t^{b}\tilde{u}^{\prime}(t)\tilde{v}(t)-\int^{t}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}=-\int^{t}_{0}x^{b}_{n}f(\tilde{v})\tilde{v}dx_{n},$
(4.16)
$t^{b}\tilde{u}(t)\tilde{v}^{\prime}(t)-\int^{t}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}=-\int^{t}_{0}x^{b}_{n}g(\tilde{u})\tilde{u}dx_{n}.$
(4.17)

Using the fact that (4.14) and (4.15), we deduce that

$\lim_{t\rightarrow\infty}t^{b}\tilde{u}^{\prime}(t)\tilde{v}(t)=\lim_{t\rightarrow\infty}t^{b}\tilde{u}(t)\tilde{v}^{\prime}(t)=0,$
(4.18)
$\int^{\infty}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}<\infty.$

Hence by (4.16) and (4.17), we have

$\int^{\infty}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}=\int^{\infty}_{0}x^{b}_{n}f(\tilde{v})\tilde{v}dx_{n}=\int^{\infty}_{0}x^{b}_{n}g(\tilde{u})\tilde{u}dx_{n}.$
(4.19)

We define $F(t)=\int^{t}_{0}f(s)ds$ and $G(t)=\int^{t}_{0}g(s)ds$. By
multiplying (4.10) by $x_{n}\tilde{v}^{\prime}$ and (4.11) by
$x_{n}\tilde{u}^{\prime}$ and integrating by parts on $(0,t)$, we obtain

$t^{b+1}\tilde{u}^{\prime}(t)\tilde{v}^{\prime}(t)-\int^{t}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}-\int^{t}_{0}x^{b+1}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime\prime}dx_{n}=-t^{b+1}F(\tilde{v})+(b+1)\int^{t}_{0}x^{b}_{n}F(\tilde{v})dx_{n},$
$\int^{t}_{0}x^{b+1}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime\prime}dx_{n}+b\int^{t}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}=-t^{b+1}G(\tilde{u})+(b+1)\int^{t}_{0}x^{b}_{n}G(\tilde{u})dx_{n}.$

Hence, we get

$t^{b+1}\tilde{u}^{\prime}(t)\tilde{v}^{\prime}(t)+(b-1)\int^{t}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}+t^{b+1}(F(\tilde{v})+G(\tilde{u}))-(b+1)\int^{t}_{0}x^{b}_{n}(F(\tilde{v})+G(\tilde{u}))dx_{n}=0.$

As in the proof of Proposition 3.1, there is a sequence
$t_{k}\rightarrow\infty$ such that

$t^{b+1}_{k}F(\tilde{v}(t_{k}))\rightarrow 0,\qquad
t^{b+1}_{k}G(\tilde{u}(t_{k}))\rightarrow 0.$ (4.20)

Using the fact that (4.14) and (4.15), we have

$\lim_{t\rightarrow\infty}t^{b+1}\tilde{u}^{\prime}(t)\tilde{v}^{\prime}(t)=0.$
(4.21)

And it is easy to see that

$F(\tilde{v}(t))\geq\frac{1}{\tilde{\tau}+1}f(\tilde{v}(t))\tilde{v}(t),\qquad
G(\tilde{u}(t))\geq\frac{1}{\tilde{\tau}+1}g(\tilde{u}(t))\tilde{u}(t).$
(4.22)

By taking $t=t_{k}$, we have

$t^{b+1}_{k}\tilde{u}^{\prime}(t_{k})\tilde{v}^{\prime}(t_{k})+(b-1)\int^{t_{k}}_{0}x^{b}_{n}\tilde{u}^{\prime}\tilde{v}^{\prime}dx_{n}+t^{b+1}_{k}(F(\tilde{v})+G(\tilde{u}))-(b+1)\int^{t_{k}}_{0}x^{b}_{n}(F(\tilde{v})+G(\tilde{u}))dx_{n}=0.$

Letting $t_{k}\rightarrow\infty$ and using (4.19), (4.20), (4.21) and (4.22),
we get

$\left(\frac{b-1}{2}-\frac{b+1}{\tilde{\tau}+1}\right)\int^{\infty}_{0}(x^{b}_{n}f(\tilde{v})\tilde{v}+x^{b}_{n}g(\tilde{u})\tilde{u})dx_{n}\geq
0.$

Since $\frac{b-1}{2}-\frac{b+1}{\tilde{\tau}+1}<0$, we get a contradiction. We
obtain $u(x)\equiv 0$ and $v(x)\equiv 0$. $\Box$

Acknowledgments. The work of the author is sponsored by Shanghai Rising-Star
Program 19QA1400900. The author would like to thank Professor Huang Genggeng
for his valuable suggestions. The author was supported by National Natural
Science Foundation of China under Grant 11871160.

## References

  * [1] Wenxiong Chen, Yanqin Fang, and Ray Yang.  Liouville theorems involving the fractional Laplacian on a half space.  Adv. Math., 274:167–198, 2015.
  * [2] Wenxiong Chen and Congming Li.  Classification of solutions of some nonlinear elliptic equations.  Duke Math. J., 63(3):615–622, 1991.
  * [3] Wenxiong Chen and Congming Li.  Classification of positive solutions for nonlinear differential and integral systems with critical exponents.  Acta Math. Sci. Ser. B (Engl. Ed.), 29(4):949–960, 2009.
  * [4] Wenxiong Chen, Congming Li, and Biao Ou.  Classification of solutions for a system of integral equations.  Comm. Partial Differential Equations, 30(1-3):59–65, 2005.
  * [5] Wenxiong Chen, Congming Li, and Biao Ou.  Classification of solutions for an integral equation.  Comm. Pure Appl. Math., 59(3):330–343, 2006.
  * [6] Philippe Clément, Raúl Manásevich, and Enzo Mitidieri.  Positive solutions for a quasilinear system via blow up.  Comm. Partial Differential Equations, 18(12):2071–2106, 1993.
  * [7] D. G. de Figueiredo and P. L. Felmer.  A Liouville-type theorem for elliptic systems.  Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4), 21(3):387–397, 1994.
  * [8] Yanqin Fang and Wenxiong Chen.  A Liouville type theorem for poly-harmonic Dirichlet problems in a half space.  Adv. Math., 229(5):2835–2867, 2012.
  * [9] Basilis Gidas, Weiming Ni, and Louis Nirenberg.  Symmetry and related properties via the maximum principle.  Comm. Math. Phys., 68(3):209–243, 1979.
  * [10] Basilis Gidas, Weiming Ni, and Louis Nirenberg.  Symmetry of positive solutions of nonlinear elliptic equations in ${\mathbb{R}}^{n}$.  7:369–402, 1981.
  * [11] Yuxia Guo and Jiaquan Liu.  Liouville type theorems for positive solutions of elliptic system in $\mathbb{R}^{N}$.  Comm. Partial Differential Equations, 33(1-3):263–284, 2008.
  * [12] Yuxia Guo and Jianjun Nie.  Classification for positive solutions of degenerate elliptic system.  Discrete Contin. Dyn. Syst., 39(3):1457–1475, 2019.
  * [13] Toshio Horiuchi.  The Schauder approach to degenerate elliptic equations with homogeneous Neumann boundary condition. I.  Bull. Fac. Sci. Ibaraki Univ. Ser. A, (27):7–32, 1995.
  * [14] Genggeng Huang.  A Liouville theorem of degenerate elliptic equation and its application.  Discrete Contin. Dyn. Syst., 33(10):4549–4566, 2013.
  * [15] Genggeng Huang and Congming. Li.  A Liouville theorem for high order degenerate elliptic equations.  J. Differential Equations, 258(4):1229–1251, 2015.
  * [16] Dongyan Li and Ran Zhuo.  An integral equation on half space.  Proc. Amer. Math. Soc., 138(8):2779–2791, 2010.
  * [17] Elliott H. Lieb.  Sharp constants in the Hardy-Littlewood-Sobolev and related inequalities.  Ann. of Math. (2), 118(2):349–374, 1983.
  * [18] Jiaqun Liu, Yuxia Guo, and Yajing Zhang.  Liouville-type theorems for polyharmonic systems in $\mathbb{R}^{N}$.  J. Differential Equations, 225(2):685–709, 2006.
  * [19] Li Ma and Dezhong Chen.  A Liouville type theorem for an integral system.  Commun. Pure Appl. Anal., 5(4):855–859, 2006.
  * [20] Enzo Mitidieri.  A Rellich type identity and applications.  Comm. Partial Differential Equations, 18(1-2):125–151, 1993.
  * [21] Enzo Mitidieri.  Nonexistence of positive solutions of semilinear elliptic systems in ${\mathbb{R}}^{N}$.  Differential Integral Equations, 9(3):465–479, 1996.
  * [22] James Serrin.  A symmetry problem in potential theory.  Arch. Rational Mech. Anal., 43:304–318, 1971.
  * [23] James Serrin and Henghui Zou.  Non-existence of positive solutions of Lane-Emden systems.  Differential Integral Equations, 9(4):635–653, 1996.
  * [24] Sufang Tang and Jingbo Dou.  Liouville type theorems for a system of integral equations on upper half space.  Acta Math. Sin. (Engl. Ser.), 30(2):261–276, 2014.
  * [25] Xiaohui Yu.  Liouville type theorems for integral equations and integral systems.  Calc. Var. Partial Differential Equations, 46(1-2):75–95, 2013\.

Addresse and E-mail:

Yating Niu

School of Mathematical Sciences

Fudan University

<EMAIL_ADDRESS>