11institutetext: Graduate School of Informatics, Kyoto University

# BlindSpotNet: Seeing Where We Cannot See

Taichi Fukuda  Kotaro Hasegawa  Shinya Ishizaki
Shohei Nobuhara  Ko Nishino

###### Abstract

We introduce 2D blind spot estimation as a critical visual task for road scene
understanding. By automatically detecting road regions that are occluded from
the vehicle’s vantage point, we can proactively alert a manual driver or a
self-driving system to potential causes of accidents (_e.g_., draw attention
to a road region from which a child may spring out). Detecting blind spots in
full 3D would be challenging, as 3D reasoning on the fly even if the car is
equipped with LiDAR would be prohibitively expensive and error prone. We
instead propose to learn to estimate blind spots in 2D, just from a monocular
camera. We achieve this in two steps. We first introduce an automatic method
for generating “ground-truth” blind spot training data for arbitrary driving
videos by leveraging monocular depth estimation, semantic segmentation, and
SLAM. The key idea is to reason in 3D but from 2D images by defining blind
spots as those road regions that are currently invisible but become visible in
the near future. We construct a large-scale dataset with this automatic
offline blind spot estimation, which we refer to as Road Blind Spot (RBS)
dataset. Next, we introduce BlindSpotNet (BSN), a simple network that fully
leverages this dataset for fully automatic estimation of frame-wise blind spot
probability maps for arbitrary driving videos. Extensive experimental results
demonstrate the validity of our RBS Dataset and the effectiveness of our BSN.

###### Keywords:

autonomous driving, ADAS, road scene understanding, blind spot estimation,
accident prevention

##  1 Introduction

Fully autonomous vehicles may soon become a reality. Advanced Driver-
Assistance Systems (ADAS) have also become ubiquitous and intelligent. A large
part of these developments is built on advancements in perceptual sensing,
both in hardware and software. In particular, 3D and 2D visual sensing have
played a large role in propelling these advancements. State-of-the-art LiDAR
systems can resolve to centimeters at 300m or longer, and image understanding
networks can recognize objects 100m away. A typical autonomous vehicle is
equipped with up to ten of these sensors’ visual perception, in addition to
other modalities.

These autonomous and assistive driving systems are, however, still prone to
catastrophic errors, even when they are operating at low speeds [20]. More
sensors would unlikely eliminate these errors. In fact, we human beings have
much fewer visual sensors (just our two eyes) but can drive at least as well
as current autonomous vehicles. Why are we able to do this? We believe, one of
the primary reasons is that, although we are limited in our visual perception,
we know that it’s limited. That is, we are fully aware of when and where we
can see and when and where we cannot see. We know that we can’t see well at
night so we drive cautiously; we know that we cannot see beyond a couple of
hundred meters, so we don’t drive too fast; we know that we cannot see behind
us, so we use side and room mirrors. Most important, we know that we cannot
see behind objects, _i.e_., that our vision can be obstructed by other objects
in the scene. We are fully aware of these blind spots, so that we pay
attention to those areas on the road with anticipation that something may
spring out from them. That is, we know where to expect the unforeseen and we
preemptively prepare for those events by attending our vision and mind to
those blind spots.

Can we make computers also “see” blind spots on the road? One approach would
be to geometrically reason occlusions caused by the static and dynamic objects
on the road (_e.g_., pedestrians and cars). This requires full 3D scene
reconstruction and localization of the moving camera, which is computationally
expensive and prone to errors as it requires fragile ray traversal. Instead,
it would be desirable to directly estimate blind spots in the 2D images
without explicit 3D reconstruction or sensing. This is particularly essential
for driving safety, as we would want to detect blind spots on the fly as we
drive down the road. Given an image (_i.e_., a frame from a live video), we
would like to mark out all the road regions that are occluded by the camera,
so that a driver or the autonomous system would know where to anticipate the
unforeseen.

How can we accomplish 2D blind spot detection? Just like we likely do, we
could learn to estimate blind spots directly in 2D. The challenge for this
lies in obtaining the training data–the ground truth. Again, explicit 3D
reasoning is unrealistic as a perfect 3D reconstruction of every frame of a
video would be prohibitively expensive and error prone, especially for a
dynamic road scene in which blind spots change every frame. On the other hand,
labeling blind spots in video frames is also near impossible, as even to the
annotators, reasoning about the blind spots from 2D images is too hard a task,
particularly for a dynamic scene. How can we then, formulate blind spot
estimation as a learning task?

In this paper, we introduce a novel dataset and network for estimating
occluded road regions in a driving video. We refer to the dataset as Road
Blind Spot (RBS) Dataset and the network as BlindSpotNet (BSN). Our major
contributions are two-fold: an algorithm for automatic generation of blind
spot training data and a simple network that learns to detect blind spots from
that training data. The first contribution is realized by implicitly reasoning
3D occlusions in a driving video from its 2D image frames by fully leveraging
depth, localization, and semantic segmentation networks. This is made possible
by our key idea of defining blind spots as areas on the road that are
currently invisible but visible in the future. We refer to these as $T$-frame
blind spots, _i.e_., those 2D road regions that are currently occluded by
other objects that become visible $T$ frames later. Clearly, these form a
subset of all true 2D blind spots; we cannot estimate the blind spots that
never become seen in our video. They, however, cover a large portion of the
blind spots (_cf_. Fig. 5) that are critical in a road scene including those
caused by parked cars on the side, oncoming cars on the other lane, street
corners, pedestrians, and buildings. Most important, they allows us to derive
an automatic means for computing blind spot regions for arbitrary driving
videos.

Our offline automatic training data generation algorithm computes $T$-frame
blind spots for each frame of a driving video by playing it backwards, and by
applying monocular depth estimation, SLAM, and semantic segmentation to obtain
the depth, camera pose, semantic regions, and road regions. By computing the
visible road regions in every frame, and then subtracting the current frame’s
from that of $T$-frames ahead, we can obtain blind spots for the current
frame. A suitable value for $T$ can be determined based on the speed of the
car and the frame rate of the video. Armed with this simple yet effective
algorithm for computing blind spot maps, we construct the RBS Dataset. The
dataset consists of 231 videos with blind spot maps computed for 21,662
frames.

For on-the-fly 2D blind spot estimation, we introduce BlindSpotNet, a deep
neural network that estimates blind spot road regions for an arbitrary road
scene directly for a single 2D input video frame, which fully leverages the
newly introduced dataset. The network architecture is a fully convolutional
encoder-decoder with Atrous Spatial Pyramid Pooling which takes in a road
scene image as the input. We show that blind spot estimation can be
implemented with a light-weight network by knowledge distillation from a
semantic segmentation network. Through extensive experiments including
ablation studies to analyze the effectiveness of the network architecture, we
show that BlindSpotNet can accurately estimate the occluded road regions in
any given frame independently, but resulting in consistent blind spot maps
through a video.

To the best of our knowledge, our work is the first to offer an extensive
dataset and a baseline method for solving this important problem of 2D blind
spot estimation. These results can directly be used to heighten the safety of
autonomous driving and assisted driving, for instance, by drawing attention of
the limited computation resource or the human driver to those blind spots
(_e.g_., by sounding an alarm from the side with an oncoming large blind spot,
if the person is looking away observed from an in-car camera). We also
envision a future where BlindSpotNet helps in training better drivers both for
autonomous and manual driving. The dataset and code will be made public once
the paper is accepted to ensure timely progress towards such a future.

##  2 Related Work

Driver assistance around blind corners has been studied in the context of
augmented/diminished reality [1, 27, 17]. Barnum _et al_. proposed a video-
overlay system that presents a see-through rendering of hidden objects behind
the wall for drivers using a surveillance camera installed at corners [1].
This system realizes realtime processing, but requires explicit modeling of
blind spots beforehand. Our proposed method estimates them automatically.

Bird’s-eye view (BEV) visualization [30, 11] and scene parsing [26, 22, 16]
around the vehicle can also assist drivers to avoid collision accidents.
Sugiura and Watanabe [22] proposed a neural network that produces probable
multi-hypothesis occupancy grid maps. Mani _et al_. [16], Yang _et al_. [26],
and Liu _et al_. [11] proposed road layout estimation networks. These methods,
however, require sensitive 3D reasoning to obtain coarse blind spots at
runtime, while our BSN estimates blind spots directly in 2D without any 3D
processing.

Amodal segmentation also handles occluded regions of each instance explicitly
[31, 9, 12, 21, 7, 24, 29]. They segment the image into semantic regions while
estimating occluded portions of each region. These methods, however, do not
estimate objects that are completely invisible in the image. We can find a
side road while its road surface is not visible at all, _e.g_., due to cars
parked in the street, by looking at gaps between buildings for example.

Understanding and predicting pedestrian behavior is also a primary objective
of ADAS. Makansi _et al_. [14] trained a network that predicts pedestrians
crossing in front of the vehicle. Bertoni _et al_. [2] estimated 3D locations
of pedestrians around the subject by also modeling the uncertainty behind
them. Our method explicitly recovers blind spots caused not only by
pedestrians but also other obstacles including passing and parked cars, street
medians, poles, etc.

Figure 1: We define blind spots as road regions that are currently invisible
but visible in the next $T$-frames, which we compute by aggregating
traversable surface areas across future frames and negating the visible
traversable surface area in the current frame. These $T$-frame blind spots
form a subset of true blind spots, cover key regions of them and, most
important, can directly be computed for arbitrary driving videos (_cf_. Fig. 5
for a real example).

Watson _et al_. [25] estimated free space including the area behind objects in
the scene for robot navigation. They also generated a traversable area dataset
to train their network. Our dataset generation leverages this footprint
dataset generation algorithm to compute both visible and invisible road areas
in a road scene image (_i.e_., driving video frame). Blind spot estimation
requires more than just the area behind objects as it needs to be computed and
estimated across frames with a dynamically changing viewpoint.

##  3 Road Blind Spot Dataset

Figure 2: Overview of algorithmic steps for generating $T$-frame blind spots
for arbitrary driving video. Visible and invisible traversable surface maps
are aggregated into the current frame from the next $T$ frames, from which the
visible traversable region is negated to obtain the $T$-frame blind spots for
the current frame.

Our first goal is to establish a training dataset for learning to detect 2D
blind spots. For this, we need an algorithm that can turn an arbitrary video
into a video annotated with 2D blind spots for each and every frame. To derive
such a method, we start by defining road-scene blind spots as something that
we can compute from 2D videos offline and derive a method for computing those
for arbitrary driving videos. With this algorithm, we construct a large-scale
dataset of road blind spot videos (RBS Dataset). This data will later be used
to train a network that can estimate 2D blind spots on the fly.

###  3.1 $T$-Frame Blind Spots

In the most general form, blind spots are volumes of the 3D space that are
occluded from the viewpoint by an object in the scene. Computing these “full
blind volumes” would be prohibitively expensive, especially for any
application that requires real-time decision making. Even though our goal in
this paper is not necessarily real-time computation at this point, a full 3D
reasoning of occluded volumes would be undesirable as our target scenes are
dynamic. We, instead, aim to estimate 2D blind spots on the road. Dangerous
traffic situations are usually caused by unanticipated movements of dynamic
objects (_e.g_., bikes, pedestrians, children, _etc_.) springing out from road
areas invisible from the driver (or the camera of the car). Once we have the
2D blind spots, we can draw attention of the drivers by, for instance,
extruding it perpendicularly to the road for 3D warning. By focusing on 2D
blind spot estimation, we eliminate the need of explicit 3D geometric
reasoning, which makes it particularly suitable for autonomous driving and
ADAS applications.

As depicted in Fig. 1, given a frame of a driving video, we define our blind
spots to be the road regions that are obstructed but becomes visible in the
future. This clearly excludes blind spots that never become visible in any
frame in the future, and thus the blind spots we compute and estimate are a
subset of the true blind spots. That said, they have a good coverage and can
be reliably computed from just ego-centric driving videos as we show in
experimental results. Although we only investigate the estimation of these
blind spots from ego-centric driving videos captured with regular perspective
cameras, the wider the field of view, the better the coverage would become.

Formally, given a frame $I_{t}$, we define its blind spots as pixels
$x\in\Omega_{t}^{T}$ corresponding to regions on the road $\Omega$ that are
occluded but become visible in the next $T$ frames
$\\{I_{t+i}|i=1,\dots,T\\}$. Our goal is to compute the set of pixels in the
blind spots $\Omega_{t}$ as a binary mask of the image
$\omega(x,t;T):\mathbb{R}^{2}\times\mathbb{R}\mapsto\\{0,1\\}$. Later
BlindSpotNet will be trained to approximate this function $\omega(x,t;T)$. We
refer to the blind spots of this definition as $T$-frame blind spots.

As depicted in Fig. 2, we compute the blind spot map $\omega$ by first
computing and aggregating both visible and invisible traversable surface maps
at frames $I_{t:t+T}$ and then eliminating the visible traversable surface map
at target frame $I_{t}$. For this, similar to Watson _et al_. [25], we compute
a traversable surface map $s(x,t;T)$ by forward warping the traversable pixels
from the next $T$ frames $\\{I_{t+i}|i=1,\dots,T\\}$ to the target frame
$I_{t}$. To perform forward warping, we assume the camera intrinsics are
known, the extrinsic parameter and the depth are estimated by a visual SLAM
algorithm [23] and an image-based depth estimation network [18], respectively.

Let $r(x,t)$ denote the visually-traversable region defined as a binary mask
representing the union of the road and pavement areas estimated by a semantic
segmentation as introduced in Watson _et al_. [25]. We first project the
visually-traversable regions estimated for $I_{t+i}$ ($i=1,\dots,T)$ to
$I_{t}$ as $r^{\prime}(x,t+i;t)$, and then aggregate them as

$s(x,t;T)=r^{\prime}(x,t+1;t)\lor r^{\prime}(x,t+2;t)\lor\dots\lor
r^{\prime}(x,t+T;t)\,,$ (1)

where $\lor$ denotes the pixel-wise logical OR.

Figure 3: Visibility mask. Left: If the vehicle makes a turn, the blind spots
straight ahead across the intersection never become visible. Right: To allow
networks estimate such blind spots, we prepare a mask image to indicate the
visible area for each frame.

As blind spots are by definition invisible regions in frame $I_{t}$, the
visually-traversable region $r(x,t)$ is deleted from $s(x,t;T)$ to obtain the
final blind spots $\omega(x,t;T)$ by

$\omega(x,t;T)=s(x,t;T)\land\bar{r}(x,t)\,,$ (2)

where $\land$ and $\bar{\cdot}$ denote the pixel-wise logical AND and
negation, respectively.

Our method relies on three componential visual understanding tasks, namely
semantic segmentation, monocular depth estimation, and visual SLAM. Although
existing methods for these tasks achieve high accuracy, they can still suffer
from slight errors. In the transformation from $r$ to $r^{\prime}$, we use two
such estimates, those of the camera pose and the depth, whose errors can cause
residuals of visible road regions after Eq. 2.

We can rectify this with simple depth comparison. We first define aggregated
depth mask $d_{a}$

$d_{a}(x,t;T)=\frac{1}{M}\sum_{i=1}^{T}r^{\prime}(x,t+i;t)d^{\prime}(x,t+i;t)\qquad
M=\sum_{i=1}^{T}r^{\prime}(x,t+i;t)\,,$ (3)

where $d^{\prime}(x,t+i;t)$ is a depth map of frame $I_{t+i}$ projected onto
frame $I_{t}$, which is calculated in a similar way as calculation of
$r^{\prime}(x,t+i;t)$. $M$ is the pixel-wise count of visually-traversable
mask over $T$ frames.

When we compare the depth map $d(x)$ of frame $I_{t}$ with the aggregated
depth mask $d_{a}$, the depth difference is large in the true blind spot
region because true blind spots are occluded by foreground objects. On the
other hand, it is small in erroneous blind spot regions because the compared
depth values come from nearby pixels. Based on this observation, we remove
erroneous blind spots by setting $\omega(x,t;T)=0$ for the pixel $x$ that
satisfy $|d(x)-d_{a}(x)|<l_{d}$. Here, $l_{d}$ is a threshold value determined
empirically.

We may be left with small blind spots caused by, for instance, a shadow of a
tire. These small blind spots are not important for driving safety. We remove
these blind spots of less than 100 pixels from the final blind spot
$\Omega_{t}$.

For building our RBS Dataset, we opt for MiDAS [18] as the monocular depth
estimator, OpenVSLAM [23] for SLAM, and Panoptic-DeepLab [5] for semantic
segmentation. The scale of the depth estimated by MiDAS is linearly aligned
with least squares fitting to the sparse 3D landmarks recovered by SLAM.

We use KITTI [8], BDD100k [28], and TITAN [15] datasets to build our RBS
Dataset. By excluding videos for which the linear correlation coefficient in
the MiDAS-to-SLAM depth alignment is less than 0.7, they provide 51, 62, and
118 videos, respectively. We obtain blind spot masks for approximately 51, 34,
and 12 minutes of videos in total, respectively. The videos are resampled to 5
fps, and we set $T=5$ seconds for each video. We refer to them as KITTI-RBS,
BDD100k-RBS, and TITAN-RBS Datasets, respectively.

###  3.2 Visibility mask

$T$-frame blind spots clearly do not cover blind spots that do not become
visible through the video. For example, as illustrated in Fig. 3, consider a
frame where the vehicle is making a right turn. The blind spots straight ahead
across the intersection never become visible and hence are not included in the
dataset, while BlindSpotNet should identify them once trained. To disambiguate
such invisible regions from non-blind spots, as shown in Fig. 3, we generate a
binary mask $V_{t}$ called visibility mask in addition to the blind spots
$\Omega_{t}$.

We use semantic segmentation and the distance from the camera to define the
visibility mask $V_{t}$. For each pixel $x$, we first classify $x$ as visible,
if the semantic segmentation label is “sky.” For non-sky pixels, we classify
$x$ as visible if the minimum distance from the 3D point at distance $d(x,t)$
behind $x$ to the camera is less than a certain threshold $L$. In our
implementation, we set $L=16$ meters.

##  4 BlindSpotNet

Now that we have (and can create limitless) abundant video data with per-frame
blind spot annotations, we can formulate 2D blind spot detection as a learning
problem. We derive a novel deep neural network for estimating blind spots in
an image of a road scene. We refer to this network as BlindSpotNet and train
and test it on our newly introduced RBS Dataset.

###  4.1 Network Architecture

Figure 4: Overall architecture of BlindSpotNet. We leverage the fact that
blind spot estimation bears similarity to semantic segmentation by adopting a
light-weight network trained with knowledge distillation from a semantic
segmentation teacher network.

As we saw in Sec. 3, blind spots are mainly determined by the scene
composition of objects including road regions and their ordering in 2D. As
such, at a higher level, direct image-based estimation of blind spots shares
similarity in its task to semantic segmentation. The task is, however, not
necessarily easier, as it is 2D dense labeling but requires implicit 3D
reasoning. Nevertheless, the output is a binary map (or its probability map),
which suggests that a simpler network but with a similar inherent
representation to semantic segmentation would be suitable for blind spot
detection.

Fig. 4 depicts the network architecture of BlindSpotNet. BlindSpotNet consists
of three components: a depth estimator $\mathcal{D}$, a semantic segmentation
teacher network $\mathcal{S}$, and a blind spot estimator $\mathcal{B}$. Given
an input image $I$ of size $W{\times}H{\times}3$, the depth estimator
$\mathcal{D}$ estimates the depth map $D$ of size $W{\times}H$ from $I$. The
blind spot estimator takes both the RGB image $I$ and the estimated depth map
$D$ as inputs and generates the blind spot map $B=\\{b\in[0,1]\\}$.

The semantic segmentation network $\mathcal{S}$ serves as a teacher network to
help train the blind spot estimator $\mathcal{B}$. The blind spot estimator
$\mathcal{B}$ should be trained to reason semantic information of the scene
similar to semantic segmentation, but its output is abstracted as simple as a
single-channel map $B$. This implies that training the blind spot estimator
$\mathcal{B}$ only with the $T$-frame blind spots can easily bypass the
semantic reasoning of the scene and overfit. To mitigate this shortcut, we
introduce the semantic segmentation network $\mathcal{S}$ pretrained on road
scenes as a teacher and use its decoder output as a soft target of a
corresponding layer output in the blind spot estimator $\mathcal{B}$.

###  4.2 Knowledge Distillation

Blind spot regions are highly correlated with the semantic structure of the
scene. For instance, blind spots can appear behind vehicles and buildings, but
never in visible road regions or in the sky. The blind spot estimator
$\mathcal{B}$ should thus be able to learn useful representations from
semantic segmentation networks for parsing road scenes when looking for blind
spots. For this, we distill knowledge from a pretrained semantic segmentation
network to our blind spot estimator $\mathcal{B}$. Based on the work of Liu
_et al_. [13], we transfer the similarity between features at an intermediate
layer of each network.

Suppose we subdivide the feature map of an intermediate layer of size
$W^{\prime}\times H^{\prime}\times C$ into a set of $w^{\prime}\times
h^{\prime}$ patches. By denoting the spatial average of the features in the
$i$th patch by $\bm{f}_{i}\in\mathbb{R}^{C}$, we define the similarity of two
patches $i$ and $j$ by their cosine distance
$a_{ij}=\frac{\bm{f}_{i}^{\top}\bm{f}_{j}}{\|\bm{f}_{i}\|\|\bm{f}_{j}\|}.$
Following [13], given this pairwise similarity for patches in both the teacher
semantic segmentation network $\mathcal{S}$ and the student network, _i.e_.,
the blind spot estimator $\mathcal{B}$, as $a_{ij}^{\mathcal{S}}$ and
$a_{ij}^{\mathcal{B}}$, respectively, we introduce a pair-wise similarity
distillation loss $l_{\mathrm{KD}}$ as

$l_{\mathrm{KD}}=\frac{1}{\left(w^{\prime}\times
h^{\prime}\right)^{2}}\sum_{i\in\mathcal{R}}\sum_{j\in\mathcal{R}}\left(a_{ij}^{\mathcal{S}}-a_{ij}^{\mathcal{B}}\right)^{2}\,,$
(4)

where $\mathcal{R}=\\{1,2,\dots,w^{\prime}\times h^{\prime}\\}$ denotes the
entire set of patches. In our implementation, we opted for DeepLabV3+ [4] as
the teacher network $\mathcal{S}$.

###  4.3 Loss Function

In addition to the similarity distillation loss $l_{\mathrm{KD}}$ in Eq. 4, we
employ a binary cross entropy loss $l_{\mathrm{BCE}}$ between the output of
the blind spot estimator $\mathcal{B}$ and the $T$-frame blind spots given by
our RBS Dataset as

$l_{\mathrm{BCE}}=-\frac{1}{|V|}\sum_{x\in V}(\omega(x)\log
b(x)+(1-\omega(x))\log(1-b(x)))\,,$ (5)

where $x$ denotes the pixels in the visibility mask $V$, $\omega(x)$ and
$b(x)$ denote the $T$-frame blind spots and the estimated probabilities at
pixel $x$. $|V|$ is the total number of the pixels in $V$. The total loss
function $L$ to be optimized is defined as a weighted sum of these two loss
functions $L=l_{\mathrm{BCE}}+\lambda l_{\mathrm{KD}}$, where $\lambda$ is an
empirically determined weighting factor.

##  5 Experimental Results

We evaluate the validity of RBS Dataset and the effectiveness of BlindSpotNet
(BSN), qualitatively and quantitatively with a comprehensive set of
experiments.

###  5.1 RBS Dataset Evaluation

Table 1: Quantitative evaluation of RBS Datasets. The two numbers in each cell indicate the recall and the false-negative rate w.r.t. the ground truth blind spot. Please see Fig. 5 for visualization. The results show that our $T$-frame blind spots approximate true blind spots well. | CARLA | KITTI
---|---|---
| Rec.$\uparrow$ | FN rate$\downarrow$ | Rec.$\uparrow$ | FN rate$\downarrow$
$T$-frame BS-GT | 0.372 | 0.013 | N/A111$T$-frame BS-GT is not available since KITTI does not have ground truth semantic segmentation for most of the frames.
$T$-frame BS (ours) | 0.297 | 0.015 | 0.169 | 0.056

We first validate our $T$-frame blind spots with synthetic data generated by
CARLA [6] and with real data from KITTI [8]. How well do they capture true
blind spots? We use $360^{\circ}$ LiDAR scans to obtain ground-truth blind
spots (BS-GT). Note again that for real use such ground truth computation will
be prohibitively expensive and would require LiDAR. We also use the ground-
truth depth maps to compute $T$-frame blind spots without noise ($T$-frame BS-
GT). In computing BS-GT, we find road regions in LiDAR points by fitting the
road plane manually in 3D. For $T$-frame BS-GT, we used the ground truth
semantic labels. We compare the $T$-frame blind spots generated by our data
generation algorithm ($T$-frame BS) and $T$-frame BS-GT with BS-GT, and
evaluate its quality in terms of the recall and the false-negative rate. Tab.
1 and Fig. 5 show the results. Since BS-GT is defined by sparse LiDAR points
while $T$-frame BS-GT and $T$-frame BS use dense depth-maps, the precision and
the false-positive (type-I error) rate do not make sense. These results show
that our $T$-frame blind spots approximates the ground-truth blind spot well.

Figure 5: Comparison of $T$-frame blind spot and “ground truth” obtained from
LiDAR data. Top-left and bottom-left figures show the ground truth and
$T$-frame blind spots, respectively. Right figure shows the 3D projection of
ground truth (yellow) and $T$-frame blind spots (red) to LiDAR scan (blue).
The green cross mark indicates the camera / LiDAR position.

###  5.2 BlindSpotNet Evaluation

##### BlindSpotNet

We use MiDAS [18] and DeepLabV3+ [4] as the depth estimator $\mathcal{D}$ and
the semantic segmentation subnetwork $\mathcal{S}$, respectively. We use
ResNet-18 and Atrous Spatial Pyramid Pooling [4] for the blind spot estimator
$\mathcal{B}$ following DeepLabV3+. BlindSpotNet is trained by Adam [10] with
$\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=1\times 10^{-8}$, and weight decay
$5\times 10^{-4}$. The learning rate is initialized to $0.001$ and a
polynomial scheduler is applied. The knowledge distillation coefficient
$\lambda$ in Sec. 4.3 is fixed to $1.0$.

We divided the 10,135 frames from KITTI-RBS into training, validation, and
test sets by $8:1:1$, and the 8,872 frames from BDD100k-RBS into training and
test sets by $8:2$. We used all the 2,655 frames from TITAN-RBS for evaluation
only since each video is too short (10 to 20 seconds) to be used for training.

##### Metrics

Blind spot estimation is a binary segmentation problem. For each input frame,
our BlindSpotNet outputs the probability map of blind spots. We threshold this
probability map to obtain the final binary blind spot mask, and compare it
with the $T$-frame blind spots by IoU, recall, and precision. The threshold is
determined empirically for each dataset as it depends on the road scene. We
plan to learn this threshold as part of the network in future work. Notice
that our RBS Datasets include blind spots that becomes visible in $T$ frames
only. For IoU, recall, and precision, we only consider pixels in the
visibility mask.

##### Baselines

As discussed earlier, our work is the first for image-based 2D blind spot
estimation, and there are no other existing methods in the literature to the
best of our knowledge. For comparison, we adapt state-of-the-art traversable
region estimation [25], 2D vehicle/pedestrian/cyclist detection by semantic
segmentation [4], and 3D vehicle/pedestrian/cyclist detection [3] for 2D blind
spot estimation as baselines and refer to them as Traversable, Detection-2D,
and Detection-3D, respectively.

For Traversable we use the hidden traversable regions, estimated by the
original implementation of Watson _et al_. [25], as blind spots. Detection-2D
is a simple baseline that detects vehicle, pedestrian, and cyclist regions
estimated by DeepLabV3+ [4] as blind spots. Detection-3D utilizes a single-
image 3D detection of vehicles, pedestrians, and cyclists by Brazil _et al_.
[3]. Given their detected 3D bounding boxes, Detection-3D returns the
intersection of the projection of their far-side faces and the results by
Detection-2D as blind spots.

We also compare with BSN without depth estimation (BSN-D), and BSN without
knowledge distillation (BSN-KD) for ablation studies. In BSN-D, the depth
estimator $\mathcal{D}$ is removed from BSN, and the blind spot estimator
$\mathcal{B}$ is modified to take the original RGB image directly. BSN-KD
disables the knowledge distillation loss by setting $\lambda=0$ in Sec. 4.3 in
BSN.

####  5.2.1 Quantitative Evaluations

Table 2: Quantitative results on the test sets from KITTI-RBS, BDD100k-RBS, and TITAN-RBS. The lines with “w/ KITTI-RBS” and “w/ BDD100k-RBS” report the results by the networks trained with KITTI-RBS and BDD100k-RBS Datasets, respectively. These results show that depth estimation and knowledge distillation contribute independently to the final accuracy, and our BSN successfully estimates blind spots across different scenes (_i.e_., datasets). | KITTI-RBS | BDD100k-RBS | TITAN-RBS
---|---|---|---
Model |  IoU$\uparrow$ |  Rec.$\uparrow$ |  Prec.$\uparrow$ |  IoU$\uparrow$ |  Rec.$\uparrow$ |  Prec.$\uparrow$ |  IoU$\uparrow$ |  Rec.$\uparrow$ |  Prec.$\uparrow$
Traversable based on [25] | 0.176 | 0.462 | 0.222 | 0.088 | 0.198 | 0.136 | 0.135 | 0.303 | 0.196
Detection-2D based on [5] | 0.129 | 0.581 | 0.142 | 0.184 | 0.652 | 0.204 | 0.142 | 0.435 | 0.182
Detection-3D based on [3] | 0.182 | 0.368 | 0.265 | 0.059 | 0.067 | 0.316 | 0.048 | 0.057 | 0.216
BSN-D w/ KITTI-RBS | 0.296 | 0.601 | 0.368 | 0.295 | 0.438 | 0.475 | 0.250 | 0.474 | 0.345
BSN-KD w/ KITTI-RBS | 0.305 | 0.646 | 0.367 | 0.225 | 0.249 | 0.700 | 0.168 | 0.249 | 0.342
BSN (Ours) w/ KITTI-RBS | 0.330 | 0.563 | 0.444 | 0.283 | 0.349 | 0.599 | 0.187 | 0.280 | 0.360
BSN-D w/ BDD100k-RBS | 0.270 | 0.629 | 0.321 | 0.360 | 0.478 | 0.593 | 0.244 | 0.420 | 0.367
BSN-KD w/ BDD100k-RBS | 0.187 | 0.210 | 0.633 | 0.350 | 0.443 | 0.624 | 0.253 | 0.529 | 0.326
BSN (Ours) w/ BDD100k-RBS | 0.314 | 0.599 | 0.398 | 0.364 | 0.533 | 0.535 | 0.257 | 0.554 | 0.324

Tab. 2 shows the results on the test sets from KITTI-RBS, BDD100k-RBS, and
TITAN-RBS Datasets. The lines with “w/ KITTI-RBS” and “w/ BDD100k-RBS”
indicate the results of the networks trained with KITTI-RBS and BDD100k-RBS
Datasets, respectively. Each network, after pre-training, was fine-tuned using
20% of the training data of the target dataset to absorb strong scene biases.
It is worth mentioning that this fine-tuning is closer to self-supervision as
the $T$-frame blind spots can be automatically computed without any external
supervision for arbitrary videos. As such, BlindSpotNet can be applied to any
driving video without suffering from domain gaps, as long as a small amount of
video can be acquired before running BlindSpotNet for inference. The 20%
training data usage of the target scene simulates such a scenario. Note that
none of the test data were used and this fine-tuning was not done for TITAN-
RBS.

BSN outperforms Watson _et al_. [25], Detection-2D, and Detection-3D. These
results show that blind spot estimation cannot be achieved by simply
estimating footprint or “behind-the-vehicle/pedestrian/cyclist” regions. The
full BSN also performs better than BSN-D and BSN-KD. This suggests that both
the depth estimator and knowledge distillation contribute to its performance
independently. Furthermore, the performance of BSN w/ KITTI-RBS on BDD100k-RBS
and TITAN-RBS and that of BSN w/ BDD100k-RBS on KITTI-RBS and TITAN-RBS
demonstrate the ability of BSN to generalize across datasets.

$T$-Frame blind spots TraversableDetection-2D Detection-3D BSN-DBSN-KDBSN
Figure 6: Blind spot estimation results. From top to bottom, $T$-frame blind
spots from our RBS Dataset, the results by Traversable, Detection-2D,
Detection-3D, BSN-D, BSN-KD, and BSN, respectively. Results are shown for
BlindSpotNet successfully achieves high precision and recall for complex road
scenes (KITTI, BDD, and TITAN from left to right) by estimating nuanced blind
spots caused by parked and moving cars, intersections, buildings, poles,
gates, _etc_. In the left most column, BlindSpotNet also estimates
intersection blind spots that are even not in the “ground-truth” $T$-frame
blind spots. This demonstrates the effectiveness of the visibility and the
advantage of BlindSpotNet of being able to train on a diverse set of scenes
thanks to the fact that $T$-frame blind spots can be easily computed on
arbitrary driving videos.

####  5.2.2 Qualitative Evaluations

---
Figure 7: Failure example. Left: $T$-frame blind spot (“ground truth”) from a
frame in BDD100k-RBS. Right: Blind spot estimation results from BlindSpotNet
trained on BDD100k-RBS. Due to the bias of BDD100k-RBS, which lacks left and
right turns at intersections, BSN cannot estimate the blind spots caused by
the intersection. We plan to address these issues by employing a panoramic
driving video for pre-training.

Fig. 6 shows blind spot estimation results for the test sets. Compared with
baseline methods, our method estimates the complex blind spots arising in
these everyday road scenes with high accuracy. The ablation study results
using BSN-D and BSN-KD clearly demonstrate the importance of the depth
estimator and knowledge distillation in BlindSpotNet. It is worth noting that
BlindSpotNet correctly estimates the left and right blind spots in the left
column example, even though the “ground-truth” $T$-frame blind spots do not
capture them due to the visibility map. These results clearly demonstrate the
effectiveness of the visibility map and the training on diverse road scenes
whose $T$-frame blind spots can be automatically computed. BlindSpotNet can be
trained on arbitrary road scenes as long as $T$-frame blind spots can be
computed, _i.e_., SLAM, semantic segmentation, and monocular depth estimation
can be applied. In this sense, it is a self-supervised method.

Fig. 7 shows a failure case example. By definition of $T$-frame blind spots,
BlindSpotNet cannot estimate intersection blind spots in videos that do not
have any turns. We plan to explore the use of wider perspective videos,
including full panoramic views, to mitigate this issue.

####  5.2.3 Ablation Study

Table 3: Ablation study. Our model achieves comparable performance to larger models and its frame-rate is promising for realtime processing. Architecture | IoU | Recall | Precision | # of params | GMACS | FPS
---|---|---|---|---|---|---
U-Net based [19] | 0.289 | 0.484 | 0.417 | 17.3M | 280.2 | 139.9
Small (ours) | 0.330 | 0.563 | 0.444 | 18.1M | 47.1 | 37.5
Medium | 0.315 | 0.478 | 0.482 | 32.0M | 93.0 | 19.5
Large | 0.337 | 0.508 | 0.500 | 59.3M | 160.9 | 11.3

We compare large/medium-sized blind spot estimators as well as a simple U-Net
[19] baseline with our small-sized (light-weight) blind spot estimator. The
differences between the large/medium-sized blind spot estimators and the
small-sized one are backbone, channel size, and the number of decoder layers.
The backbones of the large/medium-sized ones are ResNet101 and ResNet50,
respectively. Tab. 3 lists the IoU, recall, precision, the number of the
parameters, the computational complexity in GMACS, and the inference speed
with a single NVIDIA TITAN V 12GB. The results show that our model achieves
comparable performance to larger models with much smaller cost and runs
sufficiently fast for real-time use.

##  6 Conclusion

We introduced a novel computer vision task for road scene understanding,
namely 2D blind spot estimation. We tackle this challenging and critical
problem for safe driving by introducing the first comprehensive dataset (RBS
Dataset) and an end-to-end learnable network which we refer to as
BlindSpotNet. By defining 2D blind spots as road regions that are invisible
from the current viewpoint but become visible in the future, we showed that we
can automatically compute them offline for arbitrary driving videos, which in
turn enables learning to detect them with a simple neural network trained with
knowledge distillation from a pre-trained semantic segmentation network. We
believe these results offer a promising means for ensuring safer manual and
autonomous driving and open new approaches to extending self-driving and ADAS
with proactive visual perception.

## References

  * [1] Barnum, P., Sheikh, Y., Datta, A., Kanade, T.: Dynamic seethroughs: Synthesizing hidden views of moving objects. In: ISMAR. pp. 111–114 (2009)
  * [2] Bertoni, L., Kreiss, S., Alahi, A.: Monoloco: Monocular 3d pedestrian localization and uncertainty estimation. In: ICCV. pp. 6861–6871 (2019)
  * [3] Brazil, G., Liu, X.: M3D-RPN: Monocular 3D Region Proposal Network for Object Detection. In: ICCV (2019)
  * [4] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In: ECCV (2018)
  * [5] Cheng, B., Collins, M.D., Zhu, Y., Liu, T., Huang, T.S., Adam, H., Chen, L.C.: Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In: CVPR (2020)
  * [6] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An Open Urban Driving Simulator. In: Conference on robot learning. pp. 1–16. PMLR (2017)
  * [7] Ehsani, K., Mottaghi, R., Farhadi, A.: Segan: Segmenting and generating the invisible. In: CVPR (2018)
  * [8] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets Robotics: The KITTI Dataset. International Journal of Robotics Research (2013)
  * [9] Guo, R., Hoiem, D.: Beyond the line of sight: Labeling the underlying surfaces. In: ECCV. pp. 761–774 (2012)
  * [10] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [11] Liu, B., Zhuang, B., Schulter, S., Ji, P., Chandraker, M.: Understanding road layout from videos as a whole. In: CVPR (2020)
  * [12] Liu, C., Kohli, P., Furukawa, Y.: Layered scene decomposition via the occlusion-crf. In: CVPR (2016)
  * [13] Liu, Y., Shu, C., Wang, J., Shen, C.: Structured Knowledge Distillation for Dense Prediction. IEEE TPAMI pp. 1–1 (2020)
  * [14] Makansi, O., Çiçek, O., Buchicchio, K., Brox, T.: Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View With a Reachability Prior. In: CVPR. pp. 4353–4362 (2020)
  * [15] Malla, S., Dariush, B., Choi, C.: TITAN: Future Forecast using Action Priors. In: CVPR. pp. 11186–11196 (2020)
  * [16] Mani, K., Daga, S., Garg, S., Narasimhan, S.S., Krishna, M., Jatavallabhula, K.M.: MonoLayout: Amodal scene layout from a single image. In: WACV (2020)
  * [17] Rameau, F., Ha, H., Joo, K., Choi, J., Park, K., Kweon, I.S.: A real-time augmented reality system to see-through cars. IEEE transactions on visualization and computer graphics 22(11), 2395–2404 (2016)
  * [18] Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer. IEEE TPAMI (2020)
  * [19] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [20] Shivdas, S., Kelly, T.: Toyota halts all self-driving e-Palette vehicles after Olympic village accident. Reuters (2021-08-28), https://www.reuters.com/business/autos-transportation/toyota-halts-all-self-driving-e-pallete-vehicles-after-olympic-village-accident-2021-08-27/
  * [21] Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. In: CVPR (2017)
  * [22] Sugiura, T., Watanabe, T.: Probable Multi-hypothesis Blind Spot Estimation for Driving Risk Prediction. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC). pp. 4295–4302 (2019)
  * [23] Sumikura, S., Shibuya, M., Sakurada, K.: Openvslam: a versatile visual slam framework. In: ACM MM. pp. 2292–2295 (2019)
  * [24] Tighe, J., Niethammer, M., Lazebnik, S.: Scene parsing with object instances and occlusion ordering. In: CVPR (2014)
  * [25] Watson, J., Firman, M., Monszpart, A., Brostow, G.J.: Footprints and Free Space From a Single Color Image. In: CVPR (2020)
  * [26] Yang, W., Li, Q., Liu, W., Yu, Y., Ma, Y., He, S., Pan, J.: Projecting Your View Attentively: Monocular Road Scene Layout Estimation via Cross-View Transformation. In: CVPR. pp. 15536–15545 (2021)
  * [27] Yasuda, H., Ohama, Y.: Toward a practical wall see-through system for drivers: How simple can it be? In: ISMAR. pp. 333–334 (2012)
  * [28] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: BDD100K: A diverse driving dataset for heterogeneous multitask learning. In: CVPR. pp. 2636–2645 (2020)
  * [29] Zhang, Y., Song, S., Tan, P., Xiao, J.: Panocontext: A whole-room 3d context model for panoramic scene understanding. In: ECCV. pp. 668–686 (2014)
  * [30] Zhu, X., Yin, Z., Shi, J., Li, H., Lin, D.: Generative Adversarial Frontal View to Bird View Synthesis. In: 3DV. pp. 454–463 (2018)
  * [31] Zhu, Y., Tian, Y., Metaxas, D., Dollar, P.: Semantic amodal segmentation. In: CVPR (2017)


# Controlling the non-linear emission of upconversion nanoparticles to enhance
super-resolution imaging performance

Simone De Camillis<EMAIL_ADDRESS>ARC Centre of excellence for
Nanoscale BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia  Peng Ren  ARC Centre of excellence for
Nanoscale BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia  CNBP, School of Engineering and Physics,
Macquarie University  Yueying Cao  ARC Centre of excellence for Nanoscale
BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia  CNBP, Department of Molecular Sciences,
Macquarie University  Martin Ploschner  ARC Centre of excellence for Nanoscale
BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia  School of Information Technology and
Electrical Engineering, The University of Queensland, QLD 4072, Australia
Denitza Denkova  ARC Centre of excellence for Nanoscale BioPhotonics (CNBP),
Department of Physics and Astronomy, Macquarie University, NSW 2109, Australia
Institute for BioEngineering of Catalonia (IBEC), 08028 Barcelona, Spain
Xianlin Zheng  ARC Centre of excellence for Nanoscale BioPhotonics (CNBP),
Department of Physics and Astronomy, Macquarie University, NSW 2109, Australia
Yiqing Lu<EMAIL_ADDRESS>ARC Centre of excellence for Nanoscale
BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia  CNBP, School of Engineering and Physics,
Macquarie University  James A. Piper  ARC Centre of excellence for Nanoscale
BioPhotonics (CNBP), Department of Physics and Astronomy, Macquarie
University, NSW 2109, Australia

###### Abstract

Upconversion nanoparticles (UCNPs) exhibit unique optical properties such as
photo-emission stability, large anti-Stokes shift, and long excited-state
lifetimes, allowing significant advances in a broad range of applications from
biomedical sensing to super-resolution microscopy. In recent years, progress
on nanoparticle synthesis led to the development of many strategies for
enhancing their upconversion luminescence, focused in particular on heavy
doping of lanthanide ions and core-shell structures. In this article, we
investigate the non-linear emission properties of fully Yb-based core-shell
UCNPs and their impact on the super-resolution performance of stimulated
excitation-depletion (STED) microscopy and super-linear excitation-emission
(uSEE) microscopy. Controlling the power-dependent emission curve enables us
to relax constraints on the doping concentrations and to reduce the excitation
power required for accessing sub-diffraction regimes. We take advantage of
this feature to implement multiplexed super-resolution imaging of a two-sample
mixture.

Lanthanide-based upconversion nanoparticles, power-dependent emission curve,
multiplexed super-resolution imaging

## Introduction

Lanthanide-based upconversion nanoparticles (UCNPs) efficiently convert
absorbed near-infrared (NIR) light into higher photo-energy emission, enabling
a large range of applications spanning biomedical imaging and sensing, drug-
delivery systems, NIR vision extension, solar-cell technology, data security,
and more Wu  _et al._ (2015); Zhu  _et al._ (2019); Balabhadra  _et al._
(2017); Yang  _et al._ (2020); Ma  _et al._ (2019); Ambapuram  _et al._
(2020). Using UCNPs as luminescent bioprobes offers key advantages including
extended photo-stability over several hours of laser exposure, excitation in
the NIR spectral region resulting in low autofluorescence background and
reduced degree of photo-toxicity for biological samples, and lifetime-based
multiplexing capability Zheng  _et al._ (2015); Wang  _et al._ (2018); Fan
_et al._ (2018).

The most common UCNP structure comprises the crystal lattice NaYF4 hosting the
trivalent ions Yb3+ and Tm3+, which together represent a suitable
sensitiser/activator pair. The Yb3+ ion absorbs NIR
($970$-$980\text{\,}\mathrm{nm}$) radiation and efficiently transfers this
energy in a step-wise manner to the Tm3+ ion, which is responsible for the
upconversion emission. The peculiar photo-physics of this class of
nanoparticles has enabled significant advances in super-resolution microscopy.
STimulated Excitation-Depletion (STED) microscopy Hell and Wichmann (1994) for
instance can be successfully applied to UCNPs, thanks to their efficient
cross-relaxation mechanism from highly-excited states to intermediate energy
levels within the Tm3+ ions Liu  _et al._ (2017); Zhan  _et al._ (2017); Peng
_et al._ (2019). This phenomenon allows the blue emission to be substantially
suppressed by efficient depletion of the intermediate energy levels via
resonant stimulated emission. Imaging in combination with an annular-shaped
depletion beam can therefore significantly improve the lateral resolution,
even down to the nanoparticle’s size. Optical investigation of the non-linear
emission properties of UCNPs has also led to the discovery of an additional
super-resolution technique Denkova  _et al._ (2019); Caillat  _et al._ (2013);
Wang  _et al._ (2016), recently named upconversion Super-linear Excitation-
Emission (uSEE) microscopy. With suitable concentrations of the lanthanide ion
Tm3+, the 455-nm emission of UCNPs shows strong non-linear dependence as
function of the excitation laser intensity, reaching power indices of $6.2$.
Such strong super-linearity results in improvement to the resolution down to
about half of the diffraction limit in both the lateral and transversal
directions, making uSEE microscopy a simpler, low-power super-resolution
technique, not requiring complex purpose-build systems or any post-processing
analysis. The ability to resolve single UCNP probes in biological samples is
important since this allows binding of individual nanoparticles to target
structures to be visualised and quantified Denkova  _et al._ (2019); Cao  _et
al._ (2020).

Despite the unique advantages of UCNPs, their relatively moderate luminescence
brightness represents a major challenge. Applications in STED nanoscopy in
particular require a critical balance between high excitation laser
intensities, which are potentially harmful for biological samples, and
excessive acquisition or processing times. The uSEE technique is also
affected, as the emission rate is limited to photon-count values of
$10\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$
Denkova  _et al._ (2019). Over the past decade, there have been major efforts
directed at refining design and synthesis of lanthanide UCNPs aimed at
enhancing their brightness Han  _et al._ (2014). This has been achieved, for
example, by optimising the lanthanide-ion concentrations, and by growing
active/inert shell structures Zheng  _et al._ (2018); Liu  _et al._ (2019);
Homann  _et al._ (2018). Higher sensitiser (Yb3+) concentrations provide a
higher density of optical centres dedicated to the collection of photons and
to the sustenance of the excitation process, leading to a stronger
luminescence signal. At the same time, however, the reduced dopant separation
facilitates ion-ion interactions and promotes long distance energy migration
to quencher sites, therefore limiting the upconversion efficiency Chen and
Wang (2020). The strategy of overcoating the nanoparticle with shells of
active and/or inactive materials has been shown effective in reducing the
influence of quenching defects, as they are largely located on the surface of
the particle Fischer  _et al._ (2016); Tian  _et al._ (2018). For instance,
the design of active sandwich structures has been adopted to physically
separate activator and sensitiser in different layers, ensuring energy
migration to the nanoparticle core Qiu  _et al._ (2014) or preventing
deleterious cross-relaxation back to the sensitiser ions Zhong  _et al._
(2014). On the other hand, the simple coating of a thin inert shell over a
single active core has more recently been demonstrated to successfully boost
the luminescence signal by one or even two orders of magnitude, specifically
in a low-excitation power regime Tian  _et al._ (2018); Shi  _et al._ (2017).
Finally, to further maximise the luminescence brightness several research
groups have attempted the synthesis of lanthanide-based lattice, i.e.
NaYbF4:Ln (Ln: lanthanide), where the inert Y element is completely replaced
by the Yb3+ ions. Despite the possible limiting factor of concentration
quenching Zheng  _et al._ (2018), significant improvements in brightness
compared to the common Y-based UCNPs have been reported Shi  _et al._ (2017);
Shen  _et al._ (2017); Tian  _et al._ (2018); Cao  _et al._ (2020).

Given the above advances in UCNP design and synthesis, the question arises
whether there are consequential improvements that can be made to super-
resolution imaging performance. For example, changes in the Yb/Tm composition
ratio are expected to affect excitation and depletion rates and the efficiency
of cross-relaxation mechanisms, leading to variations in the non-linear
emission characteristics of the excited nanoparticles. Promising results have
been reported for NaYF4:Yb,Tm nanoparticles Ma  _et al._ (2020), where
increasing the doping concentration of the Yb3+ ions from $20$ to $60\%$
showed improvements in the emission slope at the single-nanoparticle level.
However, power-dependent studies on fully Yb-based UCNPs are limited in
literature, and to the best of our knowledge a comprehensive analysis of their
emission properties and their effect on super-resolution nanoscopy of the
particles have not been reported.

In this paper, we report results of a systematic investigation of NaYbTmF4
nanoparticles at different lanthanide ion ratios and over an extensive range
of excitation powers. We demonstrate that these particles can efficiently
sustain the depletion process for the STED microscopy, while also providing
sufficient levels of super-linearity to enable uSEE microscopy over a wide
selection of lanthanide concentrations. We observe that the emission curve of
the single nanoparticle can be substantially shifted towards lower excitation
powers by varying the lanthanide ion ratio, while the super-linear slope can
be increased with the coating of an inert shell. Further, optimising UCNP
composition and structure enables state-of-the-art super-resolution imaging
for both STED and uSEE microscopy at significantly lower excitation and
depletion power densities that have been previously reported. Additionally, we
exploit this phenomenon to implement a multiplexing imaging method capable of
recognising different UCNPs in a mixture with optimal super-resolution
performance.

## Results and discussion

Figure 1: Schematic of the confocal microscope system, where PBS is a
polarising beam splitter, VP a vortex plate, NF a neutral density filter, PL
is a polariser, $\lambda/2$ a half-wave plate, PM a power meter, DM1 and DM2
respectively a long-pass and short-pass dichroic mirror, $\lambda/4$ a
quarter-wave plate, F a blue narrow-band filter, and APD an avalanche photo-
diode. Figure 2: Imaging of lanthanide-based nanoparticles NaYbxTm1-xF4, where
the Yb concentration $x=0.92$, $0.88$, $0.84$, $0.8$. We refer to each sample
with the Yb/Tm concentration ratio $\eta=92/8$, $88/12$, $84/16$, and $80/20$,
represented by the colour red, yellow, green, and blue, respectively. (a-d)
Confocal $x$-$y$ images of the monodispersed UCNPs, where each sample was
excited respectively at $I=54$, $153$, $444$, and
$573\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$, to reach the same
single-particle photo emission. Dwell time: $3\text{\,}\mathrm{ms}$. Bar
scale: $1\text{\,}\mathrm{\SIUnitSymbolMicro m}$. (e) Single-nanoparticle
$455$-$\text{\,}\mathrm{nm}$ emission as a function of the excitation peak
intensity. The dotted lines illustrate a slope of 5. Inset: The luminescence
signal overcomes the noise level of
${10}^{-2}\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$
at the intensity threshold $I_{\textrm{th}}$, which differs for each UCNP
sample. $I_{\textrm{th}}$ is represented as function of the concentration
ratio $\eta$.

A set of hexagonal NaYbxTm1-xF4 nanoparticles with Yb concentration of
$x=0.92$, $0.88$, $0.84$, and $0.8$ were synthesised following a thermal
decomposition protocol. The synthesis was carefully realised by using a
customised system for automated growth, ensuring a high degree of precision
and reliability. Within each sample, particle size was observed to be
homogeneous, with diameters $54$, $46$, $48$, and $59\text{\,}\mathrm{nm}$,
respectively (see Supplementary Figure 1). Note the concentrations of the
sensitiser and the activator species are directly related to each other, so
that increasing the former results in decreasing the latter. Therefore, we
refer to the different nanoparticle compositions with the ratio
$\eta=$Yb${}_{\%}/$Tm%, representing the proportion between the Yb and Tm
concentrations.

We performed the imaging and the optical characterisation by means of a
custom-made dual-laser confocal system Denkova  _et al._ (2019) schematically
represented in Figure 1 (full details in Experimental Methods). The UCNPs are
excited by a $976$-$\text{\,}\mathrm{nm}$ circularly-polarised laser beam,
initially produced by a single-mode CW laser diode. Subsequently, a long-pass
dichroic mirror allows an additional annular-shaped laser beam, centred at
$808\text{\,}\mathrm{nm}$, to spatially co-propagate with the primary
excitation beam. This secondary arm is unblocked only to perform STED
microscopy. The excitation and depletion beams are delivered to the UCNP
sample via a 100x microscope objective. The collected upconversion signal is
isolated from the source component by a short-pass dichroic mirror, then
filtered at the narrow wavelength range of $446\pm 13.\text{\,}\mathrm{nm}$,
and finally coupled into a multimode fibre for luminescence signal acquisition
by an avalance photo diode.

We firstly acquired confocal images for each set of nanoparticles, as
illustrated in Figure 2(a-d). A common luminescence yield of about
$17\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$ was
reached by tuning the excitation intensity. Interestingly the same response
was obtained under different excitation power densities, from
$573\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$ in the case of the lowest
value of concentration ratio $\eta=80/20$, down to only
$54\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$ for the highest value of
$\eta=92/8$. To investigate this phenomenon in more detail, we measured the
luminescence of an isolated nanoparticle over a large range of excitation
intensities. Results are reported in Figure 2(e) in a log-log graph to better
evaluate their power dependence.

All four samples show an initial strong slope at lower power, progressively
reducing to a linear proportionality and eventually reaching a saturation
regime. Generally, the slope $s$ of the emission curve is closely related to
the multi-photon nature of the excitation process and, in the particular case
of an upconversion phenomenon, it is equal to or smaller than the upconversion
multi-photon number $N$ involved Pollnau  _et al._ (2000). Here, the blue
luminescence centred at $455\text{\,}\mathrm{nm}$ and assigned to the 1D2
$\rightarrow$ 3F4 transition has been often referred to as a five-photon
process Chen  _et al._ (2007); Wang  _et al._ (2008). Our results are in good
agreement with this assumption, as illustrated in Figure 2(e) by the slope
$s=5$ represented in dotted lines. It is important to note that energy
redistribution mechanisms, such as excited state absorption and cross-
relaxation between activator ions, are critically involved in the upconversion
luminescence. Indeed, these processes, which are strongly influenced by the
concentrations of the dopant ions, are believed to be responsible for reducing
or increasing the emission slope of lanthanide-doped materials Pollnau  _et
al._ (2000); Wang  _et al._ (2008); Bednarkiewicz  _et al._ (2019).
Remarkably, despite the significant difference in the concentration ratio
$\eta$, all four types of nanoparticles showed a comparable slope $s=5$ at
relatively low excitation intensity.

Figure 3: Comparison of the single-nanoparticle emission between core-only (in
brighter colours) and core-shell (in darker colours) structures. The core-
shell curves corresponding to $\eta=92/8$, $88/12$, $84/16$, and $80/20$ are
represented in dark red, orange, dark green, and purple, respectively. The
brighter curves follow the color code as specified in Figure 2. The legend
specifies the slope values obtained from the best fit (dashed line) of the
emission curves at low laser intensities. The slope uncertainties are equal to
$\pm 0.1$, with the exception of $\pm 0.2$ for the core-shell case of (c). The
intensity threshold for each core-shell UCNP is equal to $13$, $23$, $87$, and
$121\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$, respectively.

The notion that the emission curves in Figure 2(e) have a similar super-linear
slope suggests that similar emission properties can be achieved at different
excitation intensities by simply adjusting the concentration ratio. We
quantified this shift by looking at the intensity threshold $I_{\textrm{th}}$
at which the linear best-fit of the luminescence signal overcomes the noise
level of
${10}^{-2}\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$.
This is calculated for each sample and represented in the inset of Figure
2(e), showing an almost inverse proportionality to the square root of the
concentration ratio $\eta$, within the range of parameters explored in this
analysis. By increasing $\eta$ from $80/20$ to $92/8$, we were able to lower
the power density of the excitation laser by more than an order of magnitude.
This shift can be widely attributed to the larger concentration of Yb3+ ions,
as suggested by comparing the present results with the emission curve of the
common Y-based UCNPs at the same doping level of the Tm3+ ions (Supplementary
Figure 2). Indeed, although higher Yb concentrations facilitate energy
migration to surface quenchers, overall the fully Yb-doped nanoparticle with
concentration ratio $\eta=92/8$ (NaYb0.92Tm0.08F4) shows a threefold decrease
of the intensity threshold $I_{\textrm{th}}$ with respect to the corresponding
20%-Yb-doped nanoparticle (NaY0.72F4:Yb0.20,Tm0.08) used in our previous work
Denkova  _et al._ (2019).

At higher excitation intensity we note that the emission curves in Figure 2(e)
reach a saturation region where the luminescence is almost constant as a
function of laser power. The saturation level can be expected to vary
according to the concentration of the lanthanide ions and their relative
energy transfer efficiency. For an accurate comparison of the nanoparticle
brightness at saturation the UCNP samples should have identical size, which is
not the case here. We can nevertheless observe that the normalisation of the
emission curves for the nanoparticle volume suggests a slight increase in the
saturation yield with increasing concentration of Tm3+ ions, while the slope
values are effectively unchanged (Supplementary Figure 3).

To further improve the optical performance of our UCNPs, we coated the
nanoparticles with a thin shell of NaYF4. The morphological characterisation
by TEM gives an average shell thickness of $1.5\text{\,}\mathrm{nm}$
(Supplementary Figure 4). We observed a systematic increase of the emission
slope for core-shell compared with core-only nanoparticles, as indicated by
the data of Figure 3. This is in agreement with results of previous studies of
coating effects in high-Yb-doped UCNPs Ma  _et al._ (2020). We attribute the
slope improvement to the presence of the inert shell, which successfully
reduces energy transfer from the Yb3+ ions to the surface quenchers. A
simplified energy-level model of the sensitiser/activator system confirms this
hypothesis, estimating a slope change in the order of $20\%$ (Supplementary
Note 1). Each of the four core-shell UCNP samples examined herein demonstrated
a non-linear slope value equal to or greater than $5.6$, with a maximum value
of $7.2$ for $\eta=84/16$. The experimental results also suggest a trend in
the slope improvement as function of the lanthanide concentrations, with less
evident effects of the shell coating at higher values of the concentration
ratio $\eta$. This behaviour can be rationalised by recalling that the
$455$-nm emission is considered a five-photon process, meaning that at least
five Yb3+ ions are required to excite a Tm3+ ion. Here, $\eta$ varies from $4$
(sample $80/20$) to $11.5$ (sample $92/8$), increasingly above the optimal
Yb/Tm ratio. We can expect that at a certain threshold of this ratio, the Yb3+
ions are more effective in exciting the Tm3+ ions despite the presence of
energy losses, making the shell coating improvement less evident (as in the
case of sample $92/8$). Shell coating has also shown the potential to further
reduce the intensity threshold towards lower excitation powers. The different
shift in intensity threshold is influenced by a certain degree of
inhomogeneity in the shell thickness among nanoparticles. Although these
variations have limited effect for $\eta=92/8$, larger shifts were observed
for $\eta=80/20$ (see Supplementary Figure 5). In addition, we emphasise that
the shell coating has little effect at higher power densities, despite having
a strong impact in the super-linear emission region. At high power densities,
the effective excitation rate of the Tm centres well exceeds the non-radiative
energy loss rate related to the surface quenching, thus limiting the maximum
emission rate of the single core-shell UCNPs to the same values as the
corresponding core-only nanoparticles.

Figure 4: Single-nanoparticle integrated luminescence as function of the peak
intensity of the 808-nm depletion beam. The experimental data of core-shell
samples $92/8$, $88/12$, $84/16$, and $80/20$ are represented in dark red,
orange, dark green, and purple, respectively. For a reliable comparison, the
excitation intensity for each sample ($I_{\textrm{exc}}=58$, $154$, $296$, and
$500\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$, respectively) was
selected to deliver a luminescence yield of about
$20\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$. The
continuous curves correspond to the best fit of the function
$(1+I_{\textrm{808}}/\beta)^{-2}$, where the fitting parameter $\beta$ helps
to estimate the characteristic depletion intensity
$I_{\textrm{depl}}=(\sqrt{2}-1)\cdot\beta$ (equal to $0.34\pm 0.06\text{\,}$,
$0.8\pm 0.1\text{\,}$, $1.7\pm 0.3\text{\,}$, and $2.6\pm
0.3\text{\,}\mathrm{MW}\text{\,}{\mathrm{cm}}^{-2}$, respectively) at which
the integrated emission yield is halved. Inset: Experimental data and curves
represented in logarithmic scale.

Large variations in the lanthanide ratio and the coating of an inert shell may
significantly alter the depletion efficiency induced by the 808-nm laser,
which is critical for the optimal performance of STED microscopy. We studied
the integrated luminescence of a single nanoparticle as function of the peak
depletion intensity of the $808$-nm Gaussian beam. To effectively compare the
four types of nanoparticles, each sample was imaged at excitation intensities
sufficient to deliver a luminescence yield of about
$20\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$ in
the absence of the depletion laser. Figure 4 illustrates the normalised
depletion curves of the core-shell nanoparticles, where we can estimate the
characteristic depletion intensity $I_{\textrm{depl}}$ at which the emission
yield is halved. We observe that $I_{\textrm{depl}}$ shifts towards lower
values when increasing the lanthanide ion ratio, in a similar manner than the
excitation intensity $I_{\textrm{exc}}$. Interestingly, the ratio between the
depletion and the excitation intensity $I_{\textrm{depl}}/I_{\textrm{exc}}$
for each sample assumes values between $5.2$ and $5.8$, demonstrating a
comparable depletion efficiency across each of the four nanoparticle
compositions.

Figure 5: Super-resolution imaging with core-shell nanoparticles with
$\eta=92/8$ (NaYb0.92Tm0.08F4@NaYF4). (a) Confocal $x$-$y$ image of the UCNPs
by means of the uSEE technique (peak intensity of the excitation beam:
$46\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$). Image size:
$200\times$200\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$$.
(b) Same confocal imaging scan as in (a) performed with the STED technique.
The peak power density of the 976-nm excitation CW laser and the 808-nm
depletion beam are $49\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$ and
$4.1\text{\,}\mathrm{MW}\text{\,}{\mathrm{cm}}^{-2}$, respectively. Image
size:
$250\times$250\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$$.
Dwell time: $3\text{\,}\mathrm{ms}$. Scale bar:
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$. (c) Line profile of the uSEE-imaged
nanoparticles. The Gaussian best-fit gives a FWHM of $206\pm
12.\text{\,}\mathrm{nm}$ (d) Line profile of the STED-imaged nanoparticles.
The Lorentzian best-fit returns a FWHM of $57\pm 5.\text{\,}\mathrm{nm}$. The
relative heights of the emission peaks in STED mode compared to uSEE mode are
affected by spatial sampling and image noise, as described in Supplementary
Figure 6.

The ability to shift the emission curve over a large power region, while
maintaining strong non-linear behaviour, has a great impact on super-
resolution imaging techniques such as uSEE and STED. The first advantage is to
reduce significantly the laser power required to access these super-resolution
regimes. We confirmed this milestone by studying the core-shell nanoparticles
with the highest $\eta$ (i.e. $92\%$ Yb, $8\%$ Tm), corresponding to the case
with lowest intensity threshold. By imaging with only the
$976$-$\text{\,}\mathrm{nm}$ CW laser at a power density compatible with the
super-linear emission region, we were able to access the uSEE regime and
measure a single-nanoparticle resolution of about $200\text{\,}\mathrm{nm}$
(Figure 5(a,c)). Resolutions of about half the diffraction limit by means of
uSEE nanoscopy were already attained by our group Denkova  _et al._ (2019),
but in this case we employed an excitation peak intensity as low as
$46\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$ (about
$85\text{\,}\mathrm{\SIUnitSymbolMicro W}$), nearly three times lower than
previously reported. Additionally, we tested the efficiency of the STED
technique by co-illuminating the sample with both the excitation
$976$-$\text{\,}\mathrm{nm}$ laser and the annular-shaped
$808$-$\text{\,}\mathrm{nm}$ depletion beam. In this case, we achieved single-
nanoparticle resolution (Figure 5(b,d)), demonstrating that the depletion
process in the Tm3+ ions remains effective despite the high concentration
values of the Yb3+ ion. The best resolution observed is comparable to that of
current state-of-art UCNP-based STED nanoscopy Peng  _et al._ (2019); Liu  _et
al._ (2017), though the peak intensity of the depletion beam was only
$4.1\text{\,}\mathrm{MW}\text{\,}{\mathrm{cm}}^{-2}$
($19\text{\,}\mathrm{mW}$), more than two times smaller than our previous
report Liu  _et al._ (2017). We clarify that the excitation power for the STED
technique is selected to minimise the radiation fluence delivered to the
sample, while maintaining a sufficient signal-to-noise ratio. The excitation
powers for STED could be lowered to values comparable to those used in uSEE,
which is beneficial for improving both lateral and transversal resolutions
Plöschner  _et al._ (2020).

We also analysed the optical properties of the core-shell nanoparticles with
the highest slope value, i.e. $\eta=84/16$ (as reported in Figure 3),
confirming similar resolution performance for STED and even better resolutions
for uSEE (Supplementary Figure 7). The improvement observed in uSEE nanoscopy
reflects well the steepening of the emission curve over the different
concentration ratios. Resolution enhancement of non-linear emitters with slope
$s$ can be practically estimated to be proportional to $\sqrt{s}$, giving a
relative improvement of $13\%$ when increasing the slope from $5.6$ to $7.2$
(corresponding to a change in $\eta$ from $92/8$ to $84/16$). This is in good
agreement with the measured lateral resolution, improving from $206$ to
$183\text{\,}\mathrm{nm}$ (Figure 5(c) and Supplementary Figure 7(c)).
Moreover, the reduced excitation powers required for the core-shell samples
can be clearly observed by comparing these results with the performance of the
corresponding core-only nanoparticles (see Supplementary Figure 8).

It is evident that the ability to achieve super-resolution operation at
different lanthanide concentrations ($\eta$ varying from $92/8$ to $84/16$)
largely relaxes the constraints on the crystal composition. Conventional UCNP
synthesis for STED and uSEE applications has required specific amounts of the
matrix ions and the lanthanide dopants, i.e. 20% Yb and 8% Tm. Instead, the
design of the fully lanthanide-based nanocrystal presented in this work is
simply based on the ratio between sensitiser and activator, alleviating
requirements on precise concentration values during the UCNP synthesis.

UCNP-based microscopy offers several advantages in terms of long-term
photostability and near-infrared working conditions, including significant
benefits such as low autofluorescence background, small scattering/absorption,
and low photodamage. On the other hand, the limited photon budget of the two
super-resolution techniques requires an image acquisition time in the range of
$60$-$180\text{\,}\mathrm{s}$, a few orders of magnitude slower than
fluorescence super-resolution methods Sahl  _et al._ (2017); Sigal  _et al._
(2018). However, combination with other imaging techniques can substantially
improve the current performance. For instance, the development of parallelised
imaging systems with multiple beams scanning sub-areas of the sample has
enabled a manyfold increase in the frame rate Bingen  _et al._ (2011). In
addition, non-linear SIM Ingerman  _et al._ (2019) is compatible with the
super-linear emission feature of uSEE microscopy, representing a valid
solution to enhance both imaging speed and resolution performance.

Figure 6: Multiplexed STED imaging of the core-shell UCNPs with $\eta=92/8$
and $84/16$. (a) Single-nanoparticle $455$-$\text{\,}\mathrm{nm}$ emission
curve of each sample, represented in dark red and dark green, respectively.
(b-c) STED confocal images of the two UCNP samples mixed and deposited on the
same microscope slide. We can distinguish the first type in (b), excited at
$I_{976}=$53\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$$ (peak intensity
of the depletion beam
$I_{808}=$4.3\text{\,}\mathrm{MW}\text{\,}{\mathrm{cm}}^{-2}$$). The second
UCNP type is imaged in (c) at peak intensity
$I_{976}=$252\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$$
($I_{808}=$8.6\text{\,}\mathrm{MW}\text{\,}{\mathrm{cm}}^{-2}$$), while the
former UCNP sample reaches the saturation regime. Image size:
$200\times$200\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$$.
Dwell time: $3\text{\,}\mathrm{ms}$. Scale bar:
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$.

Finally, controlling the emission curve of non-linear emitters allows more
freedom in the adaptation of nanomaterials to each specific experiment and can
lead to interesting applications. We took advantage of this effect to
implement multiplexing capabilities in super-resolution imaging of UCNPs.
Figure 3 shows that for large differences in lanthanide concentrations the
non-linear emission regions of different nanoparticles can be distanced enough
to avoid their possible overlap in excitation intensity. In Figure 6(a), we
considered the case of the core-shell UCNPs with $\eta=92/8$ and $84/16$,
where the difference of the intensity threshold exceeds
$70\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$. By opportunely selecting
the excitation power it is possible to perform super-resolution imaging of the
first type of UCNPs (with no detectable blue luminescence from the second), or
alternatively to acquire super-resolved images of the latter UCNPs while the
former is subject to a saturated response. To experimentally verify this
multiplexing technique, we deposited the two nanoparticle samples on the same
microscope slide and acquired confocal STED images at peak power densities of
the excitation beam equal to
$53\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$ and
$252\text{\,}\mathrm{kW}\text{\,}{\mathrm{cm}}^{-2}$. The two different UCNP
samples can be clearly distinguished by comparing Figures 6(b) and (c). To
visualise better the nanoparticles, the colour bar of the second image was
rescaled to match the emission amplitude of the second type of particles with
the higher intensity threshold, causing the first UCNPs to appear
oversaturated. Inevitably the oversaturation imposes a blind region for the
second type of nanoparticles, but in-plane this restriction is limited only to
an area with radius smaller than $160\text{\,}\mathrm{nm}$. In different
experimental conditions, nanoparticle aggregation may represent a limitation
to the present multiplex imaging technique. Nevertheless, the two samples
investigated here (i.e. $92/8$ and $84/16$) show a large shift in the emission
curve, ensuring a luminescence difference of more than three orders or
magnitude at the first control intensity (red strip in Figure 6(a)).
Therefore, for small and medium aggregations ($\leq 500$ nanoparticles), the
luminescence comparison at the selected control intensities would be
sufficient to discriminate the two nanoparticle types.

Shifting the upconversion luminescence to lower intensities is not only
limited to STED nanoscopy. This straightforward multiplexing modality can also
be applied to the uSEE technique (Supplementary Figure 9), as it involves the
super-linear emission regime of UCNPs.

## Conclusion

The present work has investigated the single-nanoparticle emission of fully
Yb-based UCNPs, by comparing different compositions and structures and
scanning the excitation power over more than two orders of magnitude. We have
confirmed that this class of nanoparticles, coated with a thin shell of inert
(NaYF4) material, effectively sustains the conditions for uSEE and STED super-
resolution microscopy, despite the Yb/Tm concentration ratio varying from
$80/20$ to $92/8$. In particular, the super-linear power-dependence of the
$455$-$\text{\,}\mathrm{nm}$ emission peak is characterised by slopes equal to
or greater than $5.6$, with a peak value of $7.2$, ensuring in all cases a
uSEE resolution at least twice better than the diffraction limit. Further, we
have demonstrated that depletion efficiency of the $455$-nm emission remains
effectively constant over a wide range of lanthanide concentrations, enabling
STED resolution ($\sim$60\text{\,}\mathrm{nm}$$) to be largely unaffected.

Increasing the Yb/Tm concentration ratio from $80/20$ to $92/8$ leads
primarily to a shift of the emission curve towards lower excitation powers
without affecting the non-linear optical properties of UCNPs, behaviour which
opens several opportunities for super-resolution imaging. We have demonstrated
that the laser power density required for uSEE and STED microscopy can be
lowered by at least a factor of two. Additionally, we have performed
multiplexed super-resolution imaging of a mixture of two different UCNP
samples, purely through choice of different excitation depletion conditions
for the first time.

These results will help future breakthroughs in UCNP-based bioprobes. From the
perspective of nanomaterials, the simplified particle design introduces large
flexibility in the concentration of the lanthanide ions, relaxing the
constraints usually applied in the synthesis of UCNPs for imaging purposes. In
the context of biological applications, imaging at lower excitation powers
considerably reduces the risk of photo-damage in biological samples. Moreover,
controlling the emission curve of the upconversion
$455$-$\text{\,}\mathrm{nm}$ signal enables the observation of two or more
sub-cellular target structures with single-nanoparticle resolution, paving the
way for relative location measurements and quantitative nanoscopy.

## Experimental Methods

### Materials:

Ytterbium(III) chloride hexahydrate (YbCl3, $99.99\%$), thulium(III) chloride
hexahydrate (TmCl3, $99.99\%$), yttrium(III) chloride hexahydrate (YCl3,
$99.99\%$), oleic acid (OA, $90\%$), 1-octadecene (ODE, $90\%$), sodium oleate
($\geq 82\%$, fatty acids), ammonium fluoride (NH4F, $\geq 98\%$), and sodium
hydroxide ($\geq 97\%$, pellets) were purchased from Sigma-Aldrich and used as
received without further purification.

### Synthesis of Ln-based UCNPs:

The Ln-based UCNPs were synthesised by means of a purpose-made automated
growth system, implementing a modified co-precipitation method based on
reported literature Liu  _et al._ (2017); Shi  _et al._ (2017). The first step
is the production of the optically active core of the nanoparticles
NaYbxTm1-xF4, with $x=0.92$, $0.88$, $0.84$, and $0.8$. A suitable ratio of
YbCl3 ($x$ $\text{\,}\mathrm{mmol}$) and TmCl3 ($1-x$
$\text{\,}\mathrm{mmol}$) dissolved in methanol were stirred in a flask with
OA ($6\text{\,}\mathrm{ml}$) and ODE ($15\text{\,}\mathrm{ml}$). The mixture
was placed under argon gas flow and underwent a first heating process,
consisting of $30\text{\,}\mathrm{min}$ at
$75\text{\,}\mathrm{\SIUnitSymbolCelsius}$ and $30\text{\,}\mathrm{min}$ at
$160\text{\,}\mathrm{\SIUnitSymbolCelsius}$. After a cooling down period,
$3.125\text{\,}\mathrm{mmol}$ of sodium oleate and $4\text{\,}\mathrm{mmol}$
of NH4F were added, and the resulting solution was stirred for
$30\text{\,}\mathrm{min}$ under argon gas. The mixture was subject to a second
heating process, $160\text{\,}\mathrm{\SIUnitSymbolCelsius}$ for
$1\text{\,}\mathrm{h}$ and $310\text{\,}\mathrm{\SIUnitSymbolCelsius}$ for
$40\text{\,}\mathrm{min}$, and cooled down to room temperature under argon gas
flow. The nanocrystals were isolated by dilution in ethanol and
centrifugation, washed several times with ethanol/cyclohexane, and stored in
cyclohexane.

A similar procedure is repeated a second time for the synthesis of the shell
precursor NaYF4, but starting with YCl3 ($1\text{\,}\mathrm{mmol}$) instead of
the lanthanide elements. After the first cooling down period, the solution was
stirred with NH4F ($4\text{\,}\mathrm{mmol}$) and sodium hydroxide
($2.5\text{\,}\mathrm{mmol}$), instead of sodium oleate. Moreover, the second
hearing process consisted of only a single step to
$160\text{\,}\mathrm{\SIUnitSymbolCelsius}$ kept for $1\text{\,}\mathrm{h}$.

For the nanoparticle coating, $10\text{\,}\mathrm{ml}$ of ODE and an equal
amount of OA were stirred in a flask together with $3\text{\,}\mathrm{ml}$ of
nanocrystals. The mixture was heated to
$75\text{\,}\mathrm{\SIUnitSymbolCelsius}$ under argon for
$30\text{\,}\mathrm{min}$, and then further heated to
$310\text{\,}\mathrm{\SIUnitSymbolCelsius}$. Subsequently,
$1\text{\,}\mathrm{ml}$ of as-prepared shell precursor was injected into the
reaction flask and let react with the mixture for $15\text{\,}\mathrm{min}$ at
the same temperature. The same injection and waiting procedure was repeated
several times. Finally, the products were cooled down to room temperature
under argon gas exposure and purified with a similar procedure as for the
core-only nanoparticles.

### Sample preparation:

To prepare a sample slide, a cover slip (Grale HDS, HD LD2222 $1.01$P0,
$22\times$22\text{\,}\mathrm{mm}$$, No. 1) was washed with pure ethanol,
followed by Milli-Q water, partially dried under nitrogen flow and left fully
air-dry. $50\text{\,}\mathrm{\SIUnitSymbolMicro l}$ of Poly-L-lysine solution
($0.1\%$ w/v in deionised H2O) was dropped on the front surface of the cover
slip and washed off with Milli-Q water after $35\text{\,}\mathrm{min}$. The
UCNP solution was diluted with cyclohexane to a concentration of
$0.05\text{\,}\mathrm{mg}\text{\,}{\mathrm{ml}}^{-1}$.
$20\text{\,}\mathrm{\SIUnitSymbolMicro l}$ of this solution was dropped onto
the treated surface of the cover slip and immediately washed twice by
$500\text{\,}\mathrm{\SIUnitSymbolMicro l}$ of cyclohexane. After being air-
dried, $10\text{\,}\mathrm{\SIUnitSymbolMicro l}$ of a index-matching medium
is pipetted over a clean microscopy slide (Thermo Scientific, S41014A,
$76\times$26\text{\,}\mathrm{mm}$$) and the above prepared cover slip is
gently placed on top. Slight pressure is applied to distribute the embedding
medium. Finally, the slide sample was kept at room temperature to dry for at
least $48\text{\,}\mathrm{h}$.

For the multiplexing measurement, the slide preparation followed a similar
procedure. In this case, two UCNP samples were separalely diluted with
cyclohexane to a concentration of
$0.05\text{\,}\mathrm{mg}\text{\,}{\mathrm{ml}}^{-1}$ and mixed together to
form a $20$-$\text{\,}\mathrm{\SIUnitSymbolMicro l}$ solution.

### Confocal system:

A custom-made optical system developed by our group Denkova  _et al._ (2019)
(schematic illustrated in Figure 1) was employed to acquire the confocal
images of UCNPs and measure their power-dependent emission curves.

The excitation beam is produced by a $976$-$\text{\,}\mathrm{nm}$ CW laser
diode (Thorlabs, BL976-PAG900). After collimation, the polarisation state of
the beam is fixed by a linear polariser (Thorlabs, LPVIS100-MP). A pair of
half-wave plate (Thorlabs, WPMH10M-980) and polarisation beam slitter (PBS,
Thorlabs, CCM1-PBS252/M) is introduced along the optical path to control the
laser power. The excitation beam power is varied by rotating the half-wave
plate with a motorised rotation stage, and it is actively monitored by
measuring the beam component reflected by the PBS. Moreover, a set of neutral
density filters is employed to access lower ranges of the excitation power.
The second laser beam required for the STED depletion is delivered by a
$808$-$\text{\,}\mathrm{nm}$ CW laser diode (Lumics, LU0808M250). After
collimation, a PBS fixes the linear polarisation of the beam and a vortex
plate (Holo/Or, VL-209-5-Y-A), precisely aligned along the optical path,
modifies the beam profile into an annular shape. In this case, the depletion
power is manually varied by changing the diode current.

The two NIR laser beams were spatially overlapped through a long-pass dichroic
mirror (Chroma, ZT860lpxr), reflected by a second short-pass dichroic mirror
(Chroma, T750spxrxt), and directed towards a quarter-wave plate (Thorlabs,
AQWP10M-980) to produce the circular polarisation. Then, the laser radiation
is focussed down onto the sample by a rigidly fixed microscope objective
(Olympus, UPLSAPO 100XO, NA 1.4, 100x). Imaging of the beam profile on the
focus point were performed for an accurate estimation of the intensity
delivered to the sample (Supplementary Note 2). The sample slide is mounted on
a piezo-electric 3-axis stage (Thorlabs, MAX311D/M with BPC203 piezo
controller) to perform scanning acquisitions. The light emitted, scattered or
reflected by the sample is collected back again by the objective, but only the
upconversion signal is transmitted through the above mentioned short-pass
dichroic mirror. In order to study the specific luminescence peak at
$455\text{\,}\mathrm{nm}$, a narrow band-pass filter (Semrock FF01-448/20-25)
selecting only the emission band at $446\pm 13.\text{\,}\mathrm{nm}$ is placed
through the optical path. Finally, the signal is coupled into a multi-mode
fibre (Thorlabs, M43L01, core diameter $105\text{\,}\mathrm{\SIUnitSymbolMicro
m}$) and detected by a single-photon counting photodiode (Excelitas, SPCM-
AQRH-14-FC). Precise synchronisation between the piezo-electric stage and the
photodiode is realised by means of a multifunctional I/O card (NI 6353), which
is able to deliver the required voltages for positioning the stage and to
collect the digital counting signal from the detector.

### Acquisition procedure:

A purpose-made LabVIEW program manages the entire scanning and acquisition
procedure. Confocal 2D images are produced at fixed excitation/depletion
powers, by scanning the sample and collecting the photon counts at each
position (dwell time of $3\text{\,}\mathrm{ms}$ if not specified otherwise).
The emission curves are analysed by keeping the stage at a fixed position and
by acquiring the photon signal at different excitation powers. The automated
control of the beam power reduces power uncertainties and minimises
unnecessary laser exposures. Each point of the emission curve is obtained by
acquiring the luminescence signal for $1\text{\,}\mathrm{s}$, eight
consecutive times. The values are averaged and the relative uncertainty is
assigned with a confidence level of $95\%$. An additional uncertainty
component, due to power fluctuations and estimated to change the luminescence
signal of about $10\%$, is considered. Despite the dark count of the detector
was
${10}^{-1}\text{\,}\mathrm{c}\mathrm{t}\mathrm{s}\mathrm{/}\mathrm{m}\mathrm{s}$,
the measurement sensitivity for the emission curve was lowered an order of
magnitude by subtracting the background level. The depletion curves were
obtained by acquiring mono-dispersed nanoparticle images at a fixed excitation
power and varying the power of the depletion beam (with Gaussian profile
shape). An averaged background component was subtracted to the integrated
nanoparticle signal. Each depletion curve was normalised to the integrated
luminescence yield with no depletion beam.

## Conflicts of interest

There are no conflicts to declare.

## Acknowledgements

This work has been supported by the Australian Research Council (ARC) funding
through the Centre of Excellence for Nanoscale BioPhotonics (CE140100003) and
by Macquarie University funding (Postgraduate Scholarship of Y. C., Research
Fellowship of X. Z.). M. P. and Y. L. acknowledge support through Discovery
Early Career Research Awards (DE170100241, DE170100821). D. D. is grateful to
the AGAUR (Generalitat de Catalunya) Beatriu de Pinós grant (BP00269).

## References

  * Wu  _et al._ (2015) X. Wu, G. Chen, J. Shen, Z. Li, Y. Zhang, and G. Han, Bioconjugate Chemistry 26, 166 (2015).
  * Zhu  _et al._ (2019) X. Zhu, J. Zhang, J. Liu, and Y. Zhang, Advanced Science 6, 1901358 (2019).
  * Balabhadra  _et al._ (2017) S. Balabhadra, M. L. Debasu, C. D. Brites, R. A. Ferreira, and L. D. Carlos, Journal of Physical Chemistry C 121, 13962 (2017).
  * Yang  _et al._ (2020) B. Yang, H. Chen, Z. Zheng, and G. Li, Journal of Luminescence 223, 117226 (2020).
  * Ma  _et al._ (2019) Y. Ma, J. Bao, Y. Zhang, Z. Li, X. Zhou, C. Wan, L. Huang, Y. Zhao, G. Han, and T. Xue, Cell 177, 243 (2019).
  * Ambapuram  _et al._ (2020) M. Ambapuram, R. Ramireddy, G. Maddala, S. Godugunuru, P. Venkata, S. Yerva, and R. Mitty, ACS Applied Electronic Materials 2, 962 (2020).
  * Zheng  _et al._ (2015) W. Zheng, P. Huang, D. Tu, E. Ma, H. Zhu, and X. Chen, Chemical Society Reviews 44, 1379 (2015).
  * Wang  _et al._ (2018) F. Wang, S. Wen, H. He, B. Wang, Z. Zhou, O. Shimoni, and D. Jin, Light: Science and Applications 7, 18006 (2018).
  * Fan  _et al._ (2018) Y. Fan, P. Wang, Y. Lu, R. Wang, L. Zhou, X. Zheng, X. Li, J. A. Piper, and F. Zhang, Nature Nanotechnology 13, 941 (2018).
  * Hell and Wichmann (1994) S. W. Hell and J. Wichmann, Optics Letters 19, 780 (1994).
  * Liu  _et al._ (2017) Y. Liu, Y. Lu, X. Yang, X. Zheng, S. Wen, F. Wang, X. Vidal, J. Zhao, D. Liu, Z. Zhou, C. Ma, J. Zhou, J. A. Piper, P. Xi, and D. Jin, Nature 543, 229 (2017).
  * Zhan  _et al._ (2017) Q. Zhan, H. Liu, B. Wang, Q. Wu, R. Pu, C. Zhou, B. Huang, X. Peng, H. Ågren, and S. He, Nature Communications 8, 1 (2017).
  * Peng  _et al._ (2019) X. Peng, B. Huang, R. Pu, H. Liu, T. Zhang, J. Widengren, Q. Zhan, and H. Ågren, Nanoscale 11, 1563 (2019).
  * Denkova  _et al._ (2019) D. Denkova, M. Plöschner, M. Das, L. M. Parker, X. Zheng, Y. Lu, A. Orth, N. H. Packer, and J. A. Piper, Nature Communications 10, 1 (2019).
  * Caillat  _et al._ (2013) L. Caillat, B. Hajj, V. Shynkar, L. Michely, D. Chauvat, J. Zyss, and F. Pellé, Applied Physics Letters 102, 1 (2013).
  * Wang  _et al._ (2016) B. Wang, Q. Zhan, Y. Zhao, R. Wu, J. Liu, and S. He, Optics Express 24, A302 (2016).
  * Cao  _et al._ (2020) Y. Cao, X. Zheng, S. De Camillis, B. Shi, J. A. Piper, N. H. Packer, and Y. Lu, Nano Letters (2020), 10.1021/acs.nanolett.0c02697.
  * Han  _et al._ (2014) S. Han, R. Deng, X. Xie, and X. Liu, Angewandte Chemie - International Edition 53, 11702 (2014).
  * Zheng  _et al._ (2018) X. Zheng, S. Shikha, and Y. Zhang, Nanoscale 10, 16447 (2018).
  * Liu  _et al._ (2019) L. Liu, X. Li, Y. Fan, C. Wang, A. M. El-Toni, M. S. Alhoshan, D. Zhao, and F. Zhang, Chemistry of Materials 31, 5608 (2019).
  * Homann  _et al._ (2018) C. Homann, L. Krukewitt, F. Frenzel, B. Grauel, C. Würth, U. Resch-Genger, and M. Haase, Angewandte Chemie - International Edition 57, 8765 (2018).
  * Chen and Wang (2020) B. Chen and F. Wang, Accounts of Chemical Research 53, 358 (2020).
  * Fischer  _et al._ (2016) S. Fischer, N. D. Bronstein, J. K. Swabeck, E. M. Chan, and A. P. Alivisatos, Nano Letters 16, 7241 (2016).
  * Tian  _et al._ (2018) B. Tian, A. Fernandez-Bravo, H. Najafiaghdam, N. A. Torquato, M. V. P. Altoe, A. Teitelboim, C. A. Tajon, Y. Tian, N. J. Borys, E. S. Barnard, M. Anwar, E. M. Chan, P. J. Schuck, and B. E. Cohen, Nature Communications 9, 4 (2018).
  * Qiu  _et al._ (2014) H. Qiu, C. Yang, W. Shao, J. Damasco, X. Wang, H. Ågren, P. Prasad, and G. Chen, Nanomaterials 4, 55 (2014).
  * Zhong  _et al._ (2014) Y. Zhong, G. Tian, Z. Gu, Y. Yang, L. Gu, Y. Zhao, Y. Ma, and J. Yao, Advanced Materials 26, 2831 (2014).
  * Shi  _et al._ (2017) R. Shi, X. Ling, X. Li, L. Zhang, M. Lu, X. Xie, L. Huang, and W. Huang, Nanoscale 9, 13739 (2017).
  * Shen  _et al._ (2017) B. Shen, S. Cheng, Y. Gu, D. Ni, Y. Gao, Q. Su, W. Feng, and F. Li, Nanoscale 9, 1964 (2017).
  * Ma  _et al._ (2020) C. Ma, C. Shan, K. Park, A. T. Mok, P. J. Antonick, and X. Yang, Nanophotonics 9, 1993 (2020).
  * Pollnau  _et al._ (2000) M. Pollnau, D. Gamelin, S. Luthi, and H. Gudel, Physical Review B 61, 3337 (2000).
  * Chen  _et al._ (2007) G. Y. Chen, G. Somesfalean, Z. G. Zhang, Q. Sun, and F. P. Wang, Optics Letters 32, 87 (2007).
  * Wang  _et al._ (2008) G. Wang, W. Qin, L. Wang, G. Wei, P. Zhu, and R. Kim, Optics Express 16, 11907 (2008).
  * Bednarkiewicz  _et al._ (2019) A. Bednarkiewicz, E. M. Chan, A. Kotulska, L. Marciniak, and K. Prorok, Nanoscale Horizons 4, 881 (2019).
  * Plöschner  _et al._ (2020) M. Plöschner, D. Denkova, S. De Camillis, M. Das, L. M. Parker, X. Zheng, Y. Lu, S. Ojosnegros, and J. A. Piper, Optics Express 28, 24308 (2020).
  * Sahl  _et al._ (2017) S. J. Sahl, S. W. Hell, and S. Jakobs, Nature Reviews Molecular Cell Biology 18, 685 (2017).
  * Sigal  _et al._ (2018) Y. M. Sigal, R. Zhou, and X. Zhuang, Science 361, 880 (2018).
  * Bingen  _et al._ (2011) P. Bingen, M. Reuss, J. Engelhardt, and S. W. Hell, Optics Express 19, 23716 (2011).
  * Ingerman  _et al._ (2019) E. A. Ingerman, R. A. London, R. Heintzmann, and M. G. Gustafsson, Journal of Microscopy 273, 3 (2019).