Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or self-driving vehicle. Most modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio. In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can be used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.

§ INTRODUCTION

Accurate and robust detection of moving vehicles has crucial relevance for autonomous robots operating in outdoor environments [10, 11, 25]. In the context of self-driving cars, other moving vehicles have to be detected accurately even in challenging environmental conditions since knowing their positions and velocities is highly relevant for predicting their future movements and for planning the ego-trajectory. Also, robots operating in areas reserved for pedestrians such as sidewalks or pedestrian zones, e.g., delivery robots, require precise detections of vehicles in order to assess their planned trajectories and safety clearance with regard to moving vehicles in their vicinity. In this work, we focus on detecting moving vehicles perpendicular to the robot orientation for robots operating in pedestrian areas. Such detections, for example, enable delivery robot applications to more safely cross streets [11, 14, 15] or potentially localize vehicles within pedestrian areas.

With the rise of learning-based methods, training image-based vehicle detectors in a supervised fashion has made substantial progress. However, creating large-scale datasets is an expensive and time-consuming task. Previous works have shown that the performance of detectors can break down if they are presented with image data different from samples encountered during training. For outdoor robots, this domain gap may be induced by changes in environmental conditions such as rain, fog, or low illumination during nighttime [21]. While recent approaches in the area of domain adaptation show encouraging results [21, 22], most approaches still require a large amount of hand-annotated images in order to not exhibit a large domain gap.

We use unlabeled video clips of street scenes recorded with a camera and a multi-channel microphone array and leverage the audio-visual correspondence of features from each modality to train an audio-visual vehicle detection model. The bounding boxes predicted by this model can be used for a downstream student model that does not depend on the visual domain for vehicle detection.

Due to the evident domain gap for vision-based detection systems, some works consider the auditory domain instead since it does not exhibit a domain gap between different lighting conditions. The task of sound event localization and detection (SELD) is to localize and detect sound-emitting objects using sound recordings of a multi-channel microphone array. The difference in signal volume and time difference of arrival between the microphones can be exploited to infer the location of a sound-emitting object relative to the microphones [1]. However, state-of-the-art learning-based approaches for solving the SELD problem also require hand-annotated labels for training [4], which restricts their applicability in many domains due to the need for large-scale annotated datasets. Self-supervised approaches for audio-visual object detection, in contrast, are able to localize sound-emitting objects in videos without explicit manual annotations for object positions. The majority of these approaches are based on audio-visual co-occurrence of features in both domains [3, 2, 6]. The main idea in these works is to contrast two random video frames and their corresponding audio segment with each other, leveraging the fact that the chance of randomly sampled frames from different videos containing the same object type is negligible. These works, however, only consider mono or stereo audio instead of leveraging the spatial information in multi-channel audio. Furthermore, they cannot directly be used to predict moving vehicles from videos since vehicles may be present in all videos and at any time. Therefore, the assumption that two randomly sampled videos contain different object types no longer holds true. Lastly, pre-trained object detectors in the visual domain have previously been used as teachers to supervise the detection of sound-emitting objects from the auditory domain [7, 20]. In our work, we do not require a pre-trained teacher model but self-supervise the model using auditory and visual co-occurrence of features.

To overcome these issues, we present a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. We demonstrate that no hand-annotated data is required to train this detector and demonstrate its performance on a custom dataset. We furthermore show how this detector can be distilled into a student model which leverages solely the audio modality. In summary, our contributions are as follows:

* A novel approach for self-supervised training of an audio-visual model for moving vehicle detection in video clips.

* The publicly available Freiburg Audio-Visual Vehicles dataset with over 70 minutes of time-synchronized audio and video recordings of vehicles on roads including more than 300 bounding box annotations.

* Extensive qualitative and quantitative evaluations and ablation studies of multiple variants of our student and teacher models including investigations on the influence of the number of audio channels and the resistance to audio noise.

§ RELATED WORKS

§.§ Self-Supervised Audio-Visual Sound Source Localization

The advancement of Deep Learning enabled a multitude of self-supervised approaches for localizing sounds in recent years. The line of work most relevant for the problem we are considering aims at locating sound sources in unlabeled videos [3, 6, 9, 12, 13]. Arandjelović and Zisserman [3] propose a framework for cross-modal self-supervision from video, enabling the localization of sound-emitting objects by correlating the features produced by an audio-specific and image-specific encoding network. Other works consider a triplet loss [17] for contrasting samples of different classes with each other, while Harwarth  [8] propose to learn representations that are distributed spatially within and temporally across video frames. These approaches have in common that they exploit the fact that two different videos sampled from a large dataset have a low probability of containing the same objects, while two randomly sampled snippets from the same video have a very high probability of containing the same objects. This can be used to formulate a supervised learning task where the model uses auditory and visual features to highlight regions in videos where sound-emitting objects are visible.

In contrast to the works mentioned above, we aim at localizing object instances of a single class, namely moving vehicles, and leverage those detections for a downstream audio-detector model. The assumption that two videos randomly sampled from the dataset have a low likelihood of showing the same object class no longer holds in our application since two different videos may both contain segments with and without vehicles. To circumvent this limitation, we use the audio volume to provide a robust heuristic for classifying image-audio pairs.

§.§ Audio-Supported Vehicle Detection

Multiple approaches have been proposed in the last two decades for audio-supported vehicle detection. Chellappa  [5] propose an audio-visual vehicle tracking approach that uses Markov chain Monte Carlo techniques for joint audio-visual tracking. Wang  [23, 24] propose a multimodal temporal panorama approach to extract and reconstruct moving vehicles from an audio-visual monitoring system for subsequent downstream vehicle classification tasks. Schulz  [16] leverage Direction-of-Arrival features from a MEMS acoustic array mounted on a vehicle to detect nearby moving vehicles even when they are not in direct line-of-sight. Cross-modal model distillation approaches for detecting vehicles using auditory information were recently investigated intensively [7, 20]. Gan  [7] propose an approach that leverages stereo sounds as input and a cross-modal auditory localization system that can recover the coordinates of moving vehicles in the reference frame from stereo sound and camera meta-data. The authors utilize a pre-trained YOLO-v2 object detector for images to provide the supervision for their approach. Valverde  [20] propose a multimodal approach to distill the knowledge of multiple pre-trained teacher models into an audio-only student model.

In contrast to previous work, we do not rely on manually annotated datasets to pre-train a model as a noisy label-generator for visual object detection. Instead, we leverage an audio volume-based heuristic to provide supervision to detect moving vehicles in the videos.

§ TECHNICAL APPROACH

Pivotal to the motivation of our approach is the observation that sound emitted by passing vehicles and the associated camera images have co-occurring features in their respective domains. Our framework leverages this fact to predict heatmaps for sound-emitting vehicles in the camera images in a self-supervised fashion. It uses these heatmaps to generate bounding boxes for moving vehicles. Subsequently, it uses the bounding boxes as labels for an audio-only model for estimating the direction of arrival (DoA) of the moving vehicles. Figure <ref> illustrates the core components of the approach.

We use a volume-based heuristic to classify the videos into positive, negative, and inconclusive image-spectrogram pairs. We subsequently train an audio-visual teacher model, denoted as AV-Det, on positive and negative pairs. We embed both the input image and the stacked spectrograms into a feature space using encoders $T_I$ and $T_A$, producing a heatmap that indicates spatial correspondence between the image features and the audio features. We post-process the heatmap to generate bounding boxes. These bounding boxes can be used to train an optional audio-detector model.

§.§ Learning to Detect Sound-Emitting Vehicles

To localize the sound-emitting vehicles in a given image-spectrogram pair, we employ the contrastive learning approach introduced by Arandjelović  and Zisserman [3]. We denote an image and its associated audio segment at time step $t$ as the pair $(I_t, A_t)$, where $A_t$ denotes the concatenated spectrograms obtained from the microphone signals temporally centered around the recording time-stamp of the image $I_t$. Following the formulation by Arandjelović  and Zisserman [3], we embed the image $I_t$ in a $C$-dimensional embedding space using a convolutional encoder network $T_I$, producing a feature map $f_I$ of dimensions $H \times W \times C$. Similarly, we embed the audio spectrograms $A_t$ using a convolutional encoder network $T_A$, to produce an audio feature vector $f_A$ with dimensions $1 \times 1 \times C$. We calculate the euclidean distance between each feature vector in the image feature map with the audio feature vector and thus obtain a heatmap $\mathcal{H}$. Formally, the heatmap elements $\mathcal{H}_{mn}$ for image $I$ and multi-channel spectrogram $A$ are defined as

\begin{equation}
\mathcal{H}_{mn} = || T_I(I)_{mn} - T_A(A)||_2^2.
\end{equation}

The localization network $T_I$ is encouraged to reduce the distance between a feature vector from the region in the image containing the sound-emitting object and sound embeddings from audio segments with the same time-stamp while it increases the distance to segments containing no vehicles. This formulation requires knowledge about which sound-emitting objects are visible or audible in each video segment. In previous works, a randomly selected query image-audio pair $(I_q, A_q)$ is contrasted with pairs sampled from different videos since it can be reasonably assumed, due to the large size of the dataset, that the other pairs in the batch contain different object classes than the query pair. With the problem at hand, this assumption is not valid due to the following complications: Each video contains segments both with and without moving vehicles. Thus, the assumption that each video exclusively contains a  single class is incorrect. Therefore, we also cannot assume that two different videos contain different object classes as there are sections in all videos where moving vehicles are present. To overcome this challenge, we instead classify the image-audio pairs in the dataset into three classes: Positive, Negative, and Inconclusive. The Positive class entails pairs for which we assume they contain a moving vehicle, while the Negative class entails no moving vehicles. The Inconclusive class entails pairs for which no clear association can be found. To train our model, we omit the inconclusive pairs as we cannot be certain about their class association. The details of this approach are discussed in Subsection <ref>. We can now formulate the problem as a binary classification problem similar to Arandjelovic and Zisserman [3], where we use a binary cross-entropy loss to learn heatmaps highlighting image regions containing a moving vehicle. Denoting a mini-batch containing positive samples as $B_+$ and negative samples as $B_-$, we arrive at the following per-batch loss:

\begin{eqnarray}
\mathcal{L} &=& - \frac{1}{|B_+| + |B_-|} \left[ \sum_{i=1}^{|B_+|} \log(\max \mathcal{H}_i) \right. \nonumber \\
& & \qquad + \left. \sum_{j=1}^{|B_-|} \log(1 - \max \mathcal{H}_j) \right]
\end{eqnarray}

Minimizing this loss encourages the image encoder network to highlight image regions where the image features are similar to the audio features and dissimilar otherwise.

§.§ Sample Classification Heuristic

Previously, we glanced over the fact that we require a classifier providing positive and negative samples within a batch. However, in general, we have no access to these labels in the self-supervised learning setting. To circumvent this limitation, we use an audio-volume-based heuristic to classify samples. We leverage two key observations in the data distribution for this classifier: Firstly, frames with low volume typically do not contain any moving vehicles, while louder frames tend to contain moving vehicles as long as the camera is generally directed towards the street and vehicles are located in the camera view frustum. Secondly, we do not require all frames to be classified in this fashion. Instead, we require only a subset of all pairs to be classified as long as the number is large enough to prevent over-fitting on the positive and negative samples used for training. Therefore, we label the quietest $N_{q}$ as negative, and the loudest $N_{l}$ pairs from each data collection run as positive pairs. For our experiments, we empirically selected the loudest 15% and quietest 15% of samples for all recorded sequences. We found that thresholds between 5% and 20% lead to model performances similar to the values reported in Sec. <ref>. Figure <ref> illustrates the classification of audio segments based on the audio volume. The audio volume can potentially also be affected by the ego-robot (motor noise or noise from pavement bumps), however, the system can principally be adjusted to these additional noise sources by adding audio filtering techniques in an optional post-processing step.

§.§ Heatmap Conversion to Bounding Boxes

To be able to quantify the model performance with object detection metrics, we convert the heatmap produced by our model with the following approach: We first clip the heatmap at a threshold of $0.5$. Subsequently, we extract bounding boxes from the clipped heatmap by drawing boxes around connected regions. We do not perform any filtering of or post-processing of the heatmap and also no filtering of the bounding boxes.

§.§ Audio Student Model

We adapt our audio student model architecture from the EfficientDet object detection model [18]. The input of the student model are raw channel-concatenated audio spectrograms with a resolution of $512 \times 128$ pixels. The spectrogram corresponds to an audio snippet with a duration of 1 second. We modify the first feature encoding layer to accommodate the varying numbers of input spectrograms. The student model is trained with supervision from the bounding boxes from our AV-Det teacher model as illustrated in Fig. <ref>. We use a focal loss with $\gamma = 2$ as the learning objective for our student model.

§ DATASET

We collected a real-world video dataset of moving vehicles, the Freiburg Audio-Visual Vehicles dataset, which we will make publicly available with the publication of this manuscript. We use an XMOS XUF216 microphone array with 7 microphones in total for audio recording, where six microphones are arranged circularly with a 60-degree angular distance and one microphone is located at the center. The array is mounted horizontally for maximum angular resolution for objects moving in the horizontal plane. To capture the images, we use a FLIR BlackFly S RGB camera and crop the images to a resolution of $400 \times 1200$ pixels. Images are recorded at a fixed frame rate of 5 Hz, while the audio is captured at a sampling rate of 44.1 kHz. The microphone array and the camera are mounted on top of each other with a vertical distance of ca. $\SI{5}{cm}$.

Exemplary frames from our Freiburg AudioVisual Vehicles dataset including bounding box annotations for moving vehicles. Scenes include busy suburban streets and rural roads in varying lighting conditions. The data collection sensor configuration is depicted in the top right corner.

For our dataset, we consider two scenarios: static recording platform and moving recording platform. In the static platform scenario, the recording setup is mounted on a static camera mount. In the dynamic platform scenario, the recording setup is handheld and is moved in a translational fashion (approx. $\SI{15}{cm}$ of positional range along each spatial axis) and rotated with a maximal deviation angle of $\SI{10}{\deg}$. We collected 70 minutes of audio and video footage in nine distinct scenarios with different weather conditions ranging from clear to overcast and foggy. Overall, the dataset contains more than 20k images. The recording environments entail suburban, rural, and industrial scenes. The distance of the camera to the road varies between scenes. To evaluate detection metrics with our approach, we manually annotated more than 300 randomly selected images across all scenes with bounding boxes for moving vehicles. The dataset will be made available at <http://av-vehicles.informatik.uni-freiburg.de>.

[color=blue,very thick,forget plot]

table[x expr=+1,y index=0] figures/vol_data.csv;

[thick] (axis cs:/pgfplots/xmin,0.25) – (axis cs:/pgfplots/xmax,0.25);
[thick] (axis cs:/pgfplots/xmin,0.38) – (axis cs:/pgfplots/xmax,0.38);

[below right] at (axis cs:620,0.57) Positive;
[below right] at (axis cs:620,0.17) Negative;
[below right] at (axis cs:620,0.37) Inconclusive;

Channel-averaged audio volume $V$ over the maximum magnitude $V_0$ for the first $600$ time steps of a recording. We add horizontal bars indicating the upper and lower threshold value for positive, negative, and inconclusive samples.

§ EXPERIMENTAL RESULTS

We first compare our model with two baseline models (Section <ref>). We also perform an ablation study examining the influence of each model component on the detection metrics (Section <ref>) and evaluate the performance of the audio-detector student model (Section <ref>). We furthermore investigate the influence of noise on the heuristic classification accuracy and on the audio-visual detection metrics (Section <ref>). To quantify model performance we use the AP value of our only class Vehicle. We list AP values for IoU thresholds of $0.1$, $0.2$, and $0.3$, denoted as<EMAIL_ADDRESS><EMAIL_ADDRESS>and<EMAIL_ADDRESS>respectively. Furthermore, we introduce a bounding box center distance metric, denoted as CD. This distance metric quantifies the average euclidean distance of ground-truth and predicted bounding boxes in each image with an optimal assignment of predicted bounding boxes to ground truth bounding boxes. We introduce the CD metric due to our observation that many predicted bounding boxes are not aligned with the outline of the moving vehicles but instead occupy a subset of the vehicle area or expand over the outline of the vehicles, which leads to box overlaps below the IoU threshold. The CD metric thus helps to quantify the similarity of the box centers instead of the box overlaps.

Performance of models trained and evaluated on the full Freiburg Audio-Visual Vehicles dataset and on the dynamic split. Better metric values are indicated with arrows. The best-performing model results are shown in bold text while the best-performing self-supervised model results are shown in italics.

Approach     4cFull     4cDynamic

<EMAIL_ADDRESS>$\uparrow$<EMAIL_ADDRESS>$\uparrow$<EMAIL_ADDRESS>$\uparrow$     CD $\downarrow$<EMAIL_ADDRESS>$\uparrow$<EMAIL_ADDRESS>$\uparrow$<EMAIL_ADDRESS>$\uparrow$     CD $\downarrow$

Frame Differencing      $0.3921$     $0.3411$     $0.3192$     $23.22$     $0.2192$     $0.0920$     $0.0470$     $45.6$

Optical Flow           $0.4933$     $0.4890$     $0.4828$     $19.15$     $0.1867$     $0.1671$     $0.1633$     $32.9$

Pre-trained Detector      $0.6052$      $0.6047$     $0.5949$     $21.14$     $\mathbf{0.7662}$     $\mathbf{0.7662}$     $\mathbf{0.7662}$     $\mathbf{5.1}$

Pre-trained Detector + Flow      $0.6634$      $\mathbf{0.6634}$     $\mathbf{0.6573}$     $\mathbf{12.81}$     $\mathbf{0.7662}$     $\mathbf{0.7662}$     $\mathbf{0.7662}$     $\mathbf{5.1}$

AV-Det 6-channel Oracle           $0.6054$     $0.4971$     $0.3634$     $19.76$     $0.6294$     $0.3418$     $0.1874$     $23.2$

AV-Det 1-channel     $0.4169$     $0.2662$     $0.1774$     $25.49$     $0.1882$     $0.0966$     $0.0499$     $31.1$

AV-Det 2-channel     $0.3371$	    $0.2191$    	$0.1292$    	$26.50$    	$0.3724$    	$0.3083$    	$0.1233$    	$36.4$

AV-Det 4-channel     $0.4674$	    $0.3587$    	$0.1901$    	$21.08$    	$0.4237$    	$0.1455$    	$0.0716$    	$\mathbf{19.9}$

AV-Det 6-channel     $\mathbf{0.7126}$	    $\mathit{0.6195}$    	$\mathit{0.5575}$    	$\mathit{16.47}$    	$\mathit{0.6873}$    	$\mathit{0.4305}$    	$\mathit{0.2869}$    	$33.4$

§.§ Baselines

Due to the lack of prior work for self-supervised audio-visual vehicle detection, we created a flow-based baseline using the RAFT architecture [19]. The baseline also does not require any manual annotations and does not rely on pre-trained detectors. The RAFT model was pre-trained on FlyingChairs, FlyingThings3D, Sintel, and KITTI. We use the predicted optical flow to segment moving objects from videos. To obtain bounding boxes from the optical flow model, we first calculate the optical flow between two consecutive frames and threshold the flow field. We finally draw a bounding box around each connected region with a flow value above an empirically found threshold. We also created a baseline based on frame differencing. For this baseline, we threshold the absolute difference between two consecutive frames and extract bounding boxes. We furthermore evaluate the performance of an EfficientDet detector model [18], which was pre-trained on the MS COCO dataset. We also combine the optical flow baseline with the pre-trained object detector and filter out bounding boxes predicted by the detector if the predicted flow inside a bounding box is below a threshold value. This reduces the risk of the object detector predicting a false-positive static vehicle.

§.§ Audio-Visual Teacher Evaluation

In our experiments, we investigate the influence of the number of audio channels. Furthermore, we report the influence of our classification heuristic and compare models trained with labels from the heuristic to a model trained with labels from manual annotations, denoted as Oracle. Table <ref> lists the results of each model variant. We observe that our best-performing model variant clearly outperforms the frame differencing and vanilla optical flow baselines. All other AV-Det models also mostly outperform the baselines in the dynamic split of our dataset, where many false-positive detections are reported by the frame differencing and optical flow baselines, likely due to the camera motion inducing substantial background pixel differences in consecutive frames. We furthermore report detection metrics from our best-performing AV-Det model that are mostly on par with the pre-trained EfficientDet + Flow model for the joint static and dynamic split of our dataset and shows the best recorded performance for the<EMAIL_ADDRESS>metric. For the dynamic split of the dataset, we observe a deteriorated performance of our model compared to the pretrained model performance. We note that false-positive detections of the plain pre-trained EfficientDet detector are introduced by the detections of static vehicles in the background. This effect is lessened when the pre-trained detector is combined with the optical flow model to ignore static background vehicle boxes, leading to better detection metrics. In general, we observe that using mono-audio leads to inferior performance compared to the model variants with multiple audio channels.

We also investigate the difference in performance between a model trained with the ground truth classification data and our heuristic. With the heuristic, we report an overall precision of $94.2 \%$ for samples labeled as Positive and $77.7 \%$ for samples labeled as Negative, respectively. We observe that using the manually annotated ground truth labels does not significantly improve the detection accuracy compared to the heuristic and performs worse than our best-performing AV-Det model trained on classifications obtained from our heuristic. We presume that the heuristic provides less ambiguous training data compared to the manual annotations, which classify an audio-visual sample as Positive if at least a part of a moving vehicle is visible in the frame. On one hand, this does not necessarily mean that the vehicle is clearly audible at the same time and on the other hand, a loud vehicle could just have left the camera frustum, leading to a Negative audio-video pair that contains sounds characteristic to vehicles but has no correspondence with the image content. Pairs produced with our heuristic suffer to a lesser extent from this inconsistency.

§.§ Qualitative Results

Qualitative results of our AV-Det teacher model on the test split of our dataset. We visualize the color-coded learned vehicle heatmap, the estimated bounding boxes in green color, and the ground truth bounding boxes in blue color. The two bottom rows illustrate interesting failure cases of our approach.

Figure <ref> illustrates exemplary detections of our best-performing model on our Freiburg Audio-Visual Vehicles dataset. We observe that our model generally predicts mostly accurate heatmaps and resulting bounding boxes even in scenes with a large amount of background clutter. Also, scenes with multiple moving vehicles present show high detection accuracy. While the bounding boxes generally overlap well with the ground-truth regions containing moving vehicles, we observe that in some images, the heatmap extends over the actual dimensions of vehicles or is slightly misaligned. We hypothesize that this is due to the imperfect labeling of our volume-based heuristic for assigning image-spectrogram pairs the labels Vehicle or No-Vehicle, which results in an inaccurate back-propagation of the label signal back to the respective pixel areas in the input images. We further note a failure mode where false-positive detections of vehicles are induced by incorrect highlights in the learned heatmaps. This effect may be caused by background movement correlating with the presence of moving vehicles in the foreground, producing an incorrect association of auditory and visual features. Moreover, the approach may fail to correlate visual and audio features correctly when sound-emitting objects are located outside the camera frustum since the approach assumes such objects to be observable in both domains simultaneously. In our experiments, however, we found the moving vehicles to be the dominant sound source despite audible ambient noise.

§.§ Audio-Detector Student Model Evaluation

We evaluate the detection performance of the audio-detector student model in Table <ref>. We run experiments using 1, 2, 4, and 6-channel audio. We also evaluate a student model trained on RGB images instead of audio spectrograms. All audio-detector student models perform worse than the AV-Det teacher model but exhibit an overall decent performance with<EMAIL_ADDRESS>values close to $0.4$. In contrast to the AV-Det model, we observe no strong correlation between the number of audio channels and the detection metrics. The 6-channel model version performs similar to the 1-channel version or the other versions regarding the<EMAIL_ADDRESS>and<EMAIL_ADDRESS>metrics. We hypothesize that the student model architecture, which was adapted from the EfficentDet architecture, can only insufficiently extract additional information from the multiple spectrograms for detecting the position of moving vehicles. This assumption is underlined by the fact that the student model fed with the RGB images instead of the spectrograms performs similar to its teacher model, raising the question of whether further modifications to the student model need to be made to allow for better detection of vehicles solely from multi-channel spectrograms.

Performance of audio-detector model variants.

Model<EMAIL_ADDRESS>$\uparrow$<EMAIL_ADDRESS>$\uparrow$     CD $\downarrow$

Student 1-channel     $0.3583$     $\mathbf{0.2756}$     $40.8$

Student 2-channel     $0.3472$     $0.2450$     $\mathbf{36.5}$

Student 4-channel     $0.3799$     $0.2544$     $41.5$

Student 6-channel     $\mathbf{0.3833}$     $0.2636$     $39.2$

Student RGB Image     $0.6885$     $0.5727$     $26.8$

§.§ Sensitivity to Noise

We conduct experiments regarding the noise resistance of our approach. As described in Subsection <ref>, we use a volume-based heuristic to sort the samples into the classes Positive, Negative, and Inconclusive. One shortcoming of this approach is that objects producing sounds outside the camera frustum can change the prediction of the heuristic due to the change in overall audio signal magnitude. Furthermore, audio noise pollutes the spectrogram and leads to features that are less characteristic of vehicle sounds or the lack thereof. We therefore added varying amounts of white noise to the audio samples in our dataset. We sample the noise from a Gaussian distribution with zero mean and a standard deviation according to the specific Signal-to-Noise Ratio (SNR). We then classify samples using our heuristic and train the AV-Det model with these corrupted samples. Figure <ref> visualizes the influence of noise on the precision score of our heuristic and the<EMAIL_ADDRESS>scores of the AV-Det model. An SNR of $\SI{0}{dB}$ represents noise with the same magnitude as the audio signal while an SNR of $\SI{80}{dB}$ represents a noise signal four orders of magnitude smaller than the signal. We observe that the precision of the heuristic slightly decreases with increasing amounts of noise and arrives at a precision of $0.64$ for an SNR of $\SI{0}{dB}$. The AV-Det model precision only slightly decreases for an SNR of $\SI{40}{dB}$ but is noticeably reduced to $0.27$ for an SNR of $\SI{0}{dB}$. We conclude that our approach shows acceptable performance for lower levels of noise, but its performance deteriorates with higher amounts of noise, where the precision of our sample classification heuristic decreases. Overall, we conclude that our approach can work well for practical applications where ambient noise does not exceed moderate noise levels but clearly shows improved performance when the moving vehicles are the predominant sound source in a given scene.

scale only axis,
scaled x ticks=base 10:0,
xmin=0, xmax=80

axis y line*=left,
ymin=0, ymax=1.0,
xlabel=SNR / dB,

axis y line*=right,
axis x line=none,
ymin=0, ymax=1.0,
legend style=at=(1.0,0.34),anchor=north east,nodes=right]
/pgfplots/refstyle=plot_accuracyLabel Heuristic Precision
; AV-Det<EMAIL_ADDRESS>
Precision of our classification heuristic and our model<EMAIL_ADDRESS>score on samples corrupted with audio noise.

§ CONCLUSION

In this work, we presented a self-supervised audio-visual approach for moving vehicle detection. We showed that an audio volume-based heuristic for sample classification in combination with an audio-visual model trained in a self-supervised manner produces accurate detections of moving vehicles in images. We furthermore demonstrated that the number of audio channels greatly influences the model detection performance, where more audio channels lead to improved performance compared to single-channel audio. Finally, we illustrated that the audio-visual teacher model can be distilled into an audio-only student model, bridging the domain gap inherent to models leveraging vision as the predominant modality. Possible future work includes 360 degree images for larger visual field-of-view, more diverse scenarios in the dataset, and an improved model architecture for the audio student model.

[1]
Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen.
Sound event localization and detection of overlapping sources using
convolutional recurrent neural networks.
IEEE Journal of Selected Topics in Signal Processing,
13(1):34–48, 2018.

[2]
Triantafyllos Afouras, Yuki M Asano, Francois Fagan, Andrea Vedaldi, and
Florian Metze.
Self-supervised object detection from audio-visual correspondence.
arXiv preprint arXiv:2104.06401, 2021.

[3]
Relja Arandjelovic and Andrew Zisserman.
Objects that sound.
In Proceedings of the European conference on computer vision
(ECCV), pages 435–451, 2018.

[4]
Mark D Baxendale, Martin J Pearson, Mokhtar Nibouche, Emanuele Lindo Secco, and
Anthony G Pipe.
Audio localization for robots using parallel cerebellar models.
IEEE Robotics and automation letters, 3(4):3185–3192, 2018.

[5]
Rama Chellappa, Gang Qian, and Qinfen Zheng.
Vehicle detection and tracking using acoustic and video sensors.
In 2004 IEEE International Conference on Acoustics, Speech, and
Signal Processing, volume 3, pages iii–793. IEEE, 2004.

[6]
Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi,
and Andrew Zisserman.
Localizing visual sounds the hard way.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 16867–16876, 2021.

[7]
Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba.
Self-supervised moving vehicle tracking with stereo sound.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 7053–7062, 2019.

[8]
David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio
Torralba, and James Glass.
Jointly discovering visual objects and spoken words from raw sensory
In Proceedings of the European conference on computer vision
(ECCV), pages 649–665, 2018.

[9]
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin,
and Dejing Dou.
Discriminative sounding objects localization via self-supervised
audiovisual matching.
Advances in Neural Information Processing Systems, 33, 2020.

[10]
Rainer Kümmerle, Michael Ruhnke, Bastian Steder, Cyrill Stachniss, and
Wolfram Burgard.
A navigation system for robots operating in crowded urban
In 2013 IEEE International Conference on Robotics and
Automation, pages 3225–3232. IEEE, 2013.

[11]
Rainer Kümmerle, Michael Ruhnke, Bastian Steder, Cyrill Stachniss, and
Wolfram Burgard.
Autonomous robot navigation in highly populated pedestrian zones.
Journal of Field Robotics, 32(4):565–589, 2015.

[12]
Andrew Owens and Alexei A Efros.
Audio-visual scene analysis with self-supervised multisensory
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 631–648, 2018.

[13]
Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin.
Multiple sound sources localization from coarse to fine.
In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16, pages 292–308.
Springer, 2020.

[14]
Noha Radwan, Wolfram Burgard, and Abhinav Valada.
Multimodal interaction-aware motion prediction for autonomous street
The International Journal of Robotics Research,
39(13):1567–1598, 2020.

[15]
Noha Radwan, Wera Winterhalter, Christian Dornhege, and Wolfram Burgard.
Why did the robot cross the road?—learning from multi-modal sensor
data for autonomous road crossing.
In 2017 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 4737–4742. IEEE, 2017.

[16]
Yannick Schulz, Avinash Kini Mattar, Thomas M Hehn, and Julian FP Kooij.
Hearing what you cannot see: Acoustic vehicle detection around
IEEE Robotics and Automation Letters, 6(2):2587–2594, 2021.

[17]
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon.
Learning to localize sound source in visual scenes.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 4358–4366, 2018.

[18]
Mingxing Tan, Ruoming Pang, and Quoc V Le.
Efficientdet: Scalable and efficient object detection.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 10781–10790, 2020.

[19]
Zachary Teed and Jia Deng.
Raft: Recurrent all-pairs field transforms for optical flow.
In European conference on computer vision, pages 402–419.
Springer, 2020.

[20]
Francisco Rivera Valverde, Juana Valeria Hurtado, and Abhinav Valada.
There is more than meets the eye: Self-supervised multi-object
detection and tracking with sound by distilling multimodal knowledge.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11612–11621, 2021.

[21]
Johan Vertens, Jannik Zürn, and Wolfram Burgard.
Heatnet: Bridging the day-night domain gap in semantic segmentation
with thermal images.
In 2020 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 8461–8468. IEEE, 2020.

[22]
Mei Wang and Weihong Deng.
Deep visual domain adaptation: A survey.
Neurocomputing, 312:135–153, 2018.

[23]
Tao Wang and Zhigang Zhu.
Multimodal and multi-task audio-visual vehicle detection and
In 2012 IEEE Ninth International Conference on Advanced Video
and Signal-Based Surveillance, pages 440–446. IEEE, 2012.

[24]
Tao Wang and Zhigang Zhu.
Real time moving vehicle detection and reconstruction for improving
In 2012 IEEE Workshop on the Applications of Computer Vision
(WACV), pages 497–502. IEEE, 2012.

[25]
Jannik Zürn, Wolfram Burgard, and Abhinav Valada.
Self-supervised visual terrain classification from unsupervised
acoustic feature learning.
IEEE Transactions on Robotics, 37(2):466–481, 2020.


# Coherent manipulation of nuclear spins in the strong driving regime

Dan Yudilevich  Department of Chemical and Biological Physics, Weizmann
Institute of Science, Rehovot 7610001, Israel  Alon Salhov  Racah Institute of
Physics, The Hebrew University of Jerusalem, Jerusalem 9190401, Israel  Ido
Schaefer  Racah Institute of Physics, The Hebrew University of Jerusalem,
Jerusalem 9190401, Israel  Dahlem Center for Complex Quantum Systems and
Fachbereich Physik, Freie Universität Berlin, Arnimallee 14, 14195, Berlin,
Germany  Konstantin Herb  Department of Physics, ETH Zürich, Otto Stern Weg 1,
8093 Zurich, Switzerland  Alex Retzker  Racah Institute of Physics, The Hebrew
University of Jerusalem, Jerusalem 9190401, Israel  AWS Center for Quantum
Computing, Pasadena, CA 91125, USA  Amit Finkler  Department of Chemical and
Biological Physics, Weizmann Institute of Science, Rehovot 7610001, Israel
<EMAIL_ADDRESS>
###### Abstract

Spin-based quantum information processing makes extensive use of spin-state
manipulation. This ranges from dynamical decoupling of nuclear spins in
quantum sensing experiments to applying logical gates on qubits in a quantum
processor. Fast manipulation of spin states is highly desirable for
accelerating experiments, enhancing sensitivity, and applying elaborate pulse
sequences. Strong driving using intense radio-frequency (RF) fields can,
therefore, facilitate fast manipulation and enable broadband excitation of
spin species.

In this work, we present an antenna for strong driving in quantum sensing
experiments and theoretically address challenges of the strong driving regime.
First, we designed and implemented a micron-scale planar spiral RF antenna
capable of delivering intense fields to a sample. The planar antenna is
tailored for quantum sensing experiments using the diamond’s nitrogen-vacancy
(NV) center and should be applicable to other solid-state defects. The antenna
has a broad bandwidth of 22 MHz, is compatible with scanning probes, and is
suitable for cryogenic and ultrahigh vacuum conditions. We measure the
magnetic field induced by the antenna and estimate a field-to-current ratio of
$113\pm 16$ G/A, representing a six-fold increase in efficiency compared to
the state-of-the-art, crucial for cryogenic experiments. We demonstrate the
antenna by driving Rabi oscillations in 1H spins of an organic sample on the
diamond surface and measure 1H Rabi frequencies of over 500 kHz, i.e.,
$\mathrm{\pi}$-pulses shorter than 1 $\upmu\mathrm{s}$ – faster than
previously reported in NV-based nuclear magnetic resonance (NMR).

Finally, we discuss the implications of driving spins with a field tilted from
the transverse plane in a regime where the driving amplitude is comparable to
the spin-state splitting, such that the rotating wave approximation does not
describe the dynamics well. We present a simple recipe to optimize pulse
fidelity in this regime based on a phase and offset-shifted sine drive, which
may be optimized in situ without numerical optimization procedures or precise
modeling of the experiment. We consider this approach in a range of driving
amplitudes and show that it is particularly efficient in the case of a tilted
driving field.

The results presented here constitute a foundation for implementing fast
nuclear spin control in various systems.

##  I Introduction

Quantum sensing with solid-state spin sensors, such as the nitrogen-vacancy
(NV) center in diamond, frequently involves manipulating nuclear spin states.
Nuclear spins may be part of the sample of interest, as in the case of
nanoscale nuclear magnetic resonance (NMR) spectroscopy, which relies on
sequences of radio-frequency (RF) pulses applied to the sample to recover
information on its chemical structure [1, 2, 3, 4, 5]. Solid-state nuclear
spins around the sensor are also utilized as ancilla qubits that store the
quantum state of a sensor to retrieve it repeatedly [6, 3] or to prolong the
sensing time [7].

Most experiments have relied so far on antennas that induce weak RF driving
fields, with a standard $\pi$-pulse lasting a few tens of microseconds [8, 1,
6]. These lengthy pulses imply longer measurement times and, thus, reduced
sensitivity [9]. They may also impede the application of elaborate pulse
sequences, as the sensing time in NV-based NMR is limited by the spin
relaxation time of the NV center ($T_{1}$)[10], or a nuclear memory [3].

Fast manipulation of nuclear spins by strong RF driving fields can better
utilize the limited sensing time of NV center sensors, generate broadband
excitation of the nuclear spin resonance, and enable novel sensing protocols
[11]. Previous works demonstrated strong driving for the NV center electron
spin at a rate of $\sim 1\,\mathrm{GHz}$ [12], and 13C spins in diamond at
$\sim 70\,\mathrm{kHz}$ [13]. The highest reported driving rates for protons
in NV-based NMR are $50-80\,\mathrm{kHz}$ [1, 3].

The system described below enables spin manipulation in a regime where the
driving strength $\Omega_{d}$ is close to the energy splitting (i.e.,
$\Omega_{d}\lesssim\omega_{0}$). Alongside the experimental challenges of
producing strong driving fields, working in this regime poses a theoretical
control challenge. Most experimental setups deliver linearly polarized fields
often tilted away from the transverse plane. For weak driving strengths
($\Omega_{d}\ll\omega_{0}$), the dynamics are accurately approximated by
sinusoidal state transitions, known as Rabi oscillations, under the rotating-
wave approximation (RWA). In the regime where $\Omega_{d}\lesssim\omega_{0}$,
however, the deviations from an ideal drive (specifically, a linearly
polarized drive with a longitudinal component) will markedly alter the
dynamics [12]. Without proper adaptations, this “breakdown” of the RWA results
in the deterioration of pulse fidelity. It is thus crucial to design the
signals so as to optimize an operation’s fidelity in the strong driving
regime.

The issue of strong driving has attracted interest, especially for quantum
information processing, where strong driving can accelerate operations and
increase the speed of quantum processors [14, 15]. Among others, optimal
control strategies have been employed for optimizing quantum control in the
strong driving regime. In particular, the concept of time-optimal control
fields [16] has been introduced to identify the shortest possible signals to
control a qubit state [17, 18]. Bang-bang control sequences have been shown to
be the quickest form, while bang-bang driving at rates exceeding $\omega_{0}$
has been demonstrated on solid-state qubits [19], and other optimal control
theory-derived waveforms have been demonstrated for solid-state qubit controls
[20]. Optimal control approaches require a precise description of the driving
field and the qubit, e.g., the relative orientation and magnitude, according
to which the control signal is calculated. However, errors in the estimated
parameters might deteriorate the ultimate performance compared with the
simulations [20]. Also, in situ optimization is difficult as it requires
sampling a complex parameter space.

In this work, we design and implement a micrometer-scale planar spiral RF
antenna compatible with NV magnetometry and capable of delivering intense RF
pulses to a diamond sample. We characterize the antenna’s characteristics and
performance. Demonstrating the antenna’s function by driving proton (1H)
spins, we observe spin state Rabi oscillations at frequencies surpassing 500
kHz.

We then discuss the unique characteristics of spin-state control by a strong
and tilted driving field. We propose a novel approach to optimize the fidelity
of control pulses in this regime, which is particularly suitable for driving
fields that are noisy or not fully characterized.

##  II Experimental methods

We performed experiments on a home-built room temperature confocal microscope.
NV center electron spins were excited by a 520 nm diode laser, and their
fluorescence was measured by a single-photon counting module. Low-frequency RF
signals ($\sim$1 MHz) were irradiated to the sample via our novel, custom-
designed spiral antenna (see further in Sec. III), and the NV center
electronic spins were controlled by microwave pulses delivered by a wire drawn
above the sample.

The diamond sample was a thin, single-crystal $\left[100\right]$ diamond
membrane (approximately 30 $\upmu\mathrm{m}$ thick) patterned with nanopillar
diamond waveguides. Shallow NV centers were created in the diamond by nitrogen
ion implantation. For proton sensing, a small drop of microscope immersion oil
(Cargille Type LDF) was applied to the diamond’s surface with a sterile
syringe (see further details in SI-2).

##  III Planar spiral radio-frequency antenna

We designed the RF antenna as a planar spiral. Fig. 1(a) depicts the setup
schematically. Compatibility with a typical NV magnetometry apparatus was the
fundamental design principle. The diamond is placed directly on the antenna to
enhance the magnetic field induced at the sample’s position. A small aperture
at the center of the spiral allows optical access to NV centers. An additional
wire, drawn above the sample, is dedicated to signals in the gigahertz range
for manipulating the NV center’s electron spin.

Figure 1:  Schematic of the experimental system (not to scale). (a) The RF
spiral antenna sits underneath a thin diamond sample with NV centers in
nanopillar waveguides. The NV centers are addressed optically through an
aperture in the antenna. Microwave signals to the NV center are applied with a
thin copper wire drawn above the diamond sample. Immersion oil with 1H is
placed atop the diamond. (b) The static magnetic field $B_{0}$ is aligned with
the NV center’s axis. The spiral antenna’s field is approximately
perpendicular to the diamond’s surface, inducing an RF field component along
the NV’s axis ($B_{z}$) and a component perpendicular to $B_{0}$ ($B_{x}$).

The antenna was fabricated on a polyimide flexible printed circuit, suitable
for ultra-high vacuum and cryogenic environments. The planar geometry can
accommodate a scanning probe, such as an atomic force microscope, which may be
used, for example, to carry a sample [21] or create a magnetic field gradient
[22].

In our antenna, the inner loop diameter is $600\,\upmu\mathrm{m}$, at the
center of which is a $200\,\upmu\mathrm{m}$-diameter optical aperture; the
sample’s area of interest is placed on the aperture (Fig. 2(a)). The inner
diameter was minimized to achieve the strongest field up to the fabrication
capabilities. The spiral consists of two identical layers, separated by a
$20\,\upmu\mathrm{m}$ polyimide layer and connected by a via at the center.
The number of turns and trace width of the spiral can be set to optimize its
operation; for a larger field-per-current ratio, the number of turns should be
increased. However, the bandwidth decreases with the number of turns, and the
field-per-power is maximal for a specific number of turns (see further in
SI-1). The results presented in this study were measured with a 15-turn spiral
and a $100\,\upmu\mathrm{m}$ trace width.

Figure 2: Spiral broadband RF antenna. (a) Photo of the spiral antenna. The
sample sits on the antenna, and the working region is directly above the
aperture. (b) Transmission characteristics of the antenna ($S_{21}$
parameter). (c) Finite-element simulation of the antenna’s field, focusing on
the region of interest. The image shows a cross-section along the dashed line
in (a). The color map and contours depict the magnetic field magnitude, and
the arrows show the projection of the field’s orientation on the XZ plane. The
golden polygons depict the cross-section of the spiral’s trace. The markings
inside the polygons denote the direction of the simulated current. The
sample’s position in the current experiment is marked by the semitransparent
rectangle.

We terminate the antenna with a $\sim 50~{}\Omega$ load that dissipates over
$90\%$ of the generated power. By monitoring the voltage on the load, we also
determine the current through the antenna. The 3 dB bandwidth of the antenna
is approximately 22 MHz, as observed in the transmission spectrum of the
system ($S_{21}$ parameter, Fig. 2(b)). The antenna’s bandwidth allows working
with bias fields of up to 500 mT (for detecting proton spins). Such bias
fields are required when wishing to utilize an ancilla nuclear spin in the
diamond as a quantum memory [23, 6]. Additionally, the large bandwidth enables
the transmission of pulses shorter than a microsecond without significant
distortion.

Fig. 2(c) shows a finite element simulation of the antenna’s field
distribution. The figure depicts the field along a cross-section of the
antenna’s center and where the sample sits. The simulation confirms that the
expected magnetic field is approximately uniform in magnitude and orientation
over the projected sample position. The magnetic field for a 1 A current at
the experiment’s sample position was estimated to be $136\pm 1~{}\mathrm{G}$.
The field vector was nearly perpendicular to the spiral plane, with a slight
tilt of $1\pm 0.7^{\circ}$.

We characterized the magnetic field vector emitted by the antenna using in
situ static magnetic field measurements with the NV center. We swept a direct
current through the antenna and measured the Zeeman shift of the NV center’s
levels around $B_{0}=0$ (without an additional applied field). From the
optically detected magnetic resonance (ODMR) spectra, we extracted the
dependence of the field magnitude on the current and the tilt of the applied
magnetic field to the NV axis. The result is plotted in Fig. 3. We fit the
data to a spin Hamiltonian incorporating strain and a magnetic field tilted
away from the NV center axis. Thus, the transitions are not linearly dependent
on the magnetic field (see SI-2 for further details on the analysis). From the
transitions, we obtain the DC field-to-current ratio of
$B~{}/~{}I_{\mathrm{spiral}}=113\pm 16~{}\mathrm{\frac{G}{A}}$. The field’s
angle is measured to be tilted from the plane transverse to the NV axis by
$\theta_{d}=36.5\pm 5.8^{\circ}$(corresponding to a tilt of $\sim 1.2^{\circ}$
from the normal to the spiral plane). The measured field’s magnitude agrees
with the finite element simulation. The NV center lies at an angle of $\sim
54.7^{\circ}$ to the diamond surface, parallel to the spiral plane. Thus, the
measured orientation is consistent with our expectation that the planar spiral
antenna induces a field normal to its plane.

In a sensing experiment, as discussed in the following section, there will
usually be an applied quantizing magnetic field ($\vec{B}_{0}$) along the NV
center’s axis ($\hat{z}$ in Fig. 1(b)). Under the RWA, the transverse
component ($B_{x}$ in Fig. 1(b)) drives the spins and is proportional to the
Rabi frequency. From the DC characterization, we estimate it to be
$B_{x}~{}/~{}I_{\mathrm{spiral}}=B\cos\left(\theta_{d}\right)/I_{\mathrm{spiral}}=92\pm
14~{}\mathrm{\frac{G}{A}}$ (the magnitude might be attenuated according to the
transmission at the specific frequency, as described in Fig. 2(b)).

Figure 3: Direct current magnetic field characterization. The NV center level
shifts were measured in a series of ODMR spectra with varying currents through
the spiral. The points were fit to a model incorporating the magnetic field
tilt and a strain field. The pink areas mark the confidence intervals of the
fit.

##  IV Fast 1H Rabi oscillations

We demonstrate the antenna’s function by driving Rabi oscillations in a proton
spin ensemble of an organic sample on the diamond’s surface. As a preliminary
experiment, we detect proton nuclear magnetic resonance with an XY8-N
dynamical decoupling sequence [24], employing phase randomization to exclude
spurious harmonics of 13C spins [25]. Fig. 4(b) features an XY8-10 trace with
a dip at the expected position of the proton Larmor frequency ($B_{0}\approx
652~{}\mathrm{G},\ \omega_{{}^{1}\mathrm{H}}\approx 2.78~{}\mathrm{MHz}$),
indicating that the NV senses the proton’s oscillating magnetization.

We then employ a correlation spectroscopy sequence with RF pulses [26, 3] to
observe proton Rabi oscillations, as depicted in Fig. 4(c). We use a
correlation spectroscopy sequence based on XY8-4 dynamical decoupling blocks
locked to the proton Larmor frequency to sense the phase of the proton’s
oscillation [27, 28]. The correlation delay, i.e., the spacing between the two
sensing blocks, was fixed at $20\,\mathrm{\upmu s}$. The RF pulses during the
correlation delay, tuned to the proton Larmor frequency of varying duration,
drive the nuclear magnetization, inducing a
$\ket{\uparrow}\leftrightarrow\ket{\downarrow}$ transition.

The resulting Rabi oscillations are plotted in Fig. 4(d) for several driving
powers corresponding to different spiral currents. The oscillations were
fitted to a decaying sine function, from which we extracted the driving
frequency ($\Omega_{d}$). Fig. 4(e) summarizes the observed driving
frequencies as a function of spiral currents. The driving frequency is
proportional to the driving current, as expected. We achieve a maximal driving
frequency of $530\pm 12~{}\mathrm{kHz}$, ultimately limited by the amplifier’s
saturation power.

We estimate a driving frequency-to-current ratio of
$\Omega_{d}~{}/~{}I_{\mathrm{spiral}}=463\pm 3~{}\mathrm{\frac{kHz}{A}}$. From
this ratio, we estimate the field-to-current ratio of the transverse field at
2.78 MHz to be $B_{1}~{}/~{}I_{\mathrm{spiral}}=108.8\pm
0.7~{}\mathrm{\frac{G}{A}}$; this is in good agreement with the value expected
from in situ DC measurement ($92\pm 14~{}\mathrm{\frac{G}{A}}$) presented
previously and the finite-element simulations ($111.0\pm
0.8~{}\mathrm{\frac{G}{A}}$).

Figure 4:  Fast Rabi oscillations of 1H nuclear spins. (a) A diagram of the
randomized XY8-N pulse sequence used to sense the 1H nuclear magnetic
resonance. (b) A randomized XY8-10 spectrum featuring a dip related to the 1H
Larmor precession. (c) Diagram of the pulse sequence used to observe 1H
nuclear spin Rabi oscillations. The nuclear spin precession was detected by
correlating two XY8-4 dynamical decoupling blocks tuned to the 1H frequency
found in (b). A varying radio frequency pulse tuned to the 1H frequency during
the correlation time drives the 1H spin state. (d) Rabi oscillations of the 1H
spins for different current amplitudes driven through the spiral antenna. (e)
Summary of several Rabi frequencies measured, with a linear dependence on the
current through the antenna.

##  V Manipulating spins in the strong driving regime

As our spiral antenna can indeed reach the strong driving regime
($\Omega_{d}\sim\omega_{0}/5$ in the aforementioned experiment), we describe a
straightforward approach to generate control signals in the
$\Omega_{d}\lesssim\omega_{0}$ regime for high-fidelity operations. We show
that a simple sine signal with an offset may provide sufficient fidelity in
this regime by optimizing just one or two parameters.

Our approach is particularly suitable for tilted drive signals, that is,
signals with a component along the quantization axis (hereinafter referred to
as $\hat{z}$). Tilted drives are found in various solid-state spin qubit
systems, such as the NV center in diamond as described in the previous
section, the SiV defect in diamond [29], defects in SiC [30] and in h-BN [31],
as well as superconducting flux qubits [32]. However, to our knowledge,
optimizing strong tilted drives has not been discussed in the literature.

In what follows, we motivate our approach analytically using a clear physical
picture, illustrate its validity numerically, and compare it to optimal
control-derived signals. We argue that offset-sine signals bear benefits over
optimal control-derived signals while providing similar and sufficiently high
fidelity rates. Our focus is on the optimization of the $\pi$-pulse, which is
to be reached at $t_{\pi}\sim\frac{\pi}{\Omega_{d}}$ (a precise definition
follows below).

###  V.1 Resonant offset-sine driving pulses

We consider a two-level system driven by a tilted driving field. The system is
described by the following Hamiltonian:

$\mathcal{H}=\frac{\omega_{0}}{2}\sigma_{z}+\Omega_{d}f\left(t\right)\left(\sigma_{x}+\tan\left(\theta_{d}\right)\sigma_{z}\right)$
(1)

where $\omega_{0}$ is the energy splitting of the two-level system,
$\Omega_{d}$ is the maximum driving field amplitude, $\left|f(t)\right|\leq 1$
is the waveform, and $\theta_{d}\in\left[0,\frac{\pi}{2}\right)$ is the
driving field’s tilt angle from $\hat{x}$. Under this definition, the drive
vector is not normalized; rather, the field’s magnitude depends on the angle
$\theta_{d}$.

In the weak-driving regime, conventional driving pulses are based on resonant
sine waveforms, i.e., $f(t)=\sin\left(\omega_{0}t+\varphi_{d}\right)$. The
standard analysis proceeds with the rotating-wave-approximation (RWA) [33],
which neglects the $\hat{z}$ component of the drive (i.e., assuming
$\theta_{d}=0$, see Fig. S2(a) for schematic), as well as the counter-rotating
term of the transversal component. The resulting rotating frame Hamiltonian is
$\mathcal{H}_{I}=\frac{\Omega_{d}}{2}(\sin(\varphi_{d})\sigma_{x}-\cos(\varphi_{d})\sigma_{y})$.
In this regime, the only effect of the phase, $\varphi_{d}$, is to determine
the axis of Rabi nutation, and for a $\pi$ pulse, the effect vanishes.

In the strong-driving regime, we expect the dynamics to depend on
$\varphi_{d}$ beyond the trivial dependence of the weak-driving regime. This
is based on the observation that finite-duration sine waveforms contain a DC
component. Namely, the zero-frequency Fourier component,
$\frac{2\Omega_{d}}{\pi}\int_{0}^{\pi/\Omega_{d}}\sin\left(\omega_{0}t+\varphi_{d}\right)\mathrm{dt}$,
which depends on $\varphi_{d}$, is significant for $\Omega_{d}/\omega_{0}\sim
1$. Thus, in this regime, we anticipate that varying the phase modulates the
interplay between the different terms of the Hamiltonian, offering flexibility
for pulse fidelity optimization.

We suggest utilizing the phase $\varphi_{d}$ to mitigate the effects of the
counter-rotating term in the regime of $\Omega_{d}\lesssim\omega_{0}$ and
optimize pulse fidelity rates in this regime. This may be supplemented by a DC
offset to the drive, serving as another DC component that may be controlled to
optimize the pulse. We note that the phase of the driving field was shown to
be important in the strong driving regime in NMR already more than five
decades ago [34]. More recently, the phase’s effect was shown in single solid-
state qubit experiments [35, 36] and in NMR [37]. However, the phase has not
been discussed in the context of tilted drives or in combination with a DC
offset.

The first-order correction to the rotating frame Hamiltonian $\mathcal{H}_{I}$
is the Bloch-Siegert shift [38] that acts as an effective DC field along
$\hat{z}$ in the rotating frame [39]. Thus, the DC component of the
longitudinal driving field (present for tilted field $\theta_{d}>0$) may
assist in canceling out the effects of the counter-rotating term. Let us now
consider the special case of driving at an amplitude of
$\Omega_{d}=\frac{\omega_{0}}{2\tan\left(\theta_{d}\right)}$. In this case, a
constant (DC) waveform equal to $-1$ yields an ideal driving Hamiltonian
$\mathcal{H}=\Omega_{d}\sigma_{x}$. These observations motivate us to consider
waveforms $f(t)$ based on the “offset-sine” waveform:

$f(t)\equiv\epsilon\left(t\right)\left(a+\left(1-\left|a\right|\right)\sin\left(\omega_{0}t+\varphi_{d}\right)\right)$
(2)

where the optimization parameters are $\left|a\right|\leq 1$ (the DC offset
component) and $\varphi_{d}$ (the phase). For $a=0$, we obtain a standard sine
(symmetric around 0), while for $\left|a\right|=1$ we get a constant DC drive.

In Eq. 2 we introduced the pulse’s envelope function
$0\leq\epsilon\left(t\right)\leq 1$, which is zero at the pulse edges
($\epsilon\left(t_{0}\right)=\epsilon\left(t_{\mathrm{pulse}}\right)=0$). For
weak driving, a simple rectangle function is often used as the envelope (i.e.,
rectangular pulse shape). However, as realistic transmission lines always have
limited bandwidth, a discontinuous $\epsilon\left(t\right)$ will result in a
distorted signal, and this distortion is significant for strong and short
pulses. A smooth envelope function with finite rise and fall times can fit the
signal into a prescribed bandwidth [40], and here specifically, we used an
error-function pulse envelope [41] (see Eq. S.2 and Fig. S2(b) in the SI for a
schematic pulse).

###  V.2 Optimizing control pulses based on the offset-sine waveform

We demonstrate the performance of the offset-sine waveforms by numerically
calculating the state evolution of a qubit under such a drive Hamiltonian (Eq.
1 and Eq. 2). We consider as examples driving amplitudes of
$\Omega_{d}=\frac{\omega_{0}}{10},\frac{\omega_{0}}{3},\omega_{0}$. We focus
on $\pi$-pulses, i.e., flipping the initial state $\ket{\uparrow}$ with the
goal of maximizing the probability for $\ket{\downarrow}$.

As an illustration, we choose parameters inspired by the aforementioned spiral
antenna, namely, a drive tilt of $\theta_{d}=35.3^{\circ}$, and limit the
signals to a bandwidth of $\lesssim 10\omega_{0}$ using an error-function
envelope with a rise-time $\delta t=\frac{\pi}{10\omega_{0}}$. The pulse
durations are extrapolated from the weak driving regime and set to be
$t_{\pi}=\frac{\pi}{\Omega_{d}}+2\delta t$, which accounts for the rise and
fall times of the signal (for further details, see SI-3).

We numerically calculate the pulse fidelity according to
$\mathcal{F}=\left|\braket{\psi\left(t_{\pi}\right)}{\downarrow}\right|^{2}$
under the driving field for each driving amplitude, sampling various values of
the phase ($\varphi_{d}$) and offset ($a$) of the signal. The results are
presented in Fig. 5 (top row) in terms of infidelity $1-\mathcal{F}$ to
contrast the results. Fig. 5 (center row) shows the state evolution for
various driving signals at the driving amplitude of the corresponding column.
Evolutions are shown for various phases at zero-offset ($a=0$, light gray
curves), with the zero-offset phase yielding the best (worst) pulse fidelity
marked by dashed (dotted) curves. Evolutions under an optimal offset-sine
drive are marked by red curves. The optimal offset-sines have offset and phase
corresponding to the coordinates of minimum infidelity in the diagrams of the
top row, i.e., the brightest points. The bottom row shows the waveforms
corresponding to the different state evolutions in the center row.

Figure 5:  Optimizing offset sine $\pi$-pulse drives at different driving
strengths. Top row: Calculating pulse infidelity for several driving strengths
$\Omega_{d}$ as a function of pulse drive phase $\varphi_{d}$ and offset $a$.
Center row: spin state evolution for pulses at the corresponding drive
strengths. Trajectories for many values of $\varphi_{d}$ at $a=0$ are shown in
light curves. The best (worst) $\varphi_{d}$ values for $a=0$ are marked by
dashed (dotted) curves. Evolution for pulses with both optimal phase and
offset are shown by solid red curves. Insets focus on the pulses’ ends to
highlight the final fidelity for each case. Bottom row: the waveforms
corresponding to the trajectories drawn in the center row, with matching curve
format.

The center row of Fig. 5 illustrates how increasing the driving strength
$\Omega_{d}$ from $\frac{\omega_{0}}{10}$ to $\omega_{0}$ leads to increasing
deviation from the standard sinusoidal evolution characteristic of the RWA.
For the stronger drive amplitudes, adjusting the drive phase $\varphi_{d}$ is
crucial: for the extreme case of $\Omega_{d}=\omega_{0}$, a correct choice of
drive phase $\varphi_{d}$ will yield $\mathcal{F}\approx 0.94$, while the
worst choice will yield $\mathcal{F}\approx 0$. Additional optimization of the
DC offset significantly impacts the final state fidelity for the strongest
drive amplitudes. For example, at $\Omega_{d}\lesssim\omega_{0}$, adding a
proper offset will increase the fidelity to $\mathcal{F}>0.999$, beyond the
fault-tolerance threshold for some quantum computer architectures [42, 43].

The drive phase is a single optimization parameter to optimize strong drive
pulses and obtain pulse fidelities over 0.9, sufficient for quantum sensing
tasks. The optimal choice of phase would depend on the driving amplitude and
envelope function [39].

The DC offset we introduced as a novel optimization parameter may also be
significant, particularly for a driving field tilted by $\theta_{d}$ from
$\hat{z}$. The drive tilt even serves as an additional resource: a tilted
drive can achieve higher fidelities than driving fields purely along $\hat{x}$
when both the phase and offset are optimized (see Fig. S3(a) in the SI).

###  V.3 Comparing with optimal control theory signals

We compare our strategy with control signals generated by quantum optimal
control theory (OCT) [44, 45, 46, 47, 48]. In OCT, the optimization task is
formulated as a maximization problem of a functional object by means of
variational calculus. This yields a set of control equations, which are solved
numerically by optimization algorithms. The optimization target is the
maximization of the occupation of $\ket{\downarrow}$ at a predefined final
time, $t_{\pi}=\frac{\pi}{\Omega_{d}}+2\delta t$, as defined previously.

For our experiment, additional restrictions were added to the control problem:

  1. 1.

A restriction was imposed on the total energy of the drive that effectively
kept the peak amplitude near $\Omega_{d}$. However, we emphasize that in the
OCT solution, there was no explicit restriction on the amplitude as in the
offset-sine optimization. As a result, some OCT waveforms shown below exceed
an amplitude of $\Omega_{d}$.

  2. 2.

The spectral composition of the drive was restricted to $<10.7\omega_{0}$ to
produce an experimentally realistic waveform with a smooth temporal profile.
As mentioned before, this value is inspired by the spiral antenna.

  3. 3.

Homogeneous boundary conditions were imposed on the drive and its temporal
derivative (i.e., zero drive and zero time-derivative of the drive at $t=0$
and $t=t_{\pi}$). This was done in order to obtain a realistic, smooth rise
and fall of the drive.

The optimization problem was solved for six values of $\Omega_{d}$,
corresponding to different values of $t_{\pi}$. The drive tilt was set to
$\theta_{d}=35.3^{\circ}$. We compared the OCT signals with two forms of
offset-sine signals: an approximation to the OCT signal, obtained by a least-
square fitting of the OCT signal to Eq. 2; and the optimal offset-sine,
obtained by optimizing the offset and phase for the same parameters as the OCT
signals. The offset and phase optimization was done, as previously described,
by fixing $t_{\pi}$ and the amplitude $\Omega_{d}$, and sampling a range of
offsets and phases, choosing the offset and phase set that minimizes the
infidelity.

The signals for three cases are presented in Fig. 6(a)-(c). The fitted offset-
sine approximates the OCT-generated signals well, supporting the offset-sine
approach. Conversely, the optimal offset-sine is very similar when
$\Omega_{d}\ll\omega_{0}$, but takes on a distinct shape as $\Omega_{d}$
approaches $\omega_{0}$. The difference, however, does not come at the expense
of fidelity.

Fig. 6(d) compares the pulse fidelity rates for the OCT signals, the
approximated offset-sines, and the optimized offset-sine (for clarity, the
data is presented in terms of infidelity $1-\mathcal{F}$). The optimized
offset-sine signals differ from the OCT signals but provide comparable
fidelity rates, with $\mathcal{F}>0.999$. Interestingly, for the highest
values of $\Omega_{d}$ considered here (i.e., shortest $t_{\pi}$ times), the
fidelity rate of the optimized offset-sine even surpasses that of the OCT
signal. While the difference presumably stems from the details of the
optimization procedure, it underlines the potential of the optimized offset-
sine waveforms as an alternative optimization strategy.

Figure 6:  Comparing OCT drive signals with offset sine drives. (a)-(c) Drive
signals for different drive strengths $\Omega_{d}$ denoted above the plots,
corresponding to different pulse durations
$t_{\pi}=\frac{\pi}{\Omega_{d}}+2\delta t$. The blue dashed curves are OCT
waveforms. The dotted black curves are approximations of the OCT waveform by
an offset-sine obtained by least-square fitting (OCT fit). The solid red
curves are optimized offset-sine waveforms. (d) Infidelity rates of the
driving signals for each of the shown waveforms.

Our strategy for controlling qubits in the regime of
$\Omega_{d}\lesssim\omega_{0}$ thus relies on optimizing an offset-sine
driving signal as an alternative to existing approaches for designing strong
driving pulses, namely optimal control theory [20] and bang-bang control
sequences [17]. The optimized offset-sine approach does not require precise
driving field characterization and prior numerical optimization, and it suits
a tilted driving field. Although the fidelity rates of the OCT and optimized
offset-sine waveforms differ, the real rate would likely be lower due to
deviations between the simulated and actual drive parameters. This emphasizes
the benefit of a strategy that conveniently enables in situ optimization. The
offset-sine signal may be optimized experimentally by varying over one or two
parameters, namely the drive phase and DC offset. As such, this strategy is
convenient to minimize deviations between the real and simulated conditions,
for example, due to driving noise or a limited bandwidth [20].

##  VI Summary and conclusions

We developed a broadband spiral antenna tailored for quantum sensing
experiments with NV centers, such as nanoscale NMR. The antenna’s bandwidth
suits nuclear spins at fields up to $0.5~{}\mathrm{T}$. We drive 1H spins at a
Rabi frequency of over 500 kHz, faster than previously reported. The field-to-
current ratio of the spiral antenna is three-fold better than the state-of-
the-art, and the field-to-power ratio is over ten-fold better [13]. Thus,
owing to a low field-to-current ratio, it is possible to drive spins at
appreciable driving frequencies with low power consumption, e.g., a Rabi
frequency of over 100 kHz requires less than 2.5 W input power, making it
especially appropriate for sensitive samples or cryogenic environments.

Furthermore, we discussed the issue of driving spins in a strong driving
regime where $\Omega_{d}\lesssim\omega_{0}$. We show that spins may be flipped
with high fidelity by utilizing resonant offset-sine drive pulses optimized by
varying the drive field’s phase and offset. Our approach obtains fidelity
rates comparable to optimal control-derived signals and can be conveniently
optimized in situ, which is significant in experimental settings where the
driving field is noisy or not fully characterized. Also, offset-sine signals
are especially suitable for tilted driving fields. Pulse fidelities over 0.95
may be achieved by optimizing the drive phase, while varying the offset may
bring fidelity rates over 0.999, above the fault-tolerance threshold.

###### Acknowledgements.

We thank Nicolas Staudenmaier and Nabeel Aslam for delightful discussions on
correlation spectroscopy. We thank Yonatan Vernik and Leah Fuhrman Javitt for
their contributions. A.S. gratefully acknowledges the support of the Clore
Israel Foundation Scholars Programme, the Israeli Council for Higher
Education, and the Milner Foundation. I.S. acknowledges financial support by
the German Federal Ministry of Education and Research (BMBF), project no.
13N15929 QCStack. A.F. is the incumbent of the Elaine Blond Career Development
Chair in Perpetuity and acknowledges support from the Israel Science
Foundation (ISF grants 963/19 and 419/20) as well as the Abramson Family
Center for Young Scientists, the Willner Family Leadership Institute for the
Weizmann Institute of Science and the Helen and Martin Kimmel Institute for
Magnetic Resonance Research. We are grateful for the historic generosity of
the Harold Perlman Family. A.R. acknowledges the support of ERC grant QRES,
project number 770929, Quantera grant MfQDS, ISF and the Schwartzmann
university chair.

## References

  * Mamin  _et al._ [2013] H. J. Mamin, M. Kim, M. H. Sherwood, C. T. Rettner, K. Ohno, D. D. Awschalom, and D. Rugar, Nanoscale nuclear magnetic resonance with a nitrogen-vacancy spin sensor, Science 339, 557 (2013).
  * Loretz  _et al._ [2014] M. Loretz, S. Pezzagna, J. Meijer, and C. L. Degen, Nanoscale nuclear magnetic resonance with a 1.9-nm-deep nitrogen-vacancy sensor, Appl. Phys. Lett. 104, 033102 (2014).
  * Aslam  _et al._ [2017] N. Aslam, M. Pfender, P. Neumann, R. Reuter, A. Zappe, F. Fávaro de Oliveira, A. Denisenko, H. Sumiya, S. Onoda, J. Isoya, and J. Wrachtrup, Nanoscale nuclear magnetic resonance with chemical resolution, Science 357, 67 (2017).
  * Abobeih  _et al._ [2019] M. H. Abobeih, J. Randall, C. E. Bradley, H. P. Bartling, M. A. Bakker, M. J. Degen, M. Markham, D. J. Twitchen, and T. H. Taminiau, Atomic-scale imaging of a 27-nuclear-spin cluster using a quantum sensor, Nature 576, 411 (2019).
  * Smits  _et al._ [2019] J. Smits, J. T. Damron, P. Kehayias, A. F. McDowell, N. Mosavian, I. Fescenko, N. Ristoff, A. Laraoui, A. Jarmola, and V. M. Acosta, Two-dimensional nuclear magnetic resonance spectroscopy with a microfluidic diamond quantum sensor, Sci. Adv. 5, aaw7895 (2019).
  * Lovchinsky  _et al._ [2016] I. Lovchinsky, A. O. Sushkov, E. Urbach, N. P. de Leon, S. Choi, K. De Greve, R. Evans, R. Gertner, E. Bersin, C. Müller, L. McGuinness, F. Jelezko, R. L. Walsworth, H. Park, and M. D. Lukin, Nuclear magnetic resonance detection and spectroscopy of single proteins using quantum logic, Science 351, 836 (2016).
  * Rosskopf  _et al._ [2017] T. Rosskopf, J. Zopes, J. M. Boss, and C. L. Degen, A quantum spectrum analyzer enhanced by a nuclear spin memory, npj Quantum Inf. 3, 33 (2017).
  * Smeltzer  _et al._ [2009] B. Smeltzer, J. McIntyre, and L. Childress, Robust control of individual nuclear spins in diamond, Phys. Rev. A 80, 050302 (2009).
  * Degen  _et al._ [2017] C. L. Degen, F. Reinhard, and P. Cappellaro, Quantum sensing, Rev. Mod. Phys. 89, 035002 (2017).
  * Laraoui  _et al._ [2013] A. Laraoui, F. Dolde, C. Burk, F. Reinhard, J. Wrachtrup, and C. A. Meriles, High-resolution correlation spectroscopy of 13c spins near a nitrogen-vacancy centre in diamond, Nat. Commun. 4, 1651 (2013).
  * Munuera-Javaloy  _et al._ [2023] C. Munuera-Javaloy, A. Tobalina, and J. Casanova, High-resolution NMR spectroscopy at large fields with nitrogen vacancy centers, Phys. Rev. Lett. 130, 133603 (2023).
  * Fuchs  _et al._ [2009] G. D. Fuchs, V. V. Dobrovitski, D. M. Toyli, F. J. Heremans, and D. D. Awschalom, Gigahertz dynamics of a strongly driven single quantum spin, Science 326, 1520 (2009).
  * Herb  _et al._ [2020] K. Herb, J. Zopes, K. S. Cujia, and C. L. Degen, Broadband radio-frequency transmitter for fast nuclear spin control, Rev. Sci. Instrum. 91, 113106 (2020).
  * Song  _et al._ [2016] Y. Song, J. P. Kestner, X. Wang, and S. Das Sarma, Fast control of semiconductor qubits beyond the rotating-wave approximation, Phys. Rev. A 94, 012321 (2016).
  * Yang  _et al._ [2017] Y. C. Yang, S. N. Coppersmith, and M. Friesen, Achieving high-fidelity single-qubit gates in a strongly driven silicon-quantum-dot hybrid qubit, Phys. Rev. A 95, 062321 (2017).
  * Khaneja  _et al._ [2001] N. Khaneja, R. Brockett, and S. J. Glaser, Time optimal control in spin systems, Phys. Rev. A 63, 032308 (2001).
  * Boscain and Mason [2006] U. Boscain and P. Mason, Time minimal trajectories for a spin 1/2 particle in a magnetic field, J. Math. Phys. 47, 062101 (2006).
  * Wang and Dobrovitski [2011] Z. H. Wang and V. V. Dobrovitski, Time-optimal rotation of a spin 1/2: Application to the NV center spin in diamond, Phys. Rev. B 84, 045303 (2011).
  * Avinadav  _et al._ [2014] C. Avinadav, R. Fischer, P. London, and D. Gershoni, Time-optimal universal control of two-level systems under strong driving, Phys. Rev. B 89, 245311 (2014).
  * Scheuer  _et al._ [2014] J. Scheuer, X. Kong, R. S. Said, J. Chen, A. Kurz, L. Marseglia, J. Du, P. R. Hemmer, S. Montangero, T. Calarco, B. Naydenov, and F. Jelezko, Precise qubit control beyond the rotating wave approximation, New J. Phys. 16, 093022 (2014).
  * Degen  _et al._ [2009] C. L. Degen, M. Poggio, H. J. Mamin, C. T. Rettner, and D. Rugar, Nanoscale magnetic resonance imaging, Proc. Natl. Acad. Sci. 106, 1313 (2009).
  * Grinolds  _et al._ [2014] M. S. Grinolds, M. Warner, K. De Greve, Y. Dovzhenko, L. Thiel, R. L. Walsworth, S. Hong, P. Maletinsky, and A. Yacoby, Subnanometre resolution in three-dimensional magnetic resonance imaging of individual dark spins, Nat. Nanotechnol. 9, 279 (2014).
  * Neumann  _et al._ [2010] P. Neumann, J. Beck, M. Steiner, F. Rempp, H. Fedder, P. R. Hemmer, J. Wrachtrup, and F. Jelezko, Single-shot readout of a single nuclear spin, Science 329, 542 (2010).
  * Staudacher  _et al._ [2013] T. Staudacher, F. Shi, S. Pezzagna, J. Meijer, J. Du, C. A. Meriles, F. Reinhard, and J. Wrachtrup, Nuclear magnetic resonance spectroscopy on a (5-nanometer)3 sample volume, Science 339, 561 (2013).
  * Wang  _et al._ [2019] Z.-Y. Wang, J. E. Lang, S. Schmitt, J. Lang, J. Casanova, L. McGuinness, T. S. Monteiro, F. Jelezko, and M. B. Plenio, Randomization of pulse phases for unambiguous and robust quantum sensing, Phys. Rev. Lett. 122, 200403 (2019).
  * Pfender  _et al._ [2017] M. Pfender, N. Aslam, H. Sumiya, S. Onoda, P. Neumann, J. Isoya, C. A. Meriles, and J. Wrachtrup, Nonvolatile nuclear spin memory enables sensor-unlimited nanoscale spectroscopy of small spin clusters, Nat. Commun. 8, 834 (2017).
  * Laraoui  _et al._ [2015] A. Laraoui, D. Pagliero, and C. A. Meriles, Imaging nuclear spins weakly coupled to a probe paramagnetic center, Phys. Rev. B 91, 205410 (2015).
  * Staudacher  _et al._ [2015] T. Staudacher, N. Raatz, S. Pezzagna, J. Meijer, F. Reinhard, C. A. Meriles, and J. Wrachtrup, Probing molecular dynamics at the nanoscale via an individual paramagnetic centre, Nat. Commun. 6, 8527 (2015).
  * Nguyen  _et al._ [2019] C. T. Nguyen, D. D. Sukachev, M. K. Bhaskar, B. Machielse, D. S. Levonian, E. N. Knall, P. Stroganov, C. Chia, M. J. Burek, R. Riedinger, H. Park, M. Lončar, and M. D. Lukin, An integrated nanophotonic quantum register based on silicon-vacancy spins in diamond, Phys. Rev. B 100, 165428 (2019).
  * Miao  _et al._ [2020] K. C. Miao, J. P. Blanton, C. P. Anderson, A. Bourassa, A. L. Crook, G. Wolfowicz, H. Abe, T. Ohshima, and D. D. Awschalom, Universal coherence protection in a solid-state spin qubit, Science 369, 1493 (2020).
  * Stern  _et al._ [2022] H. L. Stern, Q. Gu, J. Jarman, S. Eizagirre Barker, N. Mendelson, D. Chugh, S. Schott, H. H. Tan, H. Sirringhaus, I. Aharonovich, and M. Atatüre, Room-temperature optically detected magnetic resonance of single defects in hexagonal boron nitride, Nat. Commun. 13, 618 (2022).
  * Yan  _et al._ [2013] F. Yan, S. Gustavsson, J. Bylander, X. Jin, F. Yoshihara, D. G. Cory, Y. Nakamura, T. P. Orlando, and W. D. Oliver, Rotating-frame relaxation as a noise spectrum analyser of a superconducting qubit undergoing driven evolution, Nat. Commun. 4, 2337 (2013).
  * Slichter [1990] C. P. Slichter,  _Principles of Magnetic Resonance_, Vol. 1 (Springer Berlin Heidelberg, 1990).
  * MacLaughlin [1970] D. E. MacLaughlin, Suppression of nutation angle variation in pulsed nuclear magnetic resonance, Rev. Sci. Instrum. 41, 1202 (1970).
  * London  _et al._ [2014] P. London, P. Balasubramanian, B. Naydenov, L. P. McGuinness, and F. Jelezko, Strong driving of a single spin using arbitrarily polarized fields, Phys. Rev. A 90, 012302 (2014).
  * Rao and Suter [2017] K. R. K. Rao and D. Suter, Nonlinear dynamics of a two-level system of a single spin driven beyond the rotating-wave approximation, Phys. Rev. A 95, 053804 (2017).
  * Bidinosti  _et al._ [2022] C. P. Bidinosti, G. Tastevin, and P. J. Nacher, Generating accurate tip angles for NMR outside the rotating-wave approximation, J. Magn. Reson. 345, 107306 (2022).
  * Bloch and Siegert [1940] F. Bloch and A. Siegert, Magnetic resonance for nonrotating fields, Phys. Rev. 57, 522 (1940).
  * Zeuch  _et al._ [2020] D. Zeuch, F. Hassler, J. J. Slim, and D. P. DiVincenzo, Exact rotating wave approximation, Ann. Phys. 423, 168327 (2020).
  * Vandersypen and Chuang [2005] L. M. K. Vandersypen and I. L. Chuang, NMR techniques for quantum control and computation, Rev. Mod. Phys. 76, 1037 (2005).
  * Bradley  _et al._ [2019] C. E. Bradley, J. Randall, M. H. Abobeih, R. C. Berrevoets, M. J. Degen, M. A. Bakker, M. Markham, D. J. Twitchen, and T. H. Taminiau, A ten-qubit solid-state spin register with quantum memory up to one minute, Phys. Rev. X 9, 031045 (2019).
  * Fowler  _et al._ [2012] A. G. Fowler, M. Mariantoni, J. M. Martinis, and A. N. Cleland, Surface codes: Towards practical large-scale quantum computation, Phys. Rev. A 86, 032324 (2012).
  * Barends  _et al._ [2014] R. Barends, J. Kelly, A. Megrant, A. Veitia, D. Sank, E. Jeffrey, T. C. White, J. Mutus, A. G. Fowler, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, C. Neill, P. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, A. N. Korotkov, A. N. Cleland, and J. M. Martinis, Superconducting quantum circuits at the surface code threshold for fault tolerance, Nature 508, 500 (2014).
  * A. P. Peirce [1988] H. R. A. P. Peirce, M. A. Dahleh, Optimal control of quantum-mechanical systems: Existence, numerical approximation, and application, Phys. Rev. A 37, 4950 (1988).
  * Kosloff  _et al._ [1989] R. Kosloff, S. A. Rice, P. Gaspard, S. Tersigni, and D. Tannor, Wavepacket dancing: Achieving chemical selectivity by shaping light pulses, Chem. Phys. 139, 201 (1989).
  * Palao and Kosloff [2003] J. P. Palao and R. Kosloff, Optimal control theory for unitary transformations, Phys. Rev. A 68, 062308 (2003).
  * Werschnik and Gross [2007] J. Werschnik and E. K. U. Gross, Quantum optimal control theory, J. Phys. B: At. Mol. Opt. Phys. 40, R175 (2007).
  * Schaefer [2012] I. Schaefer,  _Quantum Optimal Control Theory of Harmonic Generation_ , Master’s thesis, The Hebrew University of Jerusalem (2012).