11institutetext: University of Lincoln, Lincoln, UK
11email: {emwilliams<EMAIL_ADDRESS>
# Pretrained Visual Representations in Reinforcement Learning

Emlyn Williams  Athanasios Polydoros

###### Abstract

Visual reinforcement learning (RL) has made significant pro-
gress in recent years, but the choice of visual feature extractor remains a
crucial design decision. This paper compares the performance of RL algorithms
that train a convolutional neural network (CNN) from scratch with those that
utilize pre-trained visual representations (PVRs). We evaluate the Dormant
Ratio Minimization (DRM) algorithm, a state-of-the-art visual RL method,
against three PVRs: ResNet18, DINOv2, and Visual Cortex (VC). We use the
Metaworld Push-v2 and Drawer-Open-v2 tasks for our comparison. Our results
show that the choice of training from scratch compared to using PVRs for
maximising performance is task-dependent, but PVRs offer advantages in terms
of reduced replay buffer size and faster training times. We also identify a
strong correlation between the dormant ratio and model performance,
highlighting the importance of exploration in visual RL. Our study provides
insights into the trade-offs between training from scratch and using PVRs,
informing the design of future visual RL algorithms.

###### Keywords:

Reinforcement Learning Computer Vision Robotics.

##  1 Introduction

Reinforcement learning (RL) has made tremendous progress in recent years, with
the development of algorithms that enable robots to learn complex behaviors
from high-dimensional sensory inputs. In visual RL, where robots learn from
raw image observations, a key challenge lies in extracting relevant features
from the visual data. Despite significant advances in computer vision over the
last decade, current state of the art models in visual RL still use simple
convolutional neural networks (CNNs) trained from scratch for different tasks.
This approach has been successfully employed in algorithms such as DRQ-V2 [14]
and DRM [13], which have solved some of the most challenging continuous
control tasks in common RL benchmarks such as the humanoid run and dog tasks
in the deepmind control suite [13].

Notwithstanding these successes, training a CNN from scratch does not leverage
the knowledge captured by pre-trained models on large datasets. Recently,
there has been growing interest in using pre-trained image encoders as feature
extractors for Imitation Learning (IL) and RL [1][5]. These pre-trained models
have been trained on massive datasets and have learned to extract rich and
generalizable features that can be useful for a wide range of downstream tasks
[16][1][9]. By leveraging pre-trained models, RL algorithms may learn more
efficiently.

Applying RL to robotics poses unique challenges however, particularly in terms
of sample efficiency and control frequency. Specifically, data collection is
often time-consuming and expensive as it requires physical interactions with
the environment. Moreover, robots typically operate at high frequencies,
requiring control signals to be generated at rates of 10-100 Hz or more. When
training a real robot from scratch, this demands RL algorithms that can learn
quickly and make decisions rapidly.

Despite the potential benefits of using pre-trained image encoders in RL,
there are limited comparisons of the two methods in surrounding literature. In
their investigation of an "artificial visual cortex" for embodied AI, Majumdar
et al. [8] compare pre-trained visual representations (PVRs) across multiple
robotic manipulation and navigation tasks, but do not compare these models to
a learning-from-scratch RL implementation. In their comparisons, they find
that no single PVR is universally dominant across multiple domains. Parisi et
al. [10] compare learning from scratch to using PVRs, however their study only
considers learning from demonstrations using imitation learning.

In this paper, we compare visual RL algorithms that train a CNN from scratch
with counterparts that utilize frozen PVRs. We investigate the performance of
these algorithms on a simulated robotic pushing task, and analyze the trade-
offs between training a CNN from scratch and using pre-trained feature
extractors.

##  2 Related work

###  2.1 Visual Reinforcement Learning

In domains such as robotics, choosing the right RL method to maximize learning
efficiency is important for minimising the risk of robot degradation. In this
context, off-policy methods such as SAC [4] and DDPG [7] are generally more
suitable than on-policy methods because off-policy methods can learn from
experiences gathered without following the same policy that is being learned.
This allows them to leverage data collected from various sources such as
demonstrations, simulations, or other robots. In contrast, on-policy methods
such as policy gradient methods require data to be collected using the same
policy that is being learned. This can be limiting in applications where data
acquisition is expensive.

More recent advances in off-policy visual RL, such as the Data-Regularized Q
(DRQ) [6] algorithm and its successor, DRQ-V2 [14], have improved the
efficiency and effectiveness of off-policy learning in visual RL tasks by
using random shift augmentations for regularisation. These methods have been
shown to achieve state-of-the-art performance in a variety of simulated
robotics tasks such as grasping and manipulation, from a relatively small
amount of data.

In this paper, we use the Dormant Ratio Minimization (DRM) [13] visual RL
algorithm as the baseline. DRM was chosen because it is the most sample
efficient RL algorithm for visual control and has exhibited very good
performance on continuous control tasks [13]. DRM builds from the observation
that there is a connection between the reduction of an agent’s dormant ratio
and the agent’s skill acquisition in visual control tasks [13]. A dormant
neuron is a neuron that has become nearly inactive, displaying a minimal
activation level relative to other neurons in a layer. The dormant ratio is
the percentage of these dormant neurons in the entire network [13]. DRM uses
the dormant ratio to guide exploration, and periodically perturbs the agent’s
weights when the dormant ratio is high.

###  2.2 Reinforcement Learning with Pre-trained Visual Representations

Yuan et al. [16] adapt the DRQ-v2 algorithm [14] to use an ImageNet pretrained
ResNet18 as a PVR. The authors find that using features from early layers
provides better performance for continuous control tasks when compared to use
of the full network. Their approach outperforms DRQ-v2 on Deepmind control
suite tasks with video backgrounds.

Xiao et al. [12] demonstrate that self-supervised pre-training from real world
images is effective for learning motor control from pixels. They train a
vision transformer (ViT) model by masked modelling of natural images. This
visual encoder is then frozen and the RL policy is trained using the frozen
encoder. Another finding of the paper is that "in the wild" images, such as
those from YouTube or egocentric videos, lead to better visual representations
for manipulation tasks when compared to ImageNet images.

###  2.3 Learning from Video Demonstrations

In recent years, there has been a growing interest in using RL as a method of
learning from video demonstrations (LfVD) [11][17]. LfVD involves learning a
policy from a set of video demonstrations, often collected from humans or
other agents. This is motivated by the fact that learning policies from expert
videos is a scalable way of learning a wide array of tasks without access to
ground truth rewards or actions [3]. By leveraging the knowledge and expertise
encoded in these demonstrations, LfVD can significantly reduce the amount of
data required to learn a task [11][17]. However, LfVD also relies on the
ability to extract relevant features from the video data. Sharing a common PVR
for both LfVD and RL could enable more efficient learning.

Zakka et al. [17] use an ImageNet pretrained ResNet18 with a self-supervised
Temporal Cycle Consistency loss to learn reward functions from video
demonstrations. They demonstrate learning robot policies from videos of human
demonstrations on the Metaworld Push environment. Whilst they use a PVR for
learning the reward, they then use state based RL for learning from the
videos. Finding a suitable PVR for visual RL could allow both the visual state
observation and reward to be given by the PVR when using reward functions
learned from videos.

##  3 Methodology

###  3.1 Benchmarking

We use the Push-v2 task from the Metaworld benchmarking suite for our
evaluation. Metaworld consists of 50 robotic manipulation tasks such as
opening and closing doors and drawers and pressing buttons [15]. The chosen
task consists of pushing a small red puck towards a green sphere, with the
initial puck and sphere positions randomised at the beginning of each episode.
An episode is considered a success when the puck is within 5cm of the target
at the final timestep. The reward function rewards minimizing the distance
from the gripper to the puck, from the puck to the target site, and gives a
bonus when the object is grasped by the robot. A maximum reward of 20 is given
when the object is within 5cm of the target site.

From the Metaworld suite, this task was chosen as it involves manipulating a
small object which could be contained entirely within one patch of a ViT. It
was also selected due to it being one of the hardest tasks to solve in the
Metaworld suite according to the original paper [15]. One explanation for the
task’s difficulty could be that the puck can fall on its side and roll away
from the target position if not carefully placed, resulting in a failed
episode even if the object temporarily reaches the target zone.

For further evaluation we use the Drawer-Open-v2 task. The task consists of
opening a drawer, with the drawer position randomised at the beginning of each
episode. The reward function rewards minimizing the distance of the gripper to
the handle, and the distance of the handle to a target position. An episode is
considered a success when the handle is less than 3cm from a target position
and the gripper is less than 3cm from the handle. This task was chosen due to
it involving a larger and more natural looking object.

(a) Push-v2

(b) Drawer-Open-v2

Figure 1: Metaworld Tasks

All experiments use an image resolution of 112x112 and maintain a consistent
third-person camera position unless otherwise noted. The default DRM
hyperparameters [13] are used for all experiments.

###  3.2 Pretrained Visual Representations

We compare the baseline DRM algorithm to three PVRs: ResNet18, DINOv2, and
Visual Cortex (VC). The PVR’s weights are frozen throughout the RL training.

####  3.2.1 ResNet18

An ImageNet pretrained ResNet18 was chosen due it being a widely used and well
established CNN that has demonstrated its efficacy in a variety of computer
vision tasks such as classification, object detection and segmentation. Other
reasons for its use are its relatively small size and its high performance in
visual RL, shown in "Pre-Trained Image Encoder for Generalizable Visual
Reinforcement Learning" (PIEG) [16].

We adopt a similar approach to PIEG, where the images are fed through the
first two layers of a ResNet18 CNN. To mitigate the high dimensionality of the
output, we flatten the CNN output and pass it through a trainable fully
connected layer with a reduced output size. Consistent with DRM’s CNN encoder,
we omit pooling layers after the CNN output to preserve spatial information.
However, this design choice comes with a limitation: higher resolutions would
result in an explosion of parameters in the fully connected layer, making it
computationally prohibitive. The output of this layer is then fed into the
actor and critic networks of DrM.

####  3.2.2 DINOv2

DINOv2 is a self-supervised vision backbone based on a ViT model. It is
trained using a student teacher approach at both a whole image level and at a
patch level. A 1 billion parameter model is trained on a curated dataset of
142 million images. This large model is then distilled down into smaller
models. The backbone gives state of the art results for self-supervised
networks in downstream tasks such as image classification, semantic
segmentation and depth estimation. In this paper we utilise the 86 million
parameter ViT-B model. DINOv2 was chosen as a PVR for comparison as it is one
of the best performing self-supervised ViT based models for downstream tasks
noted above.

Darcet et al. [1] find that artefacts in transformer feature maps correspond
to high norm patch tokens appearing in low information background areas of
images. The authors hypothesize that large, sufficiently trained models
recognize these redundant tokens and use them as places to store, process and
retrieve global information. By adding additional tokens (register tokens) to
the transformer input to store and process global information, these high norm
patch patch tokens disappear.

To use DINOv2 as a PVR, we pass the image through the model and feed the CLS
token directly into the actor and critic networks. We also compare this to
concatenating the register tokens to the CLS token and passing these
concatenated tokens into the actor and critic networks.

####  3.2.3 Visual Cortex

Visual cortex (VC) [8] is a ViT based PVR model trained using Masked Auto-
Encoding (MAE) on egocentric data, with the aim of being used for embodied
intelligence. VC was chosen for this comparison because it was designed for
use in embodied AI. It was also found to be the best performing PVR on average
in comparisons performed in [8]. As with DINOv2, we feed the CLS token
directly into the actor and critic networks.

###  3.3 Visual RL Algorithm

As we focus on the performance of the vision systems, we do not include robot
proprioception information in the state space. To compare between the baseline
and the pretrained backbones, the CNN based encoder is replaced with the
pretrained backbone.

DRM stores observations as images in the replay buffer. At each update step a
batch of these observations is augmented and passed through the image encoder,
actor and critic networks. When using the ViT based PVRs, the image
observations are passed through the frozen vision backbone at each environment
step. The CLS token, which is a learnable vector that serves as a
representation of the entire image [2], is given by the backbone and is stored
in the replay buffer. This reduces the memory footprint of the replay buffer
by over 90%, since the CLS token is a 768 dimensional float 32 vector,
compared to the 3x112x112 8 bit integer image. Passing one image through the
backbone at each time step also avoids the need to pass mini batches of 256
through the backbone at each update step.

Another advantage of storing the CLS tokens in the replay buffer is that the
size of the replay buffer does not increase with higher resolutions, since
only the CLS token is stored. The original DRM [13] paper and PIEG [16] use
image observations of size 84x84 with a replay buffer size of 1M.
Uncompressed, this results in a replay buffer size of 21.17 GB. At a
resolution of 112x112, this increases to 37.63GB, and at 224x224 (the
resolution commonly used by ViT models), this increases to 150.53GB. These
large replay buffers pose challenges to the training of CNNs from scratch for
visual RL.

Since no pooling layers are used in DRM’s image encoder, the number of
parameters in the actor and critic network increases with the input
resolution. This limits the maximum resolution that can be used while still
being able to run at a frequency suitable for robotic control. Table 1 shows
how the number of parameters differs at different resolutions.

Table 1: Trainable Parameters Model | 112x112 | 224x224
---|---|---
DRM | 15978281 | 57373481
PIEG | 107340233 | 415621577
DINOv2 | 4768713 | 4768713
DINOV2 REG | 6151113 | 6151113
VC | 4768713 | 4768713

###  3.4 Performance Metrics

We present graphs of the episode reward and success rate of the evaluation
episodes. The episode reward is the total reward obtained by the agent for an
entire episode, and the success rate is the proportion of evaluation episodes
where the agent successfully completed the task. All results are plotted as
the mean performance over 5 seeds with a standard deviation shading of $\pm$
0.5.

##  4 Experiments and Results

###  4.1 Data Augmentation

DRM applies a padding and random shift augmentation to all images before they
are passed through the encoder to regularize the input data [14][13]. We first
test whether this augmentation is beneficial when using PVRs which have been
trained on large datasets with augmentations [9]. We compare the results of
the DINOv2 CLS token on the Push-v2 task with and without the random shift
augmentation.

(a) Rewards

(b) Success Rate

Figure 2: Comparison of using plain images to using random shift
augmentations.

As shown in figure 2, the DINOv2 PVR performed better when the random shift
augmentations used in the baseline DRM algorithm were not applied. Therefore,
we do not apply these augmentations to images passed into the ViT based PVRs
that are included in this comparative study.

###  4.2 PVR Performance

(a) Rewards

(b) Success Rate

(c) Dormant Ratio

Figure 3: Mean performance on Push-v2 Metaworld task.

(a) Rewards

(b) Success Rate

(c) Dormant Ratio

Figure 4: Mean performance on Push-v2 Metaworld task with increased
resolutions for ViT PVRs.

(a) Rewards

(b) Success Rate

(c) Actor Dormant Ratio

Figure 5: Per Seed Performance of DINOv2 CLS token at 224x224 resolution

Figure 3 presents the performance of the different models on the Push-v2 task.
The baseline DRM algorithm achieves the best performance, closely followed by
the PIEG-based ResNet model. In contrast, the ViT-based models exhibit
significantly poorer performance, highlighting a substantial gap between CNN-
based and ViT-based models.

Figure 3 also reveals a strong correlation between the dormant ratio and model
performance. The top-performing models, including the DRM and PIEG-based
ResNet, demonstrate a drastic reduction in the dormant ratio throughout
training. In contrast, the poorly performing models, including the ViT-based
models, maintain a high dormant ratio throughout training.

To explore the potential benefits of using ViT-based models at higher
resolutions, we additionally tested these models at a resolution of 224x224.
Whilst the CNN-based models are unable to operate at this resolution due to
the computational constraints of the large replay buffer size, the ViT-based
models can process images at this resolution with only a marginal decrease in
control frequency.

As shown in figure 4, the performance of the ViT models improves slightly at
the higher resolution, with the DINOv2 CLS token achieving the best
performance. However, even at this higher resolution, the ViT models are still
significantly outperformed by the CNN-based models at the lower resolution.

Figure 5 illustrates the high variance in performance across different seeds
for the DINOv2 CLS model at the 224x224 resolution. Notably, the worst
performing seeds are those that fail to reduce their dormant ratio throughout
training. This trend is consistent across the ViT-based models, with the VC
model exhibiting only one convergent seed out of five, and the DINOv2 with
register tokens showing three convergent seeds out of five.

To investigate whether these results are task specific, we compare the ViT
models at 224x224 to the CNN models at 112x112 on a drawer opening task. The
results of this experiment are shown in figure 6. We find that the DINOv2 PVR
outperforms both DRM and PIEG. This could be explained by the drawer providing
more visual features due to its size and shape when compared to the rendered
cylinder and sphere in the pushing task. The large difference between the
rewards and success rate are due to some seeds ending stuck in a local minima
of grasping the handle without opening the door, indicating the need for
further exploration.

The videos of the evaluation episodes show that the robot sometimes learned
different strategies to complete the task. Some agents would try to hit the
puck towards the target with the outside of the gripper, whilst others would
use the gripper to grasp the puck and place it near the target. These
strategies are shown in figure 7.

(a) Rewards

(b) Success Rate

Figure 6: Mean performance on Drawer-Open-v2 Metaworld task with increased
resolutions for ViT PVRs.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 7: Examples of different learned approaches. Top: robot hits puck
towards target. Bottom: Robot grabs object and moves it to target.

##  5 Discussion

Our comparison reveals that the choice of training from scratch or using a PVR
for maximising performance in visual RL is task dependant. With the DRM
algorithm performing best on the pushing task and the DINOv2 PVR performing
best on the drawer opening task, we hypothesize that PVRs perform better when
the objects being manipulated are larger and more distinguishable. We also
identify several advantages of using PVRs, including a significant reduction
in replay buffer size and operating at a higher control frequency.

We find that on the push-v2 task the PIEG PVR outperforms other PVRs, and the
DINOv2 CLS token is the top-performing ViT-based PVR. Interestingly, our
results indicate that the DINOv2 ViT outperforms the Visual Cortex (VC) model,
which was specifically designed for embodied AI. Whilst Darcet et al. [1]
demonstrate the effectiveness of register tokens in training ViTs to avoid
storing image-level information in patch tokens, we find that including these
register tokens in the state space does not improve visual RL performance when
compared to sole use of the CLS token.

Furthermore, our analysis highlights the importance of the dormant ratio in
visual RL. We found that a lower dormant ratio is a strong indicator of
successful learning, and the best-performing models demonstrate a drastic
reduction in the dormant ratio throughout training. In contrast, poorly
performing models maintained a high dormant ratio, suggesting that the robot
was unable to effectively explore and learn from the environment. The PVR
based approaches resulted in a higher number of failed seeds where the dormant
ratio did not decrease throughout training. This finding has significant
implications for incorporating PVRs into visual RL algorithms, as it suggests
that incorporating alternative mechanisms to reduce the dormant ratio may be
essential for achieving successful learning outcomes.

Using PVRs also reduces the number of trainable parameters, which can lead to
faster training times and lower computational costs. This is crucial for tasks
such as robotic manipulation and navigation in the real world, where data
collection is costly and time-consuming. This trade-off between performance
and complexity is important in visual RL, where large models can be
prohibitive for real time control. While PVRs may not yet always match the
performance of training from scratch, they offer a promising approach for
balancing performance and complexity.

##  6 Conclusion

Our study provides a comprehensive analysis of the effectiveness of pre-
trained PVRs in visual RL. Our results demonstrate that the optimal
performance of PVRs or from scratch learning is task-dependent and highlight
the importance of evaluating visual RL algorithms across multiple tasks and
domains. We identify several advantages of using PVRs, including reduced
replay buffer size and computational costs. We also find that the general
purpose DINOv2 PVR outperforms the Visual Cortex PVR that was specifically
designed for use in embodied AI. Our findings suggest that mechanisms to
reduce the dormant ratio may be essential to achieve successful learning
outcomes when incorporating PVRs into visual RL algorithms.

Future work will explore alternative RL algorithms, regularization approaches
for PVRs, analysis across further tasks, and fine-tuning PVRs during training
to provide a more comprehensive understanding of the strengths and limitations
of PVRs in visual RL.

####  6.0.1 Acknowledgements

This work was supported by the Engineering and Physical Sciences Research
Council [EP/S023917/1] and Dogtooth Robotics.

## References

  * [1] Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: The Twelfth International Conference on Learning Representations (2023)
  * [2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [3] Escontrela, A., Adeniji, A., Yan, W., Jain, A., Peng, X.B., Goldberg, K., Lee, Y., Hafner, D., Abbeel, P.: Video prediction models as rewards for reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)
  * [4] Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.: Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018)
  * [5] Karamcheti, S., Nair, S., Chen, A.S., Kollar, T., Finn, C., Sadigh, D., Liang, P.: Language-Driven Representation Learning for Robotics. In: Proceedings of Robotics: Science and Systems. Daegu, Republic of Korea (July 2023). https://doi.org/10.15607/RSS.2023.XIX.032
  * [6] Kostrikov, I., Yarats, D., Fergus, R.: Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 (2020)
  * [7] Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
  * [8] Majumdar, A., Yadav, K., Arnaud, S., Ma, J., Chen, C., Silwal, S., Jain, A., Berges, V.P., Wu, T., Vakil, J., et al.: Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems 36 (2024)
  * [9] Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (2023)
  * [10] Parisi, S., Rajeswaran, A., Purushwalkam, S., Gupta, A.: The unsurprising effectiveness of pre-trained vision models for control. In: international conference on machine learning. pp. 17359–17371. PMLR (2022)
  * [11] Schmeckpeper, K., Rybkin, O., Daniilidis, K., Levine, S., Finn, C.: Reinforcement learning with videos: Combining offline observations with interaction. In: Conference on Robot Learning. pp. 339–354. PMLR (2021)
  * [12] Xiao, T., Radosavovic, I., Darrell, T., Malik, J.: Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173 (2022)
  * [13] Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al.: Drm: Mastering visual reinforcement learning through dormant ratio minimization. In: The Twelfth International Conference on Learning Representations (2023)
  * [14] Yarats, D., Fergus, R., Lazaric, A., Pinto, L.: Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 (2021)
  * [15] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., Levine, S.: Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In: Conference on robot learning. pp. 1094–1100. PMLR (2020)
  * [16] Yuan, Z., Xue, Z., Yuan, B., Wang, X., Wu, Y., Gao, Y., Xu, H.: Pre-trained image encoder for generalizable visual reinforcement learning. Advances in Neural Information Processing Systems 35, 13022–13037 (2022)
  * [17] Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg, J., Dwibedi, D.: Xirl: Cross-embodiment inverse reinforcement learning. In: Conference on Robot Learning. pp. 537–546. PMLR (2022)


# Photonic Millimeter-wave Generation Beyond the Cavity Thermal Limit

William Groman  Electrical Computer & Energy Engineering, University of
Colorado, Boulder, CO 80309, USA  Department of Physics, University of
Colorado Boulder, 440 UCB Boulder, CO 80309, USA  National Institute of
Standards and Technology, 325 Broadway, Boulder, CO 80305, USA  Igor Kudelin
Electrical Computer & Energy Engineering, University of Colorado, Boulder, CO
80309, USA  Department of Physics, University of Colorado Boulder, 440 UCB
Boulder, CO 80309, USA  National Institute of Standards and Technology, 325
Broadway, Boulder, CO 80305, USA  Alexander Lind  Electrical Computer & Energy
Engineering, University of Colorado, Boulder, CO 80309, USA  National
Institute of Standards and Technology, 325 Broadway, Boulder, CO 80305, USA
Dahyeon Lee  Department of Physics, University of Colorado Boulder, 440 UCB
Boulder, CO 80309, USA  National Institute of Standards and Technology, 325
Broadway, Boulder, CO 80305, USA  Takuma Nakamura  National Institute of
Standards and Technology, 325 Broadway, Boulder, CO 80305, USA  Yifan Liu
Department of Physics, University of Colorado Boulder, 440 UCB Boulder, CO
80309, USA  National Institute of Standards and Technology, 325 Broadway,
Boulder, CO 80305, USA  Megan L. Kelleher  Department of Physics, University
of Colorado Boulder, 440 UCB Boulder, CO 80309, USA  National Institute of
Standards and Technology, 325 Broadway, Boulder, CO 80305, USA  Charles A.
McLemore  Department of Physics, University of Colorado Boulder, 440 UCB
Boulder, CO 80309, USA  National Institute of Standards and Technology, 325
Broadway, Boulder, CO 80305, USA  Joel Guo  Department of Electrical and
Computer Engineering, University of California, Santa Barbara, Santa Barbara,
CA 93106, USA  Lue Wu  Department of Electrical and Computer Engineering,
University of California, Santa Barbara, Santa Barbara, CA 93106, USA  Warren
Jin  Department of Electrical and Computer Engineering, University of
California, Santa Barbara, Santa Barbara, CA 93106, USA  John E. Bowers
Department of Electrical and Computer Engineering, University of California,
Santa Barbara, Santa Barbara, CA 93106, USA  Franklyn Quinlan  Electrical
Computer & Energy Engineering, University of Colorado, Boulder, CO 80309, USA
National Institute of Standards and Technology, 325 Broadway, Boulder, CO
80305, USA  Scott A. Diddams  Electrical Computer & Energy Engineering,
University of Colorado, Boulder, CO 80309, USA  Department of Physics,
University of Colorado Boulder, 440 UCB Boulder, CO 80309, USA  National
Institute of Standards and Technology, 325 Broadway, Boulder, CO 80305, USA

###### Abstract

Next-generation communications, radar and navigation systems will extend and
exploit the higher bandwidth of the millimeter-wave domain for increased
communication data rates as well as radar with higher sensitivity and
increased spatial resolution. However, realizing these advantages will require
the generation of millimeter-wave signals with low phase noise in simple and
compact form-factors. The rapidly developing field of photonic integration
addresses this challenge and provides a path toward simplified and portable,
low-noise mm-wave generation for these applications. We leverage these
advances by heterodyning two silicon photonic chip lasers, phase-locked to the
same miniature Fabry-Perot (F-P) cavity to demonstrate a simple framework for
generating low-noise millimeter-waves with phase noise below the thermal limit
of the F-P cavity. Specifically, we generate 94.5 GHz and 118.1 GHz
millimeter-wave signals with phase noise of $-117$ dBc/Hz at 10 kHz offset,
decreasing to $-120$ dBc/Hz at 40 kHz offset, a record low value for such
photonic devices. We achieve this with existing technologies that can be
integrated into a platform less than $\approx 10\hskip 3.0pt\text{mL}$ in
volume. Our work illustrates the significant potential and advantages of low
size, weight, and power (SWaP) photonic-sourced mm-waves for communications
and sensing.

## Introduction

Inherent to the higher frequency of millimeter-wave carriers (mm-wave,
$\approx 30\to 300$ GHz) is the capability of higher modulation bandwidth and
faster data transfer rates Ma2018 ; Akyildiz2014 , finer radar and sensing
spatial resolution Soumya2023 ; Rappaport2019 , antenna miniaturization
Soumya2023 , and viability of atmospheric transmission Rappaport2013 . The
impact of mm-waves is already evident in cutting-edge telecommunications, with
5G utilizing carriers in both the microwave and low mm-wave range
($\lessapprox 50\hskip 3.0pt\text{GHz}$), and 6G technology pushing carrier
frequencies beyond $90\hskip 3.0pt\text{GHz}$ Hong2021 . Furthermore, mm-wave
technologies have already shown promise for safe and non-invasive biomedical
and security sensing applications Wang2023 ; Luukanen2012 , and are widely
used as radio astronomy references doeleman2009 .

In the microwave range, rack-mount-volume synthesizers dominate the
application space. These provide wide tunability, but with close-to-carrier
phase noise that, in the best case, is given by a frequency-multiplied quartz
reference or other dielectric oscillator. Such signal sources are commonly
extended to the mm-wave via carrier multiplication by $N$, but that comes with
additional degradation in phase noise power spectral density by at least
$20\log(N)$ Bara-Maillet2015 . As a result, commercial mm-wave synthesizers at
100 GHz are typically limited in phase-noise performance in the range of -100
dBc/Hz at 10 kHz offset and can struggle to meet the demands as local
oscillators (LOs) in the aforementioned radar and telecommunication
applications. On the other hand, laboratory research synthesizers provide
remarkably low noise at the cost of increasing complexity and larger volume
Bara2012 .

Figure 1: a.) Simplified schematic of the experiment, with pictures of the key
components zoomed in. b.) Illustration of the mechanism of common mode
rejection. Two signals ($\nu_{1}$ and $\nu_{2}$) are indicated by their
nominal frequencies (gray dashed lines) with their noise contributions (the
thermal cavity noise of each laser, $\sigma^{(1)}_{\text{th}}$ and
$\sigma^{(2)}_{\text{th}}$, respectively as well as PDH locking noise,
$\sigma^{(1)}_{\text{PDH}}$ and $\sigma^{(2)}_{\text{PDH}}$, respectively)
shown as jagged continuous lines. The signal that results from their
heterodyning, $\nu_{\text{RF}}$, is shown with most of the thermal cavity
noise being subtracted out. c.) Illustration of the two laser signals (red and
blue) locked to two cavity modes (dashed Gaussians), and heterodyned at 5
cavity FSRs apart, to generate a 118.1 GHz RF tone.

Within this context, the field of mm-wave photonic synthesis with narrow
linewidth lasers provides a compelling alternative to electronic approaches.
Most simply, the interference of two continuous wave (CW) lasers can be
photomixed (photodetected) to produce a mm-wave signal at the difference
frequency of the lasers. This concept was first demonstrated just a few years
after the invention of the laser, but progress has accelerated over the past
two decades with multiple techniques involving lasers, electro-optics, and
frequency combs to generate and control the relative frequency difference
between the CW lasers Waltman1996 ; fukushima2003optoelectronic ;
Nagatsuma2009 ; rolland2014narrow ; Fortier2016 ; Tetsumoto2021 ; Kittlaus2021
.

Even greater opportunities for photonic mm-waves are offered by the confluence
of recent advances in chip-integrated low-noise lasers and frequency combs and
the means to frequency-stabilize these light sources with compact, high-
finesse, and integrable high-Q optical cavities. Importantly, lasers can now
be heterogeneously integrated on a CMOS-compatible platform together with
extremely low-loss Si4N3 ring resonators for self-injection locking (SIL)
operation Jin2021 ; Xiang2023 . This leads to laser frequency noise that is
typically associated with narrow-linewidth fiber lasers, but with smaller
footprint Li2021 . In addition, miniaturization and integration of centimeter-
sized optical cavities with finesse approaching $10^{6}$ Cheng2023 ; Liu2024
transforms the realization of hertz-linewidth lasers from large laboratory
setups to the chip-scale, and in some cases even forgoing the need for vacuum
enclosures Liu2024 . Recent combinations of such integrated photonic hardware
have further utilized the powerful approach of optical frequency division
(OFD) for low-noise microwave and mm-wave generation Kudelin2024 ;
sun2024integrated .

In this work, we show that the benefits of OFD are not required to further
advance the phase noise of mm-wave generation at 100 GHz and above. Instead we
exploit the high degree of thermal noise correlation between the axial modes
of a Fabry-Perot (F-P) to strongly reject this noise in mm-wave generation. A
cavity Q-factor approaching $10^{10}$ and linewidth near 20 kHz allows us to
tightly lock the frequencies of two lasers to cavity modes with residual noise
that is well below the room-temperature cavity thermal noise. Thus, the
correlated thermal noise of the cavity close-to-carrier that is imprinted on
each laser is subtracted in the mm-wave heterodyne beat.

We implement this concept with two SIL lasers that are frequency-locked to
modes of a miniature high-Q Fabry-Perot cavity with free spectral range,
$\text{FSR}=23.6$ GHz. For modes separated by $5\times\text{FSR}=118$ GHz, the
phase noise of the heterodyne beat is $-117.6\hskip 3.0pt\text{dBc/Hz}$ at
$10\hskip 3.0pt\text{kHz}$ offset, decreasing to $-124\hskip
3.0pt\text{dBc/Hz}$ at $30\hskip 3.0pt\text{kHz}$. Importantly, our approach
uses the same components that have been shown to be compatible with chip-scale
integration, while offering phase noise levels that match those achieved with
more complicated integrated OFD approaches. And at 10 kHz offset our results
are close to the noise levels achieved in the very best photonic mm-wave
laboratory experiments.

Our straightforward approach to mm-wave generation effectively removes the
fundamental limitations imposed by cavity thermal noise. As such, we
anticipate that further improvements in phase-noise performance will be
allowed to proceed at the more rapid pace of technical innovations in low-
noise chip-integrated lasers and frequency control techniques.

## Concept & Experiment

A simplified schematic of the experiment is shown in Fig. 1a. Starting from
the left, two distributed-feedback (DFB) lasers ($\nu_{1}$ and $\nu_{2}$) are
individually coupled and self-injection locked to separate high Q
($\text{Q}\gtrapprox 10^{8}$) Si3N4 microresonators, with free spectral ranges
(FSRs) of about $135\hskip 3.0pt\text{MHz}$. The self-injection locking is
controlled by fine-tuning the facet distance (hence phase) of the DFBs
relative to their respective microresonator chips, and results in a phase
noise suppression of the free-running DFB signal by up to $50\hskip
3.0pt\text{dB}$ Guo2022 . The SIL laser output is then fiber-coupled out of
the chip for use.

This initial self-injection locking suppresses the close-to-carrier noise and
allows us to use the Pound-Drever-Hall (PDH) technique Drever1983  to
subsequently lock each laser’s frequency to an ultra-stable, temperature- and
vacuum-controlled, miniature F-P cavity. In this case, the laser frequency
inherits the stability of the cavity, but with frequency noise fundamentally
limited by thermally driven stochastic fluctuations in the cavity length
Numata2004 . Here, the cavity we used is exhibited and referred to in
kelleher2023compact  as the "all-ULE cavity". We lock both lasers to separate
F-P cavity modes such that the mm-wave carrier frequency is
$f_{\mu}=\text{n}\times\text{FSR}=\nu_{1}-\nu_{2}$, as portrayed in Fig. 1c.
Importantly, subtraction of these two PDH-locked signals, as occurs in
photodetection, results in noise cancellation of the common noise, including
thermal noise inherited by both lasers from the cavity. This yields an output
signal with phase noise below that of the cavity thermal limit.

Fig. 1b illustrates the primary noise contributions to $f_{\mu}$ and their
subtraction in the time domain. The two PDH-locked lasers have frequency
fluctuations dominated by two terms: the intrinsic cavity thermal noise
$\sigma_{th}$ and the smaller residual noise of the PDH lock
$\sigma_{\text{PDH}}$ that is related to technical servo limitations. Because
of significant overlap of the spatial modes of $\nu_{1}$ and $\nu_{2}$ in the
F-P cavity, there is strong amplitude and phase correlation between
$\sigma_{th}^{(1)}$ and $\sigma_{th}^{(2)}$. In the heterodyne these noise
terms largely subtract, as shown in the bottom of Fig. 1b. However, the PDH
lock noise $\sigma_{\text{PDH}}^{(1)}$ and $\sigma_{\text{PDH}}^{(2)}$ arises
from uncorrelated shot noise from the separate laser locks and instead adds in
quadrature in the heterodyne signal, such that
$\sigma_{\mu}^{2}=(\sigma^{(1)}_{\text{PDH}})^{2}+(\sigma^{(2)}_{\text{PDH}})^{2}$.

The extent of the rejection of the correlated (common-mode) cavity noise can
be estimated by considering that the F-P cavity itself functions as a
frequency divider. The magnitude of the thermally driven fluctuations of a
frequency mode $\nu$ of the cavity is directly linked to the fluctuations of
the cavity FSR, such that $\sigma_{\text{FSR}}=(\text{FSR}/\nu)\sigma_{\nu}$.
Here we see that the frequency fluctuations $\sigma_{\nu}$ of mode $\nu$ are
ideally reduced by the ratio of $\text{FSR}/\nu\approx 10^{-4}$. Similarly,
$\sigma_{n\times\text{FSR}}=(n\times\text{FSR}/\nu)\sigma_{\nu}$, where
$n\times\text{FSR}$ is the separation of any two modes of the cavity Liu2024 .
Considering two lasers locked to modes separated by $5\times\text{FSR}=118$
GHz, the expected common rejection of frequency fluctuations, is $6\times
10^{-4}$, which is equivalent to a reduction of $20\cdot\log(\text{118
GHz}/\text{193 THz GHz})\approx-64$ dB in phase noise power spectral density
Liu2024 . This estimate assumes perfect correlation of thermal noise in the
two cavity modes, and therefore neglects the slight variations in cavity mode
volume with wavelength. But for two nearby modes, this approximation
sufficiently justifies how the phase noise of the mm-wave beat can surpass the
cavity thermal limit. It is then limited by the PDH locking noise of the
lasers, which can be addressed through technical advances.

In the experiment, the two PDH-locked lasers are mixed with a commercial
W-band photodetector and the resulting output is a millimeter-wave ("output"
in Fig.1), having power of $\approx 1\hskip 3.0pt\text{mW}$. Here, we
demonstrate millimeter-wave signals at $4\times\text{FSR}$ and
$5\times\text{FSR}$, with $94.5\hskip 3.0pt\text{GHz}$ and $118.1\hskip
3.0pt\text{GHz}$ carriers respectively. The phase noise of the millimeter-wave
signals was measured using cross correlation techniques to remove mixer and
amplifier noise. In brief, the photodetected signal was split with a -3 dB
waveguide coupler and each copy was mixed down to the RF using harmonic mixers
driven by independent commercial synthesizers. The two resulting baseband
signals near $5\hskip 3.0pt\text{MHz}$ where then cross-correlated on a phase
noise analyzer.

## Results

Figure 2: Phase noise plot of the 1 FSR (23.6 GHz, Blue), 4 FSR (94.5 GHz,
Yellow), and 5 FSR (118.1 GHz, Red) signals generated from heterodyning two
self-injection-locked distributed feedback lasers after PDH locking to two
modes of the miniature F-P. In green, the phase noise of the cumulative noise
contributions is plotted, extrapolated from the PDH-locked laser in-loop
noise.

The result of the cross-correlation phase noise measurements of the
millimeter-wave signals is plotted in Fig. 2. We see three overlaid traces of
approximately the same phase-noise level as the 1 FSR (23.6 GHz, blue), 4 FSR
(94.5 GHz, yellow), and 5 FSR (118.1 GHz, red) signals. This indicates that
the phase noise of any millimeter-wave signal of integer-multiple FSR
frequency can be generated and will inherit the phase noise of the PDH-locked
SIL lasers, with no loss in fidelity. The only limiting factor then is the
power on the high-speed photodetector, and hence the signal-to-noise ratio, of
the heterodyned signals as the frequency difference between the two lasers is
increased. The discrepancies in the phase noise of the 118.1 GHz signal versus
the 23.6 GHz signal (near 10 kHz offset) arise from the requirement of longer
averaging times due to the noise constraints of the cross-correlation
reference signals.

Lastly and for comparison, on the same plot we have plotted (in green) the
cumulative uncorrelated noise contributions of various sources, which have
been added in quadrature. These sources include the in-loop phase noise of
each individual SIL laser during PDH-lock operation, and the SNR floor of the
PDH servo loop. The quadrature sum shows good agreement with the measured 1
FSR, 4 FSR, and 5 FSR phase noise. Consequently, lower phase noise lasers or
higher PDH-lock gain would offer even greater noise reduction.

Figure 3: (a). Phase noise plot of the PDH-locked (Red) versus unlocked (Blue)
SIL lasers. In black-dashed line, the F-P cavity thermal limit referenced from
kelleher2023compact .

In Fig. 3, we have plotted the free-running (without PDH locking) SIL laser
phase noise (blue) and the $118.1$ GHz signal phase noise (red), side-by-side,
showing that $\approx 37$ dB of noise reduction at 10 kHz offset is achieved
by PDH locking, with AOM and EOM feedback, to the F-P cavity. Here we measured
the free-running SIL noise by heterodyning the two SIL lasers on an MUTC
$\approx 10\hskip 3.0pt\text{GHz}$ apart from one another. The $10\hskip
3.0pt\text{GHz}$ signal is mixed down to DC and measured against a maser
reference on a phase noise analyzer. We make the assumption that both SIL
lasers have similar noise profiles, such that this phase noise is simply twice
(+3 dB) that of a single SIL laser.

We also plotted the calculated thermal noise floor of the F-P cavity in
dashed-black kelleher2023compact . Comparing the thermal limit to the 118.1
GHz millimeter-wave signal reveals that for offset frequencies below 10 kHz,
the phase noise of the millimeter-wave signal surpasses the thermal-limited
optical phase noise over a wide range of Fourier frequencies. To further
quantify the amount of common-mode rejection exhibited, we independently
measured the absolute (out-of-loop) phase noise of an individual PDH-locked
SIL laser (yellow in Fig. 3). As expected, we see that SIL laser noise follows
the thermal limit of the F-P cavity up to about 10 kHz. At higher frequencies,
the laser noise is limited by PDH servo gain and bandwidth.

The difference between the out-of-loop optical phase noise and the mm-wave
phase noise is the amount of noise common to both lasers and hence subtracted
(rejected) in the heterodyne signal. Thus the mm-wave signal here gains 22 dB
of noise rejection at 10 Hz offset, due to both optical references being PDH
locked to the same cavity.

To put our result into context, we have included a survey of the reported
phase noise of state-of-the-art photonic millimeter-wave synthesis. As seen in
Fig. 4, our SIL and PDH framework allows for exceptionally low-noise
millimeter-wave synthesis, largely falling well below commercial products and
existing photonic-based sources generation up until the servo bump at around
300 kHz. This does not tell the full story, however, as other millimeter-wave
generators rely on significantly more complex systems, to achieve lower and
similar phase noise.

Figure 4: Phase noise plot of the millimeter-wave generation landscape scaled
to 118.1 GHz Fortier2016 ; sun2024integrated ; Kittlaus2021 ; Song2008 .

For example, the lowest phase noise achieved and plotted in red, incorporates
full optical frequency division using an octave spanning, titanium-doped
sapphire, bulk-optic frequency comb and a filtering etalon Fortier2016 . The
power required to run such a system is thus significantly higher than the one
in this work. The lowest photonic-chip-based mm-wave noise (blue), falls
directly in line with this work’s, yet requires a microcomb to achieve OFD,
which inevitably requires a more complex locking scheme, while simultaneously
requiring two lasers PDH locking to a microresonator cavity sun2024integrated
. In this work, we’ve shown that these phase-noise levels can be achieved with
a simple, passive self-injection locking and PDH scheme, requiring low input
currents (200 mA to drive the lasers) and providing a path toward chip
integration.

The phase noise survey also includes research mm-wave synthesizers (grey)
Song2008  and a similar self-injection-locking heterodyne scheme (yellow) but
without PDH locking to a cavity.

## Conclusion and Outlook

In conclusion, we have shown a simplified, compact way to generate low phase-
noise millimeter-wave carriers, by transferring the noise of stable optical
references to any integer multiple of the F-P cavity FSR. Importantly, through
common-mode rejection, we surpass the thermal limit of the reference cavity.
In doing so, we have shown millimeter-wave phase noise of $-117\hskip
3.0pt\text{dBc/Hz}$ at 10 kHz offset, regardless of carrier frequency. This
performance is better than any chip-scale photonic mm-wave generation to date,
and is within 5 dB the highest performing full-OFD millimeter-wave source.

In addition to the phase noise performance, the low-complexity and compact
form factor of this experiment both lend themselves towards the argument of
full integratability. For example, with advances of air-gap micro- F-P
cavities Jin2022 ; Liu2024 ; Cheng2023 , integrated SIL lasers and on-chip
MUTC photodetectors Xiang2023 ; Zang2018reduction , the approach we present
can be integrated onto a single chip. And in comparison to more complex OFD
schemes Kudelin2024 ; sun2024integrated , we show an attractive option to
reduce the number of integrated components and realize on-chip, low-noise mm-
wave generation with robust and turn-key operation.

## References

  * (1) J. Ma, R. Shrestha, L. Moeller, and D. M. Mittleman, “Invited article: Channel performance for indoor and outdoor terahertz wireless links,” APL Photonics, vol. 3, 5 2018.
  * (2) I. F. Akyildiz, J. M. Jornet, and C. Han, “Terahertz band: Next frontier for wireless communications,” Physical Communication, vol. 12, pp. 16–32, 9 2014.
  * (3) A. Soumya, C. K. Mohan, and L. R. Cenkeramaddi, “Recent advances in mmwave-radar-based sensing, its applications, and machine learning techniques: A review,” Sensors, vol. 23, p. 8901, 11 2023.
  * (4) T. S. Rappaport, Y. Xing, O. Kanhere, S. Ju, A. Madanayake, S. Mandal, A. Alkhateeb, and G. C. Trichopoulos, “Wireless communications and applications above 100 GHz: Opportunities and challenges for 6G and beyond,” IEEE Access, vol. 7, pp. 78729–78757, 2019.
  * (5) T. S. Rappaport, S. Sun, R. Mayzus, H. Zhao, Y. Azar, K. Wang, G. N. Wong, J. K. Schulz, M. Samimi, and F. Gutierrez, “Millimeter wave mobile communications for 5G cellular: It will work!,” IEEE Access, vol. 1, pp. 335–349, 2013.
  * (6) W. Hong, Z. H. Jiang, C. Yu, D. Hou, H. Wang, C. Guo, Y. Hu, L. Kuai, Y. Yu, Z. Jiang, Z. Chen, J. Chen, Z. Yu, J. Zhai, N. Zhang, L. Tian, F. Wu, G. Yang, Z.-C. Hao, and J. Y. Zhou, “The role of millimeter-wave technologies in 5G/6G wireless communications,” IEEE Journal of Microwaves, vol. 1, pp. 101–122, 1 2021.
  * (7) H. Wang, L. Lu, P. Liu, J. Zhang, S. Liu, Y. Xie, T. Huo, H. Zhou, M. Xue, Y. Fang, J. Yang, and Z. Ye, “Millimeter waves in medical applications: status and prospects,” Intelligent Medicine, 12 2023.
  * (8) A. Luukanen, R. Appleby, M. Kemp, and N. Salmon, Millimeter-Wave and Terahertz Imaging in Security Applications, pp. 491–520.  Berlin, Heidelberg: Springer Berlin Heidelberg, 2013.
  * (9) S. Doeleman, “High frequency very long baseline interferometry: Frequency standards and imaging an event horizon,” in Frequency Standards And Metrology, pp. 175–183, World Scientific, 2009.
  * (10) R. Bara-Maillet, S. R. Parker, N. R. Nand, J.-M. Le Floch, and M. E. Tobar, “Microwave–to–millimeter-wave synthesis chain phase noise performance,” IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, vol. 62, no. 10, pp. 1895–1900, 2015.
  * (11) R. Bara, J.-M. L. Floch, M. E. Tobar, P. L. Stanwix, S. R. Parker, J. G. Hartnett, and E. N. Ivanov, “Generation of 103.75 GHz cw source with $5.10^{-16}$ frequency instability using cryogenic sapphire oscillators,” IEEE Microwave and Wireless Components Letters, vol. 22, no. 2, pp. 85–87, 2012.
  * (12) S. B. Waltman, L. W. Hollberg, A. K. McIntosh, and E. R. Brown, “Demonstration of a phase-lockable microwave to submillimeter wave sweeper,” in Millimeter and Submillimeter Waves and Applications III (M. N. Afsar, ed.), vol. 2842, pp. 55 – 58, International Society for Optics and Photonics, SPIE, 1996.
  * (13) S. Fukushima, C. Silva, Y. Muramoto, and A. J. Seeds, “Optoelectronic millimeter-wave synthesis using an optical frequency comb generator, optically injection locked lasers, and a unitraveling-carrier photodiode,” Journal of lightwave technology, vol. 21, no. 12, p. 3043, 2003.
  * (14) T. Nagatsuma, H. Ito, and T. Ishibashi, “High-power rf photodiodes and their applications,” Laser & Photonics Reviews, vol. 3, no. 1-2, pp. 123–137, 2009.
  * (15) A. Rolland, G. Ducournau, G. Danion, M. Brunel, A. Beck, F. Pavanello, E. Peytavit, T. Akalin, M. Zaknoune, J.-F. Lampin, et al., “Narrow linewidth tunable terahertz radiation by photomixing without servo-locking,” IEEE Transactions on Terahertz Science and Technology, vol. 4, no. 2, pp. 260–266, 2014.
  * (16) T. M. Fortier, A. Rolland, F. Quinlan, F. N. Baynes, A. J. Metcalf, A. Hati, A. D. Ludlow, N. Hinkley, M. Shimizu, T. Ishibashi, J. C. Campbell, and S. A. Diddams, “Optically referenced broadband electronic synthesizer with 15 digits of resolution,” Laser & Photonics Reviews, vol. 10, pp. 780–790, 9 2016.
  * (17) T. Tetsumoto, T. Nagatsuma, M. E. Fermann, G. Navickaite, M. Geiselmann, and A. Rolland, “Optically referenced 300 GHz millimetre-wave oscillator,” Nature Photonics, vol. 15, pp. 516–522, 7 2021.
  * (18) E. A. Kittlaus, D. Eliyahu, S. Ganji, S. Williams, A. B. Matsko, K. B. Cooper, and S. Forouhar, “A low-noise photonic heterodyne synthesizer and its application to millimeter-wave radar,” Nature Communications, vol. 12, p. 4397, 7 2021.
  * (19) W. Jin, Q.-F. Yang, L. Chang, B. Shen, H. Wang, M. A. Leal, L. Wu, M. Gao, A. Feshali, M. Paniccia, K. J. Vahala, and J. E. Bowers, “Hertz-linewidth semiconductor lasers using cmos-ready ultra-high-q microresonators,” Nature Photonics, vol. 15, pp. 346–353, 5 2021.
  * (20) C. Xiang, W. Jin, O. Terra, B. Dong, H. Wang, L. Wu, J. Guo, T. J. Morin, E. Hughes, J. Peters, Q.-X. Ji, A. Feshali, M. Paniccia, K. J. Vahala, and J. E. Bowers, “3d integration enables ultralow-noise isolator-free lasers in silicon photonics,” Nature, vol. 620, pp. 78–85, 8 2023.
  * (21) B. Li, W. Jin, L. Wu, L. Chang, H. Wang, B. Shen, Z. Yuan, A. Feshali, M. Paniccia, K. J. Vahala, and J. E. Bowers, “Reaching fiber-laser coherence in integrated photonics,” Optics Letters, vol. 46, p. 5201, 10 2021.
  * (22) H. Cheng, N. Jin, Z. Dai, C. Xiang, J. Guo, Y. Zhou, S. A. Diddams, F. Quinlan, J. Bowers, O. Miller, and P. Rakich, “A novel approach to interface high-q fabry–pérot resonators with photonic circuits,” APL Photonics, vol. 8, 11 2023.
  * (23) Y. Liu, D. Lee, T. Nakamura, N. Jin, H. Cheng, M. L. Kelleher, C. A. McLemore, I. Kudelin, W. Groman, S. A. Diddams, P. T. Rakich, and F. Quinlan, “Low-noise microwave generation with an air-gap optical reference cavity,” APL Photonics, vol. 9, 1 2024.
  * (24) I. Kudelin, W. Groman, Q.-X. Ji, J. Guo, M. L. Kelleher, D. Lee, T. Nakamura, C. A. McLemore, P. Shirmohammadi, S. Hanifi, H. Cheng, N. Jin, L. Wu, S. Halladay, Y. Luo, Z. Dai, W. Jin, J. Bai, Y. Liu, W. Zhang, C. Xiang, L. Chang, V. Iltchenko, O. Miller, A. Matsko, S. M. Bowers, P. T. Rakich, J. C. Campbell, J. E. Bowers, K. J. Vahala, F. Quinlan, and S. A. Diddams, “Photonic chip-based low-noise microwave oscillator,” Nature, vol. 627, pp. 534–539, 3 2024.
  * (25) S. Sun, M. W. Harrington, F. Tabatabaei, S. Hanifi, K. Liu, J. Wang, B. Wang, Z. Yang, R. Liu, J. S. Morgan, S. M. Bowers, P. A. Morton, K. D. Nelson, A. Beling, D. J. Blumenthal, and X. Yi, “Kerr optical frequency division with integrated photonics for stable microwave and mmwave generation,” arXiv preprint arXiv:2305.13575, 2023.
  * (26) J. Guo, C. A. McLemore, C. Xiang, D. Lee, L. Wu, W. Jin, M. Kelleher, N. Jin, D. Mason, L. Chang, A. Feshali, M. Paniccia, P. T. Rakich, K. J. Vahala, S. A. Diddams, F. Quinlan, and J. E. Bowers, “Chip-based laser with 1-hertz integrated linewidth,” Science Advances, vol. 8, 10 2022.
  * (27) R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, “Laser phase and frequency stabilization using an optical resonator,” Applied Physics B Photophysics and Laser Chemistry, vol. 31, pp. 97–105, 6 1983.
  * (28) K. Numata, A. Kemery, and J. Camp, “Thermal-noise limit in the frequency stabilization of lasers with rigid cavities,” Phys. Rev. Lett., vol. 93, p. 250602, Dec 2004.
  * (29) M. L. Kelleher, C. A. McLemore, D. Lee, J. Davila-Rodriguez, S. A. Diddams, and F. Quinlan, “Compact, portable, thermal-noise-limited optical cavity with low acceleration sensitivity,” Optics Express, vol. 31, no. 7, pp. 11954–11965, 2023.
  * (30) H.-J. Song, N. Shimizu, T. Furuta, K. Suizu, H. Ito, and T. Nagatsuma, “Broadband-frequency-tunable sub-terahertz wave generation using an optical comb, AWGs, optical switches, and a uni-traveling carrier photodiode for spectroscopic applications,” Journal of Lightwave Technology, vol. 26, pp. 2521–2530, 8 2008.
  * (31) N. Jin, C. A. McLemore, D. Mason, J. P. Hendrie, Y. Luo, M. L. Kelleher, P. Kharel, F. Quinlan, S. A. Diddams, and P. T. Rakich, “Micro-fabricated mirrors with finesse exceeding one million,” Optica, vol. 9, p. 965, 9 2022.
  * (32) J. Zang, X. Xie, Q. Yu, Z. Yang, A. Beling, and J. C. Campbell, “Reduction of amplitude-to-phase conversion in charge-compensated modified unitraveling carrier photodiodes,” Journal of Lightwave Technology, vol. 36, no. 22, pp. 5218–5223, 2018.

Corresponding authors<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
Funding: This research was supported by DARPA GRYPHON program
(HR0011-22-2-0009) and NIST.(80NM0018D0004).

Acknowledgments: Commercial equipment and trade names are identified for
scientific clarity only and does not represent an endorsement by NIST.

Author Contributions:

W.G., I.K., J.E.B., F.Q., and S.A.D. conceived the experiment and supervised
the project. W.G., I.K., and S.A.D. wrote the paper with input from all
authors. W.G., I.K., and A.L. built the experiment and performed the mm-wave
experiment. D.L., T.N., and Y.L. provided mm-wave reference sources and input
regarding the cavity and cross-correlation measurements. M.L.K and C.A.M.
built the cavity and gave crucial information regarding the cavity. J.G.,
L.W., W.J., and J.E.B provided the lasers and spiral microresonators, as well
as input regarding the operation of the lasers. All authors contributed to
discussion of the results.

Disclosures:

The authors declare no conflicts of interest.

Data and materials availability:

All data for the figures in this manuscript is available at .