Further author information: (Send correspondence to A.K.S.)
A.K.S.: E-mail<EMAIL_ADDRESS>
# CCAT-prime: RFSoC Based Readout for Frequency Multiplexed Kinetic Inductance
Detectors

Adrian K. Sinclair  Dept. of Physics and Astronomy, University of British
Columbia, Vancouver, BC, Canada  Ryan C. Stephenson  Dept. of Physics, Arizona
State University, Tempe, AZ, USA  Cody A. Roberson  School of Earth and Space
Exploration, Arizona State University, Tempe, AZ, USA  Eric L. Weeks  School
of Earth and Space Exploration, Arizona State University, Tempe, AZ, USA
James Burgoyne  Dept. of Physics and Astronomy, University of British
Columbia, Vancouver, BC, Canada  Anthony I. Huber  Dept. of Physics and
Astronomy, University of Victoria, Victoria, BC, Canada  Philip M. Mauskopf
Dept. of Physics, Arizona State University, Tempe, AZ, USA  School of Earth
and Space Exploration, Arizona State University, Tempe, AZ, USA  Scott C.
Chapman  Dept. of Physics and Astronomy, University of British Columbia,
Vancouver, BC, Canada  Dept. of Physics and Astronomy, University of Victoria,
Victoria, BC, Canada  Herzberg Astronomy and Astrophysics Research Centre,
National Research Council Canada, Victoria, BC, Canada  Jason E. Austermann
National Institute of Standards and Technology, Boulder, CO, USA  Steve K.
Choi  Dept. of Physics, Cornell University, Ithaca, NY, USA  Cody J. Duell
Dept. of Physics, Cornell University, Ithaca, NY, USA  Michel Fich  Waterloo
Centre for Astrophysics, University of Waterloo, Waterloo, ON, Canada
Christopher E. Groppi  School of Earth and Space Exploration, Arizona State
University, Tempe, AZ, USA  Zachary Huber  Dept. of Physics, Cornell
University, Ithaca, NY, USA  Michael D. Niemack  Dept. of Physics, Cornell
University, Ithaca, NY, USA  Thomas Nikola  Cornell Center for Astrophysics
and Planetary Sciences, Cornell University, Ithaca, NY, USA  Kayla M. Rossi
Cornell Center for Astrophysics and Planetary Sciences, Cornell University,
Ithaca, NY, USA  Adhitya Sriram  School of Earth and Space Exploration,
Arizona State University, Tempe, AZ, USA  Gordon J. Stacey  Dept. of
Astronomy, Cornell University, Ithaca, NY, USA  Erik Szakiel  Dept. of
Physics, Cornell University, Ithaca, NY, USA  Joel Tsuchitori  Dept. of
Physics and Astronomy, University of British Columbia, Vancouver, BC, Canada
Eve M. Vavagiakis  Dept. of Physics, Cornell University, Ithaca, NY, USA
Jordan D. Wheeler  National Institute of Standards and Technology, Boulder,
CO, USA  the CCAT-prime collaboration

###### Abstract

The Prime-Cam instrument on the Fred Young Submillimeter Telescope (FYST) is
expected to be the largest deployment of millimeter and submillimeter
sensitive kinetic inductance detectors to date. To read out these arrays
efficiently, a microwave frequency multiplexed readout has been designed to
run on the Xilinx Radio Frequency System on a Chip (RFSoC). The RFSoC has
dramatically improved every category of size, weight, power, cost, and
bandwidth over the previous generation readout systems. We describe a baseline
firmware design which can read out four independent RF networks each with 500
MHz of bandwidth and 1000 detectors for $\sim$30 W. The overall readout
architecture is a combination of hardware, gateware/firmware, software, and
network design. The requirements of the readout are driven by the 850 GHz
instrument module of the 7-module Prime-Cam instrument. These requirements
along with other constraints which have led to critical design choices are
highlighted. Preliminary measurements of the system phase noise and dynamic
range are presented.

###### keywords:

CCAT-prime, RFSoC, MKID, frequency multiplexed, kinetic inductance

##  1 Introduction

The far-infrared sky, loosely defined from $\sim$100 GHz to a few THz, has
been a challenging band of the electromagnetic spectrum to observe.
Atmospheric absorption and emission require strategic band selection and
extraordinarily dry and high altitude observation sites. In part due to these
challenges the far-infrared (far-IR) sky still has many motivating and clear
science goals[1], a few of these are highlighted below. The role of turbulence
and magnetic fields can be studied through the polarized far-IR emission of
interstellar dust grains. The variability in protostars which informs stellar
accretion and planet formation models can be studied from mid to far-IR.
Galaxy formation and star formation history can be tracked through redshifted
lines such as CO, OIII, and CII, which fall into the same band. High redshift
proto-clusters of galaxies can be probed via the kinematic Sunyaev-Zel’dovich
effect at hundreds of gigahertz. On the largest scales, the cosmic microwave
background requires well calibrated sub-millimeter observations of polarized
dust emission to remove its contaminating foregrounds. These are just a few of
the motivating science topics for the Fred Young Submillimeter Telescope and
first light instruments described next.

Seated high atop Cerro Chajnantor in the Atacama desert of Chile, the CCAT-
prime observatory will include the Prime-Cam instrument on the Fred Young
Submillimeter Telescope (FYST) and will map the sky with imaging
polarimeters[2, 3, 4] and spectrometers[5, 6]. The telescope’s wide field of
view and large optical throughput[7, 8] enables high detector count focal
planes. Mod-Cam[9] is a single instrument module receiver that is a precursor
and testbed for Prime-Cam and is planned to achieve first light in 2024. The
enabling detector technology is the Lumped Element Kinetic Inductance Detector
(LEKID) which, due to its natural multiplexing, low fabrication cost, and low
cryogenic readout complexity, can be scaled to the $>$ 100,000 detectors
required for Prime-Cam. A second enabling technology is the Xilinx Radio
Frequency System-on-Chip (RFSoC) which has dramatically improved every
category of size, weight, power, cost, and bandwidth over the previous
generation KID readout systems.

Kinetic Inductance Detectors (KIDs) were developed nearly twenty years ago
[10] to enable ultra-wideband astronomical imaging at low fabrication cost
with high multiplexing factors into the thousands. Since then, single photon
sensitivity with modest energy resolution has been achieved from X-rays to the
near-infrared [11, 12, 13]. At longer wavelengths in the millimeter and
submillimeter, photon noise limited performance has been shown in linear power
detection mode [14, 15, 16, 17, 18]. There are recent encouraging results with
sub-gap absorption KIDs[19], DC biasing [20], and bi-layers [21], opening up
even longer wavelength detection typical of CMB studies. KIDs have been
antenna coupled [22, 23], horn coupled [24, 25, 26], and packed into filled
arrays[27], each achieving high optical efficiency to allow dense focal plane
packing. A shift from distributed to a lumped element kinetic inductance
detector (LEKID) [28] design has simplified frequency placement and relaxed
simulation requirements. The majority of the current and previously deployed
mm/sub-mm imaging telescopes which utilize KIDs have adopted lumped element
designs [29, 30, 31, 32, 33, 34, 35]. Prime-Cam will employ a lumped element
design that is horn coupled capitalizing on the development experience of the
BLAST-TNG and TolTEC LEKID arrays [26].

Early KID readouts leveraged advancements in commercially available software
defined radio technology [36] and previously developed radio astronomy
spectrometers [37, 38]. Custom boards started to be developed in the early
2010s [39, 40, 41]. A reconfigurable open-architecture computing hardware
(ROACH) based system, originally designed for radio astronomy, was repurposed
by multiple groups [29, 42, 43], as well as the second generation of the board
(ROACH2) a few years later [44, 45, 46]. The open source firmware developed
for the ROACH2 based readout of BLAST-TNG[44] has enjoyed use in the following
experiments: OLIMPO[47], MUSCAT[48], SuperSpec[49], and TolTEC[34]. The RFSoC
by Xilinx arrived near the end of the decade and immediately generated
interest in the community and spurred multiple parallel development efforts
[50, 51, 52, 53, 54, 55]. This work presents an RFSoC based readout to be to
be deployed on FYST for the Mod-Cam and Prime-Cam instruments and a future on-
chip spectrometer.

##  2 Requirements

The main driver of the readout system requirements are the detectors in five
ways: Total number of detectors, number of detectors per RF network, bandwidth
per network, added noise, and crosstalk. There are more stringent requirements
for the frequency comb discussed in the gateware Sec. 3.3.1. For the 280 GHz,
350 GHz, and EoR-Spec detector arrays the baseline bandwidth of $\sim$500 MHz
with 600 resonators or less per RF network has been adopted. In order to
adhere to thermal budget constraints, Prime-Cam’s highest frequency instrument
- the 850 GHz module[56] is planning for two octaves of bandwidth (1 GHz) for
each RF network. The 850 GHz arrays push the bandwidth requirements from the
detectors to the readout while previous KID arrays developed for BLAST-TNG[15]
and TolTEC[26] have been constrained by the instantaneous bandwidth of the
ROACH2 based readout[44]. The readout noise and crosstalk requirements are
derived from detector parameters in the next two sections.

###  2.1 Readout Noise Requirement

The readout system should not contribute significantly to the noise, which
places a requirement on both the noise and dynamic range of the system. We use
the method of Sipola et al.[57] to calculate an equivalent output noise
temperature of the KID and compare this to other readout contributors such as
the cryogenic low noise amplifier (LNA) and digital to analog converter (D/A).
The detector equivalent output noise temperature is defined as,

$T=\frac{(R\times\textrm{NEP})^{2}}{kZ_{0}},$ (1)

where $R$ is the voltage responsivity to power, NEP is the detector noise
equivalent power, $k$ is Boltzmann’s constant, $Z_{0}$ is the circuit’s
characteristic impedance of 50 $\Omega$. The voltage responsivity can be
derived from the complex forward transmission of a resonator [58],

$S_{21}=1-\frac{Q_{r}}{Q_{c}}\frac{1}{1+j2Q_{r}x},$ (2)

where $Q_{r}$ and $Q_{c}$ are the total and coupling quality factors
respectively, and $x$ is the fractional frequency shift. The voltage at the
output of the resonator would be the product of the voltage incident on the
resonator with the transmission, $V_{out}=V_{in}S_{21}(x)$. Appendix A.1 shows
that the resulting voltage responsivity to power is then

$\frac{dV_{out}}{dP}=V_{in}\frac{dx}{dP}\Big{(}\frac{dRe\\{S_{21}\\}}{dx}+j\frac{dIm\\{S_{21}\\}}{dx}\Big{)},$
(3)

where $V_{in}$ is the incident voltage of the bias tone, and $dx/dP$ is the
fractional frequency responsivity to optical power.

The two main readout noise contributors considered here are the LNA and D/A.
We compare the noise at a reference plane after the resonator array and before
the LNA. This is convenient for the LNA as the measured noise temperature
reported is referred to the input of the amplifiers. The amplifiers to be used
in initial deployment have input noise temperatures of approximately 5 K
across the band [59]. The D/A noise however depends on the total attenuation
to the reference plane. The quantization noise power can be estimated from the
variance of the error first derived by Bennett[60] in 1948 as

$P_{digi}=\frac{\Delta
V^{2}}{12Z_{0}}=\frac{1}{12Z_{0}}\frac{V_{FS}^{2}}{2^{2N_{bits}}},$ (4)

where $V_{FS}$ is the full scale voltage of the digitizer, $N_{bits}$ are the
effective number of bits. This can be converted into an equivalent noise
temperature by dividing by the Nyquist bandwidth $f_{s}/2$, Boltzmann’s
constant $k$, and scaling by the total attenuation $A$ to the reference plane
to get,

$T_{digi}=A\frac{P_{digi}}{kf_{s}/2}.$ (5)

Using the measured in-flight median values for the BLAST-TNG 850 GHz array as
a stand in for the Prime-Cam 850GHz arrays, which are currently under
development, we plot the noise temperature vs shift in linewidths in Fig. 1.
The parameters used for Fig. 1 are: total quality factor $Q_{r}=36000$,
coupling quality factor $Q_{c}=58300$, center resonant frequency $f_{0}=800$
MHz, NEP = $10^{-16}$ $W/\sqrt{Hz}$, fractional frequency responsivity
$dx/dP=5.5\times 10^{-6}$ $x/pW$, $V_{FS}=0.7$ V, $N_{bits}=12$, attenuation
from D/A to detectors A = 35 dB, tone power at detectors $P_{tone}=-90$ dBm,
and a sampling frequency fs=512 MHz. Figure 1 plots both components of the
complex output noise temperature from Eq. 1 as $Re$ and $Im$. In typical
operation the complex signal is converted such that the response is mainly in
one quadrature: the output noise temperature of the detectors is $\sim$40K in
the $Im$ quadrature on resonance. As the detector shifts away from its
previous resonant frequency the output noise temperature decreases until
falling to the level of the other contributors. LNA noise and D/A noise are
approximately equal at the output of the detector arrays.

To find the required dynamic range for the readout we must find the maximum
signal to noise measured at the output of the detectors. This is calculated as
the ratio of the bias tone power scaled by the detector transmission to the
output detector noise power,

$S/N=\frac{P_{tone}|S_{21}(x)|^{2}}{kT_{det}}=\frac{|S_{21}(x)|^{2}}{\Big{(}\frac{dx}{dP}\Big{|}\frac{dS_{21}(x)}{dx}\Big{|}NEP\Big{)}^{2}}.$
(6)

The tone power drops out of the equation and the signal to noise is determined
solely by properties of the detector. Plotting Eq. 6 as a function of
linewidth with the same detector parameters from above produces the curve in
Fig. 2. The minimum signal to noise measured at the output of the detectors
occurs on resonance and increases further from this point. The readout systems
dynamic range must be greater than the curve defined by Eq. 6 to not degrade
the total signal to noise. Figure 2 shows that if the readout system has a
measured dynamic range of 100 dB then detector noise limited performance is
possible for shifts of up to $\sim 65\%$ of a linewidth. If the shifts are
greater than that however, the signal to noise will become readout noise
limited.

Figure 1: The effective noise temperature of the KID in both quadratures, LNA
and D/A noise, plotted against the shift from resonance in multiples of a
linewidth. The dashed grey line represents the total noise summing D/A, LNA,
Re, and Im. Figure 2: Signal to noise ratio from Eq. 6 measured at the output
of the detector plotted against the shift from resonance in multiples of a
linewidth. The solid orange section above the curve is considered the detector
noise limited region and anything below or outside would be limited by the
readout. For example if a readout system has a signal to noise of 100 dB
($10^{10}$) then it will be detector noise limited for shifts up to
approximately $\sim$65% of a linewidth.

###  2.2 Channel Crosstalk

Kinetic inductance detectors suffer from electrical crosstalk when their
frequency responses overlap. This is minimized by a combination of accurate
frequency placement, high quality factors, and post-fabrication editing [61,
62]. We can calculate the overlap between two identical resonators which are
frequency neighbors as,

$X_{talk}=\frac{|S_{21}^{n}(f_{0})-1|^{2}}{|S_{21}(f_{0})-1|^{2}}=\frac{1}{1+4N^{2}},$
(7)

where $S_{21}^{n}$ is the forward transmission of the neighboring resonator
evaluated at the resonant frequency $f_{0}$ of the other and normalized by the
on resonance transmission, N is the number of linewidths ($N=Q_{r}x$) of
separation between the adjacent resonators. This electrical crosstalk takes
place at the detector arrays and sets a fundamental requirement for the
digital signal processing to meet. Figure 3 plots Eq. 7 for two different
quality factors against the frequency response of the coarse and fine
channelization stages within the gateware (see Sec. 3.3). The figure shows
that a Fast Fourier Transform (FFT) based coarse channelization with a bin
width of 500 KHz shown in magenta, does not meet the spec of both quality
factors without an additional fine channelization stage. The frequency
response of a fine channelization stage with 1024 accumulations is plotted in
orange displaying sidelobes which fall below the intrinsic detector crosstalk
for both quality factors. This suggests that for high rate applications such
as pulse detection, side lobe minimizing windows or polyphase filterbanks
would be necessary to keep channel leakage below the detector crosstalk
levels.

Figure 3: Accumulated spectrum with detector crosstalk for two different
quality factors overlaid with the frequency response of the coarse and fine
channelization (see Sec. 3.3.5).

##  3 Baseline Design

###  3.1 Overall architecture

The baseline design will use the Xilinx RFSoC which contains reconfigurable
logic fabric, an ARM microprocessor (uPC), and high speed digitizers all
integrated into a single chip. The first generation development board housing
this chip (ZCU111) is the chosen platform for CCAT-prime due to its cost and
familiarity from previous efforts by the author[54, 63]. The digital signal
processing gateware which accomplishes the stimulus and demodulation of the
detectors will sit in the reconfigurable fabric. The ARM uPC will run a
lightweight version of Linux (PetaLinux) as an operating system. The software
which directly interfaces the gateware and performs various array calibration
tasks will reside on this ARM core. Many independently controlled RFSoC ZCU111
development boards will be operating simultaneously in the deployed system,
with all communications occurring through an Ethernet network. A precision and
stable time source will be distributed to each RFSoC to ensure pointing and
detector data synchronization. The overall architecture can be roughly split
into the following sections: hardware, gateware, uPC firmware, uPC software
and networking, and analysis. A block diagram illustrating these systems and
their general interconnections is shown in Fig. 4.

Figure 4: Block diagram of the overall readout architecture showing the main
systems and interconnections between each.

###  3.2 Hardware

Most hardware components of the readout are commercially available but others
require development including, an SMA breakout board for the ZCU111, a
peripheral module sync board, and a rack-mountable enclosure. An optimized
readout also depends on the cryogenic hardware such as the LNA, attenuators,
and coaxial cables. The cryogenic attenuation to the arrays must reduce the
room temperature thermal noise but not reduce the dynamic range of the tones
produced by the D/A. The optimal cryogenic gain from the detectors to the A/D
to prevent saturation should be balanced with available cooling power. See 56
for more information on this for Prime-Cam’s 850 GHz instrument.

The ZCU111 development board and an SMA breakout board will be housed in a 1U
rack-mountable enclosure with Ethernet, power, and SMA ports. Cerro Chajnantor
has half an atmosphere of pressure and the instrument cabin will not be
pressurized. This requires the RFSoC system to dissipate at high altitude
without overheating. A method which was used for the balloon-borne project
BLAST-TNG was to conduction cool with copper heat blocks with water filled
heat pipes dumping the heat to the gondola frame. In this case the heat sink
would be plates on the bottom of the enclosures and the electronics rack
itself. Each enclosure will have 10 SMA ports (x8 for A/D and D/A, x1 for
10MHz reference, x1 for sync input), two 1GbE ports (x1 commanding, x1
streaming UDP detector packets), and one power port.

To allow for analog sync inputs to the FPGA fabric on the ZCU111, a small
board which connects to the Peripheral Modules (PMOD) connector has been
designed. The analog sync could come from a pulse-per-second of a GPS
disciplined oscillator or halfwave plate or pointing sensor trigger. This
signal would be digitized and incorporated with other timing signals such as
free running counters and packets counters into the final data UDP packet.

Each RFSoC ZCU111 development board comes with the XM500 SMA breakout board,
this has a variety of different connections in an effort to allow various use
cases. However we would like to take the differential signals from the RF data
converter pins and pass all of them through low frequency baluns (10 - 3000
MHz) to edge mounted SMA connectors. A prototype PCB of the new design is
currently being tested.

###  3.3 Gateware

The gateware (FPGA firmware) is responsible for generating the detector array
stimulus and producing demuxed, averaged, and downsampled detector data. The
heritage for the design leans heavily on the design for BLAST-TNG [44], and
the pioneering designs of MAKO[29], ARCONS [43], and MUSIC [42]. The digital
signal processing chain consists of a lookup table which stores the frequency
comb, two stage channelization accomplished with a polyphase-filterbank
followed by a digital downconversion and accumulator. This section is split up
between the different modules roughly outlined by the block diagram shown in
Fig. 5.

Figure 5: Block diagram of the digital signal processing gateware and
connection to RF data converters implemented on the RFSoC programmable logic.

####  3.3.1 Frequency Comb Lookup Table

As the main stimulus for the detectors this part of the gateware design is
closely linked to the array properties and their requirements. The frequency
comb must be able to produce tones with high resolution that is significantly
less than the resonator FWHM anywhere within the Nyquist bandwidth. Additional
desirable features are quick re-writing of the waveform and to uniquely assign
phase and amplitudes to each tone. To maximize the available D/A dynamic range
the waveform crest factor and normalization method should be carefully
constructed to not lose more than a typical OFDM peak-to-average power ratio
of 12 dB.

The planned method of frequency comb generation is to utilize the programmable
logic (PL) interfacing DDR4 memory. This is a large memory bank which operates
at four times the PL fabric clock rate thus delivering four simultaneous
samples. The initial reason for this choice is that in order to avoid spurious
signals in the waveform we enforce phase continuity on the wrap around
(meaning when the memory address to read goes from the maximum address to the
minimum). A discontinuity in the time domain increases the spurious floor in
the frequency domain. This can limit the dynamic range of the system and lower
the mux factor. As an example, the baseline design has an effective Nyquist
bandwidth of 512 MHz, and to generate a phase continuous waveform with a tone
resolution of $\Delta f\sim 500$Hz it would require a lookup table size of:

$LUT=f_{s}/\Delta f\approx 1\ \ \textrm{Million.}$ (8)

This is reasonable to place within the FPGA fabric for a single DSP chain but
will not be when implementing four DSP chains. Therefore this must be placed
in an external memory such as the PL connected DDR4.

####  3.3.2 RF Data Converter Configuration - Digital Mixers

The RF data converter in the Zynq UltraScale+ XCZU28DR-2FFVG1517E RFSoC [64]
has a compile time configuration which allows for the use of internal digital
mixers with a numerically controlled oscillator (NCO) as well as decimation
and interpolation filters. This is shown as the RF Data Converter block at the
front of the diagram in Fig. 5. The high speed sampling provide direct RF
digitization without the need for analog mixers, eliminating the need for an
intermediate-frequency (IF) system with analog mixers.

To understand how the RF data converters will be used in the baseline design,
the three processing steps are shown in Fig. 6. In the first step starting on
the left is the directly digitized Nyquist bandwidth of $f_{s}/2$. The orange
block represents the band of interest that contains the resonators. Directly
after digitization the signals get passed to the digital mixer with NCO. This
NCO is tuned to match the center frequency of the band of interest thereby
mixing the band center to DC, this is shown in the middle spectrum of the same
figure. The mixer outputs a complex signal which is a combination of in-phase
and quadrature complex signals which are then filtered and decimated, shown in
the last figure as a factor of 2. This step is critical in that it allows us
to pass only the required bandwidth through the FPGA gateware reducing the
number of parallel paths and resource utilization. The same three steps are
done in reverse for the generation of the frequency comb from the D/A.

Figure 6: Three steps accomplished by the RF data converter for the A/D
starting at the left and for the D/A starting from the right. The first
spectrum is showing the Nyquist bandwidth from 0-$f_{s}/2$ with some band of
interest in orange. The second spectrum is post digital down-conversion to
some frequency at the center of the band of interest accomplished with a
numerically controlled oscillator (NCO). The third is the filtered and
downsampled spectrum. In the reverse order, interpolation and up-conversion to
the D/A.

####  3.3.3 Coarse Channelization

The first step in de-multiplexing is to perform an N-point Fast-Fourier
Transform (FFT) which splits the input timestream into N channels of bandwidth
$f_{samp}/N$, Where $f_{samp}$ is the sampling rate post RF data converter.
Each complex output from the FFT is considered a bin. The frequency response
of a single FFT bin takes a sinx/x form with $x=\pi n$ where n is the ratio of
sample rate and input frequency. This frequency response assures that the
adjacent bin centers are zero at the cost of having a loss of signal at the
bin edge and high side lobes. To prevent this loss and substantially lower the
side band leakage many groups have employed the weighted overlap and add
(WOLA) polyphase filter-bank (PFB). While the WOLA PFB recovers the bin edge
attenuation and provides superior side lobe performance, its typical
implementation requires significantly more resources than simple window
functions. Earlier in Sec. 2.2 we showed that for lower bandwidth detector
readout ( $\sim$ 500 Hz) that employs a fine channelization stage with many
accumulations the frequency response of the coarse stage becomes
insignificant. Future work will compare the PFB vs FFT baseline.

####  3.3.4 Bin Selection

The current frequency scatter for resonators with a 500MHz band centered at
750MHz assures that many will fall within the same bin ($\Delta
f_{bin}=500$kHz). A histogram of the difference between neighboring resonators
from BLAST-TNG’s 350$\mu$m array is shown in Fig. 7. The vertical magenta line
is at 500kHz which corresponds to the FFT bin width and only 50.4$\%$ of the
found resonators are above this line. Thus we require buffering the FFT bins
to select the same bin multiple times. Post-FFT fine channelization can then
be used to recover each resonator to maintain high detector yield. Future
efforts to utilize post fabrication capacitor editing have the potential to
eliminate the need for this module.

Figure 7: Neighboring resonator frequency difference from the BLAST-TNG
350$\mu$m array. The magenta vertical bar is at 500kHz which corresponds to
the FFT bin width. Only $50.4\%$ of the found resonators are above this line;
about half of the resonators share an FFT bin.

####  3.3.5 Fine Channelization

After the coarse channelization and bin selection we perform a fine
channelization in two steps: digital down conversion, and vector accumulation.
The first step is a time-division multiplexed complex multiply of an FFT bin
value with a pre-calculated waveform. This waveform is the complex conjugate
of the beat frequency between a tone and its corresponding FFT bin center
frequency,

$(f_{beat})^{*}=e^{-2\pi i(f_{tone}-f_{bin})t}.$ (9)

After this down conversion, the complex signal no longer oscillates due to
this bin center offset and can now be coherently accumulated. If this step was
not taken the signed values would accumulate quickly to zero.

The signals are then fed into a vector accumulator which accomplishes both an
averaging and downsampling to reduce the output detector bandwidth. The number
of accumulations can be programmed on the fly to allow for dynamic detector
bandwidth sampling.

####  3.3.6 UDP packetizer

The integrated detector data is collected into a dual port memory which safely
crosses a clock domain and then is passed to a UDP packetizer. User Datagram
Protocol (UDP) is a lighter Ethernet packet protocol that does not handshake
like TCP. This allows for reduced gateware complexity and high bandwidth
Ethernet streams. An open-source VHDL implementation of a UDP streaming Media
Access Controller (MAC) from Mike Field is
used111https://github.com/hamsternz/FPGA_GigabitTx. The baseline design will
use a four port 1GbE FMC card by
Opsero/FMCEthernet222https://opsero.com/product/ethernet-fmc/ to
simultaneously stream all four independent channels.

###  3.4 uPC Software and Network

A minimal implementation of the software and network architecture is shown in
Fig. 8. The software and network must be discussed together as the commanding,
diagnostics, and data capture are all occurring on different physical systems
connected via an Ethernet network. Each RFSoC contains an ARM Cortex processor
and will be configured to run the PYNQ image v2.6. The PYNQ image contains a
Linux filesystem and a Python API to easily interact with the FPGA gateware
and other hardware peripherals. Differing from the ROACH2 systems deployed in
various instruments, the CCAT-prime readout takes advantage of the integrated
microprocessors of each RFSoC. Each uPC acts as an independent computing node
performing the majority of software computation and is provided with tasks
from a command server, a model analogous to a distributed computing network.
This method scales easily and eliminates the requirement of a high performance
user terminal to perform the numerous computation tasks simultaneously. It
also cuts down on the required data rate between the user terminal and each
RFSoC. In normal operation only key value pairs will be needed to send command
strings and status bits. Redis is an in-memory database and messaging service
that uses TCP to send and receive messages in a server-client model. While
written in C, Redis has Python bindings that are easy to integrate with the
PYNQ framework. The built in messaging queue capabilities to publish and
subscribe to various channels is ideal for commanding multiple RFSoCs over the
network. The implementation is discussed in detail below.

Figure 8: A block diagram of the software and network architecture for the
CCAT-prime instrument. The software uses a server-client model where the user
terminal acts as the server and each readout ARM uP is a client. The user
command terminal is physically connected to each RFSoC via an Ethernet
network. The commanding Ethernet network is separated from the streaming UDP
packet network. The UDP data is saved to a network attached storage device
which is accessible for near real-time analysis.

####  3.4.1 commander.py & drone.py

The two main Python programs implemented to control the readout are
commander.py and drone.py. Commander.py runs on the user terminal and
publishes commands to Redis, while drone.py runs as a Redis client and
subscribes to the command channel and executes the received commands. This
requires a command list to map the received key-value pair to a command.

Each RF network on the array will be controlled by an independent instance of
drone.py. Each drone is given a unique designation that is linked to a DSP
channel (1-4 in baseline) and RFSoC number in the server rack, these will
eventually be mapped to detector array networks. The main functionality of
drone.py is to execute commands received from subscribed Redis channels and
publish status and exception flags. The required functions can be split
broadly into two main files gateware.py and detector$\\_$functions.py. The
first is dedicated to the calls that are directly interfacing with gateware on
the FPGA for example reading and writing to registers with custom drivers
utilizing the PYNQ library. The detector functions would contain most of the
functionality specific to kinetic inductance detectors, such as creating
frequency combs, find and fit resonances from sweeps, and calculate $\Delta
f$.

###  3.5 Analysis Software

Preliminary analysis software which provides useful data products and quick
looks will be required for pre-deployment tests and commissioning. The first
of which is to convert the raw in-phase and quadrature (I$\&$Q) data streams
to something proportional to power.

The shift in frequency is closely related to the power absorbed and is thus
typically the metric to estimate. There are a few different methods for doing
this and variations of each method: The IQ method which fits the resonance in
the complex plane and moves its center to the origin and phase rotates such
that the frequency response ends up in one quadrature. The change in signal
can then be scaled by the derivatives of the loop with respect to frequency to
convert the timestreams into a frequency shift. Another method finds the
frequency shift assuming small perturbations from resonance which involves
only taking derivatives of the complex sweep with respect to frequency, no
fitting required, this is derived in 65. Typical implementations of these
methods are linear approximations to the frequency shift and thus offer only
limited range of shifts up to $\sim$ 0.65 of a linewidth before accumulating
an error of greater than 10%. An equation which computes the fractional
frequency shift which follows the non-linear resonator response is,

$x(t)=-\frac{Q(t)sin\theta_{0}}{2Q_{r}(I(t)cos\theta_{0}-A)},$ (10)

where the un-modulated phase on resonance is
$\theta_{0}=arctan(Q/I)|_{f_{0}}$, $Q_{r}$ is the total quality factor, and A
is resonator depth. This equation is found by inverting the resonator forward
transmission Eq. 21(see appendix A.2). All three approaches will be explored
in a future study.

A highly desirable goal of the time-domain science is to produce maps with a
cadence of at least one day. While the maps do not need to be fully processed
the important transient information must be easily detectable. The initial
deployment data rate will be around 1 Gb/s which provides a challenging
downstream processing task. A 10 GbE connected computer with a carefully
planned memory access strategy and processing threads will serve as the
analysis and storage computer. We will aim for an approach similar to 66 for
nearly real-time processing to produce quick maps nightly.

##  4 Implementation and Resource Utilization

An important consideration when developing any digital design for an FPGA is
the resource utilization in comparison to the resources available. The four
channel baseline design was synthesised and implemented on the gen 1 RFSoC
providing the following utilization report: CLB LUTs 22$\%$, CLB Regs 18$\%$,
DSPs 6$\%$, Block RAM 17$\%$, URAM 10 $\%$. This is visually represented in
Fig. 9 in a logic fabric view of the chip. Each channel is highlighted along
with the AXI interconnects and DDR4 memory bank. Two things are immediately
apparent from this visual representation: the AXI interconnects are taking a
significant area of the fabric, and there is appears to be ample room to grow
in the north west region of the fabric. The utilization report numbers support
the visual assessment and provide encouraging proof that a doubling of the
instantaneous bandwidth from 500 MHz to 1 GHz should be possible.

Upon closer examination into the resource utilization of a single channel we
see that the PFB dominates the fabric area. Figure 10 gives the device fabric
view of the four channel design with only a single channel highlighted in
magenta and yellow. The yellow highlights solely the PFB resources and the
magenta highlights the remaining parts of the channel. The utilization report
of the PFB gives: 4303 CLB LUTs, 9844 CLB Registers, 675 CARRY8, 1407 CLB,
3375 LUTs as Logic, 928 LUTs as Memory, 27.5 Block RAM Tiles, and 52 DSPs.
From the analysis of the digital crosstalk requirements described earlier in
Sec. 2.2, the front end filterbank may not be necessary given the high number
of accumulations in the fine channelization and expected resonator crosstalk
level. This will allow for larger FFTs within the resource budget.

Figure 9: Resource utilization of the baseline design in device logic cell
view. Implemented four x 1000 channel design on the Zynq UltraScale+ ZCU28DR -
with the four independent channels highlighted along with the DDR4 memory and
AXI interconnects.  Figure 10: Device view of the resource utilization with
only channel 1 highlighted in magenta and yellow. The yellow highlights the
resources used solely for the polyphase filterbank (PFB). The remaining
magenta elements are the rest of the resources used for channel 1. Clearly the
fabric area covered by the PFB dominates the utilization of the channel.

##  5 Demonstrations

The baseline design was set up in RF loopback with an SMA cable connecting a
D/A to it’s corresponding A/D. One thousand tones spanning -250 to 250 MHz at
baseband (pre-mixer) were generated and up-converted to 750 - 1250 MHz with
the numerically controlled local oscillator (NCLO) within the digital mixer
set to 1 GHz. On the receive side after being sampled by the A/D the waveform
is down-converted to baseband by another NCLO set to 1 GHz. Each tone is
assumed to be an independent detector channel and is demodulated and
accumulated within the digital signal processing chain. The output of the last
stage of the signal processing is captured and displayed in Fig. 11. All one
thousand channels are shown and the plot is normalized to the maximum. A
frequency dependent slope is seen in the magnitudes and a roll-off on both
ends of the band. The roll-off is suspected to arise from the half-band up and
down sampling filters frequency response within the RF data converters.
Digital leveling can be achieved by applying the inverse transfer function to
the amplitudes of the tones in the arbitrary waveform.

To characterize the dynamic range and phase noise a loopback measurement was
made with a single chain of the design. The I&Q for each detector channel are
streamed in UDP packets and captured for two minutes with a sample rate of
$\sim 500$ Hz. The phase noise power spectral density is then calculated on
the time-stream either by first calculating the phase time-stream with
$\phi=arctan(Q/I)$ then finding the power spectral density, or by first
calculating the power spectral density of I&Q separately ($S_{II}$ and
$S_{QQ}$) then using the following equation,

$S_{\phi\phi}=\frac{1}{2}\frac{S_{II}+S_{QQ}}{\overline{I}^{2}+\overline{Q}^{2}},$
(11)

Where the sum of the power spectral densities are normalized by the mean
magnitude squared. Figure 12 shows the loopback phase noise power spectral
density in dBc/Hz for various numbers of tones in the arbitrary waveform and
presents the median detector channel from each. A common mode present across
each channel was removed with a single template produced with a principle
component analysis. The inverse of the phase noise power spectral density is
the maximum possible signal to noise or dynamic range achievable for that
channel and specific arbitrary waveform. For 1000 tones this system will be
detector noise limited for shifts of up to $\sim 3/4$ of a linewidth based on
Fig. 2.

Figure 11: RF loopback measurement with the numerically controlled local
oscillator (NCLO) set to 1GHz and a 1000 tone comb. Figure 12: RF loopback
phase noise measurement for various numbers of tones with a common mode
removed.

##  6 Conclusion

The RFSoC based readout for the Mod-Cam and Prime-Cam instruments was
presented. The requirements were derived from expected detector parameters.
The hardware, gateware, and software of the baseline design were described in
detail. The implementation and resource utilization for the design proved that
4x 1000 channel design fit comfortably within the device fabric. This also
revealed that there is space to increase the bandwidth for the 850 GHz
instrument. We demonstrated the operation of the design in RF loopback showing
that the digital mixers can replace analog IQ mixers and that the phase noise
is low enough with 1000 tones to ensure detector noise limited performance for
the anticipated operational range. In the near term, we expect to have an 850
GHz prototype array in our test cryostat at NRC Herzberg to inform the
optimization of all aspects of the RFSoC based system. The first field
deployment of the RFSoC readout will be with the Mod-Cam instrument in 2024.

##  Appendix A Derivations

###  A.1 Equivalent Output Noise Temperature

Starting with the equation for the complex transmission of an LRC resonator
[58],

$S_{21}(f)=1-\frac{Q_{r}}{Q_{c}}\frac{1}{1+j2Q_{r}(f-f_{0})/f_{0}},$ (12)

Where $f$ is frequency, $f_{0}$ is the resonant frequency, $Q_{r}$ is the
total quality factor, $Q_{c}$ is the coupling quality factor. Rewriting the
fractional frequency as $(f-f_{0})/f_{0}=x$ and splitting the above equation
into real and imaginary components we get,

$Re\\{S_{21}\\}=1-\frac{Q_{r}}{Q_{c}(1+4Q_{r}^{2}x^{2})},$ (13)

and

$Im\\{S_{21}\\}=-\frac{2Q_{r}^{2}x}{Q_{c}(1+4Q_{r}^{2}x^{2})}.$ (14)

Taking the derivative of each with respect to fractional frequency $x$,

$\frac{dRe\\{S_{21}\\}}{dx}=\frac{8Q_{r}^{3}x}{Q_{c}(1+4Q_{r}^{2}x^{2})^{2}},$
(15)
$\frac{dIm\\{S_{21}\\}}{dx}=\frac{2Q_{r}^{2}}{Q_{c}(1+4Q_{r}^{2}x^{2})}-\frac{16Q_{r}^{4}x^{2}}{Q_{c}(1+4Q_{r}^{2}x^{2})^{2}}.$
(16)

In typical operation the measured output voltage depends on the probe tone
input voltage and the forward transmission,

$V_{out}=V_{in}S_{21}(x).$ (17)

The absorption of optical photons will shift the resonant frequency modulating
the transmission. Taking the derivative of the above equation to find the
change in output voltage with respect to absorbed optical power or voltage
responsivity,

$\frac{dV_{out}}{dP}=V_{in}\frac{dS_{21}}{dP}=V_{in}\frac{dS_{21}}{dx}\frac{dx}{dP}.$
(18)

Expanding the complex terms,

$\frac{dV_{out}}{dP}=V_{in}\frac{dx}{dP}\Big{(}\frac{dRe\\{S_{21}\\}}{dx}+j\frac{dIm\\{S_{21}\\}}{dx}\Big{)}.$
(19)

Following [57] we can then calculate the noise temperature as,

$T=\frac{(R\times NEP)^{2}}{kZ_{0}},$ (20)

where NEP is the detectors noise equivalent power, k is boltzmann’s constant,
$Z_{0}$ is the characteristic impedance, and $R=dV_{out}/dP$ is the voltage
responsivity.

###  A.2 Fractional Frequency Shift by Inverse Method

The equation for the complex forward transmission of a shunt resonator is,

$S_{21}=1-\frac{Q_{r}}{Q_{c}}\frac{1}{1+j2Q_{r}(f-f_{0})/f_{0}},$ (21)

Where $Q_{r}$ is the total quality factor, $Q_{c}$ is the coupling quality
factor, $f_{0}$ is the unmodulated resonant frequency. The term
$(f-f_{0})/f_{0}$ is the fractional frequency shift commonly written as $x$.
The fractional frequency shift $x$ can be directly related to the power
absorbed through measurements of the responsivity. Fixed tone readout systems
estimate this value by measuring the instantaneous complex forward
transmission at the unmodulated resonant frequency $f_{0}$. These complex
values are named the in-phase $I$ and quadrature $Q$ components of the
transmission,

$S_{21}(f_{0})=Re\\{S_{21}(f_{0})\\}+jIm\\{S_{21}(f_{0})\\}=I(t)+jQ(t).$ (22)

We seek an equation which gives the fractional frequency $x$ with the measured
I/Q signals as an input. This requires inverting Eq. 21 and using above
relation to I/Q.

Equation 21 can be split into real and imaginary components as,

$S_{21}=1-\frac{Q_{r}}{Q_{c}(1+4Q_{r}^{2}x^{2})}+j\frac{2Q_{r}^{2}x}{Q_{c}(1+4Q_{r}^{2}x^{2})}.$
(23)

This makes it easier to see that subtracting 1 from the real term and dividing
by the imaginary term simplifies the equation to,

$\frac{Re\\{S_{21}\\}-1}{Im\\{S_{21}\\}}=-\frac{1}{2Q_{r}x}.$ (24)

Now using the relation from Eq. 22 and solving for x,

$x=-\frac{Q(t)}{2Q_{r}(I(t)-1)}.$ (25)

This equation assumes that a shift in resonant frequency is equivalent to a
shift of stimulus tone frequency in the opposite direction. It’s also an
idealized symmetric resonator which has been normalized and phase rotated such
that the imaginary component on resonance is zero.

In practice the resonator measurements have some arbitrary magnitude scaling
and phase rotation angle. Including these effects the fractional frequency
shift equation becomes,

$x=-\frac{Q(t)sin\theta_{0}}{2Q_{r}(I(t)cos\theta_{0}-A)}.$ (26)

Where A is the resonator depth, $\theta_{0}$ is the phase angle at the
resonant frequency $f_{0}$ found from $\theta_{0}=arctan(Q/I)$.

###### Acknowledgements.

The CCAT-prime project, FYST and Prime-Cam instrument have been supported by
generous contributions from the Fred M. Young, Jr. Charitable Trust, Cornell
University, and the Canada Foundation for Innovation and the Provinces of
Ontario, Alberta, and British Columbia. The construction of the FYST telescope
was supported by the Großgeräte-Programm of the German Science Foundation
(Deutsche Forschungsgemeinschaft, DFG) under grant INST 216/733-1 FUGG, as
well as funding from Universität zu Köln, Universität Bonn and the Max Planck
Institut für Astrophysik, Garching. Prime-Cam: The construction of EoR-Spec is
supported by NSF grant AST-2009767. The construction of the 350 GHz instrument
module for Prime-Cam is supported by NSF grant AST-2117631. Chai: The CHAI
instrument is supported by DFG grant CRC 956/3, project ID 184018867 as well
as funding from Universität zu Köln. The author would also like to thank Sam
Rowe, Jack Hickish, Jenny Smith, Jeb Bailey III, Leandro Stefanazzi, and
Jozsef (Mazsi) Imrek for collaborative readout discussions, Ed Chapin, Lewis
Knee, Doug Henke, Frank Jiang, and Doug Johnstone of the Herzberg Astrophysics
and Astronomy Research Centre for use of the mm-wave lab equipment and sage
advise.

## References

  * [1] CCAT-Prime collaboration, Aravena, M., Austermann, J. E., Basu, K., Battaglia, N., Beringue, B., Bertoldi, F., Bigiel, F., Bond, J. R., Breysse, P. C., Broughton, C., Bustos, R., Chapman, S. C., Charmetant, M., Choi, S. K., Chung, D. T., Clark, S. E., Cothard, N. F., Crites, A. T., Dev, A., Douglas, K., Duell, C. J., Ebina, H., Erler, J., Fich, M., Fissel, L. M., Foreman, S., Gao, J., García, P., Giovanelli, R., Haynes, M. P., Hensley, B., Herter, T., Higgins, R., Huber, Z., Hubmayr, J., Johnstone, D., Karoumpis, C., Keating, L. C., Komatsu, E., Li, Y., Magnelli, B., Matthews, B. C., Meerburg, P. D., Meyers, J., Muralidhara, V., Murray, N. W., Niemack, M. D., Nikola, T., Okada, Y., Riechers, D. A., Rosolowsky, E., Roy, A., Sadavoy, S. I., Schaaf, R., Schilke, P., Scott, D., Simon, R., Sinclair, A. K., Sivakoff, G. R., Stacey, G. J., Stutz, A. M., Stutzki, J., Tahani, M., Thanjavur, K., Timmermann, R. A., Ullom, J. N., van Engelen, A., Vavagiakis, E. M., Vissers, M. R., Wheeler, J. D., White, S. D. M., Zhu, Y., and Zou, B., “CCAT-prime Collaboration: Science Goals and Forecasts with Prime-Cam on the Fred Young Submillimeter Telescope,” arXiv e-prints , arXiv:2107.10364 (July 2021).
  * [2] Vavagiakis, E. M., Ahmed, Z., Ali, A., Basu, K., Battaglia, N., Bertoldi, F., Bond, R., Bustos, R., Chapman, S. C., Chung, D., Coppi, G., Cothard, N. F., Dicker, S., Duell, C. J., Duff, S. M., Erler, J., Fich, M., Galitzki, N., Gallardo, P. A., Henderson, S. W., Herter, T. L., Hilton, G., Hubmayr, J., Irwin, K. D., Koopman, B. J., McMahon, J., Murray, N., Niemack, M. D., Nikola, T., Nolta, M., Orlowski-Scherer, J., Parshley, S. C., Riechers, D. A., Rossi, K., Scott, D., Sierra, C., Silva-Feaver, M., Simon, S. M., Stacey, G. J., Stevens, J. R., Ullom, J. N., Vissers, M. R., Walker, S., Wollack, E. J., Xu, Z., and Zhu, N., “Prime-Cam: a first-light instrument for the CCAT-prime telescope,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy IX ], Zmuidzinas, J. and Gao, J.-R., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10708, 107081U (July 2018).
  * [3] Choi, S. K., Duell, C. J., Austermann, J., Cothard, N. F., Gao, J., Freundt, R. G., Groppi, C., Herter, T., Hubmayr, J., Huber, Z. B., Keller, B., Li, Y., Mauskopf, P., Niemack, M. D., Nikola, T., Rossi, K., Sinclair, A., Stacey, G. J., Vavagiakis, E. M., Vissers, M., Tucker, C., Weeks, E., and Wheeler, J., “CCAT-prime: Characterization of the First 280 GHz MKID Array for Prime-Cam,” arXiv e-prints , arXiv:2111.01055 (Nov. 2021).
  * [4] Chapman, S.C. et al., “CCAT-prime: The 850GHz camera for Prime-Cam on FYST.”  SPIE 2022, Paper 12190-7.
  * [5] Cothard, N. F., Choi, S. K., Duell, C. J., Herter, T., Hubmayr, J., McMahon, J., Niemack, M. D., Nikola, T., Sierra, C., Stacey, G. J., Vavagiakis, E. M., Wollack, E. J., and Zou, B., “The design of the CCAT-prime epoch of reionization spectrometer instrument,” Journal of Low Temperature Physics 199, 898–907 (jan 2020).
  * [6] Nikola, T. et al., “CCAT-prime: The Epoch of Reionization Spectrometer for Prime-Cam on FYST.”  SPIE 2022, Paper 12190-26.
  * [7] Niemack, M. D., “Designs for a large-aperture telescope to map the cmb 10x faster,” Appl. Opt. 55, 1688–1696 (Mar 2016).
  * [8] Parshley, S. C., Kronshage, J., Blair, J., Herter, T., Nolta, M., Stacey, G. J., Bazarko, A., Bertoldi, F., Bustos, R., Campbell, D. B., Chapman, S., Cothard, N., Devlin, M., Erler, J., Fich, M., Gallardo, P. A., Giovanelli, R., Graf, U., Gramke, S., Haynes, M. P., Hills, R., Limon, M., Mangum, J. G., McMahon, J., Niemack, M. D., Nikola, T., Omlor, M., Riechers, D. A., Steeger, K., Stutzki, J., and Vavagiakis, E. M., “CCAT-prime: a novel telescope for sub-millimeter astronomy,” in [Ground-based and Airborne Telescopes VII ], Marshall, H. K. and Spyromilio, J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10700, 107005X (July 2018).
  * [9] Duell, C. J., Vavagiakis, E. M., Austermann, J., Chapman, S. C., Choi, S. K., Cothard, N. F., Dober, B., Gallardo, P., Gao, J., Groppi, C., Herter, T. L., Stacey, G. J., Huber, Z., Hubmayr, J., Johnstone, D., Li, Y., Mauskopf, P., McMahon, J., Niemack, M. D., Nikola, T., Rossi, K., Simon, S., Sinclair, A. K., Vissers, M., Wheeler, J., and Zou, B., “CCAT-prime: Designs and status of the first light 280 GHz MKID array and mod-cam receiver,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11453, 114531F (Dec. 2020).
  * [10] Day, P. K., LeDuc, H. G., Mazin, B. A., Vayonakis, A., and Zmuidzinas, J., “A broadband superconducting detector suitable for use in large arrays,” Nature 425, 817–821 (Oct. 2003).
  * [11] Mazin, B. A., Bumble, B., Day, P. K., Eckart, M. E., Golwala, S., Zmuidzinas, J., and Harrison, F. A., “Position sensitive x-ray spectrophotometer using microwave kinetic inductance detectors,” Applied Physics Letters 89(22), 222507 (2006).
  * [12] Szypryt, P., Development of Microwave Kinetic Inductance Detectors for Applications in Optical to Near-IR Astronomy, PhD thesis, University of California, Santa Barbara (Jan. 2017).
  * [13] de Visser, P. J., de Rooij, S. A., Murugesan, V., Thoen, D. J., and Baselmans, J. J., “Phonon-trapping-enhanced energy resolution in superconducting single-photon detectors,” Phys. Rev. Applied 16, 034051 (Sep 2021).
  * [14] Yates, S. J. C., Baselmans, J. J. A., Endo, A., Janssen, R. M. J., Ferrari, L., Diener, P., and Baryshev, A. M., “Photon noise limited radiation detection with lens-antenna coupled microwave kinetic inductance detectors,” Applied Physics Letters 99, 073505 (Aug. 2011).
  * [15] Hubmayr, J., Beall, J. A., Becker, D., Brevik, J. A., Cho, H. M., Che, G., Devlin, M., Dober, B., Gao, J., Galitzki, N., Hilton, G. C., Irwin, K. D., Li, D., Mauskopf, P., Pappas, D. P., Van Lanen, J., and Vissers, M. R., “Dual-Polarization-Sensitive Kinetic Inductance Detectors for Balloon-borne Sub-millimeter Polarimetry,” Journal of Low Temperature Physics 176, 490–496 (Aug. 2014).
  * [16] Mauskopf, P. D., Doyle, S., Barry, P., Rowe, S., Bidead, A., Ade, P. A. R., Tucker, C., Castillo, E., Monfardini, A., Goupy, J., and Calvo, M., “Photon-Noise Limited Performance in Aluminum LEKIDs,” Journal of Low Temperature Physics 176, 545–552 (Aug. 2014).
  * [17] Dober, B., Austermann, J. A., Beall, J. A., Becker, D., Che, G., Cho, H. M., Devlin, M., Duff, S. M., Galitzki, N., Gao, J., Groppi, C., Hilton, G. C., Hubmayr, J., Irwin, K. D., McKenney, C. M., Li, D., Lourie, N., Mauskopf, P., Vissers, M. R., and Wang, Y., “Optical Demonstration of THz, Dual-Polarization Sensitive Microwave Kinetic Inductance Detectors,” Journal of Low Temperature Physics 184, 173–179 (July 2016).
  * [18] Flanigan, D., McCarrick, H., Jones, G., Johnson, B. R., Abitbol, M. H., Ade, P., Araujo, D., Bradford, K., Cantor, R., Che, G., Day, P., Doyle, S., Kjellstrand, C. B., Leduc, H., Limon, M., Luu, V., Mauskopf, P., Miller, A., Mroczkowski, T., Tucker, C., and Zmuidzinas, J., “Photon noise from chaotic and coherent millimeter-wave sources measured with horn-coupled, aluminum lumped-element kinetic inductance detectors,” Applied Physics Letters 108(8), 083504 (2016).
  * [19] Levy-Bertrand, F., Benoît, A., Bourrion, O., Calvo, M., Catalano, A., Goupy, J., Valenti, F., Maleeva, N., Grünhaupt, L., Pop, I. M., and Monfardini, A., “Subgap Kinetic Inductance Detector Sensitive to 85-GHz Radiation,” Physical Review Applied 15, 044002 (Mar. 2021).
  * [20] Zhao, S., Withington, S., Goldie, D. J., and Thomas, C. N., “Suppressed-gap millimetre wave kinetic inductance detectors using DC-bias current,” Journal of Physics D Applied Physics 53, 345301 (Aug. 2020).
  * [21] Aja, B., de Ory, M. C., Fuente, L. d. l., Artal, E., Pascual, J. P., Magaz, M. T., Granados, D., and Gomez, A., “Analysis and performance of lumped-element kinetic inductance detectors for w-band,” IEEE Transactions on Microwave Theory and Techniques 69(1), 578–589 (2021).
  * [22] Day, P., Leduc, H., Goldin, A., Vayonakis, T., Mazin, B., Kumar, S., Gao, J., and Zmuidzinas, J., “Antenna-coupled microwave kinetic inductance detectors,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 559(2), 561–563 (2006).  Proceedings of the 11th International Workshop on Low Temperature Detectors.
  * [23] Barry, P. S., Doyle, S., Hornsby, A. L., Kofman, A., Mayer, E., Nadolski, A., Tang, Q. Y., Vieira, J., and Shirokoff, E., “Design and Performance of the Antenna-Coupled Lumped-Element Kinetic Inductance Detector,” Journal of Low Temperature Physics 193, 176–183 (Nov. 2018).
  * [24] McCarrick, H., Flanigan, D., Jones, G., Johnson, B. R., Ade, P., Araujo, D., Bradford, K., Cantor, R., Che, G., Day, P., Doyle, S., Leduc, H., Limon, M., Luu, V., Mauskopf, P., Miller, A., Mroczkowski, T., Tucker, C., and Zmuidzinas, J., “Horn-coupled, commercially-fabricated aluminum lumped-element kinetic inductance detectors for millimeter wavelengths,” Review of Scientific Instruments 85(12), 123117 (2014).
  * [25] Bryan, S., Bradford, K., Che, G., Day, P., Flanigan, D., Johnson, B. R., Jones, G., Kjellstrand, B., Limon, M., Mauskopf, P., McCarrick, H., Miller, A., and Smiley, B., “Design of Dual-Polarization Horn-Coupled Kinetic Inductance Detectors for Cosmic Microwave Background Polarimetry,” arXiv e-prints , arXiv:1503.04684 (Mar. 2015).
  * [26] Austermann, J., Beall, J., Bryan, S., Dober, B., Gao, J., Hilton, G., Hubmayr, J., Mauskopf, P., McKenney, C., Simon, S., Ullom, J., Vissers, M., and Wilson, G., “Millimeter-wave polarimeters using kinetic inductance detectors for toltec and beyond,” Journal of Low Temperature Physics 193, 120–127 (Nov. 2018).  Funding Information: Acknowledgements This work is supported by the NSF of the USA through Grant MSIP-1636621 and NASA through Grant APRA13-0083. Publisher Copyright: © 2018, This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection.
  * [27] Roesch, M., Benoi, A., Bideaud, A., Boudou, N., Calvo, M., Cruciani, A., Doyle, S., Leduc, H., Monfardini, A., Swenson, L., Leclercq, S., Mauskopf, P., and Schuster, K., “Development of lumped element kinetic inductance detectors for nika,” in [22nd International Symposium on Space Terahertz Technology 2011, ISSTT 2011 ], 41–45, International Symposium on Space Terahertz Technology (2011).  22nd International Symposium on Space Terahertz Technology 2011, ISSTT 2011 ; Conference date: 25-04-2011 Through 28-04-2011.
  * [28] Doyle, S., Mauskopf, P., Naylon, J., Porch, A., and Duncombe, C., “Lumped Element Kinetic Inductance Detectors,” Journal of Low Temperature Physics 151, 530–536 (Apr. 2008).
  * [29] Swenson, L. J., Day, P. K., Dowell, C. D., Eom, B. H., Hollister, M. I., Jarnot, R., Kovács, A., Leduc, H. G., McKenney, C. M., Monroe, R., Mroczkowski, T., Nguyen, H. T., and Zmuidzinas, J., “MAKO: a pathfinder instrument for on-sky demonstration of low-cost 350 micron imaging arrays,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy VI ], Holland, W. S. and Zmuidzinas, J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 8452, 84520P (Sept. 2012).
  * [30] Monfardini, A., Benoit, A., Bideaud, A., Swenson, L., Cruciani, A., Camus, P., Hoffmann, C., Désert, F. X., Doyle, S., Ade, P., Mauskopf, P., Tucker, C., Roesch, M., Leclercq, S., Schuster, K. F., Endo, A., Baryshev, A., Baselmans, J. J. A., Ferrari, L., Yates, S. J. C., Bourrion, O., Macias-Perez, J., Vescovi, C., Calvo, M., and Giordano, C., “A Dual-band Millimeter-wave Kinetic Inductance Camera for the IRAM 30 m Telescope,” ApJS 194, 24 (June 2011).
  * [31] Adam, R., Adane, A., Ade, P. A. R., André, P., Andrianasolo, A., Aussel, H., Beelen, A., Benoît, A., Bideaud, A., Billot, N., Bourrion, O., Bracco, A., Calvo, M., Catalano, A., Coiffard, G., Comis, B., De Petris, M., Désert, F. X., Doyle, S., Driessen, E. F. C., Evans, R., Goupy, J., Kramer, C., Lagache, G., Leclercq, S., Leggeri, J. P., Lestrade, J. F., Macías-Pérez, J. F., Mauskopf, P., Mayet, F., Maury, A., Monfardini, A., Navarro, S., Pascale, E., Perotto, L., Pisano, G., Ponthieu, N., Revéret, V., Rigby, A., Ritacco, A., Romero, C., Roussel, H., Ruppin, F., Schuster, K., Sievers, A., Triqueneaux, S., Tucker, C., and Zylka, R., “The NIKA2 large-field-of-view millimetre continuum camera for the 30 m IRAM telescope,” Astronomy and Astrophysics 609, A115 (Jan. 2018).
  * [32] Coppi, G., Ade, P. A. R., Ashton, P. C., Austermann, J. E., Cox, E. G., Devlin, M. J., Dober, B. J., Fanfani, V., Fissel, L. M., Galitzki, N. B., Gao, J., Gordon, S., Groppi, C. E., Hilton, G. C., Hubmayr, J., Klein, J., Li, D., Lowe, I., Lourie, N. P., McKinney, C., Mani, H., Mauskopf, P. D., Nati, F., Novak, G., Pisano, G., Romualdez, J. L., Sinclair, A. K., Soler, J. D., Tucker, C., Ullom, J., Vissers, M., Wheeler, C., and Williams, P. A., “In-flight performance of the BLAST-TNG telescope platform,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11445, 1144526 (Dec. 2020).
  * [33] Paiella, A., Ade, P. A. R., Battistelli, E. S., Castellano, M. G., Colantoni, I., Columbro, F., Coppolecchia, A., D’Alessandro, G., de Bernardis, P., De Petris, M., Gordon, S., Lamagna, L., Magneville, C., Masi, S., Mauskopf, P., Pettinari, G., Piacentini, F., Pisano, G., Polenta, G., Presta, G., Tommasi, E., Tucker, C., Vdovin, V., Volpe, A., and Yvon, D., “In-Flight Performance of the LEKIDs of the OLIMPO Experiment,” Journal of Low Temperature Physics 199, 491–501 (Feb. 2020).
  * [34] Wilson, G. W., Abi-Saad, S., Ade, P., Aretxaga, I., Austermann, J., Ban, Y., Bardin, J., Beall, J., Berthoud, M., Bryan, S., Bussan, J., Castillo, E., Chavez, M., Contente, R., DeNigris, N. S., Dober, B., Eiben, M., Ferrusca, D., Fissel, L., Gao, J., Golec, J. E., Golina, R., Gomez, A., Gordon, S., Gutermuth, R., Hilton, G., Hosseini, M., Hubmayr, J., Hughes, D., Kuczarski, S., Lee, D., Lunde, E., Ma, Z., Mani, H., Mauskopf, P., McCrackan, M., McKenney, C., McMahon, J., Novak, G., Pisano, G., Pope, A., Ralston, A., Rodriguez, I., Sánchez-Argüelles, D., Schloerb, F. P., Simon, S., Sinclair, A., Souccar, K., Campos, A. T., Tucker, C., Ullom, J., Camp, E. V., Lanen, J. V., Velazquez, M., Vissers, M., Weeks, E., and Yun, M. S., “The TolTEC camera: an overview of the instrument and in-lab testing results,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy X ], Zmuidzinas, J. and Gao, J.-R., eds., 11453, 1 – 20, International Society for Optics and Photonics, SPIE (2020).
  * [35] Brien, T. L. R., Ade, P. A. R., Barry, P. S., Castillo-Domìnguez, E., Ferrusca, D., Gascard, T., Gómez, V., Hargrave, P. C., Hornsby, A. L., Hughes, D., Pascale, E., Parrianen, J. D. A., Perez, A., Rowe, S., Tucker, C., González, S. V., and Doyle, S. M., “MUSCAT: the Mexico-UK Sub-Millimetre Camera for AsTronomy,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy IX ], Zmuidzinas, J. and Gao, J.-R., eds., 10708, 67 – 75, International Society for Optics and Photonics, SPIE (2018).
  * [36] Mazin, B. A., Day, P. K., Irwin, K. D., Reintsema, C. D., and Zmuidzinas, J., “Digital readouts for large microwave low-temperature detector arrays,” Nuclear Instruments and Methods in Physics Research A 559, 799–801 (Apr. 2006).
  * [37] Yates, S. J. C., Baryshev, A. M., Baselmans, J. J. A., Klein, B., and Güsten, R., “FFTS readout for large arrays of Microwave Kinetic Inductance Detectors,” arXiv e-prints , arXiv:0903.2431 (Mar. 2009).
  * [38] Klein, B., Philipp, S. D., Güsten, R., Krämer, I., and Samtleben, D., “A new generation of spectrometers for radio astronomy: fast Fourier transform spectrometer,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Zmuidzinas, J., Holland, W. S., Withington, S., and Duncan, W. D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 6275, 627511 (June 2006).
  * [39] Bourrion, O., Bideaud, A., Benoit, A., Cruciani, A., Macias-Perez, J. F., Monfardini, A., Roesch, M., Swenson, L., and Vescovi, C., “Electronics and data acquisition demonstrator for a kinetic inductance camera,” Journal of Instrumentation 6, 6012 (June 2011).
  * [40] Bourrion, O., Benoit, A., Bouly, J. L., Bouvier, J., Bosson, G., Calvo, M., Catalano, A., Goupy, J., Li, C., Macías-Pérez, J. F., Monfardini, A., Tourres, D., Ponchant, N., and Vescovi, C., “NIKEL_AMC: readout electronics for the NIKA2 experiment,” Journal of Instrumentation 11, P11001 (Nov. 2016).
  * [41] van Rantwijk, J., Grim, M., van Loon, D., Yates, S., Baryshev, A., and Baselmans, J., “Multiplexed Readout for 1000-Pixel Arrays of Microwave Kinetic Inductance Detectors,” IEEE Transactions on Microwave Theory Techniques 64, 1876–1883 (June 2016).
  * [42] Duan, R. and Golwala, S., “Readout for kinetic-inductance-detector-based submillimeter radio astronomy,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11453, 114531Z (Dec. 2020).
  * [43] McHugh, S., Mazin, B. A., Serfass, B., Meeker, S., O’Brien, K., Duan, R., Raffanti, R., and Werthimer, D., “A readout for large arrays of microwave kinetic inductance detectors,” Review of Scientific Instruments 83(4), 044702 (2012).
  * [44] Gordon, S., Dober, B., Sinclair, A., Rowe, S., Bryan, S., Mauskopf, P., Austermann, J., Devlin, M., Dicker, S., Gao, J., Hilton, G. C., Hubmayr, J., Jones, G., Klein, J., Lourie, N. P., McKenney, C., Nati, F., Soler, J. D., Strader, M., and Vissers, M., “An open source, fpga-based lekid readout for blast-tng: Pre-flight results,” Journal of Astronomical Instrumentation 05(04), 1641003 (2016).
  * [45] Strader, M. J., Digitial readout for microwave kinetic inductance detectors and applications in high time resolution astronomy, PhD thesis, University of California, Santa Barbara (Jan. 2016).
  * [46] Fruitwala, N., Strader, P., Cancelo, G., Zmuda, T., Treptow, K., Wilcer, N., Stoughton, C., Walter, A. B., Zobrist, N., Collura, G., Lipartito, I., Bailey, J. I., and Mazin, B. A., “Second generation readout for large format photon counting microwave kinetic inductance detectors,” Review of Scientific Instruments 91(12), 124705 (2020).
  * [47] Paiella, A., Battistelli, E. S., Castellano, M. G., Colantoni, I., Columbro, F., Coppolecchia, A., D’Alessandro, G., de Bernardis, P., Gordon, S., Lamagna, L., Mani, H., Masi, S., Mauskopf, P., Pettinari, G., Piacentini, F., and Presta, G., “Kinetic inductance detectors and readout electronics for the OLIMPO experiment,” Journal of Physics: Conference Series 1182, 012005 (feb 2019).
  * [48] Tapia, M. B., Readout and characterization system for kinetic inductance detectors (KID) of the MUSCAT project, Master’s thesis, Instituto Nacional de Astrofísica, Óptica y Electrónica, Tonantzintla, Puebla (Dec. 2019).
  * [49] Redford, J., Barry, P. S., Bradford, C., Glenn, J., Hailey-Dunsheath, S., Janssen, R. M., Karkare, K., LeDuc, H. G., Mauskopf, P., McGeehan, R., Shirokoff, E., Wheeler, J., and Zmuidzinas, J., “SuperSpec: Device Characterization and Preparation for Telescope Deployment and Observations,” in [American Astronomical Society Meeting Abstracts ], American Astronomical Society Meeting Abstracts 53, 317.01 (June 2021).
  * [50] Smith, J. P., Bailey, J. I., Tuthill, J., Stefanazzi, L., Cancelo, G., Treptow, K., and Mazin, B. A., “A high-throughput oversampled polyphase filter bank using vivado hls and pynq on a rfsoc,” IEEE Open Journal of Circuits and Systems 2, 241–252 (2021).
  * [51] Baldwin, E., Lucia, M. D., Bracken, C., Ulbricht, G., and Ray, T., “Frequency domain multiplexing with the Xilinx ZCU111 RFSoC board,” in [X-Ray, Optical, and Infrared Detectors for Astronomy IX ], Holland, A. D. and Beletic, J., eds., 11454, International Society for Optics and Photonics, SPIE (2020).
  * [52] Stefanazzi, L., Treptow, K., Wilcer, N., Stoughton, C., Montella, S., Bradford, C., Cancelo, G., Saxena, S., Arnaldi, H., Sussman, S., Houck, A., Agrawal, A., Zhang, H., Ding, C., and Schuster, D. I., “The QICK (Quantum Instrumentation Control Kit): Readout and control for qubits and detectors,” arXiv e-prints , arXiv:2110.00557 (Oct. 2021).
  * [53] Bradley, D. C., Jamison-Hooks, T. L., Staguhn, J., Amatucci, E., Browning, T., Dipirro, M., Leisawitz, D., and Carter, R., “On the advancements of digital signal processing hardware and algorithms enabling the Origins Space Telescope,” Journal of Astronomical Telescopes, Instruments, and Systems 7, 011018 (Jan. 2021).
  * [54] Sinclair, A., Stephenson, R., Hoh, J., Gordon, S., and Mauskopf, P., “On the development of a reconfigurable readout for superconducting arrays,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy X ], Zmuidzinas, J. and Gao, J.-R., eds., Proceedings of SPIE - The International Society for Optical Engineering, SPIE (2020).  Funding Information: This work has been funded by NASA PCOS and COS grant number 18-SAT18-0025 Publisher Copyright: © COPYRIGHT SPIE. Downloading of the abstract is permitted for personal use only.; Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy X 2020 ; Conference date: 14-12-2020 Through 22-12-2020.
  * [55] Frisch, J. and O’Connor, P., “RF Electronics,” arXiv e-prints , arXiv:2204.01809 (Apr. 2022).
  * [56] Huber, A.I. et al., “Optical and cryogenic design of the 850 GHz module for Prime-Cam.”  SPIE 2022, Paper 12190-93.
  * [57] Sipola, H., Luomahaara, J., Timofeev, A., Grönberg, L., Rautiainen, A., Luukanen, A., and Hassel, J., “Multiplexed readout of kinetic inductance bolometer arrays,” Review of Scientific Instruments 90, 074702 (July 2019).
  * [58] Mauskopf, P. D., “Transition Edge Sensors and Kinetic Inductance Detectors in Astronomical Instruments,” PASP 130, 082001 (Aug. 2018).
  * [59] Mani, H. and Mauskopf, P., “A single-stage cryogenic lna with low power consumption using a commercial sige hbt,” in [2014 11th International Workshop on Low Temperature Electronics (WOLTE) ], 17–20 (2014).
  * [60] Bennett, W. R., “Spectra of quantized signals,” The Bell System Technical Journal 27(3), 446–472 (1948).
  * [61] Liu, X., Guo, W., Wang, Y., Dai, M., Wei, L. F., Dober, B., McKenney, C. M., Hilton, G. C., Hubmayr, J., Austermann, J. E., Ullom, J. N., Gao, J., and Vissers, M. R., “Superconducting micro-resonator arrays with ideal frequency spacing,” Applied Physics Letters 111(25), 252601 (2017).
  * [62] Shroyer, J. E., Nelson, M., Walters, L., and Johnson, B. R., “A scalable led module for addressing kinetic inductance detector arrays,” (2022).
  * [63] Sinclair, A., In-Flight Performance of the BLAST-TNG Balloon-Borne Far-Infrared Telescope and Development of a Reconfigurable Readout on the RFSoC Platform for Frequency Multiplexed Superconducting Detector Arrays, PhD thesis, Arizona State University (2021).
  * [64] Xilinx, “RFSoC Data Converter Evaluation Tool User Guide UG1287 (v2018.3).” Xilinx, 5 December 2018 https://www.xilinx.com/content/dam/xilinx/support/documentation/boards_and_kits/zcu111/2018_3/ug1287-zcu111-rfsoc-eval-tool.pdf.  (Accessed: 29 December 2021).
  * [65] Barry, P. S., On the development of SuperSpec: a fully integrated on-chip spectrometer for far-infrared astronomy, PhD thesis, Cardiff University, United Kingdom (Jan. 2014).
  * [66] Smith, M., Kerley, D., Chapin, E. L., Dunn, J., Herriot, G., Véran, J.-P., Boyer, C., Ellerbroek, B., Gilles, L., and Wang, L., “Thirty Meter Telescope narrow-field infrared adaptive optics system real-time controller prototyping results,” in [Adaptive Optics Systems V ], Marchetti, E., Close, L. M., and Véran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9909, 99094N (July 2016).