and B positioners (see §2) had been installed on the petals, therefore some
deterioration had occurred. The root cause, discussed in section 6.3, was not
uncovered until 2021, two years after the petals had been installed on the
telescope.

Despite these issues, overall each petal performed excellently during the lab
tests, with $>$ 97% of all installed positioners meeting requirements. In
total, prior to shipment, we found 17 broken fibers. We disconnected 32
positioners from the petal controller, mostly due to being unresponsive or
causing CAN bus issues. There were 96 positioners that were disabled but not
disconnected because they did not meet performance requirements. For the
positioners that weren’t disabled, their maximum blind move accuracy was
$\leq$200 µm and their RMS error after the third correction move was $\leq$5
µm over the 24 grid moves.

Prior to shipment, each petal underwent functional testing. This test set a
baseline for the expected functionality of the petal and was repeated several
times: pre-installation on the shipping crate, post-installation on the
shipping crate, post-shipment to Kitt Peak, and post-installation on the
telescope. The test included a grounding checkout, confirmation that all
telemetry was transferred, and that the fans were fully functional. To test
functionality of the GFAs, we took a zero second exposure to measure the noise
levels. During the test, all positioners and fiducials were powered on, and
the positioners were moved in both axes. No positioner accuracy tests were
performed.

###  4.9 FVC testing

We tested the FVC feedback system in several phases, prior to the DESI focal
plane installation (summer 2019). Early on, the camera was tested in the lab
at Yale, pointing the camera at a well-measured plate of fixed, backlit fibers
from a representative distance (Baltay et al., 2019). These tests demonstrated
feasibility. In 2016, we successfully tested the system in-situ at the Mayall
Telescope, as part of ProtoDESI. In this test, the FVC imaged a set of three
fiber positioners and 13 fiducials, looking through the pre-DESI prime focus
corrector (Fagrelius et al., 2018). In spring 2019, the system was again
tested, but now looking through the DESI corrector, at the Commissioning
Instrument and its array of fixed fiducials (Ross et al., 2018).

In parallel with these campaigns, we built significant experience with
smaller-scale FVC systems on the robot test stands, as part of our robot
development and QC testing. In these stands we tested fiducial reliability and
debugged the feedback side of the positioner control interface.

##  5 Installation

We shipped petals individually from Berkeley to Kitt Peak, and installed them
one-by-one directly into the FPR on the telescope. We decided upon this
approach (rather than integrating petals to the FPR in the lab, then shipping
and installing them as a single massive unit) to reduce both schedule and
technical risks. As opposed to a more monolithic installation strategy, our
approach entailed a higher total number of critical lifts and shipments, but
allowed simpler fixtures, repetition of procedures, elimination of critical
flipping manuevers, reduced risk per shipment, and created logistical
flexibility in scheduling of deliveries, functional tests, and integration.

During the shipping and installation phase of the project, careful tracking of
high-value subsystems and tooling was critical. The success of this approach
relied on significant travel of the FPS engineers and scientists to Kitt Peak
and thoroughly-reviewed procedures for handling the hardware.

We shipped petals to Kitt Peak beginning in spring 2019. At the same time, we
practiced mock test installations of the FPD and petals with Kitt Peak staff
in the Mayall garage. The FPD was installed to the telescope in June 2019,
followed by the ten petals, complete by the end of July (see fig. 31). The
thermal system was subsequently installed in early August (Besuner et al.,
2020).

Figure 31:  At left is a mock installation trial in the Mayall Garage. The FPD
(black ring) is mounted on a steel A-frame with a height, inclination, and
bolt interface matching the corrector barrel on the telescope (when mounted on
the southeast work platform). At right is an overview picture of the real
installation, taken during insertion of our third petal to the telescope. The
installation is done with the telescope parked at the South-East (SE) platform
of the dome.

###  5.1 Shipping to Kitt Peak

We shipped petals in individual crates from LBNL direct to Kitt Peak. Each
crate contained two carts (see fig. 32), one for the petal and the other for
that unit’s permanently-attached 45 m fiber cable and slithead. After
completing the acceptance process for a given petal assembly, we transferred
it from its integration cart to a shipping cart with stiffer members, a
protective sheet metal enclosure, and a lower center of gravity. The
permanently-attached fiber cable and slit remained on their separate fiber
cart. We then wheeled the two carts up a ramp onto a vibration-isolated platen
on the petal shipping crate. We rotated the fiber cart 90° for a lower center
of gravity during shipping. All lifts, rotations, and transfers were done
according to well-rehearsed, detailed procedures. Both carts were screwed down
to an internal wood platen, mounted to the crate base via 12 wire rope
isolators (IDC SB16-610-04-A). The crates were sized to fit 2 at a time in an
enclosed air-ride truck.

Once the shipping crates arrived at Kitt Peak, the petal and attached fiber
cable assemblies were removed from the crates, and remained on their
respective pairs of carts until installation to the telescope. In this
configuration, the petals all underwent functional testing, as described in
section 4.8, which confirmed that no items were damaged during shipment.

Figure 32:  Petals are transferred from their assembly carts to lower profile
shipping carts (left) via crane procedures. At right, the wood base and
vibration-isolated internal platen of a petal shipping crate are shown, with a
petal cart and its fiber cart mounted.

###  5.2 Installation to telescope

We inserted the petal assemblies one at a time into the telescope on a large
steel fixture with 16 degrees of freedom. The system had three major
components (see fig. 33): the South East (SE) Platform, Sled, and the Petal
Mounting Assembly (PMA). The SE Platform is an existing platform at the Mayall
Telescope’s south east annex and its height can be adjusted coarsely. The Sled
includes one pair of linear rails and an acme drive screw. The rails platform
is mounted to six manually-adjustable struts, used for alignment of the rails
to the Corrector Barrel’s central axis. The PMA includes the rotation arm used
to address the 10 petal positions. The arm has a main rotation axis and a
rotary trim stage at the end where the petal mounts. The PMA rides on the Sled
rails, and is mounted on six manually-adjustable struts, for alignment of the
central PMA axis to Barrel and Sled.

Figure 33:  Illustration and photograph of the system with which we aligned
and installed petals at the Mayall Telescope. The system is also used for
removal of petals, which was done in summer 2021 to install some electrical
upgrades.

Alignment measurements were done using a laser tracker (Shourt et al., 2020).
Ball mounted retroreflectors (BMRs) were magnetically attached to kinematic
nests, permanently glued on the FPD. We had previously measured these ball
nests accurately with respect to the Barrel central axis on a CMM in the lab.
This established the Corrector central axis. We then aligned the Sled and PMA
to this axis.

The petal assembly is lifted by a steel ‘leg’ (see fig. 34). The leg inserts
from the rear of the petal assembly, guided by a rail/bearing, and attaches
rigidly to the petal structure. The leg has lift points at both its own center
of gravity and that of the petal assembly. At its base, the leg is bolted to
the end of the PMA rotator arm. We made a special holder for small ball mounts
which we could kinematically attach to the two insertion guide spikes that
project from the front of each petal (see fig. 12). Measuring the petal BMRs
with respect to those on the FPD, we could adjust the struts such that the
items were aligned within 25 – 50 µm. We then drove the PMA down the Sled
bearings with a manual hand wheel, right-angle geared to an acme screw.

Figure 34:  Petals are handled with a steel lift leg, with attachment points
for horizontal (crane) and cantilevered (insertion fixture) attachment.

Once the petal had been aligned via laser tracker measurements, we removed the
BMRs from the guide spikes and drove it forward into the FPR. The first parts
of the petal to contact the FPR are the stainless steel guide spikes. We sized
these relatively loosely: 0.2 mm smaller on the diameter than the
corresponding steel bushings in the FPR. This gave us some play for
misalignments, in particular the dynamic shifts that occur as loads move
across the SE work platform, and as the whole platform moves with respect to
the telescope. We included load cells (Omega LC703-1K, read out with a
Graphtec GL220) in series at each of the six PMA struts, so that we could
immediately identify hard contact conditions. We would drive the petal forward
until encountering a full-stop condition against the FPR ring. This we could
quantitatively determine by the axially-oriented load cell. For fine
adjustments of the PMA struts during this final insertion step, our feedbacks
were the load cell measurements, feeler gauges, and visual inspection.

At all times during shipping, lifts, and insertion, the 45 m fiber cable and
slithead (the component at the other end of the cable, which mounts and
linearly aligns the fibers in each spectrograph) are permanently connected to
each petal assembly. Cable routing was carefully planned in CAD, including
both the final configuration when installed, as well as how to manage the
fiber carts on the C-floor during the installation process (see fig. 35).

Figure 35:  Fiber cables are lashed with hook-and-loop ties to an aluminum
hexagonal grid at the rear of the focal plane assembly. The grid is supported
by the thermal enclosure, and sufficient slack was designed into the cable
routes to ensure free hexapod motion. At right, a view of the 10 cable carts
on the C-floor, after completing installation of petals, but prior to routing
cables along the truss and lowering each slithead to the spectrograph
enclosure.

###  5.3 Functional Verification

After all ten petals were installed, we completed functional tests on each
petal, confirming no grounding issues were impacting the function of the GFAs
or petal electronics. Then began the work of getting Spotmatch to work with
the PlateMaker software to correctly identify each fiducial and fiber, given
the distortion pattern of the FVC lens and corrector. For this to work,
fiducials from all 10 petals had to be illuminated, but we backlit only one
petal at a time to avoid confusion between fibers on adjacent petals. We tuned
the power to each fiducial and the backlight LED so that all points of light
had a similar signal-to-noise on the FVC.

Once all fibers/positioners were correctly identified by Spotmatch, we
repeated the calibration and performance tests that we had done in the
laboratory prior to shipment (see §4.8). For most of this testing, the
telescope was set at zenith so that the positioners pointed downward, parallel
to gravity. It was found that some positioners that had performed well in the
laboratory were no longer performing well enough for use on-sky. All
positioners that did not meet requirements were parked and identified in a
database. The majority of the positioners ($>$ 85%) worked as expected and
were tested in a variety of dome conditions. We measured significant in-dome
turbulence due to temperature gradients during the day, exacerbated by various
fans in the dome. The most stable conditions occurred in the evening with the
dome open, after the telescope had come into equilibrium with the ambient air.
Final characterization of the positioners was completed in this configuration.

Since this was the first time that more than one petal was tested at the same
time, the team was careful to incrementally test the capacity of the thermal
system to deal with the heat load from the positioners. A minimum time between
successive moves was established so that heat from the motors could be removed
from the system before the subsequent move. While fault management was in
place at this time, the thermal system hardware interlocks had not yet been
fully installed, so vigilant supervision of the temperature of all positioners
took place whenever there was power to the focal plane.

The final step before integrating the focal plane into the larger instrument
control framework was to complete a positioning accuracy test on all operable
positioners simultaneously. The positioners that met accuracy requirements
were then enabled to receive targets from the ICS. Excluding outlier targets
(nominally defined as the the worst 0.7%), the max blind move accuracy was
120.4 µm and the RMS error was 4.7 µm, for the 87% of installed positioners
that were enabled at that time (Fagrelius et al., 2020).

###  5.4 Commissioning

On-sky testing with the focal plane began first with the GFA cameras and
stationary robots in late October 2019. Once the functionality of guiding from
the GFAs was verified, the commissioning team was able to build upon work
characterizing the telescope that had begun on the DESI Commissioning
Instrument (Ross et al., 2018). During this period of observing, with some
spectrographs enabled, we did coarse telescope dithering around bright targets
to assess our initial mapping from focal plane to sky coordinates.

The first on-sky fiber positioning with the full control loop occurred in
early December 2019, with the robots operating within a restricted patrol
radius. We avoided any use of the anticollision algorithm until we had fully
characterized the robots and debugged the feedback system. Initial sky to
fiber mappings were offset by up to 10 arcsec and little flux made it down the
fibers when targeting fainter objects. Operation of the robotic positioners
enabled fine fiber dithers around bright objects to correct smaller scales of
the focal plane to sky coordinate mapping. By March 2020 the mapping had been
sufficiently characterized and verified, and the software was sufficiently
robust, such that we made the FPS available to the operations team, to begin
the spectroscopic measurements of the Survey Validation (SV) campaign.

We measured on-sky fiber positioning accuracy with a series of “dither tiles”,
in which astrometric standard stars are offset by varying, randomized
distances between each exposure in a 13-tile sequence. Treating the
spectrographs as photometers, these random offsets produce flux profiles of
numerous bright stars simultaneously in the field of view. The peaks of the
flux profiles give true positions of the stars as projected on the focal
surface. We repeated these dithers at several telescope pointings, exploring
most of the available parameter space. Using this method, errors in focal
plane metrology and the FVC distortion model were identified. On-sky
positioning accuracy, originally $\sim$ 10 arcseconds for our first targeting
attempts, improved to $\sim$ 0.1 arcseconds (DESI Collaboration et al. 2022 in
prep.) after addressing these error sources.

Following DESI’s shutdown for the COVID-19 global pandemic, which lasted from
March 16 to November 20, 2020, the anticollision move scheduling code (§3.3)
was tested on the full instrument, and demonstrated to perform as required.
(We had tested it thoroughly on spare petals in the lab, prior to this.) We
progressively released the petals for operation of robots over their full
patrol envelopes. Petal 0 was allowed full reach starting on January 29, 2021,
followed by petals 2, 4 and 5 on February 6, 2021, petals 7 and 8 on February
28 2021, and all petals on March 6, 2021. This marked the passing of a fully
functional focal plane to the survey operations team.

##  6 Performance

Performance of the FPS is described in six sections below, covering positioner
accuracy (§6.1), fiber alignment (§6.2), positioner utilization (§6.3),
reconfiguration time (§6.4), guider performance (§6.5) and fiber view camera
performance (§6.6).

###  6.1 Positioner accuracy

The DESI requirement for positioner accuracy is that the fibers be aligned
with their commanded target locations as-measured by the FVC within 10 µm RMS.
Figure 4 plots fiber positioning accuracy over three months of survey
operations, during which over 7.8 million targets were acquired. Accuracy of
the system was 9 µm RMS, with most of the error due to measurement effects
(spot centroiding $\sim 3$ µm and dome turbulence $\sim$ 3 – 8 µm). Accuracy
was subsequently improved to 6 µm RMS (see fig. 36) after we implemented code
to subtract some of the turbulence out of the measurement (see §6.6).

Figure 36:  Targeting accuracy for DESI fiber robots over a 2.5 week period of
survey operations, during which the Focal Plane System acquired more than 1.2
million targets. During this period, we operated with code that corrects some
of the dome turbulence effect in the FVC measurement. This reduced the
positioning error to 6 µm RMS (prior performance was 9 µm RMS, as shown in
figure 4).

Dither tests, as described above, tell us the overall positioning accuracy,
since they include corrector, telescope, and atmospheric effects. However,
they consume significant telescope time, and so were largely a one-time
commissioning task. Using the dither mode, we measured total fiber positioning
accuracy on sky to be 0.19 arcsec (13 µm) 2-D RMS. These results were valid to
high fidelity and were repeatable across several nights. This achieved the
design goal for positioning accuracy, losing less than 3% of the light for
DESI Main Survey Emission Line Galaxies (ELGs) under nominal conditions of 1.1
arcsec FWHM atmospheric seeing.

###  6.2 Fiber focal and angular alignment

Over the full array, our alignment process for fiber tips (see §4.3) achieved
a focus with a standard deviation of $\sigma=14.3$ µm, corresponding to a
throughput loss of $\sim 0.1\%$. In addition to this scatter, we saw a larger
systematic offset, likely caused by wear on calibration rods (see §4.3 and
figure 28) in our fiber integration fixtures. Measured over 5,526 units, the
fiber tips had a mean position -48.6 µm below the nominal focus depth.
Fortunately, we had designed sufficient adjustability into our structure, and
our QC measurements identified the problem early enough in the process so that
we were able to remove this offset by shifting the mount position of the
petals themselves. We made this adjustment at the captive shim stacks between
the petals and forward lip of the FPR (see §2.3, 4.1). Simultaneously we
counter-shifted the GFA mount plate (§4.6) on each petal in the opposite
direction, to keep the cameras in focus.

A study of the impact of fiber angular error on total throughput showed errors
within $\pm 0.5$° would incur loss $<$ 1%, and within 1° would incur loss $<$
3%. In production, we measured the as-built angle of tilt between ferrules and
$\phi$ bearing cartridges, which controls rotation of the fiber about the
eccentric axis. For the 2,023 units we measured, the mean absolute error was
0.14°, well within our 0.3°requirement. The RMS error was 0.25°, corresponding
to $\sim$ 0.2% throughput loss. Six outlier units (0.3%) had tilt error $>$
1.0°, with one fiber strongly misaligned at 6.48 deg. Over the other 2,017
units, the RMS error was 0.17°.

###  6.3 Positioner utilization

Not all fibers are usable for survey operations. As of December 2021, DESI
operates with 85.3% of its fiber robots acquiring targets, 14.3% disabled due
to mechanical or electrical problems, and 0.4% having damaged or broken fibers
(see Table 3).

Status | Count
---|---
Nominal | 4280 | (85.3%)
Improper motion | 649 | (12.9%)
Communication failure | 70 | (1.4%)
Damaged fiber | 21 | (0.4%)
Table 3:  Fiber positioner real-world utilization. Counts are as of Dec 18,
2021.

The original survey plan was to configure a minimum of 8% of the fibers in
each exposure on blank sky positions for spectroscopic sky calibration.
Specifically, each petal is to have a minimum of 40 fibers with at least one
fiber from each slitblock looking at sky (there are 25 slitblocks per petal).
To mitigate the loss of function from disabled positioners, we use their
fibers for this purpose whenever they fall on blank sky positions, which is
typically 60% of the time. In order to provide sufficient sky positions in
each petal (and for each slitblock), it is often necessary to assign
additional fibers from enabled positioners to blank sky. The net result is
that for any given exposure, 9 – 13% of all fibers are used for sky
measurements, with 1 – 2% of these being functional positioners.

A significant fraction of the positioners have motors which move at
consistently smaller angles than commanded, often in a halting manner rather
than smoothly and continuously. In early 2021 we investigated a number of root
cause hypotheses, and eliminated those related to both electrical (such as
grounded wire or failed solder joint) and externally-determinable mechanical
failure (such as broken bond joint or slipped set screw). Our remaining
hypotheses related to internal behavior in the gearmotor. During a summer 2021
shutdown period, we removed 18 failed robots from the focal plane for direct
diagnostics.

The root cause was identified in October 2021 by high-resolution CT scans.
Inside the failed gearmotors, torque is transmitted from the motor shaft to
the gear train via a pinion gear. In the failed motors, this gear was found to
be cracked radially through its wall thickness, from top to bottom of gear. In
all items we inspected, the crack occurs at the root between two gear teeth.
The crack can be seen by optical inspection, in those cases where the gear can
be unscrewed from the motor (see fig. 37). We further inspected 13 spare
motors that had never been operated since receipt from the factory and found 3
with cracked pinions (23%).

Figure 37:  X-ray (left) and optical (right) views of a typical cracked pinion
gear. In the right hand view, one can see how the pinion has slipped down the
shaft (normally it would be flush to the shaft tip, in the upper corner left
of the image).

The method of attachment of pinion gears to motor shafts was a pressure fit.
Tolerances for such fits at the sub-millimeter size scale can be challenging
to meet, and we surmise that this fraction of motors may statistically have
been outside allowable tolerances, thereby over-stressing the gear.

Unfortunately, the failure mode has a subtle operational behavior, in that it
typically only reveals itself under a combination of axial and azimuthal
slippage, specifically when the gears are biased toward the output shaft
direction, i.e. when the robot is oriented downward. Our QC testing in mass
production was done with the robots always fixtured in a horizontal position.
Hence we missed the failures until the petals were mounted in the telescope,
where they often point downward. The failure rate has been a constant $\sim$ 1
unit per week since we installed at the telescope, and proceeds regardless of
temperature, humidity, power, or move cycles. This tends to support theories
like progressive crack propagation rather than wear mechanisms.

Of the positioners with failed gearmotors, we have found that a large number
move a roughly constant fraction of the expected angular distance, i.e. with
an output ratio like: $scale=\frac{angle~{}moved}{angle~{}commanded}<1.0$. We
have tested operating these units (with individually calibrated $scale$
factors) on-sky during engineering time, to reasonably good effect. Such
robots tend to position within $\sim$ 40 µm RMS. They take a longer time to
get to their targets, by a factor $\sim 1/scale$. We also have found that some
motors can be operated accurately by reducing the angular speed of the
magnetic field that drives the rotor. As of this writing, we have chosen not
to incorporate these back into the regular targeting yet. We currently use
such units for sky background measurements instead. With an extra FVC image,
faster FVC feedback, or tuned motor driver parameters, we suspect some may be
re-introduced for targeting, as the survey progresses.

Another significant fraction of positioners developed communication errors.
Unfortunately, a single bad actor on a CAN bus can sometimes impede
communication across the entire bus. During the summer 2021 maintenance
period, we replaced our electrical distribution boards with new ones that have
individual relays per positioner. This allowed us to recover the use of
several units, by switching their bad neighbors to a separate debug bus.

###  6.4 Focal plane reconfiguration

In order to complete the 14,000 square degree DESI survey in 5 years, the
average time to reconfigure the focal plane and to slew to a new field must be
less than 2 minutes on average. All activities during this time between
exposures are coordinated by the OCS sequencer of the Instrument Control
System (ICS). During reconfiguration of the focal plane, many activities are
done in parallel to minimize the total time between exposures. Central among
these activities are two complete positioning loop cycles.

During the first iteration (the ‘blind’ move), the robots are reconfigured to
new positions while the telescope simultaneously slews to the new field. The
reconfiguration process begins with processing a set of target requests. The
anticollision calculation is performed, and move tables are generated for each
device. The average duration of this step is 6.5 seconds. Next, we execute the
move. This takes on average 28 seconds, which includes the time to transfer
the move tables to the robots. The actual time the robots are moving is only a
fraction of this number, typically 8 – 12 seconds, depending on details of
move schedules. The blind move puts fibers $\sim$ 50 µm RMS from their target
positions.

The fiber positions are imaged by the FVC and offsets from the target
positions are derived by the PlateMaker application of the ICS. The FVC
exposure time is 2 seconds and the readout time is $\sim$ 4 seconds.
Processing of the FVC images takes on average 1.5 seconds. The extraction of
positioning offsets as well as the conversion to focal plane coordinates
typically requires another 15 seconds. With this information in hand, a second
positioning loop cycle is initiated for the ‘correction’ move. A ‘handle fvc
feedback’ function analyzes positioner performance of the recent blind move,
which takes on average 3 seconds. Position tracking data is updated and any
highly deviant positioner is disabled for the night. Thereafter the correction
move is performed. These motion paths are generally much simpler than the
blind move, since corrections are small. Average times for the planning and
the move execution steps are 3 and 20 seconds, respectively. Acquisition and
processing of the FVC image after the correction move is done in parallel with
other activities so that no further overhead is accrued.

The combination of accuracy and speed of the robot system has thus allowed
DESI to achieve the fast reconfiguration times ($<$ 2 minutes) between
exposures, needed to complete the survey on schedule.

###  6.5 GFAs on-sky performance

The six guide GFAs (§2.4) each acquire one 5 second integration at an $\sim$ 8
second cadence during standard on-sky spectroscopic exposures. The typical
10$\sigma$ point source depth achieved by the guider cameras during such a 5
second integration under “normal” observing conditions (transparency $>$ 0.9,
1${}^{\prime\prime}<$ FWHM $<1.2^{\prime\prime}$, r-band AB sky mag per square
arcsecond between 20.3 and 20.4) is r = 18.35 mag AB. This depth is sufficient
for DESI guiding purposes, as it means that there are on average more than 3
Gaia stars per guider camera with S/N $>$ 10, even at the highest Galactic
latitudes. An example guide camera image is shown in figure 38. Example
‘donut’ images from a wavefront camera are shown in figure 39.

Figure 38:  A sample guide camera image of a region with high stellar density
and dust. White squares highlight the stars automatically selected for real-
time monitoring of the atmospheric transparency and fiber acceptance fraction.
The data was captured with a 10s exposure on 3-Nov-2019. Figure 39:  Co-added
images of stars observed in each GFA from a 10 s exposure on 3-Nov-2019. The
top and bottom rows show stars observed in the right and left sides of the
four focus sensors, where the image plane is above (below) the best focus
relative to the primary mirror. The similar sizes of the resulting donut-
shaped images indicates that the focal plane is in focus. The middle row shows
stars observed in the six in-focus guide sensors, with a circle showing the
aperture of a hypothetical ø 100 µm fiber (the DESI fibers have a ø 107 µm
fiber). These images allow us to monitor the seeing ($0.81^{\prime\prime}$)
and fraction (72%) of light entering each fiber in real time during a
spectrograph exposure.

During the design phase, we combined estimates of the KPNO atmospheric
extinction, Mayall primary mirror reflectivity/area, DESI corrector
throughput, GFA r-band filter transmission, and GFA detector quantum
efficiency (QE), to predict the GFA r-band zeropoint magnitude at zenith under
clear sky conditions. During DESI commissioning, we confirmed that the
measured GFA r-band zeropoints agreed with those previously predicted at the
$\sim$ 3% level (with the measurements being $\sim$ 3% deeper than the
prediction on average); this validated both the GFA sensitivity/performance
and the total Mayall/DESI (r-band) system throughput to the focal plane. The
GFA gains are stable at the $\lesssim$ 1% level from night to night, so that
r-band transparency can be meaningfully measured from GFA photometry, with the
dominant limitation being secular trends in the Mayall primary mirror
reflectivity between cleanings.

The median r-band GFA FWHM, measured in 5 second guider integrations, is
1.11′′. This FWHM includes any effects related to the GFA detector itself,
such as the finite pixel size (15 µm). The median DESI guider FWHM of 1.11′′
very closely matches the pre-DESI median Mayall/MOSAIC R-band delivered image
quality (FWHM = 1.17′′), and the Mayall/DESI Moffat $\beta$ parameter of
$\beta=3.5$ also matches that found with Mayall/MOSAIC (Meisner et al., 2020;
Dey & Valdes, 2014). The delivered image quality of the DESI GFAs therefore
very closely matches the requirements of maintaining the performance of the
foregoing Mayall/MOSAIC.

###  6.6 FVC Performance

Key factors limiting the centroiding precision of backlit fibers and fiducials
are their signal strength and the size of their PSF. Laboratory measurements
with the FVC prior to installation (Baltay et al., 2019) and at the Mayall
telescope with ProtoDESI (Fagrelius et al., 2018) established that sufficient
precision is achieved with signal amplitudes of at least $\sim$ 10,000
electrons (signal-to-noise ratio of $\sim$ 100) and diffraction-limited PSF
with FWHM of $\sim$ 2 pixels. With DESI, these requirements are met by
suitably adjusting the strength of the illumination, the FVC exposure time,
and choosing a lens aperture at least 20 times smaller than the focal length
of the FVC lens. The resulting centroiding precision is $\sim$ 20 millipixels,
which projects to $\sim$ 3 µm on the DESI focal plane, which has a platescale
of 73 µm/arcsec. With the FVC sensor cooled to -10°C, the dark current and
readout noise of the camera are not a significant factor.

Given sufficient centroiding precision, the most significant factor affecting
fiber positioning error is dome air turbulence between the FVC lens and the
DESI corrector. Air density variations in this zone slightly shift the paths
of rays emitted by the fibers and fiducials, randomly displacing their images
on the FVC sensor. The variations occur on $\sim$ 1 sec time scales. With
careful control of the dome and mirror temperature, the dome turbulence can be
minimized, but it is usually the largest contributor to the positioning error.
Figure 40 shows maps of the turbulence, as measured by centroiding variance.

Because the turbulence is correlated across the field, it can be measured and
fit by comparing the measured fiducial positions to their known positions
(from laboratory metrology or from long-term average of their FVC
measurements). Additionally, we can use previously disabled positioners with
fibers intact as post-facto fiducials. The difference between the two reveals
the effect of the turbulence at the time of the FVC exposure. Fitting with a
smooth function and extrapolating to the fiber locations, the turbulence can
be subtracted. When the turbulence is high, this is an effective method to
reduce the effect to an acceptable level.

Figure 40:  Magnitude of the centroiding variance when the turbulence is high
(left) and low (right). The highest turbulence occurs when the DESI focal
plane is imaged with the dome closed, mirror cooling fans running, and the
dome air temperature very different from the mirror temperature. The lowest
turbulence occurs with the dome open, cooling fans off, and the mirror
temperature matching the air temperature. The small dots in these maps show
the location of each measured source in the FVC image ($\sim$ 5500 sources).
The brightness of each dot scales with the variance of the respective source
position, determined from 10 repeated observations over a short time interval.
Only the variance in the horizontal direction is represented, but the variance
in the vertical direction is similar. The high turbulence map shows regions in
the image where the variance is as high as 50 millipixels ($\sim$160 µm/pixel
at the focal surface), depending on location in the field. The median is 30
millipixels. When the turbulence is low, the peak and median variance fall to
30 and 18 millipixels, respectively.

##  7 Lessons learned

The DESI Focal Plane has been highly successful; even at this early stage in
the survey it has accurately measured more extragalactic redshifts than all
previous experiments combined. All projects have their pains, however, and we
offer here some of the key things we would do differently in hindsight:

Gearmotor internals

Specifications and QC for gearmotors should address not only externally
observable performance characteristics (i.e. torque, coil resistance,
mechanical dimensions, etc), but also internal assembly clearances and
stresses, such as those that caused gear stack compression during our pre-
production run, and latent, cracked pinion gears in final production.

Subcomponent testing

In mass-production, statistical assumptions about subcomponent quality control
can fail to account for new process variations that arise unexpectedly. We
found that every precision-manufactured part must be inspected upon arrival.
Issues within a lot must be immediately fed back to the supplier.

Diagnostic resources

Planning should include personnel, time and budget for early, deep diagnostics
of any failed robot units. In particular, high resolution 3D x-rays were found
to be invaluable.

Individual disables

Bused electronics should include switches for disabling individual units on
the bus (or moving them to a parallel debug bus), so that bad actors don’t
impair their neighbors.

Connectors / module size

Fusion splicing of fibers (as opposed to using connectors) led the modularity
of spectrographs to drive the modularization of the focal plane (i.e. 10
spectrographs $\rightarrow$ 10 petals). For future instruments, from a total
system throughput and instrument uptime perspective, the throughput loss
incurred by fiber connectorization may be worth the gains of modularizing the
focal plane in smaller rafts, with better economies of scale, easier
serviceability, and faster iteration time in production.

Robot pre-production

Planning should include a large-scale pre-production run of robots, at the
scale of 1,000 or more units. Smaller batches statistically do not uncover all
issues, and furthermore lead to a scarcity of early, disposable units for
integrated tests.

Robot electronics

Individual electronics boards for each positioner were a challenge to procure,
test, and service. Integrating electronics over a somewhat larger number of
robots (e.g. 20 – 80 units or so) with connectors to the motor wires, would
ultimately have been preferable. In this spirit, broadcasting to 502 robots
over 10 CAN buses from a single petalbox control computer proved more
complicated than initially assumed. Reducing the number of robots per module
by an order of magnitude would have thus saved significant time in integration
and test.

Encoders

For ease of functional testing, calibration, and diagnostics, it would have
been advantageous to have some position encoding of the motors built into the
robots. (Even a single digital pulse per revolution would have sufficed, given
our 337:1 gear ratio. High precision encoding is not necessary, since the FVC
does a better job than any encoder of measuring final position.)

Test orientation

In hindsight, the cracked pinion gear issue might have been found earlier if
large-scale testing of robots had been done at angles mimicking the telescope
orientation. (Testing at elevated angles was performed, but not on a large
enough sample to uncover the issue.)

Ferrule attachment

Some glass ferrules for fibers cracked under the stress of place-holding set
screws when assembling into robots. This failure mode was difficult to
completely avoid (due to variabilities in screw torque), and difficult to
observe until well after the retaining epoxy had cured.

Calibration gauges

Tribology and potential for wear should be carefully considered for any
repetitively-used mechanical alignment gauges (such as the rods used for fiber
focusing). Gauges need regular, quantitative inspection and qualification.

GFA sensors

Early GFA CCD sensor selection was driven by dark noise criteria, but this
turned out to be a relatively smaller contributor to final performance. A
cheaper CCD with better area efficiency, simpler integration, or perhaps even
a large CMOS, might have sufficed.

# GFA fiducials

We designed GFAs with 2 fiducials, considering the 2D view of the FVC. In
hindsight, a 3rd fiducial would have been helpful during metrology in the lab,
to constrain the 3D rotation component of the sensor location with respect to
the rest of the focal plane.

Simple FVC lens

We ultimately transitioned from a complex, multi-element zoom lens for the FVC
to a single-element plano convex lens. This made analysis of distortions more
straightforward.

Minimize high-friction furcation tubing

Thermoplastic polyester elastomer tubing (Hytrel), commonly-used for fiber
routing, had a problematic combination of high friction against our polyimide-
coated fibers, long stress-relaxation times, and large coefficient of thermal
expansion. Lengths of this tubing greater than a few hundred mm had the
ability, post-installation, to drive the fiber axially forward into the
robotic mechanism by significant amounts, and to bow the fiber out into the
adjacent robot’s envelope. The high friction then prevents the fiber from
returning to its nominal position. Use of such tubing should be minimized or
avoided. Polyimide tubing, while expensive, performed much better.

Service routing

Complete and accurate routing of services is entirely feasible in 3D CAD. Our
computer models of the fibers, wires, and plumbing proved essential in
planning and executing this dense system ($>61,000$ free wires and fibers,
$>11,600$ electrical connectors, in a volume $<1~{}m^{3}$).

Lower barriers to integrated tests

The modularization of the petal was useful, but it would have aided debugging
and commissioning if we had been able to put together a multi-petal system in
the lab. A less complex, smaller scale modularization would have made this
more feasible.

##  8 Summary

The DESI Focal Plane System, with 5,020 robotically mounted optical fibers, is
enabling a survey of extragalactic redshifts which over the next several years
will exceed all previous data sets by an order of magnitude. The focal plane
incorporates over 675,000 individual parts (see fig. 41) and can rapidly
position thousands of fibers in parallel to $<$ 10 µm accuracy. It is
constructed of 10 identical petals, each supporting 502 robots.

Each fiber robot has two rotational axes, driven by independent ø 4 mm DC
brushless gearmotors. We operate these in two speed modes, a fast mode of
$176.07~{}deg/s$ for general repositioning, always followed by a short, fine
precision move at $2.67~{}deg/s$. The motors do not have encoders. The robots
have precision at the µm scale, and when operated ‘blind’ are accurate to
$\sim$ 50 µm RMS, a number which reflects mostly our limits of calibration.

After each blind move we measure the positions of all robots in parallel with
a Fiber View Camera. It measures the centroids of the backlit fibers with a
precision of $\sim$ 20 millipixels, which projects to $\sim$ 3 µm at the focal
surface. Based on this measurement, we send a single set of small correction
moves to the robot array. Interspersed in the array are 123 fixed optical
point sources (fiducials), which help us to remove lens, gravitational, and
air turbulence distortions in the optical path between the camera and the
focal plane. The robots ultimately achieve their commanded targets with an
accuracy of 6 µm RMS, corresponding to 0.085 arcsec on the sky.

A guide signal to the telescope is provided by six custom GFA cameras. Each
camera averages more than 3 Gaia stars per field with S/N $>$ 10, providing a
tracking error signal better than $0.03^{\prime\prime}$. Four more GFAs are
configured with split-thickness filters for wavefront measurements, providing
a feedback signal for precise hexapod alignment of the DESI corrector and
focal plane, with respect to the Mayall primary mirror.

The focal plane instrument is housed within a thick, insulated enclosure.
Interior air temperature is maintained within $\sim$ +10 to +15°C, and a flow
of clean, dry air suppresses interior dew point to $\sim$ -50 to -20°C,
preventing condensation. Heat is extracted by Novec 7100 coolant, delivered to
the focal plane via 51 m of hose.

We assembled the 10 petals and fusion spliced them to 45 m fiber cables in the
laboratory. We shipped them as separate units from LBNL directly to Kitt Peak,
and installed them one-by-one into the Mayall Telescope.

We completed installation and commissioning of the focal plane in 2019 and
2020. Survey operations began thereafter. In the first 12 months of science
operations, from November 2020 through October 2021, the DESI focal plane
assumed 1840 field configurations, resulting in 9.2 million spectra. Excluding
calibration targets and repeat observations, this yielded 4.2 million high-
confidence, unique redshifts of galaxies and quasars, and 1.0 million unique
stellar spectra. By January 2022, DESI had measured 7.5 million galaxies,
delivering them at a rate of $\sim$ 1 million per month. We eagerly anticipate
the scientific bounties that will be discovered in this unprecedented data
set.

Figure 41:  The DESI focal plane has over 675,000 individual parts. It was
installed in 2019 on the Mayall Telescope, located atop Iolkam Du’ag (Kitt
Peak) in the Tohono O’odham Nation (near Tucson, Arizona). Commissioning took
place in 2020; survey operations began in 2021.

The DESI collaboration is indebted to the late John Donaldson, Robin Lafever,
Tammie Lavoie, and Glenn Roberts for their many contributions to the success
of this project and mourns their passing. The authors in the Barcelona-Madrid
Regional Participation Group (BCN-MAD RPG) wish to acknowledge support by
MICIN under grants PID2019-111317GB, PGC2018-094773, PGC2018-102021,
SEV-2016-0588, SEV-2016-0597, MDM-2015-0509, and CEX2020-001058-M, some of
which include ERDF funds from the European Union. IFAE is partially funded by
the CERCA program of the Generalitat de Catalunya. MA, SB, NF, FP, GGR and JS
acknowledge support from the Spanish MICINN grants MultiDark CSD2009-00064,
IFT-UAM/CSIC Severo Ochoa Award SEV-2012-0249 and AYA2014-60641-C2-1-P, CSIC-
AVS contract through MICINN grant AYA2010-21231-C02-01 and CDTI grant
IDC-20101033, and support from the Campus of International Excellence
UAM+CSIC. This research is supported by the Director, Office of Science,
Office of High Energy Physics of the U.S. Department of Energy under Contract
No. DE–AC02–05CH11231, and by the National Energy Research Scientific
Computing Center, a DOE Office of Science User Facility under the same
contract; additional support for DESI is provided by the U.S. National Science
Foundation, Division of Astronomical Sciences under Contract No. AST-0950945
to the NSF’s National Optical-Infrared Astronomy Research Laboratory; the
Science and Technologies Facilities Council of the United Kingdom; the Gordon
and Betty Moore Foundation; the Heising-Simons Foundation; the French
Alternative Energies and Atomic Energy Commission (CEA); the National Council
of Science and Technology of Mexico (CONACYT); the Ministry of Science and
Innovation of Spain (MICINN), and by the DESI Member Institutions:
https://www.desi.lbl.gov/collaborating-institutions. The authors are honored
to be permitted to conduct scientific research on Iolkam Du’ag (Kitt Peak), a
mountain with particular significance to the Tohono O’odham Nation. For more
information, visit https://desi.lbl.gov/. Source data files for all plots are
archived at https://zenodo.org/record/6286817. Mayall (DESI)

## References

  * Ahumada et al. (2020) Ahumada, R., Allende Prieto, C., Almeida, A., et al. 2020, ApJS, 249, 3, doi: 10.3847/1538-4365/ab929e
  * Baltay et al. (2019) Baltay, C., Rabinowitz, D., Besuner, R., et al. 2019, PASP, 131, 065001, doi: 10.1088/1538-3873/ab15c2
  * Besuner et al. (2020) Besuner, R., Allen, L., Baltay, C., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 1144710, doi: 10.1117/12.2561507
  * Cui et al. (2012) Cui, X.-Q., Zhao, Y.-H., Chu, Y.-Q., et al. 2012, Research in Astronomy and Astrophysics, 12, 1197, doi: 10.1088/1674-4527/12/9/003
  * Dawson et al. (2013) Dawson, K. S., Schlegel, D. J., Ahn, C. P., et al. 2013, AJ, 145, 10, doi: 10.1088/0004-6256/145/1/10
  * de Jong et al. (2012) de Jong, R. S., Bellido-Tirado, O., Chiappini, C., et al. 2012, Ground-based and Airborne Instrumentation for Astronomy IV, doi: 10.1117/12.926239
  * DESI Collaboration et al. (2016) DESI Collaboration, Aghamousa, A., Aguilar, J., et al. 2016, arXiv e-prints, arXiv:1611.00036.  https://arxiv.org/abs/1611.00036
  * Dey & Valdes (2014) Dey, A., & Valdes, F. 2014, PASP, 126, 296, doi: 10.1086/675808
  * Doel et al. (2014) Doel, P., Sholl, M. J., Liang, M., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9147, Ground-based and Airborne Instrumentation for Astronomy V, 91476R, doi: 10.1117/12.2057172
  * Duan et al. (2018) Duan, Y., Silber, J. H., Claybaugh, T. M., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, 1070643, doi: 10.1117/12.2309875
  * Fabricant et al. (2005) Fabricant, D., Fata, R., Roll, J., et al. 2005, PASP, 117, 1411, doi: 10.1086/497385
  * Fagrelius et al. (2018) Fagrelius, P., Abareshi, B., Allen, L., et al. 2018, PASP, 130, 025005, doi: 10.1088/1538-3873/aaa225
  * Fagrelius et al. (2020) Fagrelius, P., Duan, Y., Fanning, K., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114478K, doi: 10.1117/12.2561631
  * Fahim et al. (2015) Fahim, N., Prada, F., Kneib, J.-P., et al. 2015, Monthly Notices of the Royal Astronomical Society, MNRAS, 794–806, doi: 10.1093/mnras/stv541
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2018, A&A, 616, A1, doi: 10.1051/0004-6361/201833051
  * Gutierrez et al. (2018) Gutierrez, G., Besuner, R. W., Brooks, D., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10702, Proc. SPIE, 107027Y, doi: 10.1117/12.2312728
  * Honscheid et al. (2016) Honscheid, K., Elliott, A. E., Beaufore, L., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9913, Software and Cyberinfrastructure for Astronomy IV, ed. G. Chiozzi & J. C. Guzman, 99130P, doi: 10.1117/12.2229835
  * Honscheid et al. (2018) Honscheid, K., Elliott, A. E., Buckley-Geer, E., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10707, Proc. SPIE, 107071D, doi: 10.1117/12.2311927
  * Kimura et al. (2010) Kimura, M., Maihara, T., Iwamuro, F., et al. 2010, The Fibre Multi-Object Spectrograph (FMOS) for Subaru Telescope.  https://arxiv.org/abs/1006.3102
  * Leitner et al. (2018) Leitner, D., Aguilar, J., Ameel, J., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10706, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, 1070669, doi: 10.1117/12.2312228
  * Lynch et al. (2016) Lynch, K. M., Marchuk, N., & Elwin, M. L. 2016, in Embedded Computing and Mechatronics with the PIC32, ed. K. M. Lynch, N. Marchuk, & M. L. Elwin (Oxford: Newnes), 249–265, doi: https://doi.org/10.1016/B978-0-12-420165-1.00019-6
  * Meisner et al. (2020) Meisner, A. M., Abareshi, B., Dey, A., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 1144794, doi: 10.1117/12.2574776
  * Pogge et al. (2020) Pogge, R. W., Derwent, M. A., O’Brien, T. P., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 1144781, doi: 10.1117/12.2561113
  * Poppett et al. (2018) Poppett, C., Sharples, R., Edelstein, J., Schmoll, J., & Bramall, D. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10702, Proc. SPIE, 107027O, doi: 10.1117/12.2312176
  * Poppett et al. (2020) Poppett, C., Jelinsky, P., Guy, J., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 1144711, doi: 10.1117/12.2562565
  * Poppett et al. (2014) Poppett, C. L., Edelstein, J., Besuner, R., & Silber, J. H. 2014, in Ground-based and Airborne Instrumentation for Astronomy V, ed. S. K. Ramsay, I. S. McLean, & H. Takami, Vol. 9147, International Society for Optics and Photonics (SPIE), 1937 – 1946, doi: 10.1117/12.2054454
  * Ross et al. (2018) Ross, A. J., Martini, P., Coles, R., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10702, Proc. SPIE, 1070280, doi: 10.1117/12.2312885
  * Schmoll et al. (2018) Schmoll, J., Besuner, R., Bramall, D., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII, ed. C. J. Evans, L. Simard, & H. Takami, 107027N, doi: 10.1117/12.2312140
  * Schubnell et al. (2016) Schubnell, M., Ameel, J., Besuner, R. W., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9908, Ground-based and Airborne Instrumentation for Astronomy VI, 990892, doi: 10.1117/12.2233370
  * Schubnell et al. (2018) Schubnell, M., Aguilar, J., Ameel, J., et al. 2018, in Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, Vol. 10706, International Society for Optics and Photonics (SPIE), doi: 10.1117/12.2311573
  * Sharp et al. (2006) Sharp, R., Saunders, W., Smith, G., et al. 2006, Ground-based and Airborne Instrumentation for Astronomy, doi: 10.1117/12.671022
  * Shectman et al. (1996) Shectman, S. A., Landy, S. D., Oemler, A., et al. 1996, ApJ, 470, 172, doi: 10.1086/177858
  * Shourt et al. (2020) Shourt, W., Besuner, R., Silber, J., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11445, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114456J, doi: 10.1117/12.2562687
  * Silber et al. (2012) Silber, J. H., Schenk, C., Anderssen, E., et al. 2012, in Modern Technologies in Space- and Ground-based Telescopes and Instrumentation II, ed. R. Navarro, C. R. Cunningham, & E. Prieto, Vol. 8450, International Society for Optics and Photonics (SPIE), 1064 – 1076, doi: 10.1117/12.926457
  * Smee et al. (2013) Smee, S. A., Gunn, J. E., Uomoto, A., et al. 2013, The Astronomical Journal, 146, 32, doi: 10.1088/0004-6256/146/2/32
  * Wang et al. (2020) Wang, S.-Y., Huang, P.-J., Chen, H.-Y., et al. 2020, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 11447, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 114477V, doi: 10.1117/12.2561194
  * Zhang et al. (2018) Zhang, K., Silber, J. H., Heetderks, H. D., et al. 2018, in Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III, ed. R. Navarro & R. Geyl, Vol. 10706, International Society for Optics and Photonics (SPIE), 1358 – 1369, doi: 10.1117/12.2314666