# The Robotic Multi-Object Focal Plane System of the Dark Energy Spectroscopic
Instrument (DESI)

Joseph Harry Silber  Lawrence Berkeley National Laboratory (LBNL) Parker A.
Fagrelius<EMAIL_ADDRESS>Parker Fagrelius  NSF’s National
Optical-Infrared Astronomy Research Laboratory (NOIRLab)  Kevin Fanning
Department of Physics, The Ohio State University (OSU)  Center for Cosmology
and AstroParticle Physics, The Ohio State University (OSU)  Michael Schubnell
University of Michigan (UM)  Jessica Nicole Aguilar  Lawrence Berkeley
National Laboratory (LBNL)  Steven Ahlen  Boston University (BU)  Jon Ameel
University of Michigan (UM)  Otger Ballester  Institut de Física d’Altes
Energies (IFAE), The Barcelona Institute of Science and Technology (BIST)
Charles Baltay  Yale University  Chris Bebek  Lawrence Berkeley National
Laboratory (LBNL)  Dominic Benton Beard  Lawrence Berkeley National Laboratory
(LBNL)  Robert Besuner  Lawrence Berkeley National Laboratory (LBNL)  Laia
Cardiel-Sas  Institut de Física d’Altes Energies (IFAE), The Barcelona
Institute of Science and Technology (BIST)  Ricard Casas  Institut de Cìencies
de l’Espai (ICE), IEEC-CSIC  Francisco Javier Castander  Institut de Cìencies
de l’Espai (ICE), IEEC-CSIC  Todd Claybaugh  Lawrence Berkeley National
Laboratory (LBNL)  Carl Dobson  Space Sciences Laboratory (SSL), UC Berkeley
Yutong Duan  Boston University (BU)  Patrick Dunlop  NSF’s National Optical-
Infrared Astronomy Research Laboratory (NOIRLab)  Jerry Edelstein  Space
Sciences Laboratory (SSL), UC Berkeley  William T. Emmet  Yale University  Ann
Elliott  Department of Physics, The Ohio State University (OSU)  Matthew Evatt
NSF’s National Optical-Infrared Astronomy Research Laboratory (NOIRLab)  Irena
Gershkovich  University of Michigan (UM)  Julien Guy  Lawrence Berkeley
National Laboratory (LBNL)  Stu Harris  Space Sciences Laboratory (SSL), UC
Berkeley  Henry Heetderks  Lawrence Berkeley National Laboratory (LBNL)  Space
Sciences Laboratory (SSL), UC Berkeley  Ian Heetderks  Lawrence Berkeley
National Laboratory (LBNL)  Space Sciences Laboratory (SSL), UC Berkeley
Klaus Honscheid  Department of Physics, The Ohio State University (OSU)
Center for Cosmology and AstroParticle Physics, The Ohio State University
(OSU)  Jose Maria Illa  Institut de Física d’Altes Energies (IFAE), The
Barcelona Institute of Science and Technology (BIST)  Patrick Jelinsky
Lawrence Berkeley National Laboratory (LBNL)  Space Sciences Laboratory (SSL),
UC Berkeley  Sharon R. Jelinsky  Space Sciences Laboratory (SSL), UC Berkeley
Jorge Jimenez  Institut de Física d’Altes Energies (IFAE), The Barcelona
Institute of Science and Technology (BIST)  Armin Karcher  Lawrence Berkeley
National Laboratory (LBNL)  Stephen Kent  Fermi National Accelerator
Laboratory (FNAL)  David Kirkby  University of California, Irvine (UCI)  Jean-
Paul Kneib  École polytechnique fédérale de Lausanne (EPFL)  Andrew Lambert
Lawrence Berkeley National Laboratory (LBNL)  Mike Lampton  Space Sciences
Laboratory (SSL), UC Berkeley  Daniela Leitner  Lawrence Berkeley National
Laboratory (LBNL)  Michael Levi  Lawrence Berkeley National Laboratory (LBNL)
Jeremy McCauley  Space Sciences Laboratory (SSL), UC Berkeley  Aaron Meisner
NSF’s National Optical-Infrared Astronomy Research Laboratory (NOIRLab)
Timothy N. Miller  Space Sciences Laboratory (SSL), UC Berkeley  Ramon Miquel
Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of Science
and Technology (BIST)  Institució Catalana de Recerca i Estudis Avançats
(ICREA)  Juliá Mundet  Institut de Física d’Altes Energies (IFAE), The
Barcelona Institute of Science and Technology (BIST)  Claire Poppett  Space
Sciences Laboratory (SSL), UC Berkeley  David Rabinowitz  Yale University
Kevin Reil  Kavli Institute for Particle Astrophysics and Cosmology and SLAC
National Accelerator Laboratory  Stanford University  David Roman  Institut de
Física d’Altes Energies (IFAE), The Barcelona Institute of Science and
Technology (BIST)  David Schlegel  Lawrence Berkeley National Laboratory
(LBNL)  Santiago Serrano  Institut de Cìencies de l’Espai (ICE), IEEC-CSIC
William Van Shourt  Space Sciences Laboratory (SSL), UC Berkeley  David
Sprayberry  NSF’s National Optical-Infrared Astronomy Research Laboratory
(NOIRLab)  Gregory Tarlé  University of Michigan (UM)  Suk Sien Tie
Department of Astronomy, The Ohio State University (OSU)  Curtis Weaverdyck
University of Michigan (UM)  Kai Zhang  Lawrence Berkeley National Laboratory
(LBNL)  Marco Azzaro  Instituto de Astrofísica de Andalucía (IAA)  Stephen
Bailey  Lawrence Berkeley National Laboratory (LBNL)  Santiago Becerril
Instituto de Astrofísica de Andalucía (IAA)  Tami Blackwell  Lawrence Berkeley
National Laboratory (LBNL)  Mohamed Bouri  École polytechnique fédérale de
Lausanne (EPFL)  David Brooks  University College London (UCL)  Elizabeth
Buckley-Geer  Fermi National Accelerator Laboratory (FNAL)  Jose Peñate Castro
Instituto de Astrofísica de Canarias (IAC)  Mark Derwent  Department of
Astronomy, The Ohio State University (OSU)  Arjun Dey  NSF’s National Optical-
Infrared Astronomy Research Laboratory (NOIRLab)  Govinda Dhungana  Southern
Methodist University (SMU)  Peter Doel  University College London (UCL)
Daniel J. Eisenstein  Center for Astrophysics $|$ Harvard & Smithsonian  Nasib
Fahim  Grupo de Investigación HCTLab, Escuela Politécnica Superior,
Universidad Autónoma de Madrid  Juan Garcia-Bellido  Instituto de Física
Teórica (IFT) UAM/CSIC, Universidad Autónoma de Madrid  Enrique Gaztañaga
Institut de Cìencies de l’Espai (ICE), IEEC-CSIC  Satya Gontcho A Gontcho
Lawrence Berkeley National Laboratory (LBNL)  Gaston Gutierrez  Fermi National
Accelerator Laboratory (FNAL)  Philipp Hörler  École polytechnique fédérale de
Lausanne (EPFL)  Robert Kehoe  Southern Methodist University (SMU)  Theodore
Kisner  Lawrence Berkeley National Laboratory (LBNL)  Anthony Kremin  Lawrence
Berkeley National Laboratory (LBNL)  University of Michigan (UM)  Luzius
Kronig  École polytechnique fédérale de Lausanne (EPFL)  Martin Landriau
Lawrence Berkeley National Laboratory (LBNL)  Laurent Le Guillou  Sorbonne
Université, Laboratoire de Physique Nucléaire et de Hautes Energies (LPNHE)
Paul Martini  Department of Astronomy, The Ohio State University (OSU)  Center
for Cosmology and AstroParticle Physics, The Ohio State University (OSU)  John
Moustakas  Department of Physics and Astronomy, Siena College  Nathalie
Palanque-Delabrouille  Lawrence Berkeley National Laboratory (LBNL)  IRFU,
CEA, Université Paris-Saclay  Xiyan Peng  National Astronomical Observatories,
Chinese Academy of Sciences  Will Percival  Department of Physics and
Astronomy, University of Waterloo  Perimeter Institute for Theoretical Physics
Francisco Prada  Instituto de Astrofísica de Andalucía (IAA)  Carlos Allende
Prieto  Instituto de Astrofísica de Canarias (IAC)  Guillermo Gonzalez de
Rivera  Grupo de Investigación HCTLab, Escuela Politécnica Superior,
Universidad Autónoma de Madrid  Eusebio Sanchez  Centro de Investigaciones
Energéticas, Medioambientales y Tecnológicas (CIEMAT)  Justo Sanchez
Instituto de Astrofísica de Andalucía (IAA)  Ray Sharples  Centre for Advanced
Instrumentation, Department of Physics, Durham University  Institute for
Computational Cosmology, Department of Physics, Durham University  Marcelle
Soares-Santos  University of Michigan (UM)  Edward Schlafly  Lawrence
Livermore National Laboratory (LLNL)  Benjamin Alan Weaver  NSF’s National
Optical-Infrared Astronomy Research Laboratory (NOIRLab)  Zhimin Zhou
National Astronomical Observatories, Chinese Academy of Sciences  Yaling Zhu
Space Sciences Laboratory (SSL), UC Berkeley  Hu Zou  National Astronomical
Observatories, Chinese Academy of Sciences

###### Abstract

A system of 5,020 robotic fiber positioners was installed in 2019 on the
Mayall Telescope, at Kitt Peak National Observatory. The robots automatically
re-target their optical fibers every 10 – 20 minutes, each to a precision of
several microns, with a reconfiguration time less than 2 minutes. Over the
next five years, they will enable the newly-constructed Dark Energy
Spectroscopic Instrument (DESI) to measure the spectra of 35 million galaxies
and quasars. DESI will produce the largest 3D map of the universe to date and
measure the expansion history of the cosmos. In addition to the 5,020 robotic
positioners and optical fibers, DESI’s Focal Plane System includes 6 guide
cameras, 4 wavefront cameras, 123 fiducial point sources, and a metrology
camera mounted at the primary mirror. The system also includes associated
structural, thermal, and electrical systems. In all, it contains over 675,000
individual parts. We discuss the design, construction, quality control, and
integration of all these components. We include a summary of the key
requirements, the review and acceptance process, on-sky validations of
requirements, and lessons learned for future multi-object, fiber-fed
spectrographs.

fiber positioner, robot, focal plane, multi-object spectrograph

##  1 Introduction

The Dark Energy Spectroscopic Instrument (DESI) is a new multi-object, fiber-
fed spectrograph, operating in the wavelength range of $0.36-0.98\,\micron$.
It was constructed by a collaboration of hundreds of researchers around the
world, and installed on the 4m Mayall Telescope at Kitt Peak National
Observatory (KPNO). The DESI Survey will be completed over 5 years, during
which the instrument will measure the precise spectra and redshifts of tens of
millions of galaxies and quasars, up to a redshift of $\sim 3.5$. This data
set will enable the DESI collaboration to build the largest 3D map of the
universe to date, providing insight into the expansion history of the universe
and ultimately furthering our understanding of dark energy (DESI Collaboration
et al., 2016). The DESI instrument was developed by a worldwide collaboration,
led by Lawrence Berkeley National Laboratory (LBNL), and supported by the U.S.
Department of Energy, Office of Science.

The primary scientific goal of the DESI survey is to constrain possible models
of dark energy. We are measuring a record-breaking catalog of spectroscopic
redshifts, with which we will determine the Baryon Acoustic Oscillation (BAO)
scale with sub-percent precision. We will also measure anisotropies in galaxy
clustering, or Redshift Space Distortions, which facilitate the study of the
growth of structure in the universe. Additionally, DESI will contribute to the
theory of inflation, to measurements of the mass of neutrinos, and will
conduct a Milky Way Survey that will inform our understanding of the
distribution of dark matter and the assembly of our galaxy.

This paper describes one of the major DESI subsystems: the Focal Plane System
(FPS)111In fact DESI’s focal surface is an asphere, but we frequently refer to
this surface as a “plane”.. The DESI Focal Plane System contains 5,020 robotic
positioners, six guide cameras, four wavefront cameras, 123 fiducial sources,
and a metrology camera. The system also includes associated structural,
thermal, electrical systems, and software. Parallel papers describe the
overall DESI instrument (DESI Collaboration et al. 2022 in prep.), the optical
corrector (Miller et al. 2022 in prep.) to which the FPS is mounted and which
establishes the aspherical focal surface, and the fiber system (Poppett et al.
2022 in prep.) which carries the light from the FPS to the spectrographs
(Jelinsky et al. 2022 in prep.).

The key feature of the Focal Plane System is a close-packed array of fiber
positioning robots, each of which manipulates one optical fiber. For
logistical flexibility and repetition of design elements, we subdivided the
system into 10 sub-units, each with 502 robots. Mechanically, the sub-units
are 10 identical 36° wedge-shaped instruments, which together fill the
circular focal plane. We refer to these 10 units as “petals”. The fibers from
each petal are transmitted in a cable to a single DESI spectrograph. Figure 1
shows a petal in the lab.

Early conceptual designs for the instrument began in 2011, and by 2015
budgeting and planning was complete. Construction of the Focal Plane System
took place from 2016 to 2018, and it was installed in 2019 on the Mayall
Telescope (Besuner et al., 2020; Shourt et al., 2020), located atop Iolkam
Du’ag (Kitt Peak) in the Tohono O’odham Nation (near Tucson, Arizona). On-sky
commissioning spanned October 2019 to March 2020, during which we demonstrated
that all subsystems met or exceeded design requirements (Fagrelius et al.,
2020; Poppett et al., 2020; Meisner et al., 2020). Operations were interrupted
between March and November 2020 due to the global COVID-19 pandemic. We began
a five-month Survey Validation phase starting on December 14, 2020 and began
the DESI Main Survey on May 14, 2021.

Large astronomical spectroscopic surveys have long been made possible by
increasingly complex fiber-fed spectrographs. This started with the Las
Campanas Redshift Survey (LCRS) (Shectman et al., 1996) which used 100 optical
fibers to measure the redshifts of 26,000 galaxies. This capability was
expanded with the 640 fiber Sloan Digital Sky Survey (SDSS) focal plate system
(Smee et al., 2013) and the 400 fiber two degree field system (2dF) with the
AAOmega spectrograph on the Anglo-Australian Telescope (AAT) (Sharp et al.,
2006). The number of fibers on the SDSS focal plane was increased to 1,000 and
was used to measured more than 2 million galaxies and close to 1 million
quasars during the BOSS (Dawson et al., 2013) and eBOSS surveys (Ahumada et
al., 2020).

For early systems, fibers were plugged by hand into custom-drilled plates,
with a uniquely manufactured plate being used for each field of sky targets.
To reduce reconfiguration time and labor, there has been a shift toward
robotically-positioned fibers. Robotic systems have been demonstrated on both
the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) (Cui et
al., 2012) and the Fiber Multi-Object Spectrograph (FMOS) (Kimura et al.,
2010) on the Subaru telescope. With its high speed ($<$ 2 minute average
reconfiguration time, §6.4), large fiber count (5,020 fibers, see §2.2), and
tight positioning accuracy ($<$ 10 µm RMS, see §6.1), the DESI Focal Plane
System described in this paper represents a significant technological
development in the state-of-the-art for such multi-object fiber-fed designs.
By January 2022, during its first year of survey operations, DESI had measured
7.5 million unique galaxy and quasar redshifts, exceeding the 3.8 million
galaxy and quasar redshifts gathered during the first twenty years of
operation of the Sloan Digital Sky Survey (Ahumada et al., 2020).

This paper discusses the design, construction, quality control, and
integration of the Focal Plane System. We intend for this paper to be of some
utility as succeeding collaborations develop their own new technologies in the
field. We describe the hardware elements in §2 and the software in §3. We then
discuss the assembly and test of the instrument (§4), installation into the
Mayall telescope (§5), and system performance (§6). Finally, in §7 we discuss
key lessons-learned through the development process.

Figure 1:  Each of the 10 petals on DESI is a complete instrument with 502
robots, 12 – 14 fiducial point sources, a guide or wavefront camera, a
protective enclosure for fiber splices, power supplies, and control
electronics. At left, a fully-assembled petal is viewed from the front,
looking toward the robotically-controlled fiber tips. Fiducial sources are
mounted in cylindrical housings, interspersed throughout the robot array. A
guide camera (covered by a protective plate in this image) is mounted at the
upper right corner, and the whole assembly here is surrounded by a protective
metal enclosure (which is removed just prior to installation in the
telescope). At right, a side view shows the layout of fibers, electronics, and
support structure.

##  2 Hardware

The DESI Focal Plane is considered a “zonal” system, in that each robot has a
limited patrol region. This is in contrast to the 640 (upgraded to 1,000)
fibers on the Sloan Digital Sky Survey focal plane (Smee et al., 2013), which
are hand-plugged into arbitrary holes on numerous custom-machined aluminum
plates, or the 300 fibers on the MMT Hectospec (Fabricant et al., 2005), which
are positioned serially with two robots. Despite the limitation in the reach
of individual fibers, zonal systems typically offer the advantages of larger
fiber count and faster reconfiguration. A zonal system was chosen for DESI, as
it has been for other similar planned instruments, such as the Subaru Prime
Focus Spectrograph (PFS), which contains 2,400 robotically-positioned fibers
(Wang et al., 2020), the SDSS-V Focal Plane System, which will replace the
manual plug-plates with 500 zonal fiber positioners (Pogge et al., 2020), and
the 2,400 fiber 4MOST instrument on the VISTA telescope (de Jong et al.,
2012).

We installed the DESI Focal Plate Assembly (FPA) on the Mayall 4-m telescope,
mounting it to the rear flange of DESI’s prime focus corrector (Miller et al.
2022 in prep). The FPA includes the array of robotic positioners (§2.1), six
guide cameras and four wavefront cameras (§2.4), 123 fiducial sources (§2.5),
fiber splice boxes (Poppett et al. 2022 in prep.), and the structural (§2.3)
and electrical systems (§2.7) to support them.

The Focal Plane System includes the FPE, an environmental enclosure around the
FPA (§2.6), a liquid chiller system for cooling, and a metrology camera
(§2.5), used to close the control loop on fiber positioning at the end of each
reconfiguration. We mounted this Fiber View Camera (FVC) behind the primary
mirror, $\sim$ 12 m from the focal surface (see fig. 2).

Figure 2:  This diagram shows the location of the focal plane system
components as installed on the Mayall telescope. The focal plate assembly,
which contains all robotic fiber positioners, guide and focus cameras, and
fiducial sources, is mounted to the prime focus corrector. The Fiber View
Camera (FVC) is mounted behind the primary mirror and measures the positions
of the backlit fibers, as well as the fixed fiducials interspersed throughout
the array.

Our survey needs required $\geq$ 3,000 fibers on our instrument, each
positionable to within 10 µm RMS relative to the guiding sensors. During
planning, we weighed the risks of instrument downtime and the failure
probabilities of individual robots against the unit costs of robots, cables,
and spectrographs. From this analysis, we settled on 5,000 fibers222Twenty
additional robotic fibers were added at the periphery of the instrument (hence
the 5,020 total robots frequently referenced in this paper), which feed not
into science spectrographs, but rather into a separate camera for monitoring
sky background levels. as a fairly optimal target for the hardware design.
Additional key requirements on the Focal Plane hardware design were to operate
in the ambient temperature, humidity, and dust levels of the Kitt Peak
environment, and to keep the total mass of all components attached to the
optical corrector $\leq$ 870 kg.

###  2.1 Fiber positioner robots

Each fiber positioner has two degrees of freedom, driven by two independent ø
4 mm DC brushless gearmotors (see fig. 3), and patrols a nominal ø 12 mm
region with its fiber. The exact patrol diameter varies slightly from unit to
unit, depending upon the as-built dimensions. Our gearmotors have an output
ratio of $(46/14+1)^{4}\approx$ 337:1 and no encoders. The robots are mounted
in a nearly hexagonal close-packed pattern (§2.2). Minimum center-to-center
pitch is 10.4 mm. Their patrol regions overlap, preventing gaps in coverage.
For each retargeting, anticollision software (§3.3) plans out a timed sequence
of moves that each robot must perform to reach its target while avoiding
hitting its neighbors.333No collision has yet been observed to harm the
hardware, but such interference might prevent a fiber (and its neighbor) from
getting to the planned destination. Fiber targeting accuracy has been
excellent, as shown in figures 4 and 36, and discussed in §6.1.

Figure 3:  The DESI fiber positioner robot is designed for a minimum 10.4 mm
pitch between neighboring units. It has 2 rotational axes, driven by
independent ø 4 mm 337:1 gear motors. Drive electronics are integrated in a
board mounted to the aft end. The assembly consists of 22 parts and 10
fasteners. The design was developed at LBNL and SSL, and mass-produced by UM
and EPFL. Figure 4:  Targeting accuracy for DESI fiber robots over a three-
month period of survey operations in autumn 2021, during which the Focal Plane
System acquired more than 7.8 million targets. The shallower red histogram
shows ‘blind’ move accuracy, i.e. the first open-loop move made toward each
target. The sharper blue histogram shows positioning error after a single
correction move, based on feedback from the Fiber View Camera (FVC, §2.5).
Blind move RMS error is 54 µm, with 0.35% of attempted targets having error
$>$ 200 µm. After the correction move, the RMS error is 9 µm, with 0.03%
outliers $>$ 200 µm. Errors here are with respect to the commanded target
positions, and include significant contributions of measurement noise (§6.6)
due to centroiding precision limits ($\sim$ 3 µm) and air turbulence in the
dome ($\sim$ 3 – 8 µm). Some turbulence subtraction code introduced in late
2021 further reduced the error to 6 µm RMS. (see fig. 36)

Key to the alignment of the positioner in the focal plane is the central axis
($\theta$) bearing cartridge. The outer, cylindrical surface of the cartridge
is machined to ø 8.286 – 8.295 mm, and contains two ball bearings at either
end. The bearings maintain alignment of the hollow, rotating, inner shaft to
within 0.005 mm total run-out. When mounted in the matching precision holes of
the focal plate (ø 8.308 – 8.318 mm), we are ensured of both maintaining
alignment throughout the rotation range, and that the robot will not
excessively intrude into the envelopes of its neighbors (see fig. 5). The
$\theta$ bearing cartridge has a precise front flange, with M8.7 x 0.35
threads just behind it. Screwing the positioner into the focal plate with
these threads cinches the flange against shallow, machined spotfaces in the
focal plate. This sets the focus alignment of the robot, such that the fiber
tips will land on the focal surface, 86.5 mm further towards the primary
mirror.

Figure 5:  Fiber positioner robots are shown packed together on the focal
plate. In addition to the fiber ferrule, each $\phi$ arm has a light-trap hole
with a dark dye. The original concept for these light-traps was to be able to
selectively extinguish bright stars in the field; in practice they have been
most useful for identifying $\phi$ angles during hardware debugging. A
projecting tab feature on each ferrule holder provides a simple but effective
hard limit against clockwise over-extension.

Glued to the shaft of the $\theta$ cartridge is an upper housing. The
eccentric ($\phi$) motor and bearing cartridge are glued to this housing; thus
they ride on the central axis rotation. The fiber and $\phi$ motor wires route
together through the hollow shaft of the $\theta$ cartridge and hardstop
mechanism, past the $\theta$ motor, and out the back end of the positioner.
There, motor wires terminate in solder joints at the forward area of the
electronics board, while fibers continue to a clip at the back. Throughout the
positioner, the fiber is protected by a ø 0.4 mm polyimide sleeve. At the
clip, we transition to a short length of ø 0.9 mm furcation tubing, made from
a thermoplastic polyester elastomer material (often referred to by the trade
name “Hytrel”), spanning the gap between the robot and larger conduits to the
rear.444We found the furcation tubing to have quite high friction against our
polyimide-coated fibers. We initially had furcation tubing sleeves covering
each fiber all the way from the robot to the splice box, a length of 1.7 – 2.5
m, depending on location in the petal. A potentially catastrophic latent issue
was averted during production, when we discovered (after having already built
up two complete petal assemblies), that this length was sufficient to drive
fibers forward, bowing them out of one positioner and into the adjacent
envelope of its neighbor’s rotating hardware. This effect could be driven
either by relaxation of tension induced in the Hytrel during assembly, or by
thermal contraction of the Hytrel in the presence of moderately cold
temperatures. The high friction prevented the fiber from sliding back to its
nominal state, leaving it permanently bowed outward. The issue was resolved by
reducing the Hytrel length to 0.2 m, and for good measure we added a fiber-
retaining guard piece at the front end of each positioner. It was necessary to
take out all the positioners from the first two petals to do this rework.

Both the $\theta$ and $\phi$ axes have hard mechanical limits in both
directions of rotation. On the $\phi$ axis, the limits (nominally
$-5\degree<\phi<+185\degree$) are imposed with machined features which contact
the upper housing. On the $\theta$ axis, we designed a sliding idler
mechanism, to achieve a range of $-195\degree<\theta<+195\degree$ (i.e.
complete coverage of the circle). To transmit rotation from the solid $\theta$
motor shaft to its hollow bearing cartridge, while still allowing the fiber
and $\phi$ wires to pass by, we designed a ‘lollipop’-shaped mechanical
connection, with an arc slot covering 300°. Thus at the 195° extremes of
$\theta$ rotation, the fiber and wires are deflected no more than 45° by the
rotating hardware.

We took care to ensure a gentle path for the fiber through the internal
mechanisms of the robot. From early on in our prototyping process, we tested
both near and far-field fiber performance, operating the robots over their
full range of motion (Poppett et al., 2014). Direct prediction of fiber
performance based on mechanical strain is a complex subject, and so for design
purposes we set ourselves a minimum allowable fiber bend radius of 50 mm. We
carefully modeled 3D fiber paths through the complete robot mechanism, at the
extrema of its travel range, and then measured the minimum radii on these 3D
curves in CAD, to ensure margin. Wherever possible, we took care to chamfer or
radius all sharp mechanical corners which might contact and unnecessarily
constrain the fiber’s motion. Ultimately this method (and similar attention to
fiber path details throughout the system) paid off: all petals as-delivered
met their throughput requirements, with $\geq$ 99% of fibers intact and $\geq$
90% throughput from the prime focus corrector to the spectrograph (Poppett et
al., 2020).

To reduce stray light reflections off the positioners’ leading surfaces, we
specified a two-step black inorganic anodize, per ECSS-Q-70-03A, as the
surface treatment for the eccentric arm and upper housing, which are made from
6061-T6 aluminum. We also included a small black acetal washer, riding on the
$\phi$ bearing shaft, just below the eccentric arm. This washer reduces glints
off the stainless steel bearing housing below it.

The gearmotors naturally have $\sim 2\degree$ of internal backlash in the
gears. We remove the backlash by always finishing each fast rotation with a
slow rotation of 3°, always in the same direction. To maintain the stability
of the fiber, each axis has a tab applying friction sideways to its shaft. The
tabs have two layers: rubbing against the shaft is a 0.002” thick leaf of
polyimide, which is backed up by a 0.002” leaf of beryllium copper.

Calibration of the complete array of robots requires more than 40,000
individual parameters.555Eight calibration parameters specify the geometric
kinematics of an individual robot unit. We have two effective arm lengths, for
the central and eccentric axes. A Cartesian coordinate pair specifies the
location of the robot’s central axis within the reference frame of its petal.
Each of the two axes then has a calibrated angular zero point, and a total
angular travel range from hardstop-to-hardstop. Beyond these eight geometric
parameters, we also have numerous lower-level settings relating to the
behavior and tuning of each gearmotor (e.g. speed, acceleration, duty cycle,
as-soldered wiring polarity, true output ratio, etc), most of which are kept
uniform across the robot array, but which we can (and do) adjust in special
cases. These can be measured on all robots in parallel, as installed in the
focal plate, using the Fiber View Camera (§2.5). The key calibration
parameters are those used to transform between internally-tracked
$(\theta,\phi)$ shaft angles and externally-measurable Cartesian $(x,y)$
coordinates. We determine these by moving the robots in parallel to a series
of nominal commanded positions, using nominal transformation parameters, and
measuring the resulting fiber positions at each point. We then calculate best-
fit values for these parameters, minimizing error of fiber positions as
predicted by the transform with respect to the measured actual values. Any set
of points reasonably covering the patrol disk of the robot will work for this
calibration method. In practice, the most efficient pattern is to measure
independent circular arcs of points on the $\theta$ and $\phi$ axes. To ensure
good arc fits, in particular on the range-limited $\phi$ axes, we typically
measure $\sim$ 16 calibration points per arc, though in most cases half that
number is sufficient. We usually also follow up the arcs with a rectilinear
grid of 24 more calibration points, to ensure we are spatially covering the
full patrol disk (not just two arc paths). Distributions of key positioner
calibration parameters are shown in figure 6.

Figure 6:  Distributions for several key calibration parameters for the fiber
positioner robots. At left are the kinematic arm lengths of the two robot
axes. ‘R1’ is the distance from the $\theta$ rotation axis to the $\phi$
rotation axis. ‘R2’ is the distance from the $\phi$ axis to the fiber. The
maximum radius to which a given robot unit can move its fiber is its
particular value for $(R1+R2)$. At the center of each unit there is an
inaccessible zone of radius $|R1-R2|$. The middle and right plots show the
maximum travel ranges from hardstop-to-hardstop for the $\theta$ and $\phi$
axes, respectively. The $\phi$ range distribution is bimodal due to a period
of several months during robot production where $\phi$ bearings were being
glued at a spacing $\sim$ 0.5 mm further than nominal off their housings, thus
increasing the effective travel range. Other calibration parameters (not shown
here) include center position of the $\theta$ rotation axis, angular zero
points of $\theta$ and $\phi$, and effective gear output ratio for each axis
(equal to 1.0 for nominally functioning robots).

Typical calibration error distributions are given in figure 7. The overall fit
error of the calibration (6.5 µm RMS) is a combined measure of the mechanical
circularity of robot motion paths and of our measurement repeatability of the
fiber tips. Comparing this both to the blind move accuracy ($\sim$ 50 µm RMS,
see figures 4 and 36), and to the correction move accuracy (6 – 9 µm RMS) one
can see that the majority of initial blind move error comes from other
factors, which may include: imperfect prediction of the plate scale as we
slew, gravitational deflections at varying declination angles (estimated to be
$\sim$ 5 µm RMS, based on measurements of a set of non-moving robots), and
subtle non-linearities (at a length scale greater than the typical correction
move, i.e. longer than $\sim$ 50 µm $\approx$ 0.5 – 1°) in the gears. For
tests in the laboratory, with robots oriented in a fixed horizontal position
and no corrector, robots typically had blind move errors of 20 – 30 µm RMS.
Our only practical constraint on blind move accuracy is to be sufficiently
predictable for anticollision path planning (see §3.3).

Figure 7:  Best-fit errors for a calibration measurement of 4,252 positioner
robots, performed in December 2020. For each positioner, transformation
parameters between $(\theta,\phi)$ and $(x,y)$ coordinates were calculated so
as to minimize the error between measured and predicted fiber locations. The
minimization in this case was done on a data set of 62 points per robot. The
left plot shows the distribution of the transformation’s overall fit error for
each unit (6.5 µm RMS). The middle and right plots show errors on the six key
kinematic parameters independently. ‘R1’ and ‘R2’ are respectively the
kinematic arm lengths between central ($\theta$) and eccentric ($\phi$) axes,
and between eccentric axis and fiber. ‘Xo’ and ‘Yo’ are the location of the
central axis on the focal plane. ‘To’ and ‘Po’ are angular zero points for the
$\theta$ and $\phi$ axes, respectively. The Xo and Yo fit errors are
effectively a measure of FVC measurement precision.

Each positioner robot has its own microcontroller and motor drivers (see
§2.7). It is attached by two wires to a power bus, two wires to a Controller
Area Network (CAN) communications bus (Lynch et al., 2016), and by one wire to
an additional common logic line (‘SYNC’), independent from the CAN bus. In
typical operation, the pre-calculated move sequences are uploaded via CAN to
the microcontrollers, one unique move table going to each robot. Upon
receiving a synchronized start signal, all the robots execute their particular
sequences of motor rotations and intermediate pauses, each according to its
own internal clock. The synchronized start signal can be either a level shift
on the SYNC line, or alternately a broadcast CAN message sent simultaneously
to all buses on the petal.

From 2009 – 2016 we produced and tested 8 generations of 34 prototype units.
In our early units (2009 – 2012), we used ø 6 mm DC brushless motors and
“R-$\theta$” kinematics (i.e. a radial stage mounted upon a central rotation
axis). These designs had 12 mm center-to-center pitch and several variants of
geared or flexured radial stage mechanisms (Silber et al., 2012). Such designs
perform well and can naturally simplify anti-collision schemes while providing
inherent anti-backlash preloads, however they tended to have high part counts
(57 – 65 parts per unit) and assembly complexity. By 2013 several motor
vendors were producing smaller ø 4 mm gearmotors, which enabled direct
mounting of an eccentric $\phi$ motor and bearing to the rotating upper
housing. At this point we switched to a simpler $\theta$-$\phi$ design, which
despite its gear backlash and inconvenient paths of motion (both of which we
recognized could be mitigated in software), cut the part count per unit in
half, to 32. We were also able to reduce the minimum center-to-center pitch to
10.4 mm, which relaxed some constraints on the optical corrector by allowing a
small diameter focal surface. During the period 2013 – 2016 we built and
tested 19 $\theta$-$\phi$ prototypes, progressively refining the design
(Schubnell et al., 2016). Following these prototypes, from December 2016 –
March 2017 we fabricated a pre-production run of 440 units. Full production
commenced in June 2017, and concluded by October 2018, with 7,148 robots
produced (§4.2).

The design was developed at Lawrence Berkeley National Laboratory (LBNL) and
Space Sciences Laboratory (SSL), and mass-produced by the University of
Michigan (UM) and École Polytechnique Fédéral Lausanne (EPFL). To ensure the
overall success of the project, alternative designs were developed in parallel
and successfully tested by a Spanish-Swiss consortium of member institutions
within the collaboration (Fahim et al., 2015).

###  2.2 Focal plate layout

The layout of the DESI focal plate is related to the pixel size and count of
the spectrograph CCD sensors. Inside each spectrograph, fibers are arranged in
a linear “slithead”, and their individual spectra are dispersed in shallow,
parallel arcs across the CCD area, mostly orthogonal to the line of fibers. We
selected our CCD size early in project planning, and determined that 500
fibers would optimally cover the sensor. Considering construction cost, survey
time, focal surface area, and feasible miniaturization of robots, we decided
upon a total of 10 spectrographs, thus 5,000 total fibers.

To match the architecture of the 10 DESI spectrographs, we segmented the focal
plate into 10 wedges called ‘petals’, each petal being 36° wide. The fiber
positioning robots are mounted to the petals such that their fiber tips patrol
ø 12 mm disks tangent to the focal surface.

The focal surface is aspheric rather than flat or spherical. This geometry
relaxed our design constraints on DESI’s large, six-element optical corrector
(Doel et al., 2014). It came at the cost of increased complexity in designing,
machining, and inspecting the physical focal plate (§2.3, §4.7). With modern
CAD, CNC, and metrology tools, we determined this to be an acceptable trade.

Perfect hexagonal close-packing is not possible on such a curved surface. To
generate the robot layout pattern, we developed an iterative code that
maximizes the number of positioners on our ø 812 mm diameter aspheric focal
surface. A key input to the code is an axisymmetric tolerance envelope,
enclosing the maximum expected geometric deviations for the complete
manufacturing run of robots (see fig. 8). We produced this envelope by
performing a detailed Monte Carlo analysis of component manufacturing
tolerances, shaft and mounting hole alignments, and the spatial volumes swept
out during repositioning.

Figure 8:  Layout of the focal plate. Left: The envelopes were iteratively
squeezed together, with 3 key constraints: fiber tips stay on the asphere,
central axis matches the chief ray, and the pattern along one edge of the 36°
petal is a straight line (see §2.2). Right: An axisymmetric tolerance envelope
was defined by 5 conical sections.

The layout code enforced three key constraints: fiber tips must remain on the
asphere, the central axis of each positioner must align with the local chief
ray, and there must be a straight line of positioners along one radial edge of
each petal. The code first seeds the robot pattern on a wide-spaced, flat,
hexagonal grid, and projects this pattern onto the asphere with a simple shift
in Z coordinate. Then, respecting the constraints described above and the
tolerance envelope for the positioner, the code iteratively nudges robot
positions by small amounts, toward each other and toward the center of the
focal surface. Iteration halts when the nudging process converges.

A further input to the code was an envelope for the GFA camera. The code
optimizes the location of the camera to minimize loss of robots, while keeping
the CCD within the ø 812 mm boundary. We ultimately placed a single guider or
wavefront camera at a common, accessible location at the outer corner of each
petal. Their distribution among the petals is shown in figure 9.

Figure 9:  Left: We selected the pattern of guider versus wavefront cameras
(indicated by ‘G’ and ‘W’) to maximize spacing of the wavefront measurements,
since they must control the tip/tilt degrees of freedom of the hexapod. Middle
and right: At the interfaces between software systems—such as move scheduling
(§3.3) and PlateMaker (§3.5)—we use a modified polar coordinate system (Q, S),
where ‘S’ is distance along the aspheric focal surface within a plane that
intersects the optical axis and is at angle ‘Q’ with respect to the cartesian
coordinates of the focal plane (X5, Y5, Z5). By matching the natural geometry
of the focal surface, the coordinate system removed several possible sources
of ambiguity between developers.

To produce a manufacturable 3D CAD model, the petal was first modeled with no
robot mounting holes. We made a separate model of the positive knockout
geometry for a single robot’s mount hole. Then, driving the modeling program
FreeCAD with a custom Python script, we patterned this knockout geometry
according to a table of hole positions and angles from the layout code. The
resulting positive hole pattern was boolean subtracted from the petal
geometry.

Finally, we visually inspected the pattern in CAD, and several more robot
positions were manually removed, which were deemed by engineering judgment to
have too little support material for practical mounting. These positions occur
along one edge of the petal, where its 36° wide envelope cannot perfectly
align to natural $\sim$ 60° pattern lines. We found the vacancies resulting
from this mismatch to be useful locations to put accessory screw holes for
handling fixtures and temperature sensors.

On DESI, we ultimately packed 5,020 positioners at mean pitch of 10.525 mm
into the ø 812 mm aspheric focal surface. As a comparison guideline for future
projects, one can view this as an overall 84% packing efficiency, i.e. the
ultimate number of science positioners $\sim 0.84(d/p)^{2}$, where $d$ is the
focal plate diameter and $p$ is the positioner-to-positioner pitch. This value
includes all sources of spatial inefficiencies on the DESI focal plate: GFA
camera envelopes, fiducial point sources, gaps between petals, and the central
cap rings, which structurally connect the noses of all ten petals.

We provided the inner diameter of the central Front Cap Ring with an M22 x 1.5
mm thread. The thread is used during assembly to attach a guide tube. At
present, the hole is not instrumented, but could in principle hold a compact
camera or integral field unit (IFU) at the center of the focal plane, so long
as the unit fits through a ø 16 mm hole (see fig. 10).

###  2.3 Focal plate structure

The key structural component of each petal assembly is the petal itself. This
is a 36° aluminum wedge, 82 mm thick at the outer radius and 103.8 mm thick at
the nose. Each petal weighs 8.6 kg and supports a much larger mass than its
self-weight in positioner robots and support electronics. The alloy is 7075,
with a -T651 temper to strengthen the material while minimizing internal
stress. The petal has 514 CNC-machined holes, each with a precision spotface
and bore (see fig. 11). Ahead of the bore is a fine female thread. Positioners
and fiducials screw into the bores like spark plugs, eliminating the need for
additional small parts such as screws to hold each positioner in place.
Contact of the device’s flange against the spotface sets the focus position of
the optical fiber installed in the robotic positioner. Fit of the bore to the
device’s outer diameter sets the position and angle of alignment. The
rotational alignment of each positioner is not controlled with any precision,
but is measured precisely after assembly. The petals were machined by Boston
University (BU).

Figure 10:  The ten petals are structurally connected at their noses by Front
and Rear Cap Rings. Captive shims at the Front Cap Ring are selected to
equalize the as-installed heights of the 10 petals. During assembly, long
custom screw drivers are guided in from the rear of the focal plate along a
removable central tube. Installation and removal of the Rear Cap Ring stack is
facilitated by a special captivator that rides on this central tube. There is
a ø 16 mm center hole, behind which lies an M22 x 1.5 mounting thread. Figure
11:  The robots and the fiducials screw into the petal like a spark plug. This
photo shows a prototype robot, placed in a cross-sectioned aluminum test
block. The test block was fabricated by milling two faces flat, clamping them
together, and then drilling, reaming, and tapping the custom threads.

The petals are retained in a large aluminum integration ring (see fig. 12),
which ultimately connects to the steel DESI corrector barrel, via a match-
drilled and flexured steel cone called the ‘Focal Plate Adapter’ (FPD) (see
fig. 13). To accommodate thermal expansion mismatch, the FPD has 40 flexural
features around the diameter, 2 mm thick x 58 mm long, across the unsupported
gap. Each flexure clamps against the outer diameter of the Focal Plane Ring
(FPR) via 4 bolts. Fabrication and alignment of the FPD structure were done by
Fermi National Accelerator Lab (FNAL) at the same time as they produced the
Corrector Barrel (Gutierrez et al., 2018).

Figure 12:  The ten petals mount into a large integration ring (FPR), about
$1/4$ of which can be seen in the photo at left. At upper right, a petal is
shown from the side, illustrating the curved front surface. We made the back
surface flat to facilitate machining, metrology, and mounting of hardware. At
lower right, two petals are placed adjacent to each other at their separation
distance (0.6 mm) when installed in the ring. Individual robot/fiducial
mounting holes can be seen, each with a precision spotface and M8.7 x 0.35
mounting thread. Also visible (in the upper of these two petals) are accessory
mounting holes, which we located in the natural gaps which occur along the
irregular edge of each 36° petal. Figure 13:  Left: Key components of the
focal plate structure include the steel FPD and aluminum FPD Ring, FPR, and
petals. The FPD Ring is permanently bolted to the FPD via 40 flexural tabs,
which accommodate the differential thermal expansion between the dissimilar
metals of the Corrector Barrel and the Focal Plate. Right: FPD and FPR during
alignment on a coordinate measuring machine.

Subsequently, the FPD and FPR were shipped to LBNL. A system of permanent
gauge blocks and shims was used for a dry-fit alignment of each as-built
petal, installed in the ring (see fig. 14). This allowed us to achieve
consistent focus positions of the 10 positioner arrays and the
Guide/Focus/Alignment cameras with respect to one another.

Figure 14:  Each as-built petal was robotically surveyed and aligned such that
fiber tips, fiducials, and cameras, when installed, would land in focus on
DESI’s nominal aspheric focal surface. The upper left and lower right photos
show the FPR and petals being surveyed in a coordinate measuring machine (CMM)
at LBNL. The two photos at lower left show the GFA camera mount plate on a
petal, which is adjusted by shimming at its three mount screws. At upper
right, a stack of two gauge blocks and an aluminum retention/contact pad are
seen at the corner of a petal. Each petal has two of these, which contact the
FPR, and give us both stiff mounting and fine tilt adjustment. Not shown in
these photos are two additional adjustment features of the petals: these are
captive shim stacks at (1) the outer radius, to set focal position, and (2) at
the inner radius (see fig. 10), to equalize the petals with respect to each
other at their tips. Over the set of all 12 petals (10 installed plus 2
spares), we achieved 12 µm RMS, 33 µm max error in our alignment.

###  2.4 Guide, focus and alignment cameras

Ten custom Guide, Focus & Alignment (GFA) cameras are mounted in the focal
plane, one at the outer edge of each petal (see fig. 15). Six of the cameras
are dedicated to the task of guiding the telescope by centroiding on
astrometric calibration stars. They also provide real-time tracking of
atmospheric transparency and seeing. The other 4 cameras are used as wavefront
sensors, for focus and alignment of the corrector and focal plane on a 6 DOF
hexapod (Gutierrez et al. (2018); Miller et al. (2022) in prep.). The GFA
cameras were built by a consortium of four Spanish institutions: Institut de
Física d’Altes Energies (IFAE), Institute of Space Sciences (ICE-CSIC, IEEC),
Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas
(CIEMAT), and Instituto de Física Teórica (IFT-UAM/CSIC).

Figure 15:  View of focal plane looking through a side porthole. Seven of ten
petals had been installed at the time of this photograph. One guider GFA
camera is visible in the foreground, while three more cameras can be seen in
the far background. The textured posts are light blockers covering stainlesss
steel guide pins, used during petal insertion. The front central cap ring is
also visible, structurally connecting and co-aligning the 10 petals at their
tips.

The guider and wavefront cameras are identical in all respects except for
their optical filters. The guiders have a flat filter, 5 mm thick, while the
wavefront cameras have a stepped filter instead, with one half of the active
area covered by a 1.625 mm thick filter, and the other half at 8.375 mm (see
fig. 16). The filters have an index of refraction of 1.8, and the CCD is
positioned to be in focus when the 5 mm glass is bonded to the cameras.

We glued the filters and CCDs into Ti 6Al-4V plates. Titanium was chosen for
the good match of its coefficient of thermal expansion (CTE $\sim 8.6$ ppm/K)
to both the dense flint filter glass (N-SF6, $\sim$ 9.0 ppm/K) and alumina
($\sim$ 6 ppm/K) substrate on which the CCD is mounted. Below the filter
plate, the GFA housing is made from 7075 aluminum, so as to match the CTE of
the robot upper housings, thus keeping the sensor in the same surface of focus
as the fiber tips despite any ambient temperature fluctuations. The $\sim$ 15
ppm/K mismatch between titanium and aluminum causes an elastic strain,
predominantly in the relatively thin-walled (1.6 mm) aluminum housing. Because
of the importance to targeting of knowing the relative positions of the GFA
fiducials (GIFs; see §2.5) with respect to the CCD pixels, we took some care
in the design of the bolted joint between the filter plate and housing, to
ensure that the GIFs (mounted to the aluminum) would never permanently shift,
even minutely, due to joint slippage under thermal extrema. To reduce stray
light reflections, we anodized the titanium filter plates per AMS 2488 Type
II.

In the wavefront camera, the thinner and thicker filters shift the focus
positions of stars to surfaces 1.5 mm above and below the CCD, respectively.
The resulting defocused ‘donut’ shapes captured by the CCD are analyzed to
determine focus and tip/tilt corrections for the hexapod.

GFA optical filters transmit light between 570 – 717 nm (designed to be 567 –
716 nm). This red band was selected for being both in the middle of DESI’s
spectral range, while also excluding any light from the 470 nm fiber backlight
and fiducial point sources (§2.5). This latter feature allows operation of the
GFAs simultaneously while the robot positions are being measured. These custom
SDSS r'-band filters were produced by Asahi Spectra Co.

An early study of guide star density indicated the need for at least 5 guider
cameras, to ensure a sufficient number of guide stars per field. Under most
weather conditions, at least 10 high-quality astrometric standards from the
Gaia Data Release 2 (Gaia Collaboration et al., 2018) could be observed if we
assume a magnitude range of $14<R<16$. In practice, we use guide stars in the
range of $12<R<18$. We additionally need a minimum of 3 wavefront cameras,
well-separated on the focal plate, in order to control the 3 DOF of defocus
plus two directions of tilt. Given the 10 petal architecture, we therefore
found it natural to put a GFA on each petal, and enjoy the redundancy of
having a spare each of the guiders and wavefront cameras on the instrument.

The GFA CCD sensors are back-illuminated devices from e2v (CCD230-42-1-143).
There are 2048 x 2064 pixels, $15\,\micron$ square, covering an image area
30.7 mm x 30.7 mm. A significant selection criteria was the small overall
package size of 42 mm x 61 mm. A key constraint on the mechanical design of
the camera was to minimize any excursions beyond the package, thus minimizing
loss of area in the overall focal layout of the focal plane. Further selection
factors included dark current and noise versus readout speed.666During unit
testing of CCDs, we found that an error in the data sheet indicated a much
lower dark current than the devices actually can achieve at our operating
temperatures. It was still insignificant for meeting our guiding requirements.
The requirement for readout time for the GFAs was established at 0.2 Hz, with
a frame transfer time of 100 ms. The read noise was expected to remain below
20 $e^{-}$/pixel, with a goal of 20 $e^{-}$/pixel when operated at the GFA’s
nominal readout rate.

The cameras have no physical shutters. The central half of the CCD is used for
image capture, imaging a region of $3.34^{\prime}\times 7.27^{\prime}$ or
24.27 sq. arcmin when accounting for the meridional and saggital rays. The
outer zones are masked for frame transfer and the CCDs are read out from four
quadrants. Readout electronics are located physically close to the sensor, in
the volume below the CCD package. The power supplies are in a separate box,
called the GFA Controller Box (GXB), mounted behind each petal. Built into the
body of each camera are mounting points for two illuminated fiducials.
Positions of the fiducials were carefully measured with respect to the CCD
sensor on each camera (see §4.6).

Figure 16:  Guider and wavefront cameras (upper left) are identical other than
their flat versus stepped filters. A compact electronics design (lower left)
minimizes the footprint of the camera so that it consumes little more than the
size of the CCD package itself in the focal plane. Cooling air and electrical
harness pass through the petal (right).

The GFA electronic design is based on digital correlated double sampling. The
electronics consist of 4 stacked boards, each with a specific task. At the
top, an Enclustra Mercury ZX5 board contains the Xilinx Zynq-7000 SoC (“system
on-a-chip”) that implements all logic and communication. The Zynq-7000 SoC
contains 2 ARM Cortex A9 cores, which run a Linux image and a server to
receive commands and send out the acquired images through TCP sockets and a 1
Gb Ethernet link. It also has a programmable logic area in which we
implemented the logic to control all the elements of the electronics as well
as safety measures related to power to the CCD. Three custom boards were
developed to complete the stack: one for the voltages and clocks, another for
power, and one for the front end electronics that contains the 100 Msps analog
to digital converter (ADC) (TI ADS5263) and preamplifiers.

The GFA is cooled by air, flowing through the camera in the direction from the
front to the rear of the focal plane. The pressure differential driving the
flow has two sources: (1) positive pressure between the C4 lens and the FPA,
produced by the single, large FPD fan (see §2.6) and (2) suction produced by
10 smaller GFA fans (one per camera) mounted via 3D printed ducts, 225 mm to
the rear. The suction fans are two-stage, counter-rotating models (SanAce
9CRA0312P4K03) to ensure sufficient pressure to push the air through the
densely-packed readout electronics. The ducts (as well as many other
lightweight airflow adapters, brackets, and tie points in the FPS) were
printed by Selective Laser Sintering (SLS) in unfilled, white Nylon 12. We
mounted a thermoelectric cooler to the underside of each CCD package, with
cooling fins in the air stream, but in practice have not found it necessary to
turn the coolers on.

###  2.5 Fiber View Camera and Illuminated fiducials

The Fiber View Camera (FVC) closes the control loop when repositioning the
robots. The robot motors themselves are driven open loop. For each
retargeting, the positioners are initially moved open-loop to their rough
target positions, with typical accuracy of $\sim$ 50 µm RMS (see fig. 4). LEDs
at the spectrographs777These are LED strips behind a blue filter and diffuser.
The strips are mounted on the forward face of the spectrograph shutter, so
that they shine directly into the linear array of fibers at the slithead. are
illuminated, backlighting the fibers. Additionally, an array of 123 fiducial
devices with well-determined fixed positions on the focal surface are
illuminated. In the FVC image, pixel centroids are measured for fibers and
fiducials. A set of small, open-loop correction moves is then sent to the
positioners, based on the FVC measurement. Because the FVC guides the robots
to their final target positions, and because in practical terms the motors
have infinitesimal resolution, the limiting factors on positioning accuracy
are measurement precision of the FVC, turbulence in the dome between FVC and
prime focus, and the accuracy of our optical model of the corrector and FVC
lens when converting between camera pixels and focal surface locations.

The FVC is mounted behind the primary mirror in the Cassegrain cage, in the
central obscuration. The camera is fit with its own 25.4 mm diameter lens with
600 mm focal length, and looks through the optical corrector toward the focal
plane 12.2 m away (see fig. 2). The first 3 m of this air gap are baffled by a
ø 0.7 m tube with ø 0.3 m aperture at its end. This reduces the length over
which air turbulence in the dome can affect the measurement. The FVC is
mounted with a fixture that allows us to rotate the camera, thus allowing us
to map out distortions of the lens.

The Fiber View Camera is a FLI Microline 50100 using a Kodak KAP-50100 CCD
camera with 6132 x 8176 pixels at 6 µm pitch. Initially, a 600 mm f/4 Canon
telephoto Lens was used, as was tested during ProtoDESI (Fagrelius et al.,
2018). During commissioning of the focal plane, we found that the non-
axisymmetric distortions in this lens were sufficiently complex to be
difficult to take out in image processing. Therefore, we decided to replace
the telephoto lens with a simpler singlet design. The newer lens system was
built using Thorlabs lens tubes and a single Newport Plano-Convex BK7 600 mm
focal length lens.

We intentionally image the sources with a small aperture so that projected
sizes of the sources are smaller than the Airy disk. This yields a uniform PSF
across the FVC field, minimizing centroiding biases. Figure 17 gives several
views of the focal plane as taken by the FVC. The camera does have a shutter,
which we have had to replace several times due to mechanical failures.

Figure 17:  Images taken by the Fiber View Camera (FVC). At left and upper
right are bright images of the focal plane, taken with side-lighting LEDs
turned on. The lower right image shows the same region as in the upper right
panel, except with the fiber tips and fiducials illuminated for centroid
measurements of their positions. Note that there are four illuminated pinholes
per fiducial.

There are a total of 123 fiducials (see fig. 18) on the focal plane: 10 for
each petal, except for one petal with 11 fiducials and another with 12. The
extra fiducials break symmetry in the image seen by the FVC, to remove any
ambiguities as to orientation. Each fiducial contains a glass disk, masked
with chrome on one side and frosted on the other. On the forward-facing,
masked side, four ø 10 µm pinholes are etched. The rear of the disk is
illuminated by a 470 nm LED, the same wavelength as the backlit fibers,
mounted inside the fiducial housing. The housing mimics the length and
material of the fiber positioners, such that the face of the glass lands in
the same focal surface as the fiber tips. The fiducials mount to the petal via
the same spark plug design as the robotic positioners. The locations of the
pinholes were measured relative to the outer diameter of the fiducials with
1.5 µm precision using a CMM (Coordinate Measuring Machine) prior to shipment
to LBL for installation.

Two fiducials on each petal were reserved for integration with the GFA cameras
and are referred to as GFA Illuminated Fiducials (GIFs), while the remaining
are referred to as Field Illuminated Fiducials (FIFs). The two types of
fiducials are identical at the front end, containing the LED and masked glass
disk. Toward the rear, however, the FIFs have a mechanical body that mimics
the fiber positioner’s interface to petal and FIPOS boards, while the GIFs
screw directly into the GFA housing. FIPOS boards are mounted at the rear of
the petal to drive the GIF backlights. The FVC and fiducials were all
delivered to the project by Yale University (Baltay et al., 2019).

Figure 18:  Fiducial point sources, embedded throughout the array, constrain
optical plate scale and distortion polynomials. Each fiducial has four etched
pinholes and is backlit by a 470 nm LED. Field Illuminated Fiducials (FIFs)
are mounted directly to the petal like the positioners. Two fiducials per
petal are mounted to the GFA housing (GIFs; not pictured).

###  2.6 Thermal management

The focal plate assembly is surrounded by an insulating enclosure, made of 3
inch thick, rigid polyurethane foam, with fiberglass facings (see fig. 19).
The enclosure serves two purposes:

  1. 1.

Thermally isolate the focal plane from ambient conditions, to prevent air
turbulence in the dome (which can spoil astronomical seeing).

  2. 2.

Provide a controlled temperature and humidity environment for focal plane
components.

When spinning thousands of motors simultaneously, the instantaneous power
dissipated by the FPA can be 5 – 10 kW. This occurs over a brief period of
time, generally less than $\sim$ 10 seconds. For a typical observation duty
cycle of 1000 seconds, the average dissipated power is $\sim$ 1.2 kW.

Early on during planning of the system, we recognized dome turbulence as a
significant risk to science performance. We set ourselves a requirement to
keep exterior surfaces of the enclosure to within $\pm$ 1°C of ambient. This
value came primarily from operational experience at the Mayall Telescope.888We
additionally noted that for a ø 1.8 m cylinder in air at $\sim$ 300 K and Kitt
Peak altitude, a surface temperature 1°C above ambient indicates a Rayleigh
number $Ra\sim 10^{8}$. Typically $Ra\sim 10^{9}$ is considered to be onset of
turbulence. We selected the thickness and material of our enclosure wall to
insulate the internal loads at this level.

The corrector and focal plate assembly move on a hexapod with respect to the
enclosure, up to $\sim 10$ mm in any direction. To accommodate this motion,
the annular gap between the focal plate and the thermal enclosure is closed
out by a flexible air skirt, rather than a rigid ring. This skirt is made of 4
layers of 0.26 inch thick fabric-covered neoprene, with a final vapor barrier
layer on top, of 0.002 inch thick Airtech Stretchlon 800 Bagging Material.

Figure 19:  The thermal enclosure, shown with side panels off (left and
center) and on (right). The heat exchanger can be seen in the upper center
photo, surrounded by white MERV filter holders. The duct just below the heat
exchanger wraps around the focal plane, injecting diffuse air flows at several
ports into the volume between the C4 lens and the robots. At lower center, the
flexible air skirt is visible, adjacent to an air duct feeding into the FPD
(the support structure that surrounds the volume between the C4 lens and the
robot array).

A chiller unit (Thermonics LC-5) located on the telescope control floor
(“C-floor”) delivers 3M Novec 7100 coolant to the focal plane via 51 m of hose
(see fig. 20). The chiller is rated to deliver coolant within a range of -40°C
to +50°C, at a flow rate of up to 4 gpm at 50 psi, and can remove $>3000$ W at
-15 °C. The unit is itself cooled by a Mayall facility loop of 50/50 ethylene
glycol/water (EGW). DESI purchased two chiller units, both of which are kept
operable on the C-floor. We can readily plumb in the spare as-needed, and have
done so several times for routine maintenance and software upgrades. The
chiller is powered by a 480 VAC uninterruptible power supply (UPS).

Figure 20:  Cooling system for the focal plane. Blue and red arrows indicate
flows of colder and warmer fluids, respectively. Within the Focal Plane
Enclosure (FPE) there are two fans. One blows air through a heat exchanger
(HX), thus transferring energy from the FPE’s interior air to the liquid
colant. This fan is oriented to blow air in an annular route around the
outside of FPD and petals. A second fan blows air via four ducts into the FPD,
where it travels axially through the GFAs and positioners, exhausting past the
electronics, fibers, and routing structures (located in the ‘OTHER BODIES’
zone of the diagram). A fraction of the cooled output air from the heat
exchanger is snorkeled into the upper zone of the FPE to ensure good thermal
mixing.

Novec 7100, like other hydrofluoroether (HFE) fluids, is a relatively
expensive coolant. It has a very low viscosity, increasing its ability to leak
through poor fittings, and may affect soft, rubbery seal materials. However,
we found these to be relatively minor issues in comparison to several key
benefits. It is non-flammable, non-conductive, low toxicity, has a low Global
Warming Potential, and zero ozone depletion potential. It is commonly used as
a cleaning agent and has a low flash point. Should any spill occur, there is
no risk of either staining the optics or shorting the electronics, and no
clean-up is required. We do, however, have to regularly top off our chillers
with more Novec ($\sim 1.5$ liters every two weeks), due to its ease of
evaporation. One potential issue we considered is that as a low-viscosity
solvent, it has the ability to remove oil from the gearboxes of motors.999We
tested this by submerging a positioner in the fluid. It operated well for some
time until binding up. When we submerged a positioner in 50/50 EGW, its
electronics instantly died. However, our robots are well-separated from the
coolant system by barriers of metal, and so we saw little risk of this
occurring in practice.

Once inside the enclosure, the coolant passes through a tube-fin heat
exchanger (Lytron 4310G10). Air is forced through the heat exchanger (HXA) by
a 172 mm wide 48 VDC counter-rotating fan (San-Ace 9CR5748P9G001). The fan is
rated for flow rates up to 636 CFM and static pressure of 5.62 in H2O. To
eliminate dust in the system, we drive the air through two MERV-13 filters,
positioned immediately ahead of and behind the heat exchanger. At the inlet we
additionally have a charcoal filter to pick up volatile organic compounds. For
ease of replacement, we made special holding cartridges for the filters, and
located them near a small access panel in the enclosure. In practice, at duty
cycles of 30 to 50%, the HXA fan drives 306 to 365 CFM through the heat
exchanger and filters, drawing 135 to 251 W of electrical power. Testing with
and without the filters in place, we found they reduce flow rate by only 2 to
5%.

A separate fan of the same make, referred to as the FPD fan, takes air from
the perimeter zone inside the enclosure and drives it through 4 large
portholes into the volume between the C4 lens and the robot array. A MERV-13
filter is mounted immediately after the FPD fan. After passing through the
filter, the air routes around the outer perimeter of the FPD structure via a ø
150 mm aluminum duct. The portholes are also ø 150 mm and each have a filter
and diffuser screen at their outlets. Any time the enclosure is opened for
maintenance, after closing it we run the HXA fan for $\sim$ 1/2 hour prior to
operating the FPD fan. This cleans up any dust in the air prior to driving it
into the volume where the optical surfaces reside.

From the C4-FPD volume, the air takes three paths. Approximately 1/3 of the
flow goes through the GFA cameras, boosted by the suction fans mounted to
each. The remainder of the flow goes either through the robots (which have
hollow central shafts) and past their electronic boards behind the petal, or
else through the 0.6 mm interstitial gaps between petals.

Clean, dry air is delivered to the enclosure at a rate of 5 – 10 CFM, and is
provided by a system installed in the Mayall facility. Humidity inside the
thermal enclosure is not actively controlled. Rather, the enclosure is
sufficiently well-sealed such that the dry air input drives dew point well
below our operating threshold of -15°C. Readout of temperature and humidity
sensors, control of fans, and temperature and flow set points of the liquid
chiller are controlled by the FXC computer (see §2.7 and §3.7). Environmental
conditions and performance of the system are summarized in figure 21.

Figure 21:  Focal plane environmental measurements over the initial 22 months
of operations, including both commissioning and regular survey ops, which
started in May, 2021. Grayed zones are seasonal, maintenance, and Covid-19
shutdown times, during which temperature and humidity are allowed to track
exterior ambient conditions. Left: While exterior air temperature varies from
-7 to +30°C, air inside the enclosure stays within $\sim$ +10 to +15°C.
Middle: While ambient dew point may drift as high as +17°C (much higher than
the hardware), dew point inside is suppressed to $\sim$ -50 to -20°C,
preventing condensation. Right: The system controls temperature of the FPR
(the large aluminum ring to which petals are mounted) to a set point of 11°C.
Coolant is typically about 5 to 8° lower, removing heat load of both the focal
plane equipment itself, as well as parasitic heat absorbed along the 50 m hose
run from the chiller to the enclosure.

###  2.7 Electrical system

Each robot has an individual electronics board (FIPOS) screwed to the back of
it (see fig. 22). These were built by UM. The boards can be powered with input
voltage anywhere in the range 5 – 12 VDC. The input voltage gets passed
directly to the motor coils by CMOS switches (ADG1636), and is regulated down
to 3.3 VDC for the digital components. These include a 32-bit, 72 MHz, single
core, Cortex-M3 processor (STM32F103), and a CAN transceiver (MAX3051) capable
of 1 Mbps communication rates. A small resistor and amplifier (MAX9634) are
connected to each of the two motors, inline with the input voltage. The signal
is digitized with the microprocessor’s built-in ADCs. Each board is 86 mm long
x 7.5 mm wide. The narrow aspect ratio allows the board to pass through the
8.3 mm mounting hole in the petal. The boards are made of 6 layers, and
include a temperature sensor. A 400 mm color-coded pigtail is soldered to the
end of the board, and is terminated at the other end by a small 5-pin
connector (Samtec S1SS-05-28).

Figure 22:  Electronics board for the fiber positioner robots, from the
University of Michigan (UM).

Maximum power consumption of the robot is 3.23 W with both motors spinning at
100% duty cycle, but we typically operate motors at 70% duty. During a typical
reconfiguration sequence, each motor physically rotates for a total of only
$\sim$ 5 seconds. The robot consumes 180 mW when idle, which occurs primarily
during communication and processing downtimes of the reconfiguration period
($\sim$ 2 minutes, see §6.4) between exposures. Otherwise we put the FIPOS
board into a lower power sleep mode, in which 15 mW is consumed.

Power is delivered to the robots by twenty 600 W supplies (Mean Well
HRPG-600-7.5). Two supplies are mounted to each petal, each of which delivers
power to 250 or 252 robots (500 or 504 total motors per supply). The power
supplies are fed 120 VAC and output 7.5 VDC. We tested the supplies early on
to ensure they were capable of ramping current fast enough for the switching
speed (18 kHz) at which we pulse-width modulate our motor coils. Given the
constraints of move scheduling with anticollision (see §3.3) we typically
operate $\sim$ 250 – 300 motors per supply at any given time during a fiber
reconfiguration.

A large printed circuit board runs radially from the power supplies toward the
tip of the petal, carrying power, CAN, and SYNC bus traces. From this radial
board, seven transverse circuit boards project orthogonally across the robot
array (see fig. 23). Each petal has 10 CAN communication buses, which are
carried on these seven boards. The boards are mounted to stiff aluminum
panels, making them robust for connector mating and routing of fibers.

At the back ends of the robots, fibers and electrical pigtails are gathered in
bundles of 12 – 14 units. The pigtails plug into the transverse boards in
patches of 14 Samtec connectors. Between these connector patches, we mount
plastic wire and fiber guides. The fibers bypass these connector patches
through the guides, and are gathered into conduits of 25 fibers each. The 20
conduits then transmit the fibers to the spool box. Fusion splices to the 45 m
fiber cable are protected in the spool box. Inside the box the fibers are
arranged in combs, with the fibers sandwiched by soft foam at the junction,
and provided with $\sim 1$ m of strain-relieving take-up length on either side
(Poppett et al., 2020).

Figure 23:  Key components of the petal assembly are illustrated at right. A
petal is shown during splicing at left.

Most hardware on each petal is controlled by a small Linux computer
(Beaglebone Black Industrial). This computer is packaged together with two CAN
control boards (sysWORXX USB-CANmodul8101010https://www2.systec-
electronic.com) and three 12 VDC power supplies in the ‘petalbox’. The
petalbox controls CAN communications to robots and fiducials, state of robot
and GFA power supplies, readout of temperature sensors, GFA thermoelectric
cooler enables, and the GFA fan. It does not control operation of the GFA
camera itself (which has a direct ethernet connection to the ICS). The
petalbox interfaces to the Instrument Control System (ICS) via ethernet, and
is powered by 120 VAC.

The GFA camera on each petal is supplied with DC voltages and an ethernet
pass-through by the ‘GXB’ electronics box. The GXB additionally provides
interlocks to shut off power to the GFA, and hardware for control of the
thermoelectric cooler inside the camera. The bias voltages on the GFA camera’s
CCD are automatically powered down by its firmware when the telemetry
(humidity, electronics temperature) drift outside of safe operating limits.
The GXB hardware interlock shuts off the GFA power completely if the
electronics overheat.

Environmental management is handled by the ‘FXC’, an electronics box
containing a similar Linux computer to those in the petalboxes. The FXC has
interfaces to the main air circulation fans, as well as temperature, humidity,
and smoke sensors. A key function of the FXC is control of the liquid chiller.
Commands to the chiller are sent via ethernet. The computer in the FXC is
powered by a 24 VDC line, independent from the other interlocked components.
In the event of tripping an environmental interlock (shutting off nearly
everything else inside the focal plane) the FXC computer maintains power (see
§2.8).

The 120 VAC lines to each of the petalboxes and the fan power supplies are on
independent, ethernet-switchable outlets. We used two rack power distribution
units (Raritan PX3-5219-N1), both mounted inside the thermal enclosure. For
each petal, 120 VAC is delivered to the two robot power supplies on a single
line. All of these 120 VAC lines ultimately terminate at a relay box on the
telescope facility main floor (“M-floor”). All 12 of the focal plane power
lines (10 for the robot power supplies and 2 for the Raritan PDUs) are
connected to a 220 VAC power panel with UPS backup.

###  2.8 Hardware interlocks

The focal plane has numerous automated safety features. We broadly classify
these as either hardware interlocks or fault management. They are designed
primarily to prevent damage to hardware, and secondarily to prevent
unnecessary loss of survey time.

Hardware Interlocks

Intended to be fail safe and are implemented in hardware (i.e., not software
that could easily be changed). ‘Hardware’ can include single-purpose, low
level processors or FPGA-based logic implementation.

Fault Management

Implemented in software and responds to an error condition with more specific
actions, triggered before any interlock would come into play. This is
discussed in §3.8.

The focal plane has an environmental Power Interlock. This is a hardwired
circuit to a set of relays on the M-floor of the telescope facility. Interlock
thresholds are shown in Table 1. When smoke is sensed or dewpoint exceeds its
threshold, power is immediately killed to nearly all components of the focal
plane system: fans, petalboxes, GFA cameras, robots, and fiducials. This
condition also triggers an inhibit line on the Novec chiller, which causes the
chiller to cease operation. When an over-temperature event occurs, power is
shut off to the FIPOS power supplies and petalboxes. The relays can also be
tripped by manually pressing an emergency stop button on the electrical panel
on the M-floor. This disables power to the FXC as well, which means that while
off, there is no information about the internal environment in the FPE.

Table 1: Sensors and thresholds to trigger hardware Power Interlock. Sensor | Units | Lower Limit | Upper Limit
---|---|---|---
Internal air temperature | °C | 0 | +35
Internal humidity | RH% | 0 | 85
Dewpoint | °C | -128 | -2
Smoke detected | 0 or 1 | n/a | 1
HXC or FPD fan failure | rpm |  $<546$ for $\geq 2$ s | n/a

Each GFA CCD camera has two additional levels of hardware interlock.
Internally, an FPGA monitors temperature of itself, the CCD, and air in
proximity to the CCD. If a limit is exceeded (50°C for the hardware or 45°C
for the air in the camera), the FPGA turns off CCD biases and sends a request
via ethernet to the petalbox, for the camera’s power supply (GXB) to be shut
down. Another temperature sensor is mounted on the GFA’s hottest component,
the A/D converter. This sensor is monitored by hardware logic in the GXB. If
the sensor exceeds 75°C, the GXB turns off GFA power immediately.

##  3 Software

Software for DESI’s focal plane is distributed among several independent
applications. The entire system contains 22 hardware controllers, 21 software
applications running on the DESI computer cluster, as well as custom firmware
operating on the FIPOS boards that drive the fiber positioner robots and
fiducials. The applications largely fall into four categories: petal and
positioner control, GFA control, FVC control, and environmental control. Most
of these categories have one hardware controller for fault management and low
level operations and one application for higher level functions like move
scheduling and image processing.

These pieces work together with the Instrument Control System (ICS) to perform
the _positioning loop_ (DESI Collaboration et al. 2022 in prep.). This process
begins with the GFAs imaging a field, allowing ICS to guide and focus DESI. At
the same time, fiber positioner robots move to pre-determined positions
corresponding to targets on the sky. The FVC then measures the positions of
the robots. ICS uses these measurements to issue an additional correction move
to the robots. Finally the FVC measures the fibers a second time, to determine
their final positions.

ICS provides many important pieces of this process which go beyond the scope
of this paper. This includes PlateMaker, which is responsible for astrometry
calculations as well as coordinate transformations between the focal plane,
sky and fiber view camera. ICS also interfaces with the telescope control
system (TCS) and hexapod for slewing and focusing on target fields. Finally,
ICS controls LEDs inside DESI’s spectrographs which back illuminate the fiber
positioners for measurement by the fiber view camera (DESI Collaboration et
al. 2022 in prep.).

###  3.1 Control of petal hardware

All petal hardware, other than the GFA camera, is controlled by software on
the petalbox computer. This software, called PetalController, is written in
Python and based on a framework provided by the ICS. PetalController handles
CAN communications with fiber positioner robots and fiducials. It receives
move schedule tables as Python dictionaries, converts them to CAN messages,
and sends them to the robots. PetalController reads temperatures and voltages,
monitors for error conditions, and publishes telemetry data to the ICS. It
controls the state of power supplies for the robots, fiducials, GFA camera and
thermoelectric cooler, and the speed of the cooling fan for the GFA camera.

Commands to PetalController are made by a single Petal application, running on
the instrument control Linux cluster. Support for alarms and notifications,
telemetry archiving, message passing, and command interfaces are all provided
by standard ICS libraries.

###  3.2 Positioner firmware

The fiber positioner microcontroller is loaded with custom firmware, written
at UM, LBNL, and EPFL. The firmware controls all aspects of robot operation,
including motor control, interpretation and response to CAN messages,
thermistor readout, and motor coil current sensing.

Motors are commutated open-loop, with pulse-width modulation at 18 kHz,
corresponding to 4,000 clock cycles per second at the microcontroller’s 72 MHz
speed. The motors have 3 coils, spaced 120° apart. At each 55 $\mu$s timer
update interval, each coil is assigned a duration for which its switch should
be closed. The durations are calculated by a cosine lookup table, with the
coils phase-shifted by 120°. Frequency of the cosine table determines rotation
speed of the magnetic field. The pulse duration is multiplied by a user-
configurable ‘duty cycle’ parameter (0 – 100%), thus controlling the net
current.

As of December 2021, the firmware currently supports 2 hard-coded speeds,
cruise (9,900 rpm at rotor = $176.07~{}deg/s$ at the output shaft) and creep
(150 rpm at rotor = $2.67~{}deg/s$ at the output shaft). We anticipate future
firmware upgrades to allow selection of speeds in-between.

Upon power-up, a bootloader routine waits 2 seconds. During this period, the
unit may be signaled by CAN message to go into a reprogramming mode. In that
mode, new firmware may be sent to each robot via CAN.

###  3.3 Move scheduling and anticollision

The fiber robots run open-loop and have overlapping patrol regions. They
therefore require precalculated motion paths (see fig. 24), in order to avoid
colliding with one another. These ‘move schedules’ are timed sequences of
motor rotations and pauses, such that each robot can get to its target while
avoiding its neighbors.

Figure 24:  Motion paths of positioners, imaged by leaving the FVC shutter
open and the fiber backlit during the move. Higher image saturation occurs at
pause points during the path. Arcs of motion are naturally dashed due to pulse
width modulation of the fiber backlight source. Positioner id numbers and
motion paths (thin solid lines) are overlaid.

The move schedule calculation is performed off-instrument, in a modern multi-
core computer. Our computation takes advantage of the natural parallelism of
the petal architecture: the move tables for the 10 petals can be calculated
simultaneously in 10 separate cores, with the constraint that we don’t allow
robots along the edges to exceed their respective petal boundaries.
Calculation time is typically 5 – 6 sec for blind moves and 2 – 3 sec for
corrections.

Prior to moving, each robot receives its particular move table and stores it
in local memory. Then upon receiving a synchronized start signal, all robots
execute their scheduled motions on their own local clocks. The synchronization
requirement between positioners’ motion start times is $\sim$ 2 ms. Our
electronics include the capability for synchronization signals via either a
dedicated hardware line or with broadcast commands sent in close succession on
the CAN buses. In practice we’ve found that either method works.

In software we model each positioner as a pair of 2D polygons, encompassing
the nominal mechanical outlines of the eccentric and central bodies (see fig.
25). The cruise motion of every positioner is simulated discretely, quantized
at 0.02 second time steps. The polygons are slightly expanded, to include
margin for variations in mechanical build tolerances, start signal
synchronization, quantization of the simulator, and small final anti-backlash
motions.

Figure 25:  2D polygon keep-out zones drawn around the two moving bodies of
each positioner. Additional polygons (not shown) gracefully provide the system
with fixed boundaries at the petal-to-petal edges and around the GFA cameras.

We’ve coined our algorithm ‘Retract, Rotate, Extend’. All robots are first
retracted ($\phi$ motion only) to within non-colliding circular envelopes,
then rotated about the central $\theta$ axis, and finally extended to their
target positions ($\phi$). This avoids the majority of would-be collisions.
Each of these three stages of motion are then ‘annealed’ (spreading out the
different positioners’ moves in time), to reduce instantaneous power.
Annealing also further reduces the number of would-be collisions. Finally, to
deal with any remaining collision cases, small off-path adjustment moves and
delays are iteratively tested and incorporated into the move schedules as
necessary. In rare cases where the algorithm cannot find any safe path to a
given target (for example if there is an interfering, disabled neighboring
positioner), the robot in question is labeled ‘frozen’ in the log for that
move. Similarly, in cases where a requested target position overlaps a
neighboring positioner or boundary, or where a request is made of a disabled
positioner, that request is rejected and labeled in the log.

The collision avoidance algorithm is an interesting (and well-bounded)
problem, and is also a relatively easy problem to provably solve in
simulation. In practice we have found that the most important design criterion
for the algorithm is architectural flexibility, to handle prosaic real-world
constraints such as:

  * •

Avoiding disabled neighbors, adjacent cameras, and petal-to-petal boundaries.

  * •

Obtaining correct (to $\sim 5$ µm) calibrations of kinematic dimensions (i.e.
center position, arm lengths). For DESI, this totals $>40,000$ parameters.

  * •

Avoiding positioners with broken fibers, whose precise locations are difficult
to measure.

  * •

Restricting extension of questionably-performing robots to a non-colliding
smaller patrol zone.

  * •

Limited selection of discrete motor speeds (e.g. ‘cruise’ and ‘creep’).

  * •

Feedback is not real-time. The FVC system in DESI takes 5+ sec per
measurement.

  * •

Feedback is strictly 2D. Having one centroided spot per positioner gives
Cartesian (x,y), but the transform from (x,y) to robot arm angles is not
unique, because the $\theta$ and $\phi$ travel ranges exceed 360° and
180°.111111For example, on a robot with central axis travel range
$-190\degree\leq\theta\leq+190\degree$ from hardstop to hardstop, there is a
critical difference between asserting $\theta=+185\degree$ versus
$\theta=-175\degree$, based on its singular measured fiber location in
Cartesian space. Similarly, there is an important physical difference between
asserting for example $(\theta,\phi)=(-5\degree,+5\degree)$ versus
$(+5\degree,-5\degree)$, again based on a single Cartesian fiber location.

  * •

Accommodating unreliable communicators. Some robots are reliable enough on the
CAN bus that we generally use them, however in rare cases they fail to
properly communicate. In such cases we must be able to avoid their unplanned
motion (or stasis).

  * •

Variations in output gear speed, i.e. cases where
$\frac{angle~{}moved}{angle~{}commanded}<1.0$ (see §6.3).

  * •

Power supplies, though large (1200 W available per petal), cannot support all
502 robots simultaneously moving both motors at 100% duty cycle.

  * •

FVC measurement error can make positioners that are very close to each other
appear to move in and out of a collision state with their neighbors.

For ease of software integration and maintenance, an early goal was set in the
DESI project to maximize usage of Python as the instrument control language.
To achieve sufficient speed of the move scheduling calculation, we took three
approaches. First, the polygon collision calculation code was isolated in a
standalone module. It was prototyped in Python, then converted to
Cython121212https://cython.org/, and finally optimized for speed. At this low
level (translating and rotating polygons, checking for intersections), data
structures are simple and easily statically typed. Second, the remainder of
the move scheduling calculation suite was highly cached. At this higher level
of abstraction (representation of hundreds of uniquely calibrated robots
swinging around each other in a scheduled dance), more complex data structures
were required. Data storage and retrieval in Python are fast, while loop
executions are relatively slow; hence we put a premium here on never
calculating the same thing twice. This policy produced reasonably fast code,
but came at some cost of complexity, putting a premium on keeping all cached
data up-to-date. Finally, and most importantly, we wrote ourselves a detailed
test harness, with which to repeatably test and optimize in simulation every
possible corner of the move scheduling system.

###  3.4 Offline software tools

Every move and action commanded to the positioners is logged in a database
which takes a snapshot of the current software state of the robots. This
database can be queried offline to analyze positioners for calibration of
parameters or positioning performance.
Desimeter131313https://github.com/desihub/desimeter/ is a software suite for
processing positioner information offline. This includes making batch queries
to the positioner database, processing raw FVC images, and transformations
between sky, FVC, focal plane, and local robot coordinate systems. We wrote
the desimeter transformation code in isolation from PlateMaker (used online
during observing) so that we can cross-check between the two.

Positioner calibration parameters (kinematic ‘arm lengths’, central axis
location, angular zero points, effective gear ratios) are calculated by making
best-fits between commanded angular motions of the motors and measured
locations of the fibers. This can be done either with a random scattering of
points, or (often more efficiently) by independent arcs of points on the
$\theta$ and $\phi$ axes. This optimization code is also a part of the
desimeter suite.

Many of these tools are also used by DESI’s
fiberassign141414https://fiberassign.readthedocs.io software, which generates
configurations of the positioners, called tiles, to observe targets on the
sky. There is a daily one-way sync from the instrument control model of the
positioners to the model used by fiberassign. This enables tiles generated on
the fly during the night to remain aware of potential changes in the
instrument model of the positioners, such as changes in the usability of
robots, calibration parameters, or synthesized parameters like keep-out
polygons. Letting these parameters go out of sync causes tiles to include
unreachable target requests, which the online control system must then reject.
Therefore keeping this information up-to-date is important for aligning the
maximum number of fibers with targets and thus maintaining survey homogeneity.

###  3.5 FVC centroiding and spot matching

As described in §2.5, the FVC is used to close the control loop with the
positioners. Centroids of spots in the FVC image provide a measure, in pixels,
of the relative locations of the fibers and the fiducial sources on the focal
surface. The transformation of these measurements from pixels at the FVC image
plane to physical coordinates at the focal surface is done by a software
module called PlateMaker (Honscheid et al., 2016), which is not discussed in
detail here.

The FVC software is separated into two parts: ‘fiberview’ and ‘spotmatch’.
Fiberview records FVC images, detects spots above a fixed brightness threshold
with intensity profiles matching the diffraction-limited PSF (this filters
cosmic ray hits and hot pixels), and precisely measures pixel centroids of
each source. Spotmatch associates each fiber spot to its respective location
as predicted by PlateMaker. Because the predicted locations are sometimes far
from their actual locations, due to robot positioning errors, and because of
intermittent errors illuminating fiducials, spot matching is not always
trivial. To ensure correct associations, PlateMaker also provides a prediction
for the location in the FVC image of each robot’s center of rotation. For
spots that inadvertently appear far from their expected location, their
proximity to one of the robot centers is usually enough to make a secure
match. For normal cases, proximity to the nearest predicted source is usually
the correct association (after correcting for small errors in scale,
translation, and rotation of the PlateMaker model). Once each centroid is
matched with its predicted source, this information is fed back to PlateMaker,
which translates the measured pixel locations to the reference frame of the
focal surface and determines any correction moves required to align the fibers
with their sky targets.

###  3.6 GFA camera operation

To fit within a small form factor of each GFA housing, the electronics are
distributed across several stacked printed-circuit boards (see §2.4). A hybrid
field programmable gate array (FPGA) / computer processing unit (CPU) ties all
operations together. The CPU runs a Linux operating system supporting high-
level functionality, such as standard protocols for network communication,
data transfer, and command processing. At a lower level, the FPGA logic is
programmed for hardware-level operations, like control of the CCD,
environmental sensors, a thermoelectric cooler, and the analog-to-digital
converter required for signal digitization. With this system, the masked area
of the CCD (2K x 1K pixels) can be read out to memory by four amplifiers in
2.5 sec, with the option of exposing a new image simultaneously. Noise and
dark current depend on the CCD operating temperature, but levels acceptable
for telescope guiding (at a duty cycle of $\sim$ 5 sec) are achieved without
any additional cooling, beyond that which is normally maintained in the focal
plane enclosure ($\sim$ 10 – 15°C, as shown in fig. 21).

Typically each GFA is operated independently and remotely. A dedicated client
application (‘camera.py’) sends commands over the network to each GFA, where a
dedicated command server (‘gfaserver’) receives and processes the commands.
There are two channels for communication: one for command and status, another
for asynchronous data transfer. Except during image readout, telemetry data
are continuously updated.

Each client program is responsible for: (1) configuring the respective GFA at
power up (setting bias levels and clocking options); (2) reconfiguring the GFA
as necessary for each exposure (setting exposure time and readout mode); (3)
monitoring the telemetry; (4) starting each exposure; and (5) receiving,
repackaging, and saving the image data when they are sent back by the GFA.
Each client program also acts as a command server to a higher level
controlling process, executing GFA operations as requested. In normal
operation while guiding the telescope, $\sim$ 5 second exposures are
continuously recorded. If there is an occasional readout error (e.g. dropped
data), the client can automatically recover by reconfiguring the GFA before
the next exposure command is received.

###  3.7 Environmental control

Environmental control software runs on the FXC, a small Linux computer housed
within the thermal enclosure (see §2.7). The FXC reads temperature and
humidity sensors positioned at numerous points inside and just outside the
thermal enclosure. The FXC controls the fans blowing air through the heat
exchanger (HXA), into the FPD volume, and a small exhaust fan for driving air
out of the system, a feature which in practice we have never used.

Every 60 seconds, the FXC updates the temperature set point for the liquid
chiller, which is physically located some 50 m away from the focal plane ‘as
the coolant flows’, on the telescope’s C-floor. The set point is calculated
with a PID loop, controlling on the measured temperature of the FPR (the large
aluminum ring to which the 10 petals are mounted). Separately, the chiller
runs its own internal PID loop, controlling its heating and refrigeration
components to achieve the requested coolant set point. This second PID loop is
built into the chiller vendor’s firmware. The FXC can also set other key
chiller parameters, such as target flow rate. These parameters are stored in
chiller non-volatile RAM so they are typically only set when the chiller is
initially configured.

###  3.8 Fault management

In addition to hardware interlocks (see §2.8), we implement numerous fault
management actions in software. Some actions only raise warnings to the
operator, while others may selectively disable components. Whereas the
tripping of a hardware interlock may incur significant instrument down-time,
fault management responses tend to be quicker to recover from.

In §2.8 we described the global Power Interlock relays, which can be triggered
by hardware logic. The FXC computer is also provided with the ability to trip
these same relays, based on information known only in software, including
chiller and dry air supply flow rates, fan speeds, differential air pressure
across the FPD boundary (ensuring sufficient pressure exists to push air
through the robots), structure temperature, air temperature, humidity, and dew
point.

Software limits may be quite restrictive, guaranteeing the instrument is not
only in a safe envelope, but in our intended operational mode. For example, in
the hardware interlock, the lower limit on air temperature and upper limit on
dew point are set respectively at 0°C and -2°C (see Table 1). In software
however, we trigger a violation at +5°C air or -15°C dew point. The hardware
limit is a final fail safe against frost conditions; the software limit allows
additional protective actions prior to reaching that point.

The FXC software requires any violation to persist $>20$ seconds before
triggering, to allow time for the relevant components to self-correct. If
conditions have not improved, it alerts the ICS, so that other actions can be
taken (for example shutting down GFA CCDs). After a 30 second window, the FXC
then trips the relays.

PetalController (see §3.1) provides the primary fault management on each
petal. It monitors numerous temperatures, voltages, fan speeds, power supply
and interlock states, and the responsiveness of all robots and fiducials on
the CAN bus. It gathers regular temperature telemetry from all responsive
robots, giving us a granular thermal map of the array. In response to these
conditions, the petalbox can enable or disable power to any of these
components. We also put a software rate limiter on the sending of move tables
to positioners. The rate-limiter prevents unnecessary build-up of heat or
mechanical cycles, for example if an erroneous external script were to
repeatedly request moves.

##  4 Assembly and Test

We built key sub-assemblies at several institutions. These included the FPD at
FNAL, the GFAs by the Spanish Consortium, machined petals provided by BU, FVC
and fiducials from Yale, and positioner robots from UM and EPFL. These items
were tested and sent to LBNL for system level integration. At LBNL we
installed short lengths of ferrulized, anti-reflection coated fiber (PFAs)
into the robots. Then we installed the robots into each petal, and finally
spliced the PFAs to the fiber cables (Poppett et al. 2022 in prep), which were
provided by Durham University.

When the petals were fully assembled, they underwent precise metrology,
measuring the locations of the GFAs with respect to the fiducials and petal
structure. We required that the fiducials be measured to 10 µm accuracy with
respect to the focal surface, and to 5 µm accuracy with respect to the GFAs.
Furthermore, we required that the fiber tips be located within $\pm$100 µm of
an ideal focal surface.

The fully-assembled petals each underwent a formal acceptance process before
being shipped individually to Kitt Peak for installation. The subsystem
acceptance package included demonstration of compliance with key requirements,
the compilation of a data set including the metrology measurements, and the
completion of a full functional test.

###  4.1 Petal alignment to mounting ring

Practical machining tolerances would have been insufficient to align the
petals to our desired level of accuracy in the telescope. We therefore
included alignment features at the petal’s mounting interfaces, consisting of
captive shims and gauge blocks.

Each petal has three precision tooling balls, permanently mounted via pins and
epoxy. Upon completion of machining each petal, BU contracted with ZEISS to
measure all robot/fiducial mounting holes and spotfaces with respect to these
balls. DESI’s nominal aspheric focal surface was best-fit to the measured hole
array, as projected 86.5 mm ahead, the length of a robot from mounting flange
to fiber tip. This complex measurement was done both as a quality control test
and to establish the exact location of the focal surface on each petal, with
respect to a simple set of datums (i.e. the tooling balls).

Separately, the FPR and FPD were aligned by FNAL with respect to the Corrector
Barrel. An array of 11 tooling balls around the edge of the FPR could thus be
tied to the expected final position of the Corrector. At LBNL, we mounted the
petals into the FPR, established a global coordinate system with the FPR
tooling balls, and then aligned each petal to that system using its 3 local
tooling balls.

Alignment was done via a system of shims (see §2.3). We achieved a focus
alignment of 12 µm RMS of the tooling balls, with worst case error 33 µm at
the tip of one petal. For logistical flexibility, we also checked whether
petals could be satisfactorily installed in alternate positions about the
ring. We dismounted the petals, shuffled their locations, and re-installed
them. In the shuffled positions, RMS error was 16 µm, and worst case error was
59 µm (on the same petal as before). After completing the alignment with the
tooling balls, we also back-calculated the overall error of all 5,020 holes
with respect to the nominal focal surface. The overall alignment error was
$\pm 15$ µm RMS by geometric metrics, translating to $\sim 0.12\%$ reduction
in throughput (Duan et al., 2018).

###  4.2 Positioner assembly and test

The robots were designed at LBNL, SSL, and UM, as discussed in §2.1. Early
prototypes were built at LBNL, and volume production was completed at UM. EPFL
provided key precision parts, consultation, and independent testing.
Production of gearmotors and bearings was by Namiki Precision Jewel Co.

A fiber positioner robot is composed of seven sequential sub-assemblies,
illustrated in figure 26. We established quality control checks at each step,
to minimize loss of parts and time (see §4.5). Upon completion of the robot,
we installed a temporary, 1 m long, cleaved fiber, and mounted the unit
horizontally on a test stand with a calibrated metrology camera (SBIG STF8300,
monochrome CCD). Each test stand could hold 40 positioners simultaneously. To
ensure good centroids, the fibers were diffusely illuminated in a custom
integrating chamber, 3D printed in white nylon, holding a board of blue LEDs.
The positioning robots were tested for accuracy and reliability in a standard
grid of 192 points. To ensure rejection of early-failing robots, the test
sequence included $>6,000$ total retargetings, and running to the hard stops
50 times. From these tests, we were able to give each positioner a grade (see
Table 2). Only those positioners that received A and B grades were shipped to
LBNL for further integration.

Figure 26:  We built the positioners in seven sequential sub-assemblies, with quality control checks at each step. Table 2: Grading scheme for positioners, and measured yield for the first 4,000 units we produced. Grade | Max. blind move error | Max. corrected move error | RMS corrected move error | Yield
---|---|---|---|---
A | 100% $\leq 100\micron$ | 100% $\leq 15\micron$ | 100% $\leq 5\micron$ | 96.1%
B | 100% $\leq 250\micron$ | 100% $\leq 25\micron$, 95% $\leq 15\micron$ | 100% $\leq 10\micron$, 95% $\leq 5\micron$ | 0.8%
C | 100% $\leq 250\micron$ | 100% $\leq 50\micron$, 95% $\leq 25\micron$ | 100% $\leq 20\micron$, 95% $\leq 10\micron$ | 0.6%
D | 100% $\leq 500\micron$ | 100% $\leq 50\micron$, 95% $\leq 25\micron$ | 100% $\leq 20\micron$, 95% $\leq 10\micron$ | 0%
F | Does not meet any of the above | 2.4%

Lifetime tests of $\geq$ 100,000 moves were done on 78 positioners. Our
requirement was a 90% survival rate after 107,000 moves. In these sequences,
we would move the positioner open-loop to a series of 2,000 – 10,000 random
points, followed by a closed-loop FVC test grid of 192 measurement points. Of
the 78 units life-tested, 23 were operated past 1,000,000 cycles, to improve
the statistical power of our distribution (see fig. 27). A Weibull reliability
fit indicated 98% survival after 107,000 moves and 90% after 1,200,000 moves
(Schubnell et al., 2018).

These lifetime tests did not reveal a significant class of motor failures,
which we discovered only later when the positioners were installed on the
telescope (see §6.3). We currently hypothesize that the tests missed this
because the positioners were oriented horizontally when tested in the lab,
rather than pointing downward as they do in the telescope.

Figure 27:  Summary of robot lifetime tests during design qualification. Each
cycle is a move at cruise speed to a random position within the robot’s patrol
radius. Failure was defined by a positioner becoming unable to reach its
commanded targets, as measured by the test stand FVC.

Robot performance at thermal extremes was qualified in a dry, thermal chamber,
at temperatures ranging from -20 to +35°C. This testing was done on 83 robot
units and several fiducials, all of which survived the thermal extremes.

An array of 15 robots and a fiducial was tested for stability under the
influence of air flows of several speeds, in horizontal, vertical, and face-on
directions. This was done in consideration of the cooling air which we pass
through the focal plate (Zhang et al., 2018).

###  4.3 Fiber integration to positioners

Our design required integration of fibers to robots prior to installation into
petals. It also required installation of the robot to the petal from the front
side. We therefore installed a short section of fiber (the Positioner Fiber
Assembly, or ‘PFA’) into each robot, and subsequently fusion-spliced the PFAs
to the $\sim$ 45 m cables. A significant logistical benefit of splicing was to
decouple cable and slit fabrication in Durham, UK (Schmoll et al., 2018) from
petal assembly in Berkeley, California. The PFAs were spliced to cables only
after all positioners had been installed and tested in a given petal.

Each PFA is composed of a single $\sim 3.1$ m strand of Polymicro FBP optical
fiber with a core diameter of 107 µm and a thin polyimide coating. This is the
same fiber as used in the fiber cables. One end of the fiber was cleaved and
bonded into a custom ø 1.25 mm borosilicate glass ferrule, then shipped to
Infinite Optics for anti-reflection coating. For strain relief and protection
through the working mechanisms of the robot, we bonded a ø 0.4 mm polyimide
sleeve at the end of each ferrule. In addition to general robustness, the
sleeve reduces strain-induced Focal Ratio Degradation (FRD), by preventing
direct contact points to the fiber within the fiber positioner (Poppett et
al., 2018).

We installed the PFAs into robots from the front side. To ease the
installation, we would first install a temporary length of Hytrel furcation
tube from the rear. With this guide tube in place, the fiber was smoothly
pushed through the positioner.

The nominal distance from the robot mount flange to the focal surface is 86.5
mm. We chose to set the axial position of the fibers optically rather than by
a mechanical stop. This relieved us of any concerns about damaging the fiber
tips. It also naturally built traceable test data (automated optical focus
scans) directly into the alignment process. After inserting the PFA, we
clamped the robot into alignment fixtures like that shown in figure 28. The
ferrule was manually adjusted to an approximate focus location. Then, at the
rear, the fiber was lightly clamped by soft foam to a motorized stage
(Thorlabs MT1/M-Z8), which was axially independent of the positioner. The
fiber was backlit with an LED, and the tip was imaged by an Edmond Optics CMOS
Color USB Camera (EO-3112C), with a Mitutoyo 10x Objective lens.

An automated script stepped the fiber through several axial positions, imaging
the fiber at each step. A parabola was fit to the imaged spot widths, with its
minimum indicating best focus. The stage returned the fiber to the best focus
position, completing the program. We used epoxy to retain the ferrule at the
front of the robot. A small set screw inside the robot’s eccentric axis arm
held the ferrule in place while the glue cured.151515When the set screw was
over-tightened, it would crack the glass ferrule. Even with calibrated
wrenches, torque can be challenging to finely regulate when fastening such
small screws (size 0 – 80). We lost several fibers to this failure mode.

In the aft section of the robot, where the fiber emerges, we loosely retained
it in an axial groove, alongside the $\phi$ motor wires, using three phosphor-
bronze clips. At the rear of the electronics board, we transitioned from the ø
0.4 mm polyimide sleeve to ø 0.9 mm Hytrel furcation tube. We made the
transition between these tubes by gluing them into either side of a PEEK
‘hardpoint’, which was snapped into a clip on the electronics board. The fiber
was free to slide there; we axially restrained it only at the ferrule and at
the far end of the PFA (inside the spool box, where there is 0.5 – 1 m of free
slack prior to the soft foam mount pads that retain the splice joint).

The remaining $\sim$ 2.9 m of fiber, inside its furcation tube, was then
spooled on to a custom injection-molded plastic holster. The holster had an
integrated trough designed to hold the robot snugly, and a clamshell cover. In
this state the robot and fiber assembly was safe and convenient for handling
and storage.

Calibration of the focus alignment fixture was done with an undersized,
machined aluminum rod, to which we wrung gauge blocks as necessary to achieve
the exact final length. The rod had mounting features and center of gravity to
match a fiber positioner. Aluminum was chosen for thermal expansion matching
to the robot’s upper housing. To calibrate, we installed the rod in the
fixture and focused the camera on the face of the front gauge block.

Figure 28:  The fiber alignment fixture (left) has a camera and microscope
mounted at one end on a manual micrometer stage and a motorized fiber
adjustment stage at the other. Between them, the robot mounts in a vee-and-
flange, which we designed for functional similarity to the true focal plate
interface, while allowing installation and removal using a simple toggle
clamp, with no rotation of the assembly required along its screw threads. At
the motorized stage, the fiber is gripped by another toggle clamp actuating a
pivot arm with a soft foam grip. Temporary ø 0.9 mm furcation tubing is used
to assist us in sliding the fiber through the mechanical assembly. At upper
right is a closer view of the spring-loaded vee-and-flange clamping interface.
At lower right, a calibration rod is loaded in the clamp.

We found it challenging at times to maintain perfect calibration of the
installation fixture’s focus position. In order to match the mounting of fiber
positioner robots, we had designed the calibration rod with a similarly small
flange (which sets focus position when the robot is screwed into the focal
plate). However, a single positioner only undergoes one mechanical cycle in
and out of the fixture, whereas the calibration rod experienced hundreds.
Furthermore, we made the rod out of aluminum, to match the CTE of the robot’s
upper housing. The result was wear damage on the flange feature of the rod.
Asymmetries in the damage profile on the rod’s flange caused the focus point
to vary slightly from day to day, depending on what rotational angle the rod
had been installed in the fixture. These problems were identified midway
through production, tracked, and mitigated (see §6.2).

###  4.4 Positioner integration to petals

Prior to installation into a petal, we visually inspected each positioner,
cleaned it with ionized air, and commanded it to move. The motion was observed
by eye as a check of basic function. After checking that the fiber had not
been broken and could transmit light, the fiber was unspooled from its custom
holster and fed through a given mounting hole in the petal. The fiber was laid
as straight as possible on a long table behind the petal and then the
electrical connector and pigtail wires were fed through the hole. Low-strength
anaerobic threadlocker (Loctite 222) was applied to the M8.7 threads below the
flat mating surface (see fig. 11) and the positioner was lightly screwed in by
hand. Prior to cure of the threadlocker, the unit was torqued in to 0.8 N-m,
using a custom pin spanner wrench. We had designed 5 radial holes, ø 1.6 x 1.0
mm deep, into the flange of the theta bearing cartridge. The pin wrench
reached around the fiber arm and upper housing of the robot to these holes,
engaging them with a pin feature. Some complicated geometries were needed to
ensure reliable engagement and capture of the flange, so we had the wrenches
3D printed in 17-4PH steel, hardened to H900, using the 20 µm direct-metal
laser sintering process from Protolabs, Inc. This proved strong and effective.

The need to access each positioner first by hand and then with the torque
wrench meant that the positioners had to be installed serially, working across
the petal. Each step of installation was tracked in a Google Sheets
“traveler”, where we recorded information including positioner ID number,
installation hole number, and confirmation of electrical function. Both motors
($\phi$ and $\theta$) were moved to confirm basic mechanical function. The
positioner was then moved to a ‘parked’ location, with the $\phi$ motor fully
retracted and the $\theta$ motor in a neutral position, about halfway between
its hard limits. Fiducials were also installed in this sequence, in a similar
way to the positioners. We were able to install up to 50 positioners into a
petal per day.

At the transverse electronics boards, fibers were laid in groups of 12 or 13
into plastic guides, which separated them at this point from their robots’
corresponding electrical pigtails (see fig. 1). The 12 or 13 pigtails were
secured in guides which snapped on top of the fiber guides. These guide stacks
were mounted to the transverse board, adjacent to patches of 14 right-angle
electrical connectors. After 25 positioners had been installed and connected,
their fibers were gathered to the rear and fed into a black plastic conduit
for protection. This conduit was eventually secured to the large radial
aluminum plate which carries the radial electronics board. Once all
positioners were installed, the PFAs were ready to be spliced to the fiber
cables, as discussed in Poppett et al. 2022 (in prep).

###  4.5 Inventory and quality control

We had a pre-production phase for robot assembly, during which 440 positioners
were built at UM. During this period, we developed and refined our quality
control (QC) protocols, procedures, tooling, and work spaces. We built and
tested a SQL database for parts QC and inventory and to provide continuous
production statistics. The database allowed us to identify production problems
in a timely manner, and to mitigate these issues with the supplier or the
assembly teams as soon as possible. With 32 parts per positioner (22 custom
parts + 10 fasteners), and seven subassembly steps, the database helped
maintain production rates with a reliable part count and clear QC interfaces.
In addition to the SQL database, we made extensive use of Apple iOS
applications and Google Sheets. These provided flexible and accessible real-
time data management. A similar approach was used to track positioners as they
were shipped, stored, and moved through the final integration steps and QC,
including PFA installation and splicing. By the conclusion of the pre-
production phase, given sufficient parts supply, the UM team was capable of a
build rate of 50 positioners per day (see fig. 29).

The tolerances required for the positioners and the irreversibility of glued
assemblies necessitated 100% QC on each part, and regular evaluation of
tooling accuracy. We put considerable effort into developing QC processes that
were satisfactorily prognostic while remaining as simple as possible. Typical
inspection tools were dial indicators, granite surfaces, machined gauges, and
V-blocks. Visual inspections were performed on each part for burrs and
defects, on each subassembly for adhesive under- or over-fill, and for screw
engagement. Each bearing was manually turned to feel for friction, and a
microscope was used for a final inspection (Leitner et al., 2018). Each
subassembly was individually qualified to minimize part loss.

Figure 29:  Mass production of fiber positioners is plotted at left. We
produced a total of 7,148 robots. A pre-production run of 440 units (not shown
here) lasted from December 2016 – March 2017. Tooling and procedures were
finalized by the time of a May 2017 manufacturing readiness review. Full
production began in June 2017. By fall 2017, all manufacturing startup issues
had been resolved, and production accelerated to eventually exceed the planned
cadence (500 units per month) by mid-2018. Upper right: Members of the
production team at UM in the positioner assembly clean room. Lower right: Some
subassembly bonding fixtures.

In addition, the subassembly sequence was optimized to minimize loss of the
most expensive and long lead time components. During the pre-production phase
we determined our real-world production yields, and this defined the number of
parts required to yield at least 6000 production positioners (sufficient for
10 production petals and 2 spares). Our modular assembly procedure (having
clearly defined, discrete subassemblies, and continuous training of staff)
allowed us to continue steady production in circumstances when particular
component deliveries were delayed or available parts were non-compliant. We
would flexibly shift our workers to produce more subassemblies of the parts we
had in stock.

During the pre-production phase, we identified an issue where motor gear
trains could be compressed to a state of inoperability by a slight, nearly
zero force, axial motion of the output shaft. This was found by inspection of
CT scans of the positioner done at Los Alamos National Laboratory. Working
closely with our supplier we modified the motor assembly specifications and
adjusted our assembly technique to bias the assembly in order to reliably
achieve the required gear spacing of the motor assembly.

For a one-time mass production run, the rate of assembly and the yield were
excellent. UM’s diverse team of students, technicians, and engineers produced
150 – 200 units per week. Yields on the seven sub-assemblies ranged between 96
– 100%. For complete positioners, the yield was $98\%$. The biggest challenges
were (1) resolving the early problem with the motor gears that impeded our
pre-production run, and (2) maintaining sufficient parts inventory to keep the
production rolling.

###  4.6 GFA integration to petals

The GFA is mounted such that the planar active sensor area intersects the
curved focal surface (as shifted by the GFA filter), with 1/2 of the sensor
area above and 1/2 below focus. In the nominal mounting position, deviations
from focus due to this mismatch between the planar CCD sensor and the aspheric
DESI focal surface are $\sim$ 48 µm peak-to-valley across the sensor. We aimed
to mount all sensors to within 50 µm of this nominal position.

In each camera, IFAE measured the CCD sensor plane optically, and aligned it
to within 20 µm with respect to the mechanical base. Between the petal and the
camera, we placed an intermediate mounting plate, to which the camera would
interface. The mounting plate was attached to the petal with three M3 screws.
A shim stack beneath each screw provided the necessary adjustment of piston,
tip, and tilt.

We aligned the mount plates at LBNL, long before any actual GFA cameras had
been built. The locations of the plates were measured relative to the three
tooling balls on each petal with a touch CMM (see §4.7 and figure 14). Shims
were iteratively changed to achieve correct spatial alignment. It typically
took 2 – 3 measurement and adjustment iterations. The worst case error among
the mount plates was 7 µm.

Upon receipt of cameras at LBNL, we installed two GIFs (fiducials with mount
features specific to the GFA housing). A shim stack at each GIF mounting
location allowed us to independently align them to the focal surface. Prior to
installing cameras on petals, we measured the GIF dot positions relative to
the camera sensor. For this measurement we built a custom metrology rig,
consisting of a spot projector mounted on X-Y motorized stages, together with
a beam splitter and a Thorlabs DCU223M camera. First we would image the GIF
pinholes, then translate the stage to a pattern of positions over the CCD. The
spot would be projected through the GFA filter and read out with the camera.
Using this rig, the locations of the GFA sensors were measured relative to the
GIF pinholes with a lateral accuracy of 5 µm.

The metrology rig was also capable of imaging the reflection of its own
projected spot. We mounted the GFA on a separate, orthogonal stage with a
manually operated micrometer, to adjust the focal distance between the GFA and
spot projector. Around the GFA, we mounted and precisely measured three gauge
blocks, to establish a datum plane. Translating the camera with this stage, we
could determine the relative Z position of key features of the camera as
determined either by the image of GIF dots, the spot as read out on the
sensor, or as reflected off of exterior mechanical datum features.

After integration of robots, fibers, and electronics, the GFA was mounted on
the petal. The cooling fan and duct were installed, the camera was connected
to the GXB and Petal Controller, and a grounding wire was attached between the
GFA and the petal structure. Dark current and noise measurements were repeated
after installation to confirm that no grounding issues were present, keeping
care that a cover was always on the GFA sensor to avoid any saturation events.
We measured an average read noise of 18 e- RMS with a gain of 3.7 e-/ADU,
within the requirement of 25 e-/pixel.

###  4.7 Metrology of completed petals

The DESI operating model relies on being able to link the reference frames of
the GFA cameras and the fiber positioners, enabling alignment of the fibers
with targets based on their RA/DEC location (Honscheid et al., 2018). The
illuminated fiducials (FIFs and GIFs) provide the mutual reference points
between them. In the previous section we described how we linked the location
of the GFA sensors to the GIFs. Once all positioners and the GFA were
installed on a petal, we measured the locations of the FIFs and GIFs relative
to each other.

For this measurement, we used the ZEISS CMM at LBNL, which has an optical
probe in addition to the standard touch probe. The lateral position of the
optical probe was cross-calibrated to the touch probe by both imaging and
touching a reference ball. To cross-calibrate the focal position of the
optical probe, we used a flat surface on the GFA housing that could both be
touched and optically imaged. Due to possible tilts in the GFA, calibration at
this point induced an error in the focus measurements of $\pm$10 µm.

Our operator guided the CMM to each of 4 pinholes on each fiducial (FIFs and
GIFs), and used the optical probe, to measure their X, Y and Z locations. The
four corners of the GFA were measured with both the touch and optical probes,
and the face of one fiducial was additionally checked with the touch probe.
These measurements were repeated four times on each petal. To get statistics
on fiber focus, we also backlit all fibers and used the optical probe to
measure the locations of $\sim$ 50 fibers, repeating for a total of 3
measurements each.

###  4.8 Full petal tests in the lab

Following fiducial and fiber metrology, final functional tests and positioner
accuracy tests were completed in the lab at LBNL before shipping to Kitt Peak.
The main performance metric we wanted to confirm was that we could place the
fiber tips within 10 µm RMS of their nominal lateral target positions. This
was the first time that we were able to test much of the system software,
relying on interaction between the petal controller, move scheduling,
PlateMaker, FVC centroiding and spot matching, and fault management.

The tests took place in a Class 1,000 clean room at LBNL. The petal was set on
an integration cart with the positioners oriented horizontally (see fig. 30).
The fiber slit was installed in a temporary illumination system and an FVC was
positioned at the other end of the room, approximately 12.5 meters away. When
a petal was first set up, the fiducials were used by the FVC and PlateMaker to
establish a distortion map of the FVC lens and solve for the focal plate
coordinate system. When the positioners were backlit with the illuminator, the
FVC could measure the centroid of each positioner. Given the solved focal
plate coordinate system, the positioners were identified by Spotmatch. A
challenge in laboratory testing was that without the DESI Corrector, the chief
rays of the fibers along the curved focal surface did not point directly at
the camera. Depending on how the petal was oriented relative to the FVC, some
fibers and fiducials appeared dimmer than others. Once all fibers were
correctly identified by Spotmatch and the plate scale had been determined, the
positioners were calibrated by measuring arcs of points about their $\theta$
and $\phi$ axes.

Figure 30:  Petal testing in a clean room at LBNL. The metrology camera sits
$\sim$ 12.5 m from the petal, which is connected to a fiber cable on an
adjacent cart. The fiber slit is illuminated by an LED. The positioners lay
horizontally, pointing at the FVC metrology camera. Credit: Marilyn Chung/LBNL

During the lab tests, the anticollision software was not yet mature enough to
incorporate. We kept each positioner within a restricted patrol envelope, to
ensure that it could not overlap with its neighbors. We tested as-installed
positioner performance by moving all robots along square grids of 24 targets,
5 x 5 with no center target. The grids were $\sim$ 3 x 3 mm in size. We tested
up to 3 move iterations.

These were our first tests moving all 502 robots in a petal simultaneously. We
found several positioners with communication errors that would bring down the
CAN network for all devices on their bus. These were identified and unplugged
from their respective buses. The positioners remained mechanically in place,
and their fibers were left intact. Several robots started to show performance
issues that hadn’t been identified during production QC testing. Some of these
motors, mostly $\phi$, were found to be “sticky”, meaning that their motion
was slow and lurching. Other motors were seen to not move at all. Only grade A