# The atmospheric dispersion corrector for the Large Sky Area Multi–object
Fibre Spectroscopic Telescope (LAMOST)

Ding-qiang Su1,2,3, Peng Jia1,2,3 and Genrong Liu3
1Department of Astronomy, Nanjing University, 22 Hankou Road, Nanjing 210093,
China
2Key Laboratory of Modern Astronomy and Astrophysics (Nanjing University),
Ministry of Education, Nanjing 210093, China
3National Astronomical Observatories / Nanjing Institute of Astronomical
Optics & Technology (NIAOT), Chinese Academy of Science,
188 Bancang Street, Nanjing 210042, China E-mail<EMAIL_ADDRESS>
###### Abstract

The Large Sky Area Multi–object Fibre Spectroscopic Telescope (LAMOST) is the
largest (aperture 4 $\mathrm{m}$) wide field of view (FOV) telescope and is
equipped with the largest amount (4000) of optical fibres in the world. For
the LAMOST North and the LAMOST South the FOV are 5 ∘ and 3.5 ∘, the linear
diameters are 1.75 $\mathrm{m}$ and 1.22 $\mathrm{m}$, respectively. A new
kind of atmospheric dispersion corrector (ADC) is put forward and designed for
LAMOST. It is a segmented lens which consists of many lens–prism strips.
Although it is very big, its thickness is only 12 $\mathrm{mm}$. Thus the
difficulty of obtaining big optical glass is avoided, and the aberration
caused by the ADC is small. Moving this segmented lens along the optical axis,
the different dispersions can be obtained. The effects of ADC’s slits on the
diffraction energy distribution and on the obstruction of light are discussed.
The aberration caused by ADC is calculated and discussed. All these results
are acceptable. Such an ADC could also be used for other optical fibre
spectroscopic telescopes, especially those which a have very large FOV.

###### keywords:

Telescopes – Instrumentation: spectrographs – Atmospheric effects

††pagerange: The atmospheric dispersion corrector for the Large Sky Area
Multi–object Fibre Spectroscopic Telescope (LAMOST)–14††pubyear: 2011

##  1 Introduction

Large Sky Area Multi–object Fibre Spectroscopic Telescope (LAMOST) is a new
type telescope (Wang et al., 1996; Cui et al., 2000; Su & Cui, 2004). The main
parameters of LAMOST are the following: clear aperture 4 $\mathrm{m}$
(average), f–ratio 5, and field of view (FOV) 5 ∘. The linear diameter of FOV
is 1.75 $\mathrm{m}$. 4000 optical fibres (Xing et al., 1998), which introduce
the light of different celestial objects to 16 spectrographs (Zhu et al.,
2006), are put on such a big focal surface. The dedication ceremony of LAMOST
was held on 2008 October 16. We call it the LAMOST North. At present, another
telescope of this kind, the LAMOST South designed to survey the southern sky,
is under consideration by China and other countries, with a view to
international cooperation (Cui et al., 2010). The clear aperture and f–ratio
of the LAMOST South are still 4 $\mathrm{m}$ and $5$. But its FOV will be
reduced to 3.5 ∘. The linear diameter of FOV of this telescope is 1.22
$\mathrm{m}$. Many years ago some atmospheric dispersion correctors for small
FOV had been designed and used. Since 1980’s some atmospheric dispersion
correctors for larger FOV have been designed (Epps et al., 1984; Su, 1986; Su
& Liang, 1986; Wynne, 1986; Willstrop, 1987; Liang & Su, 1988; Su et al.,
1988; Bingham, 1988; Wynne, 1988; Wang & Su, 1990). In these correctors there
is a pair of prisms or lens–prisms (lensms), each of which is a cemented lens
with a tilted cemented surface and consists of two different glasses. Rotating
these two lens–prisms, we could obtain different dispersions. Using this
method in LAMOST would require very big lens–prisms, and it is difficult to
obtain optical glass of such large size and to support the big lens–prisms.
Liu & Yuan have designed several kinds of small corrector each for an optical
fibre (Liu & Yuan, 2005). But it is difficult to install and move them for
4000 optical fibres. In this paper, as an example a detailed design and
discussion of this ADC are given for the LAMOST South. It is a big but thin
segmented lens which consists of many lens–prism strips. Thus the difficulty
of obtaining such big optical glass is avoided, and since it is thin, the
introduced aberration is small. Moving this segmented lens along the optical
axis, we could obtain different dispersions. The entire design of this ADC
could be used for the LAMOST North too—we only need to extend its diameter to
about 1.78 $\mathrm{m}$. In this ADC the clear aperture of a portion of
celestial objects is divided into two parts, i.e., a slit is added in it. Here
the word “silt” means silt plus chamfer at the edge of each lens–prism strip
and they are covered with black paint, i.e., in this paper “slit” means a
light-obstructing black belt with a width of 1 $\mathrm{mm}$. In this case the
diffraction spot is enlarged. But in LAMOST, as the following discussion will
show, it is not serious and it is acceptable. Apart from this, there is some
loss from light obstruction by the silt. For different celestial objects, the
loss is different in the whole FOV, and also different when ADC is at
different positions. For these two reasons mentioned above, such a segmented
ADC is not suitable for the diffraction–limited high resolution telescope and
photometry. Such a segmented ADC is mainly used for the optical fibre
spectroscopic telescope, especially if this telescope has a very large FOV.

##  2 A brief introduction of LAMOST

LAMOST is shown in Figure 1. It lies on the ground along the South–North
direction. Mb is a spherical mirror. Ma is an aspherical reflecting plate,
which corrects the spherical aberration of Mb and reflects the light of
celestial objects to the Mb. Ma and Mb consists of 24 and 37 hexagonal mirrors
respectively and each mirror has diagonal 1.1 $\mathrm{m}$. In a segmented
mirror telescope PSF is enlarged, especially when a segmented ADC is added. At
a good astronomical site, if we do not use adaptive optics, atmospheric seeing
only allows a telescope with an aperture under 30 $\mathrm{cm}$ to obtain the
diffraction–limited image. Because adaptive optics cannot be used for a large
FOV telescope, therefore, in this case for a segmented mirror telescope a sub-
area of diameter 50–60 $\mathrm{cm}$ is enough. The other reason that LAMOST
North uses two segmented mirrors is to save the cost. By the way, it is
possible that LAMOST South will use one segmented mirror consisting of 1.1
$\mathrm{m}$ hexagonal sub–mirrors. On the other hand, in many optical fibre
spectroscopic telescopes the diameter of optical fibre is more than 1.5
$\mathrm{arcsec}$. Compared with the seeing and diameter of the optical fibre,
the effect of enlarged PSF in LAMOST including segmented ADC is minor and
acceptable. During the observation, for a particular observation direction (a
particular sky area), LAMOST is a reflecting Schmidt telescope. But when
observing different directions (i.e. different celestial objects or same
celestial objects in different times), LAMOST should be different reflecting
Schmidt telescopes. This means the shape of Ma should be different for
correcting the spherical aberration. Traditionally, such an optical system can
not be realized. Active optics is a key technology for correcting the
telescope errors–gravitation deformation and thermal deformation. This
technology was developed mainly by Wilson and Lemaitre et.al. (Wilson, 1999;
Lemaitre, 2009) for the thin mirror, and by Nelson and Mast et.al. (Nelson,
1980; Mast & Nelson, 1980, 1982) for the segmented mirror. Chinese experts
have creatively applied active optics technology to Ma to make such a
telescope possible and developed the thin–mirror and segmented–mirror combined
active optics (Su et al., 1986, 1998; Cui et al., 2000; Su & Cui, 2004; Cui,
2008). LAMOST is a wide FOV telescope with the largest aperture and has the
strongest fibre spectroscopic obtaining capability in the world. The observing
sky area of LAMOST is $-10^{\circ}\leqslant\delta\leqslant+90^{\circ}$. The
celestial objects are observed for an average of 1.5 hours before and after
they pass through the meridian. During observation only the mounting of Ma
does the tracking and the focal surface does the rotation. LAMOST North was
set up in Xing Long Station (latitude 40.4 ∘ N, height above sea level 900
$\mathrm{m}$), National Astronomical Observatories, Chinese Academy of
Sciences. The biggest zenith distance (with the celestial object in the
meridian, it is the same in the following text) is 50.4 ∘. For waveband $380$
– $1000\mathrm{nm}$ the atmospheric dispersion is 2.2 $\mathrm{arcsec}$. At
Xing Long Station FWHM of seeing is about 2 $\mathrm{arcsec}$. The biggest
spread of aberration is 1.84 $\mathrm{arcsec}$. The diameter of optical fibre
adopted is 3.3 $\mathrm{arcsec}$. In this situation, although it is better to
correct the atmospheric dispersion, leaving it uncorrected will not present
serious problem. Through the development of the LAMOST North, Chinese experts
found that the size of each optical fibre positioner could be significantly
reduced, so in the LAMOST South the FOV will be reduced to 3.5 ∘, the linear
diameter of it is 1.22 $\mathrm{m}$, and 6000 optical fibres could be put on
this focal surface. The observing sky area of the LAMOST South is
$0^{\circ}\geqslant\delta\geqslant-90^{\circ}$. The LAMOST South may be
installed on the the NOAO Las Campanas Observatory or the ESO Paranal
Observatory. NOAO Las Campanas Observatory is situated at latitude 29.9 ∘ S,
altitude 2400 $\mathrm{m}$ above the sea level and the biggest zenith distance
observed is 60.1 ∘. ESO Paranal Observatory is situated at latitude 24.6 ∘ S,
altitude 2635 $\mathrm{m}$ above the sea level and the biggest zenith distance
observed is 65.4 ∘. For waveband $380$ – 1000 $\mathrm{nm}$ the corresponding
atmospheric dispersion at these two observatories are 2.75 $\mathrm{arcsec}$
and 3.35 $\mathrm{arcsec}$ respectively. Since image quality (due to the
reduction of FOV to 3.5 ∘) and two observatories’seeing are better than the
LAMOST North, the diameter of optical fibre used will be 1.6
$\mathrm{arcsec}$. In the LAMOST South the atmospheric dispersion should be
corrected. Since at ESO Paranal Observatory the biggest atmospheric dispersion
is bigger, it is chosen as an example in this paper.

##  3 The structure of this ADC

The layout and specification of this ADC are shown in Figure 2,3 and Table 1.
Each lens–prism strip consists of Schott glass PSK3 and LLF1. The refractive
index of PSK3 and LLF1 are given in Table 2. In $\lambda$ $=$ 441.8
$\mathrm{nm}$ the refractive indices of the two kinds of glass are the same.
In LAMOST the Mb and the focal surface are concentric. The radius of focal
surface is 20 $\mathrm{m}$. When ADC is at the farthest position, we take its
two outside surfaces and the focal surface to be concentric, i.e. the radius
of the outside surface equals 20 $\mathrm{m}$ \+ 250 $\mathrm{mm}$. We require
that in this position for waveband $380$ – 1000 $\mathrm{nm}$ this ADC can
produce dispersion of 3.35 $\mathrm{arcsec}$. From it the tilt angle of
lens–prism strip can be obtained which is 6.89 ∘. We set the width of each
lens–prism strip to be 50 $\mathrm{mm}$ which equals to the light beam
diameter of each celestial object on this ADC. Thus in this position only one
slit is added for each object. Since the thickness difference of one lens of
lens–prism strip is 6.04 $\mathrm{mm}$, we take the total thickness of ADC is
12 $\mathrm{mm}$. Since the f–ratio of LAMOST is 5, the diameter of ADC should
equal the linear FOV diameter plus 50 $\mathrm{mm}$ $(250/5=50)$, i.e., 1.27
$\mathrm{m}$ for the LAMOST South. So the maximum length of the strip is also
1.27 $\mathrm{m}$ for the LAMOST South. A similar result for the LAMOST North
is 1.78 $\mathrm{m}$. If one feels that some lens–prism strips are too long in
the LAMOST North, those strips longer than 1 $\mathrm{m}$ could be divided
into two parts. In this case, only about $1/20$ celestial objects will meet
two slits when ADC is at the farthest position. Although such an ADC is very
big, it is very thin and only includes one segmented lens. As it is moved
along the optical axis its dispersion is changed. Thus the atmospheric
dispersion for $z<65.4$ ∘ celestial objects can be compensated. The dispersion
produced by this ADC is direct proportion to the distance from it to focal
surface. And in these situations only one slit is met for a part of celestial
objects. When the atmospheric dispersion is small enough which needs not to be
compensated, this ADC can be moved out easily. With regard to the
manufacturing of the ADC, we plan to glue these lens–prism stripes together
(with removable glue) to form a disk, then grind and polish it. We have
already had some experience with such a method. Since the maximum light beam
of each celestial object on ADC is only 50 $\mathrm{mm}$, the figure tolerance
of lens–prism strip is loose. As a whole this ADC’s tolerance of position, tip
and tilts are loose. All lens–prism strips are fixed on the edge of the frame.
So long as each lens-prism strip is well fixed, its tip and tilts will be
small. Liquid glue may be used for the cemented surface to reduce the
reflecting loss. Nevertheless, moderate difficulties do exist for the
manufacturing and mounting of the ADC, thus it is still necessary to conduct
more research and testing in this respect.

##  4 Some discussions on special topics

###  4.1 The effect of ADC’s slit on diffraction energy distribution

In LAMOST both Ma and Mb consist of hexagonal mirrors each with a diagonal of
1.1 $\mathrm{m}$. The surface area of such a hexagonal mirror equals to a
circular area with a diameter of 1 $\mathrm{m}$. For different celestial
objects these sub–mirrors of Ma and Mb are covered by each other in the clear
aperture with different states. Since in LAMOST all sub–mirrors only co–focus,
the light from different sub–areas is non–coherent and these shapes of
sub–areas divided by edges of hexagonal mirrors are complex. First we discuss
the case when both Ma and Mb are approximately perpendicular to the optical
axis, i.e., the angle between the light of the celestial object and the
optical axis pointing to Mb is small. It can be found that each hexagonal
sub–mirror is mainly divided into 3 to 4 sub–areas. If we assume 4 sub–areas,
an important conclusion can be obtained: the average surface area of sub–area
equals a circular area with a diameter of 0.5 $\mathrm{m}$. As a rough
estimate we think that the diffraction energy distribution of LAMOST in this
case is like a circular hole with a diameter of 0.5 $\mathrm{m}$, i.e., 84
$\mathrm{percent}$ of the light energy spreads in about 0.5 $\mathrm{arcsec}$
area (Airy disk). Xu made a detailed calculation for four special situations,
and a similar conclusion was obtained (Xu, 1997). Since LAMOST’s clear
aperture is 4 $\mathrm{m}$, its surface area is 64 times of a 0.5 $\mathrm{m}$
circular area. We could think that about 64 sub–areas are included in LAMOST
clear aperture. For a particular celestial object in the worst situation its
clear aperture is divided by an ADC’s slit along the diameter direction, thus
about eight sub–areas are divided. As an average result the diffraction energy
of the eight sub–areas will distribute in two times its original length in
perpendicular direction to the slit, i.e., the 84 $\mathrm{percent}$
diffraction energy will spread in an area 0.5 $\mathrm{arcsec}$ wide and 1
$\mathrm{arcsec}$ long, i.e., half of the energy will disperse beyond the 0.5
$\mathrm{arcsec}$ area. Thus in the 0.5 $\mathrm{arcsec}$ area the light
energy will reduce by $4/64$, i.e., about 6 $\mathrm{percent}$. The energy
loss is small and it still distributes in 1 $\mathrm{arcsec}$ area. Given that
the diameter of optical fibre is 1.6 $\mathrm{arcsec}$, the energy loss is
acceptable. Secondly, we discuss the situation where the celestial objects
observed are near the celestial pole. In this case, in a plane perpendicular
to the optical axis the projective width of sub–mirrors of Ma will reduce to
about 1/2, but the projective width of sub–mirrors of Mb is unchangeable.
Considering these two factors and using a method of analysis similar to the
above, we obtain the following conclusion: in this case in LAMOST 84
$\mathrm{percent}$ of the light energy spreads in an area about 0.5
$\mathrm{arcsec}$ wide and 0.75 $\mathrm{arcsec}$ long and in this area the
light energy will reduce about 4.5 $\mathrm{percent}$ due to an ADC’s slit.
The energy loss is small and it distributes in a 1.5 $\mathrm{arcsec}$ area.
Given that the diameter of optical fibre is 1.6 $\mathrm{arcsec}$, the energy
loss is acceptable. In LAMOST South either Ma or Mb may adopt a non–segmented
(monolithic) mirror. It is easy to find that if either Ma or Mb is a
non–segmented mirror or consists of larger sub–mirrors, with such a segmented
ADC the total diffraction energy distribution will be more concentrated than
in the above situation.

###  4.2 The probability of objects meet slit

In the largest zenith distance $\mathrm{z}$ = 65.4 ∘, each object will meet a
slit of strips. When $\mathrm{z}<$65.4 ∘, i.e., the distance between ADC and
focal surface is less than 250 $\mathrm{mm}$, only a part of objects meet a
slit of lens–prism strips. For example, if the distance between ADC and focal
surface is 100 $\mathrm{mm}$ (according to $\mathrm{z}$ = 39.6 ∘, see section
5 and Table 3) only $2/5$ objects meet a slit and if the distance between ADC
and focal surface is 50 $\mathrm{mm}$ (according to $\mathrm{z}=$20.6 ∘), only
$1/5$ objects meet a slit.

###  4.3 The light obstructed by slits of ADC

According to a technical requirement, the edge of strips of ADC should be
chamfered to a projective width of 0.5 $\mathrm{mm}$. In order to reduce
scattered light, all chamfers and slits of ADC should be covered with black
paint. Thus all slits will become black belts with a width of 1 $\mathrm{mm}$
between two strips. These slits will obstruct light. The width of each strip
is 50 $\mathrm{mm}$. The approximate average obstructed ratio equals 1/50 = 2
$\mathrm{percent}$. The thickness of ADC is 12 $\mathrm{mm}$. The f–ratio of
LAMOST is 5 and the maximum inclination angle of ray is 1/10 to ADC’s surface.
Considering these situations, we obtain that about an average of 2.5
$\mathrm{percent}$ of light will be obstructed. This is the average light
loss. For a specific celestial object, this loss may be zero when the object
does not meet a slit, and it may be several times the average loss when ADC is
near the focal surface and the object meets a slit. Since the loss of light
obstruction by a slit is uneven, such a segmented ADC is not suitable for
photometry.

###  4.4 The ADC’s orientation

Both the atmospheric dispersion and ADC’s dispersion are vectors. The
compensating error equals ADC dispersion vector plus the atmospheric
dispersion vector. For compensation not only the amounts of the two vectors
should be equal but also their directions should be opposite to each other. In
a telescope the ADC should be rotated to make its dispersion direction
opposite to the atmospheric dispersion direction. From spherical astronomy the
ADC’s orientation formula can be derived easily. The tolerance of ADC’s
orientation angular is loose.

###  4.5 The aberrations

First, we ignore the tilt of cemented surface of ADC. Since the radii of ADC
is very large, it can be considered as a parallel glass plate. From the
third–order aberration formula the least circle of spherical aberration is
only 0.01 $\mathrm{arcsec}$ and it could be corrected by active optics. From
the chromatic aberration formula the spread circle of it is 0.17
$\mathrm{arcsec}$ at the extreme wavelengths 380 $\mathrm{nm}$ and 1000
$\mathrm{nm}$. For a glass parallel plate the spherical aberration and
chromatic aberration are unchanged when it moved along optical axis. When the
tilt of cemented surface of ADC is considered the aberration mainly coma will
be added. It increases with the increasing difference of the refractive index
of two glasses, i.e., it is maximum in two extreme wavelengths. And this
aberration is direct proportion to the distance from ADC to the focal surface.
Here we do not use formula to calculate it. In next section by using Zemax
software all aberrations, indicated by spot diagrams, will be given including
this aberration.

###  4.6 The compensation error of atmospheric dispersion

Due to the difference between the atmosphere and the glass dispersion, the
compensation error is existent. It is just like as secondary spectrum in an
achromatic optical system. Apparently it is direct proportion to the amount of
the compensated atmospheric dispersion.

##  5 The calculation and following discussion

The Zemax is used for the following calculations.
In this paper only the aberrations caused by the ADC are analyzed i.e. the
aberrations of original optical system are ignored.
We take the two outside surfaces of the ADC to be concentric with the focal
surface when this ADC is at the farthest position. In this situation if we
ignore the structure of strip in the whole FOV the images are the same. We
only need to calculate and discuss the case where the object is at the centre
of FOV but with different relations to the strips. For centre objects we take
two states: (1) the light beam area of object is at the middle of a lens–prism
strip; (2) a slit of two lens–prism strips is at the centre of the light beam
area of this object, i.e., the light of this object is half in one strip and
half in another strip. In this section all spot diagrams are calculated for
these two states. For state (1) we adjust the tilt angle of the cemented
surface of lens–prism strip to make the image centroids of $\lambda$ $=$ 380
$\mathrm{nm}$ and $\lambda$ $=$ 1000 $\mathrm{nm}$ coincide at $\mathrm{z}$
$=$ 65.4 ∘, i.e., to remove the atmospheric dispersion at these two extreme
wavelengths. Thus the tilt angle 6.89 ∘ mentioned above is obtained. In this
situation we found at wavelength about 500 $\mathrm{nm}$, the image centroid
is farthest from the co–centre of $\lambda$ $=$ 380 $\mathrm{nm}$ and
$\lambda$ = $1000$ $\mathrm{nm}$, the angular distance of them is the maximum
compensation error. Since all images should be in the one same focal surface
in calculation of state (2) the focal surface position of state (1) is used.
Since the radii of ADC are fixed when the distance between ADC and the focal
surface is less than 250 $\mathrm{mm}$, the two outside surfaces of ADC are
not concentric with the focal surface. In this situation image quality in
centre and off–axis of FOV are different. But the difference is small. We
still only calculate and discuss where the object is at the FOV centre and in
the above two states, i.e., (1) the light beam area of object is at the middle
of a lens–prism strip. For each decided position of ADC the zenith distance is
chosen to make the image centroids of $\lambda$ = 380 $\mathrm{nm}$ and
$\lambda$ $=$ 1000 $\mathrm{nm}$ coinciding, i.e., eliminate the atmospheric
dispersion at these two extreme wavelengths. (2) a slit of two lens–prism
strips is at the centre of the light beam area of this object. In this state
the focal surface position of state (1) is used.
These main calculation results are shown in Figure 4, 5, 6, 7, 8 and Table
3,4,5. It is clear that the predictions in 4.5 and 4.6 are proved and some
specific values are obtained: the largest monochromatic image at extreme
wavelength is about 0.18 $\mathrm{arcsec}$, mainly is chromatic aberration.
This figure shows no apparent change as the ADC moves along the optical axis.
From Figure 4, 5, 6, 7 and 8 we can find some coma induced by tilt cemented
surface and it reduces with the ADC’s moves towards the focal surface. The
maximum compensation error of atmospheric dispersion is 0.29 $\mathrm{arcsec}$
for the atmospheric dispersion 3.35 $\mathrm{arcsec}$, i.e., the compensation
error is about $1/12$ of the atmospheric dispersion. The total spread in the
waveband between $\lambda$ $=$ 380 $\mathrm{nm}$ and $\lambda$ = 1000
$\mathrm{nm}$ is about 0.4 $\mathrm{arcsec}$. In Figure 4, 5, 6, 7 and 8, the
same wavelength images are separated about 0.04 $\mathrm{arcsec}$ because the
light of this object is half in one strip and half in the other strip. These
two tilt cemented surfaces have different distances to focal surface. They
produce different dispersions, one bigger and the other smaller than the
average dispersion. By the way, the most part of one semi–circle of lens–prism
strip is LLF1, the other most part is PSK3, they produce the difference
chromatic aberration, so the sizes of chromatic spread are different. Even
though a 0.4 $\mathrm{arcsec}$ geometrical aberration is brought, it is worthy
to use such a simple ADC to correct 3.35 $\mathrm{arcsec}$ atmospheric
dispersion. Based on these analysis and calculation above, some extending
results could be obtained. For example if the thickness of ADC increases to 18
$\mathrm{mm}$, the chromatic aberration increase about 0.085 $\mathrm{arcsec}$
and the total spread from waveband $\lambda$ $=$ 380 $\mathrm{nm}$ to
$\lambda$ $=$ 1000 $\mathrm{nm}$ increases to about 0.45 $\mathrm{arcsec}$. In
this situation the wide of each strip can increase to 75 $\mathrm{mm}$ or even
100 $\mathrm{mm}$, thus ADC at the farthest position only $2/3$ or even $1/2$
objects will meet a slit.

##  6 Conclusion

In this paper a new kind of atmospheric dispersion corrector (ADC) is put
forward. It is a segmented lens which consists of many lens–prism strips. As
an example, such an ADC is discussed and designed for the LAMOST South. Its
linear diameter of FOV is 1.22 $\mathrm{m}$ and its f–ratio is 5. Although
this ADC’s diameter is 1.27 $\mathrm{m}$, its thickness is only 12
$\mathrm{mm}$. Thus the difficulty of obtaining big optical glass is avoided,
and the aberration caused by the ADC is small. When we move this segmented
lens along the optical axis, the different dispersions can be obtained. These
slits of ADC will produce about 2.5 $\mathrm{percent}$ average obstruction
loss of light. In the largest zenith distance each celestial object’s light
only meets one slit of this ADC, and at other zenith distance only a part
object light meet a slit. The effects of ADC slits on the diffraction energy
distribution are discussed. Since LAMOST is used for optical fibre
spectroscopic observation and the diameter of optical fibre is 1.6
$\mathrm{arcsec}$ (LAMOST South) and 3.3 $\mathrm{arcsec}$ (LAMOST North),
these effects are acceptable. From waveband $\lambda$ $=$ 380 $\mathrm{nm}$ to
$\lambda$ $=$ 1000 $\mathrm{nm}$ a total spread of aberration 0.4
$\mathrm{arcsec}$, which mainly is the compensation error of dispersions and
achromatic aberration, will be brought when we use such an ADC for
compensating 3.35 $\mathrm{arcsec}$ atmospheric dispersion. In this segmented
ADC the diffraction spot is enlarged and the loss of light obstruction by slit
is uneven. For these two reasons, such a segmented ADC is not suitable for the
diffraction-limited high resolution telescope and photometry. It is mainly
used for the optical fibre spectroscopic telescope. Since this ADC is thin,
the aberration caused is small, there is no difficulty for obtaining its
glasses, its thickness does not increase with the enlarging of FOV, and it has
almost the same image quality for the center and off–axis of FOV. Such an ADC
is especially suitable for very large FOV optical fibre spectroscopic
telescopes.

## Acknowledgments

Thanks to Professor Xiangqun Cui for her enthusiastic support and helpful
discussion, and to Professor Xiangyan Yuan for her helpful discussion and
assistance during calculation.

## References

  * Bingham (1988) Bingham R. G., 1988, in Proceedings of ESO Conference On Very Large Telescopes and their Instrumentation, ed. M.-H. Ulrich, 1167
  * Cui et al. (2000) Cui X., Su D., Wang Y., 2000, in Optical Design, Materials, Fabrication, and Maintenance, ed. P. Dierickx, Proc. SPIE Vol. 4003, 347
  * Cui (2008) Cui X., 2008, in Ground-based and Airborne Telescopes II, eds. L. M. Stepp, and R. Gilmozzi, Proc. SPIE Vol. 7012, 701204
  * Cui et al. (2010) Cui X., Wang S., Su D., Zhao Y., Wang Y., Chu Y., Li G., 2010, in Ground-based and Airborne Telescopes III, eds. L. M. Stepp, R. Gilmozzi, and H. J. Hall, Proc. SPIE Vol. 7733, 77330B
  * Epps et al. (1984) Epps H. W., Angel J. R. P., Anderson E., 1984, in Very Large Telescopes their Instrumentation and Programs, eds. M.-H. Ulrich, and K. Kjar, Proc. IAU Coll., 79, 519
  * Lemaitre (2009) Lemaitre G. R., 2009, Astronomical Optics and Elasticity Theory, Springer–Verlag Berlin Heidelberg
  * Liang & Su (1988) Liang M., Su D., 1988, in Proceedings of ESO Conference on Very Large Telescopes and their Instrumentation, ed. M.-H. Ulrich, 237
  * Liu & Yuan (2005) Liu G., Yuan X., 2005, Acta Astron. Sin., 50, 224
  * Mast & Nelson (1980) Mast T., and Nelson J., 1980, in Proceedings of Kitt Peak National Observatory Conference on Optical and Infrared Telescopes for the 1990s, ed. A. Hewitt, 508
  * Mast & Nelson (1982) Mast T. S., Nelson J. E., 1982, Appl. Opt., 21, 2631
  * Nelson (1980) Nelson J. E., 1980, in Proceedings of Kitt Peak National Observatory Conference on Optical and Infrared Telescopes for the 1990s, ed. A. Hewitt, 631
  * Su (1986) Su D., 1986, A&A, 156, 381
  * Su & Liang (1986) Su D., Liang M., 1986, in Proceedings of SPIE Conference on Advanced Technology Optical Telescopes III, ed. L.D. Barr, 628, 479
  * Su et al. (1986) Su D., Cao C., Liang M., 1986, in Proceedings of SPIE Conference on Advanced Technology Optical Telescopes III, ed. L.D. Barr, 628,479
  * Su et al. (1988) Su D., Wang Y., Yi M., 1988, in Proceedings of ESO Conference on Very Large Telescopes and their Instrumentation, ed. M.-H. Ulrich, 247
  * Su et al. (1998) Su D., Cui X., Wang Y., Yao Z., 1998, in SPIE Conference on Advanced Technology Optical/IR Telescopes IV,ed. L. M. Stepp, Vol .3352, 76
  * Su & Cui (2004) Su D., Cui X., 2004, Chinese J. of Astron. and Astrophys., 4, 1
  * Wang et al. (1996) Wang S., Su D., Chu Y., Cui X., Wang Y., 1996, Appl. Opt., 35, 5155
  * Wang & Su (1990) Wang Y., Su D., 1990, A&A, 232, 589
  * Willstrop (1987) Willstrop R. V., 1987, MNRAS, 225, 187
  * Wilson (1999) Wilson R. N., 1999, Reflecting Telescope Optics II, Springer–Verlag Berlin Heidelberg
  * Wynne (1986) Wynne C. G., 1986, Observatory, 106, 163
  * Wynne (1988) Wynne C. G., 1988, MNRAS, 230, 457
  * Xing et al. (1998) Xing X., Zhai C., Du H., Li W., Hu H., Wang R., Shi D., 1998, in SPIE Conference on Advanced Technology Optical/IR Telescopes IV, ed. L. M. Stepp, Vol. 3352, 839
  * Xu (1997) Xu W., 1997, Acta Astrophysica Sinica, 17, 96
  * Zhu et al. (2006) Zhu Y., Hu Z., Zhang Q., Wang L., Wang J., 2006, in Ground-based and Airborne Instrumentation for Astronomy, eds. I. S. McLean, and M. Iye, Proc. SPIE 6269, 62690M–1

Figure 1: The LAMOST and the position of the ADC in LAMOST

Figure 2: A sectional drawing of the ADC and focal surface. In this figure the
radii of ADC and focal surface have been reduced (i.e., more bended), and the
thickness of the ADC has been enlarged.

Figure 3: The figure (a) shows the light beam area of the object is at the
middle of a lens–prism strip and the figure (b) shows a slit of two lens–prism
strips is at the centre of the light beam area of the object.

Table 1: The structure parameters of the ADC Surface | Radius ($\mathrm{mm}$) | Separation ($\mathrm{mm}$) | Glass | Tilt Angle (∘)
---|---|---|---|---
1 | 20250 |  |  |
|  | 6 | LLF1 |
2 | 20244 |  |  | 6.89
|  | 6 | PSK3 |
3 | 20238 |  |  |
|  | 238 |  |
Focal Surface | 20000 |  |  |
Table 2: The refractive index of the Schott glass PSK3 and LLF1 $\lambda$ ($\mathrm{nm}$) | 380 | 441.8 | 500 | 587.6 | 656.3 | 1000
---|---|---|---|---|---|---
LLF1 | 1.574977 | 1.562383 | 1.555066 | 1.548138 | 1.544564 | 1.535632
PSK3 | 1.570682 | 1.562383 | 1.557313 | 1.552320 | 1.549650 | 1.542388

Figure 4: The spot diagram when the distance from the first surface of the ADC
to the focal surface is 250 $\mathrm{mm}$. The upper figure shows the spot
diagram when the light beam area of the object is at the middle of a
lens–prism strip and the lower figure shows when a slit of two lens–prism
strips is at the centre of the light beam area of the object. The length of
the scale line (the line beside the spot diagram) represents 40
$\mu\mathrm{m}$ (0.413 $\mathrm{arcsec}$). The different colour of the
diagrams in the picture represent different wavelength: blue, $\lambda$ 380
$\mathrm{nm}$, the spot diagram in the top of the upper figure; green,
$\lambda$ 500 $\mathrm{nm}$, the spot diagram in the lowermost of the upper
figure; red, $\lambda$ 1000 $\mathrm{nm}$, the spot diagram in the middle of
the upper figure.

Figure 5: The spot diagram when the distance from the first surface of ADC to
the focal surface is 200 $\mathrm{mm}$. The rest of explanation is the same as
in Figure 4.

Figure 6: The spot diagram when the distance from the first surface of ADC to
the focal surface is 150 $\mathrm{mm}$. The rest of explanation is the same as
in Figure 4.

Figure 7: The spot diagram when the distance from the first surface of ADC to
the focal surface is 100 $\mathrm{mm}$. The rest of explanation is the same as
in Figure 4.

Figure 8: The spot diagram when the distance from the first surface of ADC to
the focal surface is 50 $\mathrm{mm}$. The rest of explanation is the same as
in Figure 4.

Table 3: The distance from the first surface of ADC to the focal surface, corresponding to the object’s zenith distance at Paranal Observatory and the diameter of the light beam area of the object on the ADC. Distance from the first surface |  |  |  |  |
---|---|---|---|---|---
of ADC to the focal surface ($\mathrm{mm}$) | 250 | 200 | 150 | 100 | 50
The object’s zenith distance |  |  |  |  |
at Paranal Observatory (∘) | 65.4 | 59.9 | 51.9 | 39.6 | 20.6
The diameter of the light beam area of |  |  |  |  |
the object on the ADC ($\mathrm{mm}$) | 50 | 40 | 30 | 20 | 10
Table 4:  The correction results of the ADC when the light beam area of the object is at the middle of a lens–prism strip. Largest spread of spot diagram and compensation error are in $\mathrm{arcsec}$. Distance from the first surface |  |  |  |  |
---|---|---|---|---|---
of ADC to focal surface ($\mathrm{mm}$) | 250 | 200 | 150 | 100 | 50
Largest spread of spot diagram $\lambda$ 380 $\mathrm{nm}$ | 0.180 | 0.180 | 0.182 | 0.183 | 0.185
Largest spread of spot diagram $\lambda$ 500 $\mathrm{nm}$ | 0.033 | 0.029 | 0.027 | 0.024 | 0.021
Largest spread of spot diagram $\lambda$ 1000 $\mathrm{nm}$ | 0.172 | 0.168 | 0.165 | 0.162 | 0.160
Largest spread of spot diagram |  |  |  |  |
of the whole waveband $\lambda$ 380 $\mathrm{nm}$ – 1000 $\mathrm{nm}$ | 0.401 | 0.343 | 0.279 | 0.215 | 0.185
Compensation error | 0.289 | 0.227 | 0.175 | 0.113 | 0.041
Table 5:  The correction results of the ADC when a slit of two lens–prism strips is at the centre of the light beam area of the object. Largest spread of spot diagram and compensation error are in $\mathrm{arcsec}$. Distance from the first surface |  |  |  |  |
---|---|---|---|---|---
of ADC to focal surface ($\mathrm{mm}$) | 250 | 200 | 150 | 100 | 50
Largest spread of spot diagram $\lambda$ 380 $\mathrm{nm}$ | 0.192 | 0.193 | 0.194 | 0.196 | 0.200
Largest spread of spot diagram $\lambda$ 500 $\mathrm{nm}$ | 0.028 | 0.025 | 0.023 | 0.026 | 0.027
Largest spread of spot diagram $\lambda$ 1000 $\mathrm{nm}$ | 0.188 | 0.187 | 0.185 | 0.183 | 0.184
Largest spread of spot diagram |  |  |  |  |
of the whole waveband $\lambda$ 380 $\mathrm{nm}$ – 1000 $\mathrm{nm}$ | 0.397 | 0.339 | 0.276 | 0.212 | 0.200
Compensation error | 0.289 | 0.227 | 0.175 | 0.108 | 0.046


# S-Flow GAN

Yakov Miron , Yona Coscas
Elbit Systems Aerospace
{yakov.miron<EMAIL_ADDRESS>

###### Abstract

Our work offers a new method for domain translation from semantic label maps
and Computer Graphic (CG) simulation edge map images to photo-realistic
images. We train a Generative Adversarial Network (GAN) in a conditional way
to generate a photo-realistic version of a given CG scene. Existing
architectures of GANs still lack the photo-realism capabilities needed to
train DNNs for computer vision tasks, we address this issue by embedding edge
maps, and training it in an adversarial mode 1. We also offer an extension to
our model that uses our GAN architecture to create visually appealing and
temporally coherent videos.

##  1 Introduction

#####

The topic of image to image translation and more generally video to video
translation is of major importance for training autonomous systems. It is
beneficial to train an autonomous agent in real environments, but not
practical, since enough data cannot be gathered [7]. However, using simulated
scenes for training might lack details since a synthetic image will not be
photo-realistic and will lack the variability and randomness of real images,
causing training to succeed up to a certain point. This gap is also referred
to as “the reality gap” [7]. By combining a non photo-realistic, simulated
model with an available dataset, we can generate diverse scenes containing
numerous types of objects, lightning conditions, colorization etc. [6].

#####

In this paper, we depict a new approach to generate images from a semantic
label map and a flexible Deep Convolution Neural Network (DCNN) we called Deep
Neural Edge Detector (DNED) which embed edge maps. we combine embedded edge
maps which act as a skeleton with a semantic map as input to our model (fig
2), The model outputs a photo-realistic version of that scene. Using the
skeleton by itself will generate images that lack variability as it restricts
the representation to that specific skeleton itself. Instead, we learn to
represent skeletons by a neural network and at test time, we sample the
closest appropriate skeleton the network has seen at training. Moreover, we
have extended this idea to generate photo-realistic videos (i.e. sequence of
images) with a novel loss that uses the optical flow algorithm for pixel
coherency between consecutive images.

Figure 1: in this paper we propose a method for generating photo-realistic
images from semantic labels of a simulator scene. This figure provides images
related to the Synthia dataset [26]. Left - semantic map of the scene. Middle
- generated image from pix2pixHD [37]. Right - Our generated image. The
texture and color space in our generated image is more natural giving the
image the desired photo-realism.

#####

Recent works in the field of image generation include pix2pix [18] offering
image generation from semantic maps, cascaded refinement networks [6] using
networks refining different resolutions in a cascade manner, pix2pixHD [37]
can generate HD images in a conditional manner using multi-scale discriminator
and an dual generator used as a super resolution generator. L1 loss for image
generation is known to generate low quality images as the generated images are
blurred and lack details [10]. Instead, [11], [19] are using a modified
version of the perceptual loss, allowing generation of finer details in an
image. Pix2pixHD [37] and CRN [6] are using a perceptual loss as well for
training their networks, e.g. VGGnet [31]. Moreover, pix2pixHD are using
instance maps as well as label maps to enable the generator to separate
several objects of the same semantics. This is of high importance when
synthesizing images having many instances of the same semantics in a single
frame.

#####

As for video generation the loss used by [36], [30] tend to be computationally
expensive while our approach is simpler. We are using two generators of the
same architecture, and they are mutually trained using our new optical flow
based loss that is fed by dense optical flow estimation. Our evaluation method
is FID [15] and FVD [34] as it is a common metric being used for image and
video generation schemes. We call this work s-Flow GAN since we embed Spatial
information obtained from dense optical flow in a neural network as a prior
for image generation and flow maps for video coherency. This optical flow is
available since the simulated image is accessible at test time in the case of
CG2real scheme.

#####

We make Three major contributions: First, our model can generate visually
appealing photo-realistic images from semantic maps having high definition
details. Second, we incorporate a neural network to embed edge maps, thus
allowing generation of diverse versions of the same scenes. Third, we offer a
new loss function for generating natural looking videos using the above
mentioned image generation scheme. please refer to this link for videos and
comparison to related work.

##  2 Related Work

###  2.1 Generative Adversarial Networks

Generative Adversarial Networks (GAN) were introduced in 2014 [13]. This
method generate images that look authentic to human observers. They do so by
having two neural networks, one generating candidates while the other acts as
a critique and tries to evaluate the generation quality
[2],[25],[42],[43],[28]. GANs are widely used for image generation; some image
synthesis schemes are used to generate low resolution images e.g. 32x32 [18]
while [4] were able to generate higher resolution images (up to 512x512). In
addition, [37] were able to generate even higher resolution images using
coarse-to-fine generators. The reason generating high resolution images is
challenging is the high dimensionality of the image generation task and the
need to provide queues for high resolution [24], [20]. We offer queues as an
edge map skeletons generated by our proposed DNED module. During training the
DNED is trained to learn the representations of real image edge maps. During
test the DNED is shown a CG (Computer Graphics) edge map, finds its best
representation and provides the generator with an appropriate generated edge
map sampled from real image edge maps distribution.

Figure 2: example of the images for training the model. Left is the semantic
map. Middle is the edge map extracted from the real image. Right is the real
image being used by the discriminator for adversarial training. Please note
that the real image is not used by the generator neither at training nor at
test time, but only its edge map. The main issue in the CG2real model compared
to the image to image models is that the simulator’s image is available to the
generator at test time. We thus use the simulator’s image to extract the edge
map allowing the generator to generate the necessary fine details in the
output image.

###  2.2 Image synthesis

####  2.2.1 Image to image translation

In the pix2pix setting, they used a Conditional GAN [23], where the network’s
input is a semantic map of the scene, and while training in adversarial mode,
a fake version of the real image is given to the discriminator to distinguish.
In the CG2real setting in addition to the semantic map we also have access to
the simulated image. Using the CG image as is, might be counter productive
since it will be trained to reconstruct CG images and not photo-realistic
ones. Conversely some of the underlying CG information correlates with the
real world and can provide meaningful prior to the synthesis. Since the
relevant information lies in the image high frequencies [5], we learn the
distribution of edge maps in real images (high resolution details), and
provide representation of it to the generator at test time. Some image
generation tasks use label maps only, e.g. [18]. The label maps provide only
information about the class of a given pixel. In order to generate photo-
realistic images, some use instance maps as well [37], This way, they can
differentiate several adjacent objects of the same class. Nonetheless, while
most datasets provide object level information about classes like cars,
pedestrians, etc. they do not provide that information about vegetation and
buildings. As a result, the generated images might not correctly separate
those adjacent objects, thus degrading photo-realism.

####  2.2.2  Learning edges by a neural network

Generating edge maps using neural networks is a well established method.
Holistically-Nested Edge Detection (HED) provides holistic image training and
prediction for multi-scale and multi-level feature learning [39]. They use a
composition of generated edge maps to learn a fine description of the edge
scene. Inspired by their work, we train a neural network to learn edge maps of
real images.

As mentioned before, our generator requires an edge map as input. we get the
edge map using a spacial Laplacian operator with threshold. Providing the
generator with deterministic edge map will produce the same scene, so we train
the DNED to take as input that deterministic edge map, learn its
representation and produce a variant of that edge map, as a superposition of
edges seen in real datasets. This way the generator will be able to produce a
varaiaty of photorealistic images for the same scene.

Since our approach (using edge maps) is not class dependent, we do not need
instance map information to generate several adjacent instances of the same
semantics. Moreover, this approach addresses the problem of generating fine
details within a class like buildings and vegetation as can bee seen in fig 5.

###  2.3 Video to video synthesis

Generating temporally coherent image sequences is a known challenge. Recent
works use GANs to generate videos in an unconditional setting [27],[33],[35],
by sampling from a random vector, but don’t provide the generator with
temporal constrains, thus generating non coherent sequences of images. Other
works like video matting [3] and video inpainting [38] translate videos to
videos but rely on problem specific constrains and designs. A recent work
named vid2vid [36] offers to conditionally generate video from video and is
considered to one of the best approaches to date. Using FlowNet 2.0 [17] they
predict the optical flow of the next image. In addition, they use a mask to
differentiate between two parts; the hallucinated image generated from
instance-level semantic segmentation masks and the predicted image from the
previous frame. By adding these two parts, this method can combine the
predicted details from the previously generated image, with the details from
the newly generated image. Inspired by [36], we are using flow maps of
consecutive images to generate temporally coherent videos. Contrary to [36] we
are not using a CNN to predict the flow maps or a sequence generator, but a
classical Computer vision approach. This is since a pre-trained network
(trained on real datasets) failed to generalize and infer on simulated
datasets e.g. Synthia. This enables better temporal coherency and improve
video generation robustness.

##  3 Model

Our CG2real model aims to learn the conditional distribution of an image given
a semantic map. Our video generation model aims to use this learned
distribution for generating temporally coherent videos using the generated
images from the CG2real scheme. We first depict the image generation scheme,
then we review our video generation model.

###  3.1 Image generation

We use a conditional GAN to generate images from semantic maps as in [18]. In
order to generate images, the generator receives the semantic segmentation
images $s_{i}$ and maps it to photo-realistic images $x_{i}$. In parallel, the
discriminator takes two images, The real image $x_{i}$ (ground truth) and the
generated image ${f_{i}}$ and learns to distinguish between them. This
supervised learning scheme is trained in the well-known min max game
[13],[28]:

$\ \min_{G}\ \max_{D}\mathcal{L}_{GAN}(D,G)\ $ (1)

$\mathcal{L}_{GAN(D,G)}=E_{(x,s)}[log(x,s)]+E_{(s\sim
p_{data}(s))}[log(1-D(s,G(s)))]$ (2)

###  3.2 Embedding edge maps

In order to generate photo-realistic visually appealing images containing fine
details, we provide a learnt representation of an edge map to the generator
(fig 2), allowing it to learn the conditional distribution of real images
given semantic maps and edge maps, i.e.:

$\mathcal{L}_{GAN(D,G,e)}=E_{(x,s)}[log(x,s)]+E_{((s,e)\sim
p_{data}(s,e))}[log(1-D(s,G(s,e)))]$ (3)

During training, given an example image ${x_{i}}$, we can estimate its edge
map by the well-known spatial Laplacian operator [14],[9]. This edge map is
concatenated to the semantic label map and both are given as priors to the
generator for adversarial training of the fake image ${f_{i}}$ vs. the real
image ${x_{i}}$. To allow a stable training we begin training our GAN with the
edge maps from the Laplacian operator. After stabilization of the generator
and discriminator, we provide our generator with edge maps from the DNED. We
then jointly train the GAN with the DNED.

The DNED architecture is a modified version of HED [39]. In HED, they generate
several sized versions of the edge map, each having a different receptive
field. The purpose is to create an ensemble of edge maps, each allowing
different level of details in the image. When superimposing all, the resulting
edge map will have coarse-to-fine level of details in the generated edge map
image. By changing the weights of that ensemble, we can generate the desired
variability in the generated edge map, thus allowing us to generate diverse
versions of the output. To conclude, the loss function for training the DNED
is:

$\mathcal{L}_{DNED}\colon=\mathcal{L}_{DNED}(E(x))=\sum_{i=1}^{N}a_{i}*BCE(d_{i}(x),E(x))$
(4)

Where: $d_{i}(x)$, $i=0:5$ is the $i^{th}$ side output of a single scale,
$E(x)$ is the classic edge map generated by the spatial Laplacian operator,
BCE is the binary cross entropy loss. $N=6$ in our case. $a_{i}$ is the
contribution of the $i^{th}$ scale to the ensemble.

#####

Increasing the resolution of the image might be challenging for GAN training.
In other methods the discriminator needs a large receptive field
[18],[29],[31],[22], requiring a deeper network or larger convolution kernels.
Using a deeper network is prone to overfitting and in the case of GAN
training, and might cause training to be unstably. This challenge is usually
addressed by the multi-scale approach [12],[9],[16],[20],[40]. Since the DNED
embed a learnt representation of skeletons, our architecture performs very
well on higher resolution images. Our original generated images were of size
[512x256]. We have successfully trained our model to generate images of size
[768x384] , i.e. 1.5 times larger in each dimension without changing the model
while using a single discriminator (see 3).

Figure 3:  comparison of 768x384 pix images generated by pix2pixHD (Left) and
our model (Right). Our model can generate lower level details in the image,
thus improving its photo-realism. This figure provides an example comparing
(768x384 pix) resolution images of pix2pixHD (left side) compared to our model
(right).

We showed that generating high quality images when using a single
discriminator is feasible and training is stable. We provide comparison using
our method with multi-scale discriminator 4. the FM loss is computed with k=1
for single layer discriminator and k=3 for multi layer one:

Figure 4: Comparison of generated test images, when training with a single
discriminator and a multi-scale one. The left image is generated when the
generator was trained with a single discriminator, while the right image while
using a multi-scale one. This figure demonstrates that when using our model
(with our skeleton), training with a single discriminator might be enough.

$\mathcal{L}_{FM_{m}}^{k}\colon=\mathcal{L}_{FM_{m}}^{k}(D_{k},G,e)=\sum_{i=1}^{T}\frac{1}{N_{i}}E_{{(x,s,e)}\sim
p_{data}(s,x,e)}\mathcal{L}_{1}(D_{k}^{i}(s,x)-D_{k}^{i}(s,G(s,e))$ (5)

Figure 5:  By using edge maps, the model learns to separate objects of the
same semantics. The most dominant example is buildings. Unlike cars,
pedestrians or bicycle riders, that are separable using the instance map,
buildings are not. The semantic label provides the pixels in which the
building exists. Considering the fact that a scene of adjacent buildings is
somewhat common, the ability to separate them is of high value. Left - the
label map. Middle - generated image by [37]. Right - our generated image. Our
model can generate unique adjacent buildings from the semantic label maps of
better quality compared to [37].

In addition, following [10],[11],[19],[44] we are using the perceptual loss
for improved visual performance and to encourage the discriminator distinguish
real or fake samples using a pre traind VGGnet [21].

$\mathcal{L}_{percep}\colon=\mathcal{L}_{percep}(x,G(s,e))=\frac{1}{P}\sum_{i=1}^{P}\mathcal{L}_{1}(FL_{VGG_{i}}(x)-FL_{VGG_{i}}(G(s,e)))$
(6)

Where, P is the number of slices from a pre-trained VGG network and
$FL_{VGG_{i}}$are the features extracted by the VGG network from the $i^{th}$
layer of the real and generated images respectively. To conclude, our overall
objective for generating photo-realistic, diverse images in the CG2real
setting is to minimize $L_{CG2real}$:

$L_{CG2real}=\ \min_{G}\ \max_{D_{k},\
k=1:l_{m}}\sum_{l=1}^{l_{m}}\mathcal{L}_{GAN}(D_{k},G,e)+\lambda_{1}\sum_{l=1}^{l_{m}}\mathcal{L}_{FM_{m}}^{k}+\lambda_{2}\mathcal{L}_{percep}+\lambda_{3}\mathcal{L}_{NNED}$
(7)

Figure 6: previous work test images [37] (Top) compared to our model test
images (Bottom). The images generated by our model contain low level details,
allowing the desired photo-realism

###  3.3 Video generation

Using pre trained CG2real networks, we generate two consecutive images, and
then estimate two flow maps. The first flow map is between $x_{i},x_{i+1}$,
where $x_{i}$ and $x_{i+1}$ are two consecutive real images. The second flow
map is between $G(s_{i},e_{i}),G(s_{i+1},e_{i+1})$ , where $G(s_{i},e_{i})$
and $G(s_{i+1},e_{i+1})$ are two consecutive generated (fake) images. Note
that the generation of $G(s_{i},e_{i}),G(s_{i+1},e_{i+1})$ is done
independently, meaning we apply our CG2real method twice, without any
modifications. To conclude we enforce temporal coherency by using the
following loss:

$\mathcal{L}_{flow}=\mathcal{L}_{1}(\mathcal{F}_{real},\mathcal{F}_{fake})$
(8)

Where $\mathcal{F}_{real}=\mathcal{F}(x_{i},x_{i+1})$,
$\mathcal{F}_{fake}=\mathcal{F}(G(s_{i},e_{i}),G(s_{i+1},e_{i+1}))$ and
$\mathcal{F}(*)$ is the optical flow operator. This formulation eliminates the
need of using a sequential generator as in [36], allowing us not only using
our image generation model twice, which adds more constrains to the video
generation scheme, but also avoid errors accumulation arising from positive
feedback by feeding a generated image to the generator, as can be seen in
figure 7 and in this video.

Figure 7:  block diagram of the video generation model. Two identical CG2real
models generate Fake image (t) and Fake image (t+1). The two consecutive fake
images are fed to the flow-fake estimator, while two consecutive real images
are fed to the flow-real estimator. Both real and fake flow maps are trained
using $L_{1}(F_{real},F_{fake})$ loss. This enables the pre-trained CG2real
models to learn the required coherency for generating photo-realistic videos.

By adding $L_{flow}$ to the $L_{CG2real}$ loss, the network learns to generate
$G(s_{i+1},e_{i+1})$ taking the flow maps into account, thus generating
temporally coherent images as depicted in 7.

$\mathcal{L}_{videogen}=\mathcal{L}_{flow}+\mathcal{L}_{CG2real}$ (9)

##  4 Results

Our goal is to generate photo-realistic images. In (fig 6) we can find some
examples from the CG2real image synthesis task, and in (fig 8) present
consecutive images depicting the video to video synthesis. We use the same
evaluation methods as used by previous image to image works ,e.g. pix2pix [18]
, pix2pixHD [37] and others. The evaluation process consist of performing
semantic segmentation with a pre-trained seamntic segmentation network [41] on
synthesized images produces by our model, then calculating the semantic pixel
accuracy and the mean intersection over union (mIoU) over the classes in the
dataset. As shown in tables 1, 2 bellow, our network outperforms previous
works. The ground-truth results are the pixel accuracy and mIoU when
performing the same semantic segmentation with the real images (Oracle).

Furthermore, to evaluate the image generation quality, we used another metric
to evaluate distances between datasets called FID (Fréchet Inception Distance)
[15],[1]. It is a very common metric for generative models as it correlates
well with the visual quality of generated samples [36]. FID calculates the
distance between two multivariate Gaussians real and generated respectively;
where $X_{r}\sim N(\mu_{r},\Sigma_{r}$) and $X_{g}\sim N(\mu_{g},\Sigma_{g}$)
are the 2048-dimensional activations of the Inception-v3 pool3 layer [32], and
$FID=\|\mu_{r}-\mu_{g}\|^{2}+Tr(\Sigma_{r}+\Sigma_{g}-2({\Sigma_{r}\Sigma_{g}})^{1/2})$
is the score for image distributions $X_{r}$ and $X_{g}$. Lower FID score is
better, meaning higher similarity between real and generated samples.

Cityscapes | Pix2pix | Pix2pixHD | Ours | Oracle
---|---|---|---|---
Pixel accuracy [%] | 0.7279 | 0.81 | 0.83 | 0.86
Mean IoU [%] | 0.5324 | 0.67 | 0.69 | 0.701
Table 1:  semantic segmentation results on the cityscapes [8] validation set Synthia | Pix2pix | Pix2pixHD | Ours | Oracle
---|---|---|---|---
Pixel accuracy [%] | 0.54 | 0.79944 | 0.860753 | 0.913132
Mean IoU [%] | 0.36 | 0.55955 | 0.740040 | 0.8419
Table 2:  semantic segmentation results on the Synthia [26] dataset FID,FVD | Pix2pix | Pix2pixHD | Vid2vid | Ours-img | Ours-vid
---|---|---|---|---|---
FID | 116.69 | 71.21 | 154.36 | 69.25 | 69.81
FVD | - | - | 0.706 | - | 0.326
Table 3:  FID and FVD metric comparisson between pix2pix, pix2pixHD vid2vid
and Ours.

As can be seen in tables 1, 2, pix2pixHD’s results are better than pix2pix for
pixel accuracy and mIoU. Our results are better than pix2pixHD, and almost
meet the oracle’s results on both Synthia [26] and cityscapes [8]. In table 3,
we compare the FID score for all the four image generation models w.r.t the
Oracle. Ours-img (Our image generation model) outperforms both pix2pix and
pix2pixHD. Moreover, adding a temporal consistency constrain to the image
generation process degrades image quality. Vid2vid uses pix2pixHD as its image
generation model imposes a substantial degradation in the image quality (71.21
to 154.36). Our video generation uses our CG2real model had a marginal effect
on the FID score of Ours-vid (69.25 to 69.81 and even outperformed pix2pixHD)
and did not degrade generated images quality (fig 8).

Our video generation evaluation method is FVD (Fréchet Video Distance)
proposed by [34]. FVD is a metric for video generation models evaluation and
it uses a modified version of FID. we calculated the FVD score for our
generated video (Ours-vid) w.r.t. the Oracle (real video) and did the same for
vid2vid w.r.t the same Oracle. Our FVD score on the video test set is 0.326
while vid2vid’s is 0.706 meaning our videos are more than twice similar to the
oracle. we suggest that this substantial margin stems from the errors
accumulated in the video generation model of vid2vid (fig 8). As mentioned,
Our video generation model uses our flow loss therefore does not encounter
this phenomena.

Figure 8: Comparison of video generation. Up - images generated by vid2vid
[36]. Down - images generated by our video generation model. Our generated
images are temporally coherent and visually appealing In our images sky is
more natural, road signs are clearer and buildings have finer level of
details. This example emphasizes the error propagation of vid2vid’s model wile
our model does not accumulate errors (see street lights in upper right corner
of each image). The main objective of the video generation model is to enable
generating non flickering images by giving objects in consecutive images the
same color and texture, i.e. sample from the same area in the latent spaces.
full videos can be seen here .

##  5 Summary

We present a CG2real conditional image generation as well as a conditional
video synthesis. We offer to use a network learning the distribution of edge
maps from real images and integrate it into a generator (DNED). We were able
to generate highly detailed and diverse images thus enabling better photo-
realism. Using the DNED enable generating diverse yet photo-realistic
realizations of the same desired scene without using instance maps. As for
video generation, we offer a new scheme that utilizes flow maps allowing
better temporal coherence in videos. We compared our model to recent works and
found that it outperforms both current quantitative results and more
importantly generates appealing images. Furthermore, our video generation
model generates temporally coherent and consistent videos.

## References

  * Adler & Lunz [2018] Jonas Adler and Sebastian Lunz.  Banach wasserstein gan.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems 31_ , pp. 6754–6763. Curran Associates, Inc., 2018.  URL http://papers.nips.cc/paper/7909-banach-wasserstein-gan.pdf.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein gan.  _arXiv preprint arXiv:1701.07875_ , 2017.
  * Bai et al. [2009] Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro.  Video snapcut: Robust video object cutout using localized classifiers.  In _ACM SIGGRAPH 2009 Papers_ , SIGGRAPH ’09, pp. 70:1–70:11, New York, NY, USA, 2009. ACM.  ISBN 978-1-60558-726-4.  doi: 10.1145/1576246.1531376.  URL http://doi.acm.org/10.1145/1576246.1531376.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Burt & Adelson [1983] Peter Burt and Edward Adelson.  The laplacian pyramid as a compact image code.  _IEEE Transactions on communications_ , 31(4):532–540, 1983.
  * Chen & Koltun [2017] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1511–1520, 2017.
  * Collins et al. [2018] Jack Collins, David Howard, and Jürgen Leitner.  Quantifying the reality gap in robotic manipulation tasks.  _arXiv preprint arXiv:1811.01484_ , 2018.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Denton et al. [2015] Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus.  Deep generative image models using a laplacian pyramid of adversarial networks, 2015.
  * Dosovitskiy & Brox [2016] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks, 2016.
  * Gatys et al. [2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2414–2423, 2016.
  * Ghiasi & Fowlkes [2016] Golnaz Ghiasi and Charless C. Fowlkes.  Laplacian pyramid reconstruction and refinement for semantic segmentation.  _Lecture Notes in Computer Science_ , pp. 519–534, 2016.  ISSN 1611-3349.  doi: 10.1007/978-3-319-46487-9_32.  URL http://dx.doi.org/10.1007/978-3-319-46487-9_32.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * Guattery & Miller [2000] Stephen Guattery and Gary L. Miller.  Graph embedding and laplacian eigenvalues.  _SIAM J. Matrix Anal. Appl._ , 21(3):703–723, 2000.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Advances in Neural Information Processing Systems_ , pp. 6626–6637, 2017.
  * Huang et al. [2017] Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie.  Stacked generative adversarial networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jul 2017.  doi: 10.1109/cvpr.2017.202.  URL http://dx.doi.org/10.1109/CVPR.2017.202.
  * Ilg et al. [2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jul 2017.  doi: 10.1109/cvpr.2017.179.  URL http://dx.doi.org/10.1109/CVPR.2017.179.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1125–1134, 2017.
  * Johnson et al. [2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  _Lecture Notes in Computer Science_ , pp. 694–711, 2016.  ISSN 1611-3349.  doi: 10.1007/978-3-319-46475-6_43.  URL http://dx.doi.org/10.1007/978-3-319-46475-6_43.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation, 2017.
  * Ledig et al. [2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4681–4690, 2017.
  * Luo et al. [2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In _Advances in neural information processing systems_ , pp. 4898–4906, 2016.
  * Mirza & Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets, 2014.
  * Odena et al. [2017] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pp. 2642–2651. JMLR. org, 2017.
  * Radford et al. [2015] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
  * Ros et al. [2016] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Saito et al. [2017] Masaki Saito, Eiichi Matsumoto, and Shunta Saito.  Temporal generative adversarial nets with singular value clipping.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.  doi: 10.1109/iccv.2017.308.  URL http://dx.doi.org/10.1109/ICCV.2017.308.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In _Advances in neural information processing systems_ , pp. 2234–2242, 2016.
  * Seif & Androutsos [2018] George Seif and Dimitrios Androutsos.  Large receptive field networks for high-scale image super-resolution.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 763–772, 2018.
  * Shahar et al. [2011] Oded Shahar, Alon Faktor, and Michal Irani.  Super-resolution from a single video.  In _CVPR_ , 2011.  URL http://www.wisdom.weizmann.ac.il/~vision/SingleVideoSR.html.
  * Simonyan & Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun 2016.  doi: 10.1109/cvpr.2016.308.  URL http://dx.doi.org/10.1109/CVPR.2016.308.
  * Tulyakov et al. [2018] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz.  Mocogan: Decomposing motion and content for video generation.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , Jun 2018.  doi: 10.1109/cvpr.2018.00165.  URL http://dx.doi.org/10.1109/CVPR.2018.00165.
  * Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  _arXiv preprint arXiv:1812.01717_ , 2018.
  * Vondrick et al. [2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.  Generating videos with scene dynamics, 2016.
  * Wang et al. [2018a] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis, 2018a.
  * Wang et al. [2018b] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018b.
  * Wexler et al. [2004] Yonatan Wexler, Eli Shechtman, and Michal Irani.  Space-time video completion.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 1, pp. I–I. IEEE, 2004.
  * Xie & Tu [2017] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  _International Journal of Computer Vision_ , 125(1-3):3–18, Mar 2017.  ISSN 1573-1405.  doi: 10.1007/s11263-017-1004-z.  URL http://dx.doi.org/10.1007/s11263-017-1004-z.
  * Zhang et al. [2017] Han Zhang, Tao Xu, and Hongsheng Li.  Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.  doi: 10.1109/iccv.2017.629.  URL http://dx.doi.org/10.1109/ICCV.2017.629.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jul 2017.  doi: 10.1109/cvpr.2017.660.  URL http://dx.doi.org/10.1109/CVPR.2017.660.
  * Zhao et al. [2016] Junbo Zhao, Michael Mathieu, and Yann LeCun.  Energy-based generative adversarial network, 2016.
  * Zhu et al. [2016] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros.  Generative visual manipulation on the natural image manifold.  _Lecture Notes in Computer Science_ , pp. 597–613, 2016.  ISSN 1611-3349.  doi: 10.1007/978-3-319-46454-1_36.  URL http://dx.doi.org/10.1007/978-3-319-46454-1_36.
  * Zhu et al. [2017] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman.  Toward multimodal image-to-image translation.  In _Advances in Neural Information Processing Systems_ , pp. 465–476, 2017.

##  Appendix A Appendix

Figure 9: Additional Test images from Cityscapes Dataset. Left - pix2pixHD.
Right - Ours. These images further demonstrate the photo-realism achieved by
our model. Figure 10: Additional test images on Synthia dataset. Left -
pix2pixHD. Right - Ours. These images demonstrate improved image quality,
better and finer details in the generated objects, buildings and vegetation.
Figure 11: Additional test images on Synthia dataset. Left - pix2pixHD. Right
- Ours. These images demonstrate improved image quality, better and finer
details in the generated objects, buildings and vegetation. Figure 12:  Test
video on CityScapes. Left - vid2vid. Right - Ours video gen model. These
images demonstrate better temporal coherency in the generated images.
Moreover, in the top left corner of the left video, we notice the error
propagates. Better yet, the buildings in the right video are more reasonable,
w.r.t. windows, shades, general texture, etc. As for image quality, the road
signs in the right video are better emphasized. Figure 13: Test video on
CityScapes. Left - vid2vid. Right - Ours video gen model. This figure provides
more images from the same video presented in fig 8. Pay attention to the error
propagation on the top right images of vid2vid. Again, our model demonstrates
finer road signs and higher level of details in the generated buildings.