# First light of the Integral Field Unit of GRIS on the GREGOR solar telescope

C. Dominguez-Tagle1,2∗ M. Collados1,2 R. Lopez1,2 J. J. Vaz Cedillo1,2 M.A.
Esteves 3 O. Grassin 3
N. Vega1,2 A. Mato1,2 J. Quintero1,2 H. Rodriguez 1,2 S. Regalado1,2 F.
Gonzalez1,2 1Instituto de Astrofisica de Canarias, C/Via Lactea, 38205
Tenerife, Spain.
2Departamento de Astrofisica, Universidad de La Laguna, 38206, Tenerife, Spain
3Leibniz-Institute für Sonnenphysik, Schöneckstrasse 6, 79104 Freiburg,
Germany

((to be inserted by publisher); (to be inserted by publisher); (to be inserted
by publisher))

###### Abstract

An Integral Field Unit (IFU) based on image-slicers has been added to the
GREGOR Infrared Spectrograph (GRIS). This upgrade to the instrument makes
possible 2D spectropolarimetry in the near-infrared by simultaneously
recording the full Stokes profiles of spectral lines (in a given spectral
interval) at all the points in the field of view. It provides high-cadence
spectropolarimetric observations at the instrument’s high spatial resolution
and high polarization sensitivity at the GREGOR solar telescope. The IFU is
ideal for observing the polarized spectrum of fast-evolving solar features at
high spatial and spectral resolutions. The high observing cadence opens the
possibility of time-series observations. The analysis of observations to this
level of accuracy is essential for understanding the complex dynamics and
interactions of solar plasma and magnetic fields. The image slicer of the IFU
has eight slices of width 100 $\mu$m, covering a total field of view of 6′′
$\times$ 3′′. It was designed and built within the framework of the European
projects SOLARNET and GREST, as a prototype for future instruments of the
European Solar Telescope (EST) and was integrated into GRIS. After two
commissioning campaigns in 2017 and 2018, the IFU was finally installed at the
end of September 2018 and offered to all observers who use the telescope.

###### keywords:

Integral Field Unit; IFU; GREGOR; GRIS; spectropolarimetry; solar.

∗Corresponding author.

; ; ;

##  1 Introduction

Integral field units (IFUs) based on image-slicers are commonly used in
astrophysical night-time slit-spectroscopy. They were initially proposed by
Bowen (1938) and have been developed since the work of Weitzel96 and Content
(1997). The working principle of image slicers is to slice a region of the
focal-plane image into a series of small rectangles and rearrange them in the
form of a long slit. This slit is composed of a given number of mini-slits
(equal to the number of slicer mirrors). The width of the individual slicer
mirrors determines the slit width that feeds the spectrograph, into which the
IFU is integrated.

In solar physics, IFUs based on this technology can be used to record fast-
evolving solar features at high spatial and spectral resolution, even at near-
infrared (NIR) wavelengths. As described in a review of instrumentation for
solar spectropolarimetry by Iglesias & Feller (2019), there are few
instruments currently in operation (see Socas-Navarro _et al._ , 2006; Jaeggli
_et al._ , 2010; Cao _et al._ , 2012; Collados _et al._ , 2012; Jaeggli, 2022)
that can perform solar spectropolarimetry in the NIR (i.e., up to 1.7 $\mu$m).

In the case of GRIS (GREGOR Infrared Spectrograph, see Collados _et al._ ,
2012), adding an IFU to the instrument makes possible to obtain spectra at all
the points in a 2D field of view (FOV) and get the full Stokes profiles of the
spectral lines in a given spectral interval. This makes possible to observe,
at high spatial and spectral resolutions, the polarized spectrum of fast-
evolving solar features at a cadence of a few seconds. The IFU added to GRIS
was designed at the Instituto de Astrofísica de Canarias (IAC) as a first
approach to the conceptual instrument called “MUSICA” Calcines _et al._
(2013b), for EST instrumentation Calcines _et al._ (2013a). The IFU was built
in collaboration with Winlight Systems (formerly Winlight Optics) within the
framework of the SOLARNET and Getting Ready for EST (GREST) European projects.
The IFU was taken to the GREGOR solar telescope Schmidt _et al._ (2012) and
integrated into GRIS in 2017. From the first tests, it was clear that some
work still needed to be done, and a second commissioning campaign was carried
out in 2018 Dominguez-Tagle _et al._ (2018). During the scientific validation’
procedure, a number of spectropolarimetric measurements at the solar disk
center were recorded. After these campaigns, the IFU in GRIS was made
available to the entire GREGOR observing community.

Between 2018 and 2019, more than 15 scientific teams observed with the IFU. It
was in use for more than 70% of the telescope’s observing time. More than 100
days of scientific data have been archived (i.e., selecting those obtained in
good observing conditions). In 2020 and the first semester of 2021, the IFU
was not offered because of the pandemic and changes in the telescope optics.
Four scientific articles with results based on IFU observations have already
been published Kaithakkal _et al._ (2020); Campbell _et al._ (2021); Anan _et
al._ (2021); Nelson _et al._ (2021).

The instrument and the IFU are described in Section 2, including an analysis
of performance. Some first light science observations and other examples are
presented in Section 3. Our conclusions are given in Section 4.

##  2 Instrument description

###  2.1 GRIS

GRIS Collados _et al._ (2012) is a spectropolarimeter consisting of a 6 m
focal-length Czerny–Turner spectrograph and a polarimeter built from the
heritage of TIP-II Collados _et al._ (2007). GRIS works in the wavelength
range 1.0–2.3 $\mu$m. Its 1k $\times$ 1k NIR detector can be read at up to 36
fps. The polarimeter has two sets of Ferroelectric Liquid Crystals, one
optimized to work at 1.0–1.3 $\mu$m and the other at 1.5–1.8 $\mu$m. GRIS has
two observing modes, spectroscopy and spectropolarimetry, both with a long
slit (0.26′′ width $\times$ 60′′ length). It has a Slit Scan Unit (SSU) to do
scans perpendicular to the slit. The SSU allows the scanning of a FOV up to
60′′ $\times$ 64′′ to be covered. The polarimeter is placed immediately after
the SSU. As described in Collados _et al._ (2012), the slit mask in the SSU is
inclined $15^{\circ}$ with respect to the entrance beam in order to reflect
the light that arrives out of the slit towards a slit-jaw imaging system. This
inclination does not alter the operation of the spectrograph and allows a
field context image to be generated. Table 2.1 shows the specifications of the
slit mode in the first column. The IFU mode is discussed in the next section.

[ht] GRIS slit and IFU specifications Slit mode IFU mode Number of slits: 1
Number of slices: 8 Slit length: $<$ 60′′ Slice length: 6′′ Slit width: 0.26′′
Slice width: 0.375′′ (100 $\mu$m) Slit Scan (SSU) 2D Scan (FOVSS) Moves in 1
direction Moves in 2 directions Max. FOV: $60^{\prime\prime}\times
64^{\prime\prime}$ Max. FOV: $60^{\prime\prime}\times 60^{\prime\prime}$
Double sampling mode: half slit width (see Section 3) same Spectral wavelength
ranges: 18 Å at 10830 Å same 40 Å at 15650 Å same Pixel scale: 0.135′′ same
Simultaneous slit-jaw images acquisition same Detector: 1k $\times$ 1k same
Well depth: 16384 ADUs (14 bits) Gain: 19 e-/ADU same

###  2.2 IFU

The IFU has been coupled to GRIS to add an additional observing mode.
Currently, it is possible to exchange the slit and IFU modes in a single day
Vega-Reyes _et al._ (2016). In the slit mode, the slit is placed at the focal
plane of the telescope and is part of the SSU. The light path exits the SSU
and goes through the polarimeter to the rest of the spectrograph. In IFU mode,
the arrangement is different because of the available space and the size of
the optical components of the polarimeter. Figure 1 shows the layout of the
IFU mode. An entrance mask is placed at the telescope’s focal plane to limit
the FOV to the size of the image slicer. The mask is the first element of the
Field of View Scan System (FOVSS). This new scanning system is described in
Section 2.3. A reimaging system is incorporated in order to place the IFU in a
new focal plane. The reimaging system is based on a classic collimator-camera
design and includes two mirrors (RS1 and RS2). The image slicer of the IFU
(shown at the top of Figure 1) is at the focal plane generated by the RS2
mirror.

Figure 1: Light path in the IFU mode. The telescope focus is at the bottom of
the figure (Mask). The light goes through the FOVSS and the reimaging system
up to the IFU and then continues towards the polarimeter. The light path
distances ($a,b,c,d$) in the FOVSS are shown next to its components, the
entrance mask and the three SMs. The components for the reimaging system are
the two RS mirrors. Those for the IFU are the image slicer and the arrays of
collimator and camera mirrors. The position of the output mini-slits is also
shown.

The IFU covers a FOV of $6^{\prime\prime}\times 3^{\prime\prime}$ in a single
exposure. It is based on image-slicer technology with eight slicer mirrors of
size 1.8 mm $\times$ 0.1 mm, each. These mirrors cut a rectangular region of
the image at the focal plane and reorganize it into a long slit formed by
eight mini-slits. The whole IFU body is fabricated in Zerodur to reduce
thermal sensitivity because the instrument works at room temperature. The
reorganization of the image from the image slicer towards the output slit is
accomplished by a reimaging system, which has a classic collimator-camera
design. There is a pair of collimator and camera mirrors for each slice. The
optical design evolved from the U-path described in Calcines _et al._ (2014)
to a Z-path, using spherical mirrors for the collimator and camera. The system
is designed to be telecentric and includes a mask at the pupil plane to reduce
straylight. The eight beams have the pupil in the same position so that one
pupil mask works for all of them. The output slit is arranged in two rows of
mini-slits in order to minimize geometrical distortion and optical
aberrations. The optical design is shown in Figure 2. The insert shows the
output mini-slits illuminated with sunlight. There is red light at the mini-
slits because the IFU is fed by wavelengths over 650 nm.

Figure 2: IFU optical layout. The light paths generated by every slice of the
image are represented in different colors. The insert shows a real image of
the output mini-slits. The layout orientation is left-right reversed, compared
to that in Figure 1, in order to show the mini-slits projected towards the
reader.

The collimator and camera mirrors are grouped into arrays and their numbering
is shown in Figure 3. It can be seen that the central mirrors in the image
slicer correspond to the edges of the collimator and mini-slit arrays. After
the mini-slits, the light path continues to the polarimeter and the rest of
the spectrograph. Two flat mirrors are inserted in the path after the
polarimeter in order to take the beam back to the spectrograph optical axis
and to compensate for the difference in the path length with respect to the
slit mode.

Figure 3: Slicer, collimator and mini-slit numbering for identification
purposes.

From an operational point of view, observations in IFU mode are conducted in a
similar way to those in slit mode, with the exception that the IFU provides
the opportunity to observe a 2D region in one shot. In addition, it has
scanning capabilities with the FOVSS to record a larger solar area. Both the
SSU and the FOVSS can cover roughly the same area, although the IFU is most
useful for fast small-area scans. The second column of Table 2.1 shows the
specifications of the IFU mode.

Figure 4: Pictures of the entrance mask in the IFU mode (left) and the slit
mode (right). The IFU mode is showed with sunlight over the mask. Here the
FOVSS, the IFU (Zerodur block) and the polarimeter (metallic cylinder) are
partially visible. In the slit-mode picture the SSU and the polarimeter are
clearly visible.

As described before, the entrance mask defines the FOV with precision in order
to illuminate only the useful part of the image slicer. The mask is placed at
the entrance of the FOVSS (see Figure 4) and has the same inclination as the
slit mask ($15^{\circ}$). During the first campaigns it was a rectangle cut on
the aluminum coating over an SiO2 window. However the small particles of dust
present over the inclined glass contaminated the spectra. Regular cleaning of
the mask was required. It was replaced in 2019 by a true-hole mask fabricated
in a Si wafer coated with protected aluminum. The wafers are as good as any
optical mirror since they have an intrinsic flatness better than 2 $\mu$m over
150 mm and the rectangle, machined using etching, has errors of the order of a
few microns. This new mask improved the quality of the spectra.

###  2.3 FOVSS

The FOVSS was designed to do 2D scans covering the total FOV up to
$60^{\prime\prime}\times 60^{\prime\prime}$ Esteves _et al._ (2018). The scan
size is configurable by the user. The FOVSS keeps the optical path length
constant, regardless of the scanning position over the FOV. This condition is
satisfied by means of three Scanning Mirrors (SM), which change their
positions using three motors. Figure 1 includes the layout of the FOVSS
configuration and its components. The entire assembly, with the exception of
RS1, is mounted on a common translation stage (TS) which moves the system
along the X-axis. The mask, plus the SM1 and SM2 set are mechanically
connected to a second TS, which moves this set along the Y-axis. Finally, the
SM2 and SM3 set is mechanically connected to a third TS which allows movement
along the X-axis independently of the first TS. TS1 and TS2 allow the movement
of the mask along the X and Y axes, respectively, to enable 2D scanning. TS3
uses the SM2 and SM3 set to compensate for the optical path length as a
function of the system movement while scanning. The total optical path length,
determined by the sum of segments $b+c+d$ (shown in Figure 1), is always
constant.

Figure 5 shows a representation of the image slicer and some possible scan
positions over the FOV, which is illustrated by a real-scale image from the
Sun (SDO/HMI continuum). The image slicer is represented by a red rectangle,
including the eight slicer mirrors. The left frame represents a single
exposure over an area of 6′′ $\times$ 3′′. The black rectangles in the right
frame represent an example of a 3 $\times$ 7 scan. The available scan
movements are represented by the arrows.

Figure 5: Example of the IFU mode in a single position (left frame). The image
slicer is represented by a red rectangle (FOV of 6′′ $\times$ 3′′). The right
frame is an example of a 3 $\times$ 7 scan (18′′ $\times$ 21′′). The available
scan movements are represented by the arrows, and a FOV up to
$60^{\prime\prime}$ $\times$ $60^{\prime\prime}$ can be covered. The
background image is from SDO/HMI continuum.

The FOVSS allows scans to be made following different patterns, such as those
shown in Figure 6. The patterns called “-vertical” are also available
horizontally, to give transposed scanning patterns. The initial scan position
can be chosen between the geometrical center or one corner of the full scan
pattern. Observers can choose the pattern that better fits the solar-feature
dynamics they are studying.

Figure 6:  Examples of the FOVSS scan patterns: (a) Raster-vertical, (b)
Snake-vertical and (c) Spiral-horizontal. The arrows represent the movements
that define the scan positions.

###  2.4 Performance of the IFU

As described in Section 2.2, the IFU has a set of collimator and camera
mirrors to arrange the FOV into a slit. The mirror arrangement and numbering
(illustrated in Figure 3) show that the central mirrors in the image slicer
correspond to the edges of the collimator and mini-slit arrays. This layout
produces some light contamination on the adjacent mirrors. Figure 7 shows the
IFU in operation and gives an idea of the small dimensions of its components.
The size of each collimator and camera mirror is 4.2 mm $\times$ 4.5 mm. One
possible source of that contamination has been evaluated by studying the
diffraction effects seen on the collimator mirrors due to the thin dimensions
of the slicer mirror (100 $\mu$m). The tests were done at a wavelength of
15650 Å, by illuminating only one slicer mirror at a time and measuring the
light from all the output mini-slits. Table 7 shows the result of the tests.
The measurements are represented as percentages, normalized to the self-
illumination of each slice (diagonal values). The background level at the
detector is subtracted. The maximum value of the slice-to-slice contamination
is 6.3% for the neighbors of slice number 7, and the median value is 3.5%. The
pupil mask reduces this value significantly. The same tests done without the
mask give a median value of 4.8%. Another effect, specific to the geometry of
the mini-slits and collimator arrays (seen in Figure 2), is “intra-row” light
contamination, resulting from the layout of the collimator array in two rows.
The values that are affected for each row are underlined (see Table 7). The
maximum value of the intra-row contamination is 1.5% from slice number 4 to
number 8 and the median value is 1.2%. These measured values are compared with
the modeled behavior (Regalado et al., in prep.), which estimates that at
least 80% of the energy (from the first diffraction ring) is contained within
a single slicer mirror. This value is lower than that obtained in the study by
Calcines _et al._ (2014), because that work was done at a wavelength of 10000
Å, which is a case with less diffraction. An additional mask at the mini-slit
plane could help to reduce some straylight in the direction perpendicular to
the slits. The inclusion of this mask has been considered; the manufacturing
and fixing, however, are difficult owing to the discontinued form of mini-
slits and the limited availability of space. It is planned for the future
prototypes.

Figure 7: Pictures of the IFU in operation. The IFU is the glass block
(Zerodur). The visible elements in the left picture are, from left to right:
the RS2 mirror, the array of collimator mirrors, the M1 folding mirror and the
array of camera mirrors. The right picture shows the mini-slits projected onto
a piece of paper. The polarimeter has been moved out of the light path to take
the pictures.

[ht] Light contamination measurements represented as percentages. Rows (I)
denote the illuminated slicer mirror number and columns (M) the measured
mirror number. The values that are affected for each row are underlined. The
numbering is shown in Figure 3.  Mirrors 4 3 2 1 5 6 7 8 (I $\setminus$ M) 4
100 3.0 0.8 0.3 0.1 0.1 0.4 1.5 3 3.5 100 3.3 1.0 0.5 0.5 1.1 1.2 2 0.5 3.4
100 3.3 0.9 1.5 1.2 0.3 1 0.1 0.8 3.4 100 4.2 1.6 0.5 0.3 5 0.0 0.2 0.8 3.5
100 4.7 0.8 0.5 6 0.1 0.4 1.6 1.5 3.3 100 5.2 0.8 7 0.2 1.2 1.2 0.5 0.6 3.7
100 6.3 8 1.1 1.1 0.4 0.1 0.5 0.6 2.8 100

##  3 Science observations

Typical observations in IFU mode are similar to those carried out in slit
mode: the observer has to choose the exposure time and the number of
accumulations. The scan option provides the possibility of covering a bigger
FOV by controlling the number of scan steps. The only difference is that in
IFU mode there are two dimensions for the movements, as explained previously.
The scan can be automatically repeated in time to obtain temporal series. In
order to calibrate the observations, flat-field images have to be taken every
1.5 h (maximum) and a series of polarimetric calibration images has to be
recorded at least once per day. The latter is performed using the Polarimetric
Calibration Unit Hofmann _et al._ (2012).

The spatial sampling can be done in single- or double-mode. Single sampling is
equivalent to taking a single exposure at every position in the FOV. In double
sampling mode, two exposures are taken at every position, one displaced half-
slice-width with respect to the other. With it, the spatial resolution due to
the sampling is doubled. The two spectral images are adequately interlaced by
the reduction pipeline. Figure 8 illustrates how the double-sampling step size
is 50 $\mu$m for the slice width of 100 $\mu$m ($0.375^{\prime\prime}$).

Figure 8: Representation of single sampling (left) and double sampling
(right). The rectangles represent the slice mirrors and their apparent
position with respect to the movements. Only two slice mirrors have been drawn
for simplicity.

$t_{\rm scan}\simeq[4*nac*(t_{\rm int}+t_{\rm readout})+t_{\rm
mov}]*sampling*nv*nh$ (1)

The scan time can be estimated using Eq. 1, where:

$nac$ = number of accumulations
$t_{\rm int}$ = integration time
$t_{\rm readout}$ = 30 ms
$t_{\rm mov}$ = 1.3 s approx.
$sampling$ = 1 for single, 2 for double sampling (see the text)
$nv$ = number of vertical steps (running perpendicular to the slices)
$nh$ = number of horizontal steps (running parallel to the slices)

Figure 9: Example of the raw image of IFU spectra in spectropolarimetric mode
centered at 15650 Å. The wavelength range is 40 Å and Fe i absorption lines
are visible. The most intense one is Fe i 15662.0 Å. Two series of eight
spectra are shown for two orthogonal states of polarization, one at the top of
the raw image and the other at the bottom. The apparent shift in wavelength
between two consecutive spectra is an optical shift due to the arrangement of
the mini-slits in two rows.

###  3.1 Reduction pipeline

The reduction pipeline Collados (2003) was updated to recognize the IFU mode
and to calibrate the data in any observing mode. It is based on the procedures
described in Collados (1999). The current version, GRIS_V8 is used for the
data reduction process with both slit and IFU modes. The pipeline does the
dark current and flat corrections, as well as the wavelength and polarimetric
calibrations. The pipeline also reorganizes the image of the spectra, from
what is seen in the detector (see Figure 9) into a 3D data cube containing two
spatial directions plus the spectral direction. The resulting file contains
two additional dimensions, one for the Stokes parameters ($I$, $Q$, $U$, and
$V$) and one for time, in the case of temporal series. The data has a spatial
sampling of $0.135^{\prime\prime}\times~{}0.1875^{\prime\prime}$, where the
first number is the pixel scale and the second is, in double sampling mode,
the slice width divided by 2. In single sampling mode it is
$0.135^{\prime\prime}\times~{}0.375^{\prime\prime}$.

Figure 9 shows an example of the raw image of IFU spectra seen at the
detector. It was taken in spectropolarimetric mode, so there are two groups of
eight spectra, corresponding to two orthogonal states of polarization. The
central wavelength is 15650 Å and the range is 40 Å. There is an apparent
shift in wavelength between two consecutive spectra. This effect is actually
an optical shift due to the arrangement of the mini-slits in two rows (as seen
in the insert of Figure 2). Some bad pixels and the readout structure
(vertical lines) of the detector are also present in this raw image. The
pipeline removes all these artifacts and corrects the optical shift. Five
spectral absorption lines are visible, Fe i 15662.0 Å being the most intense
one. The wavelengths of the other Fe i absorption lines are: 15645.0 Å,
15648.5 Å, 15652.9 Å and 15665.2 Å.

###  3.2 Examples of observations

Some examples of the data obtained with the IFU are described below in order
to show the instrument capabilities, versatility and performance.

####  3.2.1 1 $\times$ 2 scan of solar granulation

A small region at the center of the solar disk was observed after the
commissioning on September 2018. Figure 10 shows an example of the time-series
observations with a 1 $\times$ 2 scan
($6^{\prime\prime}\times~{}6^{\prime\prime}$). This series shows the
reconstructed images of the continuum intensity near 15650 Å, observed with a
cadence of 18 s. The total duration of the series is about 90 minutes. The
evolution of the granulation is clearly seen over the selected time range for
the figure. The number at the top of each image is the elapsed time relative
to the start of this temporal series. The formation of one exploding granule
is distinguishable at the lower-left corner of the image at 273 s, and its
evolution can be followed over the subsequent images. The analysis of the
evolving magnetic fields is in progress (Dominguez-Tagle et al., in prep.).

Figure 10: Example of a series of continuum intensity images near 15650 Å of
the solar granulation, observed with a 1 $\times$ 2 scan
($6^{\prime\prime}\times~{}6^{\prime\prime}$). The cadence is 18 s and the
elapsed time relative to the start of this temporal series is printed at the
top of each image.  Figure 11: Monochromatic images of parts of active region
12665 observed with a 3 $\times$ 3 scan
($18^{\prime\prime}\times~{}9^{\prime\prime}$). The images are, from left to
right and top to bottom, the continuum intensity and the Q, U, V monochromatic
images at a wavelength located 0.5 Å to the blue from the center of the Fe i
15648.52 Å spectral line.

####  3.2.2 3 $\times$ 3 scan of solar active region

Active region 12665 (located at solar coordinates S06E24) was observed on July
2017, with a 3 $\times$ 3 scan ($18^{\prime\prime}\times~{}9^{\prime\prime}$).
It was observed over 24 minutes in a temporal series with a cadence of 40 s.
This FOV covers a region with a small umbra with a spiral-shaped penumbra.
Figure 11 shows the continuum intensity map and the Q, U, V monochromatic
images extracted from the polarized spectrum at 0.5 Å to the blue from the
center of the Fe i 15648.52 Å spectral line. This spectral line is specially
suited for magnetic studies, because of its large magnetic sensitivity (Landé
factor $g_{\rm eff}=2$). The data sets have been obtained with three
modulation cycles. In each of these, a set of four images is recorded, from
which the four Stokes parameters I, Q, U, and V can be retrieved. All images
corresponding to the same modulation step are summed up in real time. The
signal to noise ratio achieved a value of 700 in the Q, U, and V continua
($I_{c}/\sigma_{Q,U,V}$, where $I_{c}$ is the continuum intensity and
$\sigma_{Q,U,V}$ is the Q, U, V continuum noise level -one standard
deviation-).

Figure 12: Example of polarized spectral profiles in a small wavelength range
around the center of the Fe i 15648.52 Å spectral line ($\lambda_{0}$). The
images correspond to the points indicated with a blue cross in Fig. 11. The
Zeeman splitting of the intensity profiles is apparent, as well as the
amplitude and separation of the Zeeman components in the polarized profiles.

Figure 12 shows the I, Q, U, V profiles at the point indicated by the blue
cross in Figure 11. Only a spectral range of 4 Å around the Fe i 15648.52 Å
spectral line is shown, whereas the recorded spectral range is 40 Å. The
Zeeman splitting in these selected points corresponds to about 2300 G. The
amplitudes of the linearly polarized profiles indicate a non-negligible
inclination of the magnetic field. The red dashed lines in Figure 11 are used
to display equivalent horizontal long-slit I, Q, U and V spectral images (see
Figure 13). The spatial variations of the field strength, polarity and
orientation are clearly detected, as indicated by the separation and sign of
the Zeeman components in the Q, U and V images.

Figure 13: Equivalent horizontal long-slit spectral images, corresponding to
the red dashed lines displayed in Fig. 11. The four images displays, from left
to right and top to bottom, the I, Q, U, and V profiles, normalized to the
continuum intensity of the quiet Sun at the solar disk center. The X-axis is
centered on the wavelength of the Fe i 15648.52 Å spectral line
($\lambda_{0}$), and the Y-axis represents the horizontal length of each map.

####  3.2.3 Published results based on IFU observations

Quiet-Sun magnetic flux cancellations were studied by Kaithakkal _et al._
(2020), from observations of quiet-Sun granulation performed on November 2018.
The 40-minute time-series observations have a cadence of 26.4 s, for a 1
$\times$ 2 scan ($6^{\prime\prime}\times~{}6^{\prime\prime}$). The spectral
range was centered on the Si i 10827.108 Å spectral line, which is
magnetically sensitive with a Landé factor $g_{\rm eff}=1.5$.

A plage region in active region 12723 was observed on October 2018, and the
analysis of the magnetic field structures, published by Anan _et al._ (2021)
is partially based on observations with GRIS/IFU. Three temporal series of 1
$\times$ 2 scan ($6^{\prime\prime}\times~{}6^{\prime\prime}$) with a cadence
of 26 s were performed by the IFU. The observations were centered on a
spectral region containing the He i triplet at 10830 Å and Si i 10827.108 Å
spectral line.

Magnetic fields in photospheric small-scale network regions were studied by
Campbell _et al._ (2021), from observations performed on May 2019. A 3
$\times$ 3 scan ($18^{\prime\prime}\times~{}9^{\prime\prime}$) of quiet Sun
inter-network regions, very close to the disk center, was used for the
observations. Their analysis is based on five spectral absorption lines,
including Fe i 15648.52 Å and Fe i 15652.87 Å.

On September 2019, a 3 $\times$ 3 scan
($18^{\prime\prime}\times~{}9^{\prime\prime}$) over the lead pore of active
region 12748, was conducted by Nelson _et al._ (2021) to study the line-of-
sight magnetic field strength in pores. Their research is mostly based on a
GRIS/IFU time-series observations of Fe i 15648.52 Å and Fe i 15652.87 Å
spectral lines, with a cadence of 67 s.

##  4 Conclusions

An IFU was designed and successfully commissioned on the GRIS spectrograph
installed at the GREGOR solar telescope of the Observatorio del Teide
(Tenerife). The IFU, based on image slicers, opens up new possibilities for
NIR observations of fast-evolving features in the Sun. It has very good
optical quality with low straylight, and gives unique performance, allowing
faster small-area scans than a traditional slit spectropolarimeter.

Based on the success of the first campaigns, the IFU has been offered to all
observers who can be granted telescope observing time. So far, more than two
years of operations have been completed and more than 15 teams of scientists
have observed with this instrument. The response of observers has been very
positive in terms of both feedback comments and the high demand for the
instrument. The first scientific results of observations with the IFU are
beginning to be published.

### Acknowledgments

This work was carried out with the funding of the Projects SOLARNET (FP7;
funded by the European Commission’s 7th Framework Program under grant
agreement no. 312495), GREST (funded by the European Commission’s H2020
Program under grant agreement no. 653982) and SOLARNET (H2020; funded by the
European Commission’s H2020 Program under grant agreement no. 824135). SDO
data are provided by the Joint Science Operations Center-Science Data
Processing.

The 1.5 m GREGOR solar telescope was built by a German consortium under the
leadership of the Leibniz-Institut für Sonnenphysik in Freiburg with the
Leibniz-Institut für Astrophysik Potsdam, the Institut für Astrophysik
Göttingen, and the Max-Planck-Institut für Sonnensystemforschung in Göttingen
as partners, and with contributions by the Instituto de Astrofísica de
Canarias and the Astronomical Institute of the Academy of Sciences of the
Czech Republic.

Preprint of an article submitted for consideration in Journal of Astronomical
Instrumentation © 2022 [copyright World Scientific Publishing Company]
https://doi.org/10.1142/S2251171722500143

## References

  * Anan _et al._ (2021) Anan, T., Schad, T. A., Kitai, R., Dima, G. I., Jaeggli, S. A., Tarr, L. A., Collados, M., Dominguez-Tagle, C. & Kleint, L. [2021] _Astrophysical Journal_ 921, 39, 10.3847/1538-4357/ac1b9c.
  * Bowen (1938) Bowen, I. S. [1938] _Astrophysical Journal_ 88, 113, 10.1086/143964.
  * Calcines _et al._ (2013a) Calcines, A., Lopez, R. L. & Collados, M. [2013a] _Journal of Astronomical Instrumentation_ 2, 1350007, 10.1142/S2251171713500074.
  * Calcines _et al._ (2013b) Calcines, A., Lopez, R. L. & Collados, M. [2013b] _Journal of Astronomical Instrumentation_ 2, 1350009, 10.1142/S2251171713500098.
  * Calcines _et al._ (2014) Calcines, A., Lopez, R. L., Collados, M. & Vega Reyes, N. [2014] “MuSICa image slicer prototype at 1.5-m GREGOR solar telescope,” _Ground-based and Airborne Instrumentation for Astronomy V_ , eds. Ramsay, S. K., McLean, I. S. & Takami, H., p. 91473I, 10.1117/12.2053577.
  * Campbell _et al._ (2021) Campbell, R. J., Mathioudakis, M., Collados, M., Keys, P. H., Asensio Ramos, A., Nelson, C. J., Kuridze, D. & Reid, A. [2021] _Astronomy and Astrophysics_ 647, A182, 10.1051/0004-6361/202040028.
  * Cao _et al._ (2012) Cao, W., Goode, P. R., Ahn, K., Gorceix, N., Schmidt, W. & Lin, H. [2012] “NIRIS: The Second Generation Near-Infrared Imaging Spectro-polarimeter for the 1.6 Meter New Solar Telescope,” _Second ATST-EAST Meeting: Magnetic Fields from the Photosphere to the Corona._ , eds. Rimmele, T. R., Tritschler, A., Wöger, F., Collados Vera, M., Socas-Navarro, H., Schlichenmaier, R., Carlsson, M., Berger, T., Cadavid, A., Gilbert, P. R., Goode, P. R. & Knölker, M., p. 291.
  * Collados (1999) Collados, M. [1999] “High Resolution Spectropolarimetry and Magnetography,” _Third Advances in Solar Physics Euroconference: Magnetic Fields and Oscillations_ , eds. Schmieder, B., Hofmann, A. & Staude, J., p. 3.
  * Collados (2003) Collados, M. [2003] “Stokes polarimeters in the near-infrared,” _Polarimetry in Astronomy_ , ed. Fineschi, S., p. 55, 10.1117/12.459370.
  * Collados _et al._ (2007) Collados, M., Lagg, A., Diaz Garcia, J. J., Hernandez Suarez, E., Lopez Lopez, R., Paez, E. & Solanki, S. K. [2007] “Tenerife Infrared Polarimeter II,” _The Physics of Chromospheric Plasmas_ , eds. Heinzel, P., Dorotovič, I. & Rutten, R. J., p. 611.
  * Collados _et al._ (2012) Collados, M., Lopez, R., Paez, E., Hernandez, E., Reyes, M., Calcines, A., Ballesteros, E., Diaz, J. J., Denker, C., Lagg, A., Schlichenmaier, R., Schmidt, W., Solanki, S. K., Strassmeier, K. G., von der Luhe, O. & Volkmer, R. [2012] _Astronomische Nachrichten_ 333, 872, 10.1002/asna.201211738.
  * Content (1997) Content, R. [1997] “New design for integral field spectroscopy with 8-m telescopes,” _Optical Telescopes of Today and Tomorrow_ , ed. Ardeberg, A. L., p. 1295, 10.1117/12.269020.
  * Dominguez-Tagle _et al._ (2018) Dominguez-Tagle, C., Collados, M., Esteves, M. A., Vega Reyes, N., Vaz Cedillo, J. J., Lopez, R. L., Paez, E., Grassin, O. & Gracia, F. [2018] “Commissioning tests of an Integral Field Unit (IFU) at GREGOR solar telescope,” _Ground-based and Airborne Instrumentation for Astronomy VII_ , eds. Evans, C. J., Simard, L. & Takami, H., p. 107022I, 10.1117/12.2312525.
  * Esteves _et al._ (2018) Esteves, M. A., Dominguez-Tagle, C., Vaz Cedillo, J. J., Grassin, O., Vega Reyes, N., Gracia Temich, F. & Collados Vera, M. [2018] “Development of a field-of-view scanning system (FoV-SS): test results and lessons learned,” _Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation III_ , eds. Navarro, R. & Geyl, R., p. 107066I, 10.1117/12.2313385.
  * Hofmann _et al._ (2012) Hofmann, A., Arlt, K., Balthasar, H., Bauer, S. M., Bittner, W., Paschke, J., Popow, E., Rendtel, J., Soltau, D. & Waldmann, T. [2012] _Astronomische Nachrichten_ 333, 854, 10.1002/asna.201211733.
  * Iglesias & Feller (2019) Iglesias, F. A. & Feller, A. [2019] _Optical Engineering_ 58, 082417, 10.1117/1.OE.58.8.082417.
  * Jaeggli (2022) Jaeggli, S. A. [2022] “The DKIST diffraction-limited near-infrared spectro polarimeter,” In preparation.
  * Jaeggli _et al._ (2010) Jaeggli, S. A., Lin, H., Mickey, D. L., Kuhn, J. R., Hegwer, S. L., Rimmele, T. R. & Penn, M. J. [2010] _Memorie della Societa Astronomica Italiana_ 81, 763.
  * Kaithakkal _et al._ (2020) Kaithakkal, A. J., Borrero, J. M., Fischer, C. E., Dominguez-Tagle, C. & Collados, M. [2020] _Astronomy and Astrophysics_ 634, A131, 10.1051/0004-6361/201936941.
  * Löfdahl (2002) Löfdahl, M. G. [2002] “Multi-frame blind deconvolution with linear equality constraints,” _Image Reconstruction from Incomplete Data_ , eds. Bones, P. J., Fiddy, M. A. & Millane, R. P., p. 146, 10.1117/12.451791.
  * Nelson _et al._ (2021) Nelson, C. J., Campbell, R. J. & Mathioudakis, M. [2021] _Astronomy and Astrophysics_ 654, A50, 10.1051/0004-6361/202141368.
  * Schmidt _et al._ (2012) Schmidt, W., von der Lühe, O., Volkmer, R., Denker, C., Solanki, S. K., Balthasar, H., Bello Gonzalez, N., Berkefeld, T., Collados, M., Fischer, A., Halbgewachs, C., Heidecke, F., Hofmann, A., Kneer, F., Lagg, A., Nicklas, H., Popow, E., Puschmann, K. G., Schmidt, D., Sigwarth, M., Sobotka, M., Soltau, D., Staude, J., Strassmeier, K. G. & Waldmann , T. A. [2012] _Astronomische Nachrichten_ 333, 796, 10.1002/asna.201211725.
  * Socas-Navarro _et al._ (2006) Socas-Navarro, H., Elmore, D., Pietarila, A., Darnell, A., Lites, B. W., Tomczyk, S. & Hegwer, S. [2006] _Solar Physics_ 235, 55, 10.1007/s11207-006-0020-x.
  * Vega-Reyes _et al._ (2016) Vega-Reyes, N., Esteves, M. A., Sanchez-Capuchino, J., Salaun, Y., Lopez, R. L., Gracia, F., Herrera, P. E., Grivel, C., Cedillo, J. J. V. & Collados, M. [2016] “Opto-mechanical design of an image slicer for the GRIS spectrograph at GREGOR,” _Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation II_ , eds. Navarro, R. & Burge, J. H., International Society for Optics and Photonics (SPIE), p. 1702 , 10.1117/12.2232615, URL https://doi.org/10.1117/12.2232615.


# C3LLM: Conditional Multimodal Content Generation Using Large Language Models

Zixuan Wang*
HKUST
<EMAIL_ADDRESS>
Qinkai Duan*
HKUST
<EMAIL_ADDRESS>
Yu-Wing Tai
Dartmouth College
<EMAIL_ADDRESS>Chi-Keung Tang
HKUST
<EMAIL_ADDRESS>

###### Abstract

We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a
novel framework combining three tasks of video-to-audio, audio-to-text, and
text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure
as a bridge for aligning different modalities, synthesizing the given
conditional information, and making multimodal generation in a discrete
manner. Our contributions are as follows. First, we adapt a hierarchical
structure for audio generation tasks with pre-trained audio codebooks.
Specifically, we train the LLM to generate audio semantic tokens from the
given conditions, and further use a non-autoregressive transformer to generate
different levels of acoustic tokens in layers to better enhance the fidelity
of the generated audio. Second, based on the intuition that LLMs were
originally designed for discrete tasks with the next-word prediction method,
we use the discrete representation for audio generation and compress their
semantic meanings into acoustic tokens, similar to adding “acoustic
vocabulary” to LLM. Third, our method combines the previous tasks of audio
understanding, video-to-audio generation, and text-to-audio generation
together into one unified model, providing more versatility in an end-to-end
fashion. Our C3LLM achieves improved results through various automated
evaluation metrics, providing better semantic alignment compared to previous
methods.

##  1 Introduction

Conditional multimodal generation is the task of generating output that
incorporates different modalities, such as text, image, video, and audio [24,
48, 21, 22]. Essentially, this multimodal task can be seen as a translation
task among different modalities, and thus, challenges arise for making
inferences from cross-modal representations and dealing with potential
modality gaps [29].

Figure 1: C3LLM is capable of video-to-audio, audio-to-text, and text-to-
audio. Examples of different tasks are illustrated in arrows of different
colors.

Multimodal Large Language Models (MM-LLMs) have recently gained significant
interest in research due to their ability to understand and follow user
instructions. Most work focuses on contextual understanding across various
modalities like video-to-text [52, 40], audio-to-text [20, 25] and image-to-
text [31]. However, the area of audio generation, particularly video-to-audio
[26, 10] or image-to-audio generation [39], remains underexplored. This is
partly because video contains excessive visual information not always needed
for audio generation, while images lack the temporal information crucial for
audio. Video-to-audio tasks also face synchronization challenges, with recent
solutions like temporal masking [50] proving inadequate for complex scenarios.
Additionally, current methods often encode video features by extracting a few
random frames [50, 7], which hinders learning temporal information.

Audio generation is less intuitive compared to other tasks, as it is harder to
precisely measure the generated sound quality using human ears. Additionally,
previous works mainly focus on generating music-related audio, which is more
structured compared to naturally occurring audio [6, 32]. Few works have
focused on generating visually guided open-domain audio clips [4, 55].
Moreover, most existing models [26, 51] are only limited to generating audio-
only content, consequently constrained to specific downstream tasks. While
contemporary work CoDi [44] achieves some form of any-to-any generation, their
result simply uses linear interpolations in the latent space. C3Net [53]
adapted three ControlNet [54] architectures on top of CoDi’s design, still it
also relies on interpolation when predicting the final output.

In this connection, we aim at utilizing the versatility of LLM to align
between different modalities. With sufficient data, transformers or LLMs have
shown to be effective in serving as a unified backbone on different modality
tasks even with simple design [42]. We thus propose Conditioned-On-Three-
Modalities Large Language Model, or C3LLM in short. C3LLM mainly comprises a
LLM backbone serving as the bridge among three different modalities, and a
hierarchy audio tokenizer adapted from Encodec [11] for decoding. Our model
first encodes the respective modality, either audio, text or video, to be
processed by the LLM backbone. For video, we extract the dense information and
project it to the LLM’s embedding space. For audio, we use the audio tokenizer
to convert the information into discrete representation and translate the
corresponding indices from the tokenizer codebooks into LLM special tokens,
which are extended as part of LLM’s vocabulary beforehand. The semantic
information is further processed by the LLM. For tasks involving audio
generation, We treat the LLM prediction as coarse acoustic tokens. The
preliminary result is further extended to fine-grained acoustic tokens and
combined to generate the final audio output, and thus multimodal output not
limited to a single modality.

To sum up, our contributions are as follows: 1) C3LLM utilizes the versatility
of LLM for conditional multimodal generation tasks, where the LLM treats
encoded audio information as additional acoustic vocabulary; 2) C3LLM uses a
discrete tokenizer for modeling audio representation hierarchically, which
better suits the nature of LLM while preserving the quality of the generated
audio; 3) Consequently, this paper provides a uniform model for three
different tasks involving video, audio and text, see Figure 1. Through
extensive evaluation, our work demonstrates on-par results compared to the
state-of-the-art models in their respective domains, providing further insight
into conditional multimodal tasks.

Figure 2: Overview of the video encoding part and the LLM. On the right is the
non-autoregressive (NAR) transformer to further extend the coarse acoustic
tokens into fine-grained acoustic tokens. We freeze the encoder, train a MLP,
and finetune LLM using LoRA. For text condition, we directly input them into
LLM after tokenization and also treat them as the condition for NAR
transformer decoding.

##  2 Related Work

Multimodal Alignment Mokady et al. [33] has utilized the powerful Contrastive
Language-Image Pre-Training (CLIP) [37] model to project the image into an
image-text shared latent space. Given image embedding, they further apply a
GPT-2 [38] to generate caption. Wu et al. [17] trained CLAP model on LAION-
Audio-630K, a large collection of 633,526 audio-text pairs from different data
sources, to obtain a robust and general result on all types of audio clips.
Similar to them, we encode video, audio and text data into a shared latent
space using contrastive learning and projection. This approach enables an
audio encoder to project audio into a space shared with image and text,
aligning more closely with the hidden space of a large language model. For
long audio understanding, local features and overall information are fused.

Multimodal LLM LLM is prevailing in the AI community. ChatGPT [34] and GPT-4
[35] have shown great power in understanding and reasoning tasks. Other open
source LLMs, such as LLama [36], Vicuna [5], Alpaca [45], and Gemma [16] have
greatly contributed to the research community and inspired many innovations.
With these advances, progress has been made in using LLMs for understanding
multimodal information, improving the performance of MM-LLMs [31].

We notice that most LLMs understand multimodal information with the help of a
well-trained encoder to bridge the gap between modality information and LLM
hidden space. For example, PandaGPT [41] utilizes ImageBind [19] to align
multiple modalities. Video-LLama [52] applies two Q-formers to transform video
and audio information. Llava [31] directly applies a linear projection layer
to give LLM image information. MiniGPT-4 [56] follows this approach, using
linear layers to align Vicuna and Q-former. To fit the discrete and auto-
regressive nature of LLMs, Large World Model [30] achieves long-context video
understanding by combining VQGAN [12] to tokenize each frame of the video.
However, these LLMs lack the capability to generate modalities other than
language.

Multimodal Generation In multimodal generation, the goal is to generate
various modalities like audio, text, images, and videos interchangeably.
State-of-the-art approaches, such as Composable Diffusion (CoDi) [44] and
C3Net [53], produce diverse modality combinations conditioned on other
modalities. NExT-GPT [49] and CoDi2 [43] leverage LLMs to process and
synthesize semantic information from different modalities, providing a wide
range of meaningful conditions for the diffusion model. Diffusion models are
crucial for generating and refining high-quality content, typically encoding
each modality into a shared latent space and using MLPs to project information
between the LLM hidden space and the diffusion latent condition space.

However, these methods often overlook the distinct features of each modality.
For instance, NExT-GPT uses ImageBind [19] to encode videos into image space,
losing temporal features. In audio generation, the diffusion model struggles
to map audio timing accurately to corresponding video frames, making it
challenging to incorporate the temporal dimension of video modality.

Figure 3: Overview of audio encoding part and the LLM. We use a pre-trained
CLAP encoder and an MLP layer to align the audio information with LLM
embedding.

Conditioned Audio Generation To better focus on the time domain of audio, the
transformer architecture [46] is well-suited due to its attention mechanism.
In the context of audio generation, SpecVQGAN [26] has successfully employed a
neural codebook to represent audio information, enabling the use of a
transformer to predict discrete audio tokens based on video features.

Recently, VALL-E [47] and MusicLM [13] use multiple codebooks and Residual
Vector Quantization (RVQ) [11] to create diverse audio representations. An
auto-regressive transformer is employed to predict the initial layers of audio
indices based on the textual conditions, and a non-auto-regressive (NAR)
transformer is followed to generate the subsequent layers, taking into account
the previously generated tokens and conditions. It has been observed that the
first audio index layer captures crucial information for overall audio
quality, like rhythm and intonation, while subsequent layers add detailed
information, enhancing richness.

##  3 Method

C3LLM comprises three major components: 1) Tokenizers for encoding
corresponding modality information to be comprehended by LLM; 2) a multimodal
LLM that serves as a powerful backbone that connects corresponding modalities.
Especially for audio generation tasks, we have 3) a non-autoregressive (NAR)
Transformer that further refines the generated coarse acoustic tokens from the
LLM. Our model structure is illustrated in Figure 2 and Figure 3.

###  3.1 Audio Tokenizer and Video Tokenizer

For audio generation, we employ the EnCodec [11] pretrained codebooks and
decoder, which utilizes the RVQ method to obtain eight codebooks that enable
high-bandwidth audio reconstruction. To streamline the process and conserve
computational resources, we focus on the first two codebooks. We leverage the
LLM to predict the first layer, taking into account text or visual conditions,
and we train the NAR transformer decoder specifically to predict the second
layer of audio tokens. After the prediction of the first two layers of audio
tokens, the pre-trained decoder decodes them back into audio waveform.

For the audio-to-text task, we leverage the powerful audio encoder of CLAP to
convert audio waveforms into a shared space that is more similar to the hidden
space of the LLM. We freeze the audio encoder and introduce an MLP to project
the audio vectors into the hidden space of the LLM. This allows for a more
effective integration of audio information into the text-generation process.
By utilizing the capabilities of CLAP and LLM together, we achieve improved
performance in generating detailed descriptions from audio inputs.

When it comes to the video-to-audio task, we draw inspiration from the
successful approach employed by SpecVQGAN [26]. To efficiently capture both
visual and temporal information while compressing the video data, we employ a
frame-wise feature extractor denoted as $H$. This feature extractor extracts
RGB features ($f^{r}$) and optical flow features ($f^{o}$) from each frame of
the video. By applying frame-wise concatenation, we obtain the video feature
representation $F=\\{f_{i}^{r},f_{i}^{o}\\}_{i=1}^{N}$, where $N$ represents
the total number of frames in the video. To further enhance the integration of
visual information, we introduce an additional MLP that transforms the
concatenated video features into embeddings suitable for the LLM.

Contrary to the discrete audio tokenization method, we choose to use the
continuous representation for video. During our experiment, we observe that
video involves too much visual information that audio generation may not
necessarily need. A similar discrete method that uses VQGAN [12] to process
frame-level information will result in excessive visual tokens, making
learning visual information inefficient. The MLP projection layer that we
employed will project the continuous video feature into LLM embedding space,
similar to previous methods [31]. For text input, we directly use the LLM’s
tokenizer for tokenization.

###  3.2 Autoregressive Generation of Coarse Audio Tokens

Employing the EnCodec audio tokenizer allows us to represent continuous audio
information in discrete form. We denote the continuous audio input as
$a\in\mathbb{R}^{C\times L}$, where $C$ is the number of channels and $L$ is
the time of the audio clip times sample rate. First, the audio input is
encoded in a smaller representation in the form of
$z=E(a)\in\mathbb{R}^{C\times N\times D\times Q}$, with $Q$ denoting the
number of quantizers used during the encoding process and $D$ is the dimension
of the codebooks. Our next step is to convert the representation into LLM-
aware acoustic tokens. Specifically, we obtain the indices
$s\in\mathbb{R}^{C\times N\times Q}$from the encoded audio by comparing with
the quantizer codebook.

To jointly model different modalities in a unified model, we further extended
the LLM’s text vocabulary $V_{t}=\\{v_{i}\\}_{i=1}^{N_{t}}$with acoustic
vocabulary $V_{a}=\\{v_{j}\\}_{j=1}^{N_{a}}$. The extended audio-text
vocabulary now becomes $V=\\{V_{t},V_{a}\\}$. Contrary to previous audio
generation models [26, 10] that involve the generation of a single modality,
our method equipped the LLM backbone with the ability to understand and
generate both audio and text information with a unified vocabulary.

To better differentiate the three kinds of modalities that condition the
autoregressive generation, we further wrap the encoded feature with special
tokens as modality indicators. To be more specific, we wrap the audio tokens
with <Audio>,</Audio> indicators and video embedding in an embedded sequence
of <Video>,</Video> indicators. In doing so, we avoid the possibility of
confusing the LLM with different kinds of information.

To further elaborate on the conditional generation tasks performed by LLM: for
audio-to-text and text-to-audio tasks, the source input
$X_{a,t}=\\{x_{t}^{i}\\}_{i=1}^{N}$ is a sequence of either acoustic/text
tokens. Here $N$ is the number of tokens we have and $x_{t}\in V$; for video-
to-audio task, the source input $X_{v}=\\{x_{e}^{i}\\}_{i=1}^{N}$ is a
sequence of embeddings and $x_{e}\in\mathbb{R}^{D}$, where $D$ is the
embedding dimension of LLM. After LLM’s tokenization, the input tokens for
audio and text will become input embeddings and fed into the LLM. Our LLM
backbone is a decoder-only structure with the next token prediction method.
The distribution of the predicted token in the first layer is given by
$p_{\theta_{LLM}}(\mathbf{C}_{1}|X)=\prod_{i}p_{\theta_{LLM}}(c_{1}^{i}|X,\mathbf{C}_{1}^{<i})$
autoregressively. The objective has thus become:

$\mathcal{L}_{\mathit{LLM}}=-\sum_{i=1}^{T^{\prime}}\log
p_{\theta_{\mathit{LLM}}}(c_{1}^{i}|X,\mathbf{C}_{1}^{<i}),$ (1)

where $T^{\prime}$ is the number of acoustic tokens generated by LLM,
$\theta_{\mathit{LLM}}$ is the parameter of LLM, $c_{1}^{i}$ is the token
generated at step $i$, $\mathbf{C}_{1}^{<i}$ are previous tokens, and $X$ is
the text or video condition.

During inference, the LLM will autoregressively predict the next token until
<eos> is generated. Our LLM thus serves as the bridge for connecting between
different pairs of modalities. The generated output will be decoded
subsequently.

Due to the limited computation resource available, we use Gemma-2B [16], a
lightweight open-source LLM developed by Google, which is claimed to have
comparable performance with LLaMA-2-7B [14] on many QA and reasoning tasks. We
use Low-Rank Adaptor (LoRA) [23] to finetune Gemma to make it understand
vision/text conditions and generate audio tokens.

###  3.3 Non-Autoregressive Transformer for Audio Refinement

In C3LLM, we propose an audio refinement method to further ensure the
generated audio fidelity. Inspired by [47], we utilize a non-autoregressive
Transformer (NAR) to transform coarse acoustic tokens from LLM’s output to
fine-grained acoustic tokens. We treat the video embedding or text input as
condition and concatenate it with the generated coarse acoustic tokens. In the
original paper, the NAR is used to predict seven layers of acoustic tokens
given by the first layer. However, we find this design very slow to converge
during our experiment. We adopt a simpler design to only utilize two layers of
codebooks, and train the NAR to predict the second layer given the first layer
prediction generated by LLM. Thus probability distribution for the next layer
is given by
$p_{\theta_{\mathit{NAR}}}(\mathbf{C}_{2}|\mathbf{X},\mathbf{C}_{1})$, and we
want to minimize the objective function:

$\mathcal{L}_{\mathit{NAR}}=-\log
p_{\theta_{\mathit{NAR}}}(\mathbf{C}_{2}|\mathbf{X},\mathbf{C}_{1})=-\sum_{i=1}^{T^{\prime}}\log
p_{\theta_{\mathit{NAR}}}(c_{2}^{i}|\mathbf{X},\mathbf{C}_{1}).$ (2)

###  3.4 Detokenization for High Fidelity Output

The decoder combines LSTM and CNN architectures. The LSTM component emphasizes
temporal consistency, while the CNN component reconstructs frequency
information. The model employs a combination of L1 loss for the time domain
and a set of L1 and L2 losses for the Mel-spectrogram in the frequency domain,
across various time scales. To effectively preserve audio information, the
model incorporates two strategies: 1) Utilizing a greater number of tokens to
represent each second of audio, thereby increasing the sample rate. 2)
Employing multiple codebooks to capture a wider bandwidth, enhancing the
representation.

The audio waveform can experience interference from multiple sources. By
employing more codebooks, the model can effectively decompose overlapping
signals, with each codebook capturing audio of different frequencies.

##  4 Experiments

###  4.1 Training Datasets

We apply corresponding datasets for the three tasks. For video-to-audio task,
we finetune our model on the VGGSound [3] dataset, which contains over 310
classes of 200,000+ videos, capturing challenging real-world acoustic
scenarios. Concerning the large size, we use around half of the common version
of VGGSound containing 164 classes. The resulting number of training video
samples is 63,853.

For audio-to-text and text-to-audio tasks, we use the AudioCaps [8] dataset.
AudioCaps [8] dataset is a large-scale dataset of about 46K audio clips paired
with human-written text collected via crowdsourcing on the AudioSet [18]
dataset. The dataset contains a diverse range of 10-second audio samples from
various natural sources, including vehicles, animals, weather, etc. We filter
all the failed links and produce 45,028 sound files in train split. For audio-
to-test task, we notice that in some papers such as EnCLAP [27], multiple
referencing ground truth are used for evaluation. As the original AudioCaps
paper [8] only contains one caption per audio clip, we choose to only use one
caption as the referencing ground truth.

###  4.2 Evaluation Metrics

The evaluation metrics are summarized as follows: For video-to-audio and text-
to-audio tasks, we use the Inception score (ISc) and Frechet audio distance
(FAD) to evaluate audio fidelity. For audio-video relevance, we utilize the
MKL metric [26] and we use KL for text-to-audio task. For audio-to-text task,
we use the CIDEr (Consensus-based Image Description Evaluation), SPIDEr
(SPeech-to-Image Description Evaluation), and SPICE (Semantic Propositional
Image Caption Evaluation). Furthermore, to evaluate the synchronization of the
generated audio in the video-to-audio setting, we use the same evaluation
metrics as CondFoleyGen [10], namely # Onset Accuracy [10], and Onset AP [10].

###  4.3 Evaluation and comparison

We mainly compare our model with CoDi [44], which is the current state-of-the-
art model combining different multimodal content generation tasks. We download
the pretrained fp16 version of CoDi [44] model and evaluate on the same test
set. The training and evaluation are conducted on NVIDIA GeForce RTX 4090. The
main result is presented in Table 1

Figure 4:  Comparison with baseline for video-to-audio generation task. CoDi failed to generate semantic-aligned audio and the generation is not clean, often mixed with human speaking or noise. Our method can produce aligned audio with clear synchronization.  Task | Method | Metric
---|---|---
V2A |  | KL$\downarrow$ | ISc$\uparrow$ | FAD$\downarrow$
Codi [44] | 3.6449 | 3.6941 | 11.5036
Ours (w/o NAR) | 3.5873 | 2.1301 | 17.1095
Ours | 3.5522 | 2.5783 | 10.2217
A2T |  | SPIDEr$\uparrow$ | CIDEr$\uparrow$ | SPICE$\uparrow$
Codi [44] | 0.0726 | 0.0812 | 0.0640
Ours | 0.3150 | 0.4721 | 0.1579
T2A |  | KL$\downarrow$ | ISc$\uparrow$ | FAD$\downarrow$
CoDi [44] | 3.1786 | 4.5047 | 10.1597
Ours (w/o NAR) | 3.4865 | 2.5732 | 22.7320
Ours | 3.6765 | 2.9606 | 18.1615
Table 1: Quantitative comparison with baseline on three tasks. Method | Metric
---|---
| # Onset Accuracy$\uparrow$ | Onset AP$\uparrow$
Codi [44] | 0.097 | 0.535
Ours | 0.142 | 0.670
Table 2: Results for evaluating video-audio synchronization on VGGSound dataset Ground truth | Our method |  CoDi [44]
---|---|---
continuous snoring of a person | a person snoring | dog sleeping on a bed
church bells ringing | bells are ringing | angel is in a coat by in red hat
a car engine is revving while driving | a vehicle engine accelerates | driving for speed going for a crossing
a telephone ringing | a telephone rings several times | phone of the phone
A cat meowing a few times | a cat is meowing |  catwoman’s cats are coming cat of  the cats in a cat calendar
spinning tires on pavement | a car is skidding wildly |  car for just avoiding a speeding car  wreck with cold scary highway blare
Table 3: Samples output obtained from AudioCap test set. We observe that CoDi
often fails to capture the semantic meaning, and the generated captions are
more like describing visual input rather than acoustic sound. Additionally,
the output is not sufficiently fluent. We thus believe LLM structure is more
capable for captioning tasks

Table 4: Additional A2T task conducted on Clotho dataset Method | Metric
---|---
| SPIDEr$\uparrow$ | CIDEr$\uparrow$ | SPICE$\uparrow$
Codi [44] | 0.0640 | 0.0766 | 0.0514
Ours | 0.2088 | 0.3097 | 0.1078

Table 5: Additional V2A task conducted on VAS dataset Method | Metric
---|---
| KL$\downarrow$ | ISc$\uparrow$ | FAD$\downarrow$
Codi [44] | 4.54874 | 3.12170 | 11.80060
Ours | 3.97517 | 2.69774 | 10.35411

For audio-to-test task, we use the open-sourced Audio Captioning metrics [28]
for evaluation. we observe that CoDi’s performance is lower by a large margin,
which might be the result that LLM is more capable of captioning tasks due to
the next-word prediction method. We include some test results in Table 3.

For video-to-audio task, our result is better than the baseline except for the
ISc metric. The ISc metric measures the diversity of the generated audio. As
our model is only fine-tuned on Gemma-2B [16] backbone, we believe our result
will be improved with more power backbone and tunable parameters, given more
training resources. Additionally, our method achieves synchronization with the
input video, as shown in Figure 4. The quantitative evaluation is presented in
Table 2

We notice that for text-to-audio, CoDi is trained on multiple datasets such as
AudioCaps [8], AudioSet [18], BBC Sound Effect, Soundnet [2], and Freesound.
On the other hand, our model is only trained on AudioCaps. The total number of
training examples for CoDi is significantly larger than ours. Furthermore,
there is a huge domain gap between text and audio. Audio waveform has time
information while text does not, so it is hard to map a sentence to a specific
audio token in each time frame. Besides, we utilize LoRA [23] to finuetune LLM
which is not capable of bridging the gap with so few trainable parameters.
These are the reasons for the comparison result.

###  4.4 Ablation Studies

We hereby conduct experiments to test how the non-autoregressive transformer
will refine the output coarse acoustic tokens. We include our results in Table
1. As shown in the table, the NAR plays a central role in further improving
the result.

To mitigate the possible effect of using only one referencing ground truth for
evaluating audio-to-text task, we tested our model on the Clotho dataset [9].
The Clotho dataset contains 4981 audio clips and 3938 clips in train split.
Audio waveforms are from 15 to 30s duration, and each audio has 5 captions
which are 8 to 20 words long. We perform similar processing as the AuidoCaps
dataset for training and evaluation. Table 5 tabulates the results.

We also conducted an additional evaluation of our model on the VAS [15]
dataset for video-to-audio task, as shown in Table 5. We obtain a similar
result as VGGSound. Our model continues to outperform the baseline.

##  5 Conclusion and Discussion

###  5.1 Limitation and Future Work

Due to limited computation power, we can only take one modality input and
generate text or audio. Next, we want to condition on two or more modalities
and generate video, and we want to generate long audio/video. Moreover, to
bridge the modality gap between text-to-audio generation, we will adopt
semantic tokens of audio to give well-aligned information. We will utilize
more powerful LLM backbones such as LLama3 [1]

###  5.2 Conclusion

In this paper, we present C3LLM, a unified structure that can perform three
tasks namely video-to-audio, audio-to-text and text-to-audio. Our model
capitalizes on the power of LLM for translating and aligning between different
modalities. We also propose a non-autoregressive transformer for audio
refinement. Through extensive experiments, we show that our model can
synthesize high-fidelity audio, ensuring semantic alignment with input,
especially synchronization with the visual condition.

Although our model demonstrates excellent results through evaluation,
challenges exist that can restrain the performance. Specifically for the
audio-to-text task, our model relies on the pre-trained CLAP encoder that
poses an upper-bound for modeling more complex scenarios. A more efficient way
for audio encoding is worthwhile in future research.

## References

  * [1] AI@Meta.  Llama 3 model card, 2024.
  * [2] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  In Advances in Neural Information Processing Systems, 2016.
  * [3] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  Vggsound: A large-scale audio-visual dataset.  In ICASSP, 2020.
  * [4] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan.  Generating visually aligned sound from videos.  IEEE Transactions on Image Processing, 2020.
  * [5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [6] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez.  Simple and controllable music generation.  arXiv:2306.05284, 2024.
  * [7] Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serra, Taylor Berg-Kirkpatrick, and Julian McAuley.  Clipsonic: Text-to-audio synthesis with unlabeled videos and pretrained languagevision models.  WASPAA, 2023.
  * [8] Kim Chris Dongjoo, Kim Byeongchang, Lee Hyunmin, and Kim Gunhee.  Audiocaps: Generating captions for audios in the wild.  In NAACL-HLT, 2019.
  * [9] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.  Clotho: an audio captioning datase.  In Proceedings of the ICASSP, pages 736––740, 2020.
  * [10] Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens.  High fidelity neural audio compression.  In CVPR, 2023.
  * [11] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  arXiv:2210.13438, 2022.
  * [12] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [13] Andrea Agostinelli et. al.  Musiclm: Generating music from text.  arXiv:2301.11325, 2023.
  * [14] Hugo Touvron et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv:2307.09288, 2023.
  * [15] Peihao Chen et al.  Generating visually aligned sound from videos.  TIP, 2020.
  * [16] Thomas Mesnard et al.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  arXiv:2403.08295, 2024.
  * [17] Yusong Wu et al.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  arXiv:2211.06687, 2022.
  * [18] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In ICASSP, 2017.
  * [19] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In CVPR, 2023.
  * [20] Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass.  Listen, think, and understand.  arXiv:2305.10790, 2023b., 2024.
  * [21] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans.  Imagen video: High definition video generation with diffusion models.  arXiv:2210.02303, 2022.
  * [22] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  arXiv:2205.15868, 2022.
  * [23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W Chen.  Lora: Low-rank adaptation of large language models.  arXiv:2106.09685, 2021.
  * [24] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao.  Visual instruction tuning.  arXiv:2304.08485, 2024.
  * [25] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  ArXiv, abs/2304.12995, 2023.
  * [26] Vladimir Iashin and Esa Rahtu.  Taming visually guided sound generation.  arXiv:2110.08791, 2021.
  * [27] Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo.  Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning.  arXiv preprint arXiv:2401.17690, 2024.
  * [28] Etienne Labbé.  aac-metrics: Metrics for evaluating automated audio captioning systems for pytorch.  https://github.com/Labbeti/aac-metrics/, 2013.
  * [29] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou.  Mind the gap: Understanding the modality gap in multimodal contrastive representation learning.  arXiv:2203.02053, 2022.
  * [30] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel.  World model on million-length video and language with ringattention.  arXiv preprint, 2024.
  * [31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv:2304.08485, 2024.
  * [32] Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herremans, and Soujanya Poria.  Mustango: Toward controllable text-to-music generation.  arXiv:2311.08355, 2023.
  * [33] R. Mokady, A. Hertz, and A. H. Bermano.  Clipcap: Clip prefix for image captioning.  arXiv:2111.09734, 2021.
  * [34] OpenAI.  Introducing chatgpt, 2022.
  * [35] OpenAI.  Gpt-4 technical report, 2023.
  * [36] OpenAI.  Llama: Open and efficient foundation language models, 2023.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners, 2019.
  * [39] Roy Sheffer and Yossi Adil.  I hear your true colors: Image guided audio generation.  In ICASSP, 2023.
  * [40] Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie.  Audio-visual llm for video understanding.  arXiv:2312.06720, 2024.
  * [41] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355, 2023.
  * [42] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Emu: Generative pretraining in multimodality.  arXiv:2307.05222, 2024.
  * [43] Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal.  Codi-2: In-context, interleaved, and interactive any-to-any generation.  arXiv:2311.18775, 2023.
  * [44] Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal.  Any-to-any generation via composable diffusion.  arXiv:2305.11846, 2023.
  * [45] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [47] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv:2301.02111, 2023.
  * [48] Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li.  Customvideo: Customizing text-to-video generation with multiple subjects.  arXiv:2401.09962, 2024.
  * [49] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  arXiv:2309.05519, 2023.
  * [50] Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li.  Sonicvisionlm: Playing sound with vision language models.  arXiv:2401.04394, 2024.
  * [51] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  arXiv:2207.09983, 2022.
  * [52] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  CoRR, abs/2306.02858, 2023.
  * [53] Juntao Zhang, Yuehuai Liu, Yu-Wing Tai, and Chi-Keung Tang.  C3net: Compound conditioned controlnet for multimodal content generation.  arXiv:2311.17951, 2023.
  * [54] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In ICCV, 2023.
  * [55] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L. Berg.  Visual to sound: Generating natural sound for videos in the wild.  In CVPR, 2018.
  * [56] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.

##  Appendix A Appendix / supplemental material

###  A.1 Audio-to-Text examples

In this section, we provide more examples of the predicted output of the
audio-to-text task in Table 6, in addition to Table 3. We also include some
examples from the Clotho dataset [9], shown in Table 7.

Ground truth | Our method |  CoDi [44]
---|---|---
a person snoring | a person snores lightly | sleeping or a baby in a skiing
a river stream of water flowing | water is rushing by | brown birds outside in the forest
clicking followed by humming noise | an engine humming and clicking | highway worker hiking cold turkey
sounds of a river with man briefly mumbling | water is flowing and a man speaks | waterfall falls in water
several goats bleat | sheep bleat nearby | the brown goat
a police siren going off | a siren is emitted |  clear weather warning direction as  water officer does traffic safety direction while a traffic stop in her direction.
Table 6: Additional samples output obtained from AudioCap test set. We again notice that words describing colors or visual scenes exist in CoDi’s output which is unusual. Ground truth 1 | Our method |  CoDi [44]
---|---|---
a radio dispatcher and an officer are communicating over the radio |  a radio is tuned, as a person speaks over the radio |  foreign radio is not time to  watch someone on the phone to communications wire
lost of people are conversing in a very busy diner |  a group of people are talking and laughing | people walking in the area
a machine is running in a humming manner while metal is buzzing |  a buzzing electric engine that is trying to start up |  pilot cable bowler take a caution  steer light not flying < speed> engine.
Table 7: Additional samples output obtained from Clotho test set. Due to the
space limit, we only include ground truth caption 1. Other referencing
groundtruth can be found in the CSV file provided by the original paper [9].