# Coadded Spectroscopic Stellar Parameters and Abundances from the LAMOST Low
Resolution Survey

Jacob H. Hamer Department of Physics and Astronomy, Johns Hopkins University,
3400 N. Charles Street, Baltimore, MD 21218, USA

###### Abstract

I combine duplicate spectroscopic stellar parameter estimates in the LAMOST
Data Release 6 Low Resolution Spectral Survey A, F, G, and K Type stellar
parameter catalog. Combining repeat measurements results in a factor of two
improvement in the precision of the spectroscopic stellar parameter estimates.
Moreover, this trivializes the process of performing coordinate-based cross-
matching with other catalogs. Similarly, I combine duplicate stellar abundance
estimates for the Xiang et al. (2019) catalog which was produced using LAMOST
Data Release 5 Low Resolution Spectral Survey data. These data have numerous
applications in stellar, galactic, and exoplanet astronomy. The catalogs I
produce are available as machine-readable tables at
https://doi.org/10.7281/T1/QISGRU.

Astronomy data analysis — Stellar abundances — Fundamental parameters of stars
— Stellar properties — Sky surveys — Astrophysics - Solar and Stellar
Astrophysics

## 1

The Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST) Data
Release 6 (DR6) contains a catalog of spectroscopic stellar parameters for
5,773,552 A, F, G, and K stars generated from the R$\sim$1800 Low Resolution
Survey (LRS) (Cui et al., 2012; Zhao et al., 2012). While the Medium
Resolution Survey (MRS) data release combines repeat observations into coadded
spectra, from which they produce stellar parameter estimates, the LRS does not
do so. In this research note I describe the production of a catalogue which
combines parameter estimates for objects with repeat observations.

The LAMOST LRS input catalog describes tname as a unique identifier for each
object in the catalog. However, matching the catalog to itself on-sky reveals
matches between objects with different tname as well as tsource, with similar
spectroscopic stellar parameters. Therefore, tname is not a sufficient
identifier to find all parameter estimates for the same star. Additionally, I
find that a number of objects from the PILOT survey for which a single tname
corresponds to multiple objects with similar declinations but drastically
different right ascensions. I therefore undertake the following steps to
combine parameter estimates for different observations of the same star.
First, I group the catalog by tname and tsource. Then, I check that the
positions of the grouped objects do not differ by more than 10 arcseconds in
order to exclude the sources from the PILOT survey that have identical tname
but different on-sky positions. Next, I check that the radial velocity
estimates are not discrepant, only combining measurements when the
significance

$t=\frac{\mathrm{RV}_{\mathrm{max}}-\mathrm{RV}_{\mathrm{min}}}{\sqrt{\sigma_{\mathrm{RV,max}}^{2}+\sigma_{\mathrm{RV,min}}^{2}}}$
(1)

is less than 3. Such a discrepancy could indicate that the objects are being
incorrectly grouped by our method, or that the star is in an unresolved
binary.

For grouped objects which pass the two tests described, I combine their
stellar parameter estimates as follows. I calculate the arithmetic weighted
mean of each parameter using the SNR in the SDSS g band as the weight:

$\bar{x}=\frac{\sum_{i}^{N}\mathrm{SNR}_{i}x_{i}}{\sum_{i}^{N}\mathrm{SNR}_{i}},$
(2)

where $x_{i}=\\{T_{\mathrm{eff}},\mathrm{[Fe/H]},\log{g},\mathrm{RV}\\}$
denotes a set of spectroscopic stellar parameters from one observation, and
SNR is the signal-to-noise ratio in the SDSS g band. I also produce a right
ascension and declination for the combined observation with the same equation.
It follows from error propagation that the uncertainty on our new parameter
estimate is

$\sigma_{\bar{x}}=\frac{\sqrt{\sum_{i}^{N}\mathrm{SNR}_{i}^{2}\sigma_{x,i}^{2}}}{\sum_{i}^{N}\mathrm{SNR}_{i}}.$
(3)

I estimate the combined SDSS g band SNR of this “coadded” observation by
adding the g band SNR of each observation in quadrature. I also report the
number of observations which go into the new parameter estimate.

At this point, I have combined observations which can be grouped based on
their shared tname and tsource, but have not yet grouped objects which differ
in these identifiers but are co-located on the sky. To identify stars which
should be combined based on their distance on-sky, I match the grouped catalog
with itself on the sky using TOPCAT. I use a radius of 0.5 arcseconds, and I
retain all matches. I use the GroupID outputted by TOPCAT as a key to identify
which observations to group, and combine them using the same methods described
above, including the check that the radial velocities do not have a
discrepancy with a significance over 3.

The catalog which results from these steps to combine multiple observations is
4,334,538 objects. 22% of these sets of spectroscopic stellar parameters are
the result of coadding. 72% are the result of 2 coadded observations and 28%
are the result of 3 or more. In addition, I produce a catalog which provides
sets of obsid which have I determined to correspond to a single object at the
listed right ascension and declination. This key allows for one to carry out
their own method of coadding repeat observations of an object.

We apply a similar methodology to the catalog of stellar abundances produced
by Xiang et al. (2019) using the LAMOST Data Release 5 LRS. I remove any
observations which are flagged as poor quality based on the chi-squared of the
spectral fit. Additionally, for each observation I use the individual element
quality flags to remove unreliable abundance estimates. As the [$\alpha$/Fe]
estimate is the weighted mean of [Mg/Fe], [Si/Fe], [Ca/Fe], and [Ti/Fe], I
remove this estimate if any of the elements are flagged as unreliable. I also
omit unreliable estimates of $T_{\mathrm{eff}}$, $\log g$, and microturbulent
velocity $V_{t}$. I combine the reliable parameter estimates to produce a
weighted mean and associated error as above. The resulting catalog contains
5,792,830 objects, 20% of which are the result of coadding multiple
observations.

This material is based upon work supported by the National Science Foundation
under grant number 2009415.

## References

  * Cui et al. (2012) Cui, X.-Q., Zhao, Y.-H., Chu, Y.-Q., et al. 2012, Research in Astronomy and Astrophysics, 12, 1197, doi: 10.1088/1674-4527/12/9/003
  * Xiang et al. (2019) Xiang, M., Ting, Y.-S., Rix, H.-W., et al. 2019, The Astrophysical Journal Supplement Series, 245, 34, doi: 10.3847/1538-4365/ab5364
  * Zhao et al. (2012) Zhao, G., Zhao, Y.-H., Chu, Y.-Q., Jing, Y.-P., & Deng, L.-C. 2012, Research in Astronomy and Astrophysics, 12, 723, doi: 10.1088/1674-4527/12/7/002


# Bulk superconductivity in FeTe1-xSex via physicochemical pumping of excess
iron

Lianyang Dong  Materials Department, University of California, Santa Barbara,
California 93106, USA  He Zhao  Department of Physics, Boston College,
Chestnut Hill, Massachusetts 02467, USA  Ilija Zeljkovic  Department of
Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA  Stephen D.
Wilson<EMAIL_ADDRESS>Materials Department, University of
California, Santa Barbara, California 93106, USA  John W. Harter
<EMAIL_ADDRESS>Materials Department, University of California, Santa Barbara,
California 93106, USA

###### Abstract

The iron-based superconductor FeTe1-xSex has attracted considerable attention
as a candidate topological superconductor owing to a unique combination of
topological surface states and bulk high-temperature superconductivity. The
superconducting properties of as-grown single crystals, however, are highly
variable and synthesis dependent due to excess interstitial iron impurities
incorporated during growth. Here we report a novel physicochemical process for
pumping this interstitial iron out of the FeTe1-xSex matrix and achieving bulk
superconductivity. Our method should have significant value for the synthesis
of high-quality single crystals of FeTe1-xSex with large superconducting
volume fractions.

##  I Introduction

The iron-based superconductor FeTe1-xSex with chemical composition $x\approx
0.5$ possesses topological spin-helical Dirac surface states proximity-coupled
to bulk $s$-wave superconductivity [1, 2, 3], realizing the celebrated Fu-Kane
mechanism of topological superconductivity [4]. Such a system hosts Majorana
zero modes that obey non-Abelian exchange statistics, a crucial ingredient in
topological quantum computing [5, 6]. The relatively high superconducting
critical temperature ($T_{c}=14.5$ K) and facile growth of large single
crystals make FeTe1-xSex a promising material platform to both generate and
study Majorana fermions [7, 8, 9]. In as-grown crystals, however, excess iron
may appear at interstitial sites in Fe1+yTe1-xSex. Depending on growth
conditions and the starting Te:Se ratio, the excess iron concentration $y$ can
reach values exceeding 10%. While a starting Te:Se ratio of 1:1 has been shown
to yield bulk superconducting crystals with $y\approx 0$, moving away from
this ratio is known to rapidly degrade superconductivity via the promotion of
interstitial iron impurities [10, 11, 12, 13, 14, 15]. Furthermore, a gradated
iron composition may also develop within a single crystal boule, rendering the
superconductivity inhomogeneous across the crystal.

To remedy this problem, a number of single-step post-growth annealing
solutions have been proposed to reduce $y$. These methods have been performed
in both an oxidizing environment and in an overpressure of Te/Se vapor [16,
17, 18, 22, 19, 20, 21]. Nevertheless, their application remains limited due
to a strong dependence on crystal morphology as well as uncertainties
concerning the presence of superconductivity within the surface-shielded core
of bulk crystals. The mechanisms underlying these annealing methods and their
ability to tune iron content throughout the bulk of Fe1+yTe1-xSex crystals
remain open questions. Of particular intrigue is the use of oxygen containing
environments to anneal Fe1+yTe1-xSex crystals. The propensity of O2 to oxidize
iron provides a strong chemical driving force for removing interstitial iron
from the FeTe1-xSex matrix, but the extent to which this process is diffusion-
limited prior to the process decomposing the FeTe1-xSex lattice itself is
unclear. In particular, the limitations on how much interstitial iron can be
cleanly removed from the matrix and over what length scale (i.e. crystal
thickness) have yet to be determined.

In this paper, we explore the use of oxygen as an annealing agent to remove
excess iron from Fe1+yTe1-xSex crystals. For crystals greater than $\sim 30$
$\mu$m in thickness, we find that a single annealing step is insufficient to
produce bulk superconductivity, instead inducing superconductivity only on the
outer shell of the sample exposed to the oxidizing environment. To mitigate
this limitation, we develop a cyclical post-annealing method to gradually
improve the superconducting volume fraction by physicochemically pumping
excess iron out of the material, eventually achieving fully bulk
superconductivity. Specifically, by using repeated cycles of oxygen annealing,
acid etching, and vacuum annealing, we demonstrate that $y\approx 0.08$ can be
reasonably removed from the bulk of an Fe1+yTe1-xSex single crystal with
thickness approaching several hundred microns. We envision this technique will
be useful for the tailored experimental exploration of the candidate
topological superconducting state in this material system.

Figure 1:  XPS measurements of an air-annealed sample of Fe1+yTe1-xSex. (a)
Survey spectrum showing the presence of iron, oxygen, and carbon on the
surface. Analysis of O 1$s$ and Fe 2$p$ peaks gives an initial Fe:O atomic
ratio of 1:2.05. (b) High-resolution scan of the O 1$s$ peak, from which it is
determined that only 73.5% of oxygen is bonded to iron, with the remainder
coming from various carbon compounds on the surface. This yields a corrected
Fe:O ratio of 1:1.51 (Fe2O3).

##  II Experimental Details

Single crystals with nominal composition FeTe0.55Se0.45 were grown using a
self-flux method similar to previous reports [23, 22]. Starting materials Fe
(99.99%), Te (99.999%), and Se (99.999%) were well mixed in the molar ratio
1:0.55:0.45 and double-sealed in quartz ampoules evacuated to high vacuum
($\sim 10^{-5}$ mbar). The sealed ampoules were placed in a furnace and heated
slowly to 1100 ∘C, held for 20 hours, cooled to 650 ∘C at a rate of 2 ∘C per
hour, and then cooled to 400 ∘C at a rate of 5 ∘C per hour. The furnace was
then powered off and cooled down to room temperature. Single crystals were
removed from the solidified flux by breaking the quartz ampoules in air.

X-ray fluorescence (XRF) measurements were performed using a Rigaku ZSX Primus
IV (WDXRF). A 1 mm probe diameter was chosen for analysis of thin plate single
crystals with sizes up to 6 mm. X-ray photoelectron spectroscopy (XPS)
measurements were conducted using a Thermo Fisher Escalab Xi+ instrument. A
monochromatic Al K$\alpha$ x-ray source was used for all samples. Pass
energies were 100 eV for survey scans and 20 eV for high-resolution scans.
Samples were measured without cleaving in vacuum in order to study the thin
blue oxide layer on the sample surface, resulting in the detection of some
surface carbon compounds which will be discussed later. Magnetization
measurements were performed using a Quantum Design MPMS3 SQUID magnetometer.
Crystals were mounted on a quartz paddle with GE varnish and magnetization
data was collected using a 10 Oe field. Powder samples (crushed crystals) were
placed inside of a plastic capsule mounted onto a brass half-tube sample
holder.

Scanning tunneling microscopy (STM) measurements were performed on single
crystals cleaved under ultra-high vacuum at 80 K and immediately inserted into
the STM head in order to expose a pristine surface free of contaminants. STM
data was acquired at a base temperature of 4.5 K using a commercial Unisoku
USM1300 system. Spectroscopic measurements were taken using standard lock-in
techniques at 915 Hz. STM tips were home-made chemically etched tungsten. Tip
sharpness was first evaluated under an optical microscope before the tip was
annealed in vacuum to a bright orange color and inserted into the STM.

##  III XRF and XPS Analysis

As-grown crystals of Fe1+yTe1-xSex were initially analyzed by XRF. Elemental
analysis of multiple samples consistently showed a tellurium-rich and iron-
rich composition, with an average Fe:Te:Se ratio of 1.08:0.59:0.41. Selenium
is more volatile than tellurium, and it is likely that a small fraction
evaporated from the melt and deposited onto the walls of the quartz ampoule
during growth. Notably, the Te:Se ratio found in these crystals rests on the
boundary where excess iron begins to be incorporated during growth and bulk
superconductivity vanishes [10]. This is the regime where post-growth
treatment of crystals becomes necessary. This as-grown Fe1.08Te0.59Se0.41
composition demonstrates the need for our approach to inducing bulk
superconductivity.

To explore the effects of annealing in an oxidizing environment, a small
crystal was cleaved from a larger single crystal boule of Fe1.08Te0.59Se0.41.
Crystals cleave with shiny metallic surfaces, but the color of the crystal
surface is known to turn blue after post-annealing in air or pure O2 [13, 15].
This suggests that the excess iron removed from the bulk crystal during the
annealing step remains on the crystal surface as an oxide. To verify this, we
carried out XPS measurements on the blue surface layer of an air-annealed
sample (300 ∘C for 4 hours). As shown in Fig. 1(a), the elements iron, oxygen,
and carbon were detected, indicating that the blue surface consists of iron
oxide and some carbon compounds. The overall Fe:O ratio was measured to be
1:2.05, but the spectral shape of the O $1s$ peak shown in Fig. 1(b) indicates
that approximately 26.5% of the oxygen was bonded to carbon, with the
remainder bonded to iron, giving a corrected Fe:O ratio of approximately
1:1.51. This analysis shows that the surface layer is mainly Fe2O3, with the
blue hue coming from thin film optical interference effects between the
crystal surface and the outer oxide layer. The fact that the color is always
blue suggests that the oxide layer that forms under moderate oxidizing
conditions is self-limiting. As we will discuss later, the Fe2O3 surface layer
that develops can be removed by briefly etching in a solution of hydrochloric
acid without damaging the underlying Fe1+yTe0.59Se0.41 crystal.

##  IV Single Post-Annealing Step

As shown in Fig. 2(a), the magnetic susceptibility of an as-grown
Fe1.08Te0.59Se0.41 crystal of thickness 75 $\mu$m does not exhibit a
superconducting diamagnetic transition because the excess iron completely
suppresses superconductivity. After annealing in air at 300 ∘C for 4 hours, a
clear superconducting transition is observed at $T_{c}\approx 14$ K with an
apparent large superconducting volume fraction. This large shielding fraction
is due to a shell effect as shallow excess iron is pulled from the crystal and
oxidized only at the surface. If this same crystal is then sealed under vacuum
and annealed at 400 ∘C for 24 hours, the apparent superconducting volume
fraction decreases significantly. This is due to the vacuum anneal causing the
remaining excess iron in the bulk of the crystal to diffuse to the surface. To
confirm this hypothesis, an air-annealed sample with a clear superconducting
transition was pulverized into a powder, after which the superconducting
volume fraction dropped to almost zero, as shown in Fig. 2(b). Both
observations strongly suggest that the superconductivity that appears after
annealing in air occurs at the surface only and that the apparent
superconducting volume fraction determined by the Meissner effect is not
indicative of the bulk.

Figure 2:  Magnetic susceptibility measured during a single post-annealing
cycle for samples with thickness $\sim 75$ $\mu$m. (a) As-grown samples do not
exhibit signs of superconductivity, but after annealing in air, a large
diamagnetic signal is measured. The diamagnetism is suppressed by subsequent
vacuum annealing. (b) The apparent superconducting volume fraction of an air-
annealed sample is significantly reduced after pulverizing.

##  V Physicochemical Pumping

Figure 3:  Evolution of magnetic susceptibility after multiple annealing
cycles for samples with thickness $\sim 75$ $\mu$m. (a) Annealing in air
consistently yields apparent superconducting volume fractions above 90%. (b) A
steady cycle-to-cycle improvement in volume fraction is observed after the
vacuum annealing steps. (c) After the final vacuum annealing step,
pulverization of a single crystal no longer reduces the apparent
superconducting volume fraction, indicating bulk superconductivity has been
achieved.

Here we show that to obtain fully bulk superconductivity, multiple annealing
cycles are required. Each cycle consists of three steps: (1) annealing in air
to oxidize shallow excess iron impurities, (2) etching in hydrochloric acid to
remove the resulting iron oxide, and (3) annealing in vacuum to redistribute
or “pump” iron impurities from the bulk to the surface via diffusion. Carrying
out several cycles of this procedure on the same sample, we observed that an
apparent superconducting volume fraction above 90% is consistently observed
after each anneal in air, as shown in Fig. 3(a). On the other hand, after
subsequent annealing in vacuum the superconducting volume fraction steadily
increased from cycle to cycle to nearly 100%, as shown in Fig. 3(b). This
behavior is expected if a small amount of excess iron is removed from the bulk
after the completion of each cycle, thereby gradually improving the
superconducting properties until a sharp transition and bulk superconductivity
are reached. To verify this end point, after performing several annealing
cycles, the magnetic susceptibilities of a sample measured before and after
pulverization were compared, as shown in Fig. 3(c). The similar near-100%
superconducting volume fractions of the powder and single crystal confirm that
bulk superconductivity is achieved after several annealing cycles. Further
verification is obtained through bulk-sensitive heat capacity measurements,
where a large anomaly at $T_{c}$ is only observed after several cycles, as
shown in Fig. 4.

Figure 4:  Heat capacity versus temperature for an as-grown (non-
superconducting) crystal, a crystal that has undergone a single post-annealing
step and exhibits a large diamagnetic transition, and a fully-cycled crystal.
Annealing once results in a small change in heat capacity at the critical
temperature, likely resulting from surface superconductivity, but only fully-
cycled crystals show a large heat capacity anomaly indicative of bulk
superconductivity.

In Fig. 5 we plot the approximate number of cycles required to reach bulk
superconductivity (no change in volume fraction after vacuum annealing) versus
crystal thickness. With increasing thickness, the number of required cycles
increases, following a linear trend. This functional relationship supports the
hypothesis that a fixed amount of excess iron per unit area is pumped from the
sample during each cycle, as would be expected for self-limiting surface
oxidation. In particular, a thicker crystal requires the removal of a
proportionally larger amount of iron before bulk superconductivity is
achieved, and therefore requires a proportionally larger number of post-
annealing cycles. This means that the cyclical post-annealing technique is
rate-limited by the oxidation step. The proportionality constant extracted
from the data suggests $\sim 54$ post-annealing cycles are required for each
millimeter of crystal thickness. XRF measurements of fully-cycled crystals
exhibiting bulk superconductivity showed that the iron content had decreased
to $(1+y)\approx 0.98-0.99$ following the final annealing step. This
demonstrates that interstitial iron levels as high as $y=0.08$ can be
successfully removed from several hundred $\mu$m thick crystals in a
manageable number ($<10$) of cycles. For example, Table 1 shows a steady
decrease in iron content for a sample that underwent four annealing cycles.

Step | Fe ($1+y$)
---|---
As-grown | 1.063
Cycle 1 | 1.040
Cycle 2 | 1.030
Cycle 3 | 1.013
Cycle 4 | 0.983
Table 1:  Iron content measured by XRF for a sample undergoing several
physicochemical pumping cycles. Each value is the average of four randomly-
chosen spots on the sample surface. Figure 5:  Number of post-annealing cycles
required to reach bulk superconductivity versus crystal thickness. An
approximately linear dependence is observed with a proportionality constant of
$\sim 54$ cycles per millimeter of thickness.

To further explore the local homogeneity and bulk superconductivity of a
fully-cycled crystal, we characterized a bulk-superconducting sample using
STM. An STM topograph is shown in Fig. 6(a), depicting a chalcogen-atom-
terminated cleaved surface with randomly distributed bright and dark spots on
a square lattice, corresponding to tellurium and selenium atoms, respectively.
Counting the spots gives a Te:Se ratio of 0.61:0.39, consistent with the XRF
results. The topograph also shows several brighter spots scattered on the
surface, corresponding to interstitial iron impurities centered at the
midpoint of four neighboring Te/Se atoms. This excess iron level is very close
to the nominal $y=0$ value, with only 5 interstitial iron sites identified
among more than 2000 unit cells. Shown in Fig. 6(b) are $dI/dV$ tunneling
spectra. All spectra exhibit a large superconducting energy gap of $\sim 2.5$
meV, demonstrating that the superconducting state is uniform across the
sample. We note here that the STM data do not resolve the smaller gap (1.5 – 2
meV) or zero bias conductance peak typically seen in other STM data [24].
These observations could be due to the different Te:Se ratio, where at higher
ratios the sample may no longer be topological.

Figure 6:  STM data from a cleaved surface of FeTe0.59Se0.41 measured at 4.5
K. (a) Representative topographic image showing tellurium atoms (bright spots)
and selenium atoms (dark spots) randomly distributed on a square lattice.
Counting the spots yields a Te:Se ratio of 0.61:0.39. Also apparent are
brighter spots representing interstitial iron impurities. (b) $dI/dV$ spectra
taken along a cut extending away from the interstitial iron impurity marked by
a red cross in panel (a), showing a superconducting energy gap of $\sim 2.5$
meV. STM setup conditions: $I_{\mathrm{set}}=100$ pA;
$V_{\mathrm{sample}}=-10$ mV; $V_{\mathrm{exc}}=0.2$ mV (zero-to-peak). Figure
7:  Illustration of the cyclical physicochemical pumping procedure. Orange
dots represent excess iron impurities. Due to the self-limiting nature of the
oxidation step, after each cycle a fixed amount of excess iron is removed from
the crystal. The process may be considered complete when the vacuum annealing
step no longer changes the volume fraction determined from magnetic
susceptibility measurements.

##  VI Discussion

A number of differing approaches have been proposed for tuning the excess iron
content within Fe1+yTe1-xSex, ranging from vacuum annealing [13], to oxygen
annealing [18, 22], to annealing in tellurium vapor [20]. Depending on the
specific processing technique used, these approaches may suffer from
diffusion-limited removal of iron deep inside bulk crystals, sample
degradation under heat treatment, or an inhomogeneous distribution of iron
across the sample. The iterative gradual removal of interstitial iron
presented here mitigates these challenges. Our cyclical post-annealing process
involves iron being physicochemically pumped from the bulk to the surface and
then removed via a three-step process of oxidation of shallow iron to Fe2O3,
acid etching to physically remove the surface oxide, and thermally-activated
diffusion to redistribute the remaining iron in the crystal. The process is
illustrated in Fig. 7. The real superconducting volume fraction is gradually
improved after each annealing cycle, and bulk superconductivity can typically
be achieved after several cycles.

While we have chosen reasonably gentle oxidizing conditions for the first step
of the cycle (300 ∘C in air), this step may be further optimized to reduce the
required total number of cycles for a given sample thickness. Earlier single-
step annealing studies of thin Fe1+yTe1-xSex crystals suggest this to be the
case [16, 15]. Intriguingly, tailoring the multistep process to realize finite
$y$ values within bulk single crystals may also allow for new avenues to study
both the competing magnetic states (stabilized via interstitial iron [24, 25])
and superconductivity as well as the impact of finite $y$ on topological
superconductivity. Exploring this process for tailored $y$ values and with
varying Te:Se ratios are appealing next steps.

##  VII Conclusion

FeTe1-xSex is a promising material platform for studying topological
superconductivity and Majorana fermions. In as-grown crystals, however,
superconductivity is usually suppressed by excess iron impurities. While these
impurities may be removed by annealing in an oxidizing environment, we
demonstrate that for crystals with thicknesses greater than $\sim 30$ $\mu$m,
a single annealing step is insufficient to remove iron from the bulk. To
address this, we present a cyclic multistep annealing/etching process for the
production of bulk superconducting crystals with $y\approx 0$. Our method
should have significant value for the synthesis of high-quality single
crystals of FeTe1-xSex with large superconducting volume fractions at Te:Se
ratios away from the commonly studied 1:1 values. We speculate that the
general physicochemical pumping procedure developed here may also be
applicable to the growth of other material systems where as-grown impurities
must be removed in order to improve physical properties.

## Acknowledgments

This work was supported by the Materials Research Science and Engineering
Centers (MRSEC) program of the National Science Foundation through Grant No.
DMR-1720256 (Seed Program). I.Z. acknowledges support from the Army Research
Office through Grant No. W911NF-17-1-0399.

## References

  * [1] Z. Wang, P. Zhang, G. Xu, L. K. Zeng, H. Miao, X. Xu, T. Qian, H. Weng, P. Richard, A. V. Fedorov, H. Ding, X. Dai, and Z. Fang. Phys. Rev. B 92, 115119 (2015).
  * [2] X. Wu, S. Qin, Y. Liang, H. Fan, and J. Hu. Phys. Rev. B 93, 115129 (2016).
  * [3] G. Xu, B. Lian, P. Tang, X.-L. Qi, and S.-C. Zhang. Phys. Rev. Lett. 117, 047001 (2016).
  * [4] L. Fu and C. L. Kane. Phys. Rev. Lett. 100, 096407 (2008).
  * [5] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma. Rev. Mod. Phys. 80, 1083-1159 (2008).
  * [6] J. Alicea. Rep. Prog. Phys. 75, 076501 (2012).
  * [7] P. Zhang, K. Yaji, T. Hashimoto, Y. Ota, T. Kondo, K. Okazaki, Z. Wang, J. Wen, G. D. Gu, H. Ding, and S. Shin. Science 360, 182-186 (2018).
  * [8] D. Wang, L. Kong, P. Fan, H. Chen, S. Zhu, W. Liu, L. Cao, Y. Sun, S. Du, J. Schneeloch, R. Zhong, G. Gu, L. Fu, H. Ding, and H.-J. Gao. Science 362, 333-335 (2018).
  * [9] T. Machida, Y. Sun, S. Pyon, S. Takeda, Y. Kohsaka, T. Hanaguri, T. Sasagawa, and T. Tamegai. Nat. Mater. 18, 811-815 (2019).
  * [10] B. C. Sales, A. S. Sefat, M. A. McGuire, R. Y. Jin, D. Mandrus, and Y. Mozharivskyj. Phys. Rev. B 79, 094521 (2009).
  * [11] K. W. Yeh, C. T. Ke, T. W. Huang, T. K. Chen, Y. L. Huang, P. M. Wu, and M. K. Wu. Cryst. Growth Des. 9, 4847-4851 (2009).
  * [12] M. Bendele, P. Babkevich, S. Katrych, S. N. Gvasaliya, E. Pomjakushina, K. Conder, B. Roessli, A. T. Boothroyd, R. Khasanov, and H. Keller. Phys. Rev. B 82, 212504 (2010).
  * [13] C. Dong, H. Wang, Z. Li, J. Chen, H. Q. Yuan, and M. Fang. Phys. Rev. B 84, 224506 (2011).
  * [14] S. Cao, S. Shen, L. Chen, S. Yuan, B. Kang, and J. Zhang. J. Appl. Phys. 110, 033914 (2011).
  * [15] Y. Sun, Y. Tsuchiya, T. Taen, T. Yamada, S. Pyon, A. Sugimoto, T. Ekino, Z. Shi, and T. Tamegai. Sci. Rep. 4, 4585 (2014).
  * [16] Y. Mizuguchi, K. Deguchi, S. Tsuda, T. Yamaguchi, and Y. Takano. EPL 90, 57002 (2010).
  * [17] Y. Kawasaki, K. Deguchi, S. Demura, T. Watanabe, H. Okazaki, T. Ozaki, T. Yamaguchi, H. Takeya, and Y. Takano. Solid State Commun. 152, 1135-1138 (2012).
  * [18] Y. Sun, T. Taen, Y. Tshuchiya, Q. Ding, S. Pyon, Z. Shi, and T. Tamegai. Appl. Phys. Express 6, 043101 (2013).
  * [19] W. Zhou, Y. Sun, S. Zhang, J. Zhuang, F. Yuan, X. Li, Z. Shi, T. Yamada, Y. Tsuchiya, and T. Tamegai. J. Phys. Soc. Jpn. 83, 064704 (2014).
  * [20] Z. Xu, J. A. Schneeloch, M. Yi, Y. Zhao, M. Matsuda, D. M. Pajerowski, S. Chi, R. J. Birgeneau, G. Gu, J. M. Tranquada, and G. Xu. Phys. Rev. B 97, 214511 (2018).
  * [21] Y. Sun, Z. Shi, and T. Tamegai. Supercond. Sci. Technol. 32, 103001 (2019).
  * [22] Y. Sun, T. Taen, Y. Tsuchiya, Z. X. Shi, and T. Tamegai. Supercond. Sci. Technol. 26, 015015 (2013).
  * [23] H. Okazaki, T. Watanabe, T. Yamaguchi, Y. Kawasaki, K. Deguchi, S. Demura, T. Ozaki, S. J. Denholme, Y. Mizuguchi, H. Takeya, and Y. Takano. J. Phys. Soc. Jpn. 81, 113707 (2012).
  * [24] J.-X. Yin, Z. Wu, J.-H. Wang, Z.-Y. Ye, J. Gong, X.-Y. Hou, L. Shan, A. Li, X.-J. Liang, X.-X. Wu, J. Li, C.-S. Ting, Z.-Q. Wang, J.-P. Hu, P.-H. Hor, H. Ding, and S. H. Pan. Nat. Phys. 11, 543-546 (2015).
  * [25] R. Viennois, E. Giannini, D. van der Marel, and R. Cerny. J. Solid State Chem. 183, 769-775 (2010).


2

# Decision Transformer as a Foundation Model for
Partially Observable Continuous Control

Xiangyuan Zhang1∗ Weichao Mao1∗ Haoran Qiu2 Tamer Başar1 ∗Equal contributions.
1Department of ECE and CSL, University of Illinois Urbana-Champaign (UIUC).
2Department of CS and CSL, UIUC. Emails: {xz7, weichao2, haoranq4,
<EMAIL_ADDRESS>
###### Abstract

Closed-loop control of nonlinear dynamical systems with partial-state
observability demands expert knowledge of a diverse, less standardized set of
theoretical tools. Moreover, it requires a delicate integration of controller
and estimator designs to achieve the desired system behavior. To establish a
general controller synthesis framework, we explore the Decision Transformer
(DT) architecture. Specifically, we first frame the control task as predicting
the current optimal action based on past observations, actions, and rewards,
eliminating the need for a separate estimator design. Then, we leverage the
pre-trained language models, i.e., the Generative Pre-trained Transformer
(GPT) series, to initialize DT and subsequently train it for control tasks
using low-rank adaptation (LoRA). Our comprehensive experiments across five
distinct control tasks, ranging from maneuvering aerospace systems to
controlling partial differential equations (PDEs), demonstrate DT’s capability
to capture the parameter-agnostic structures intrinsic to control tasks. DT
exhibits remarkable zero-shot generalization abilities for completely new
tasks and rapidly surpasses expert performance levels with a minimal amount of
demonstration data. These findings highlight the potential of DT as a
foundational controller for general control applications.

##  I Introduction

Effective closed-loop control of nonlinear systems with partial-state
observability demands a diverse spectrum of theoretical tools from adaptive
and robust control to state estimation [1, 2, 3, 4]. In addition, expert-level
domain knowledge is required to carefully integrate different components of
the control system to achieve the desired behavior in each application. To
establish an end-to-end controller synthesis framework that is agnostic to
system dynamics, control objectives, and policy constraints, Reinforcement
Learning (RL) has taken the initial step [5, 6, 7, 8]. However, as RL methods
focus on optimizing a single task, RL controllers can hardly generalize to new
tasks with different system parameters, an updated objective, or a set of new
constraints. Upon each task change, RL requires millions of new samples for
re-training, hindering its applications to systems with rapidly changing
dynamics or industries that require an adaptive controller, e.g., aerospace
[9]. Recently, meta-RL has been proposed for solving these multi-task
scenarios, where it learns a policy for handling multiple tasks
simultaneously, and adapts to unseen tasks using minimal iterations or
demonstrations [10, 11, 12]. Applying meta-RL to adaptive control and
investigating its theoretical properties have been a trending research topic
[13, 14, 15, 16].

Inspired by the success of Large Language Models (LLMs) [17, 18, 19, 20], we
explore the Transformer architecture [21] for learning controller designs,
building on the Decision Transformer (DT) literature [22, 23]. First, we
formulate the control problem as predicting the current optimal action from a
sequence of past observations, actions, and rewards. This approach leverages
Transformer’s exceptional representational capabilities and the inherent
structure of control tasks, effectively compressing the history into an
“approximate information state” [24] and eliminating the explicit state
estimation step [25, 1, 26, 27].

We initialize DT using a pre-trained GPT-2 model with $117$ millions of
parameters for language tasks [17], and train DT on a self-generated offline
control dataset via Low-Rank Adaptation (LoRA) [28]. Then, we evaluate DT’s
effectiveness through comprehensive experiments on five distinct tasks, from
maneuvering aerospace systems to controlling partial differential equations
(PDEs). In Section III, we first study the single-task control setting as a
proof of concept, showing that DT can consistently match or surpass the
unknown behavior policies and is also competitive with state-of-the-art
offline RL methods [29]. We detail a few single-task ablation studies in
Section III-B. Subsequently, Section IV investigates the multi-task setting
with significantly perturbed system dynamics, either through altering physical
parameters or injecting random noises into the system matrices. We show that
DT can zero-shot generalize to new tasks that are completely out of the
training distribution. Moreover, after seeing a minimal amount of
demonstration data (e.g., $10$ rollout trajectories), DT can quickly match or
outperform expert behavior policy in the new task. Our experimental findings
confirm DT’s ability to capture parameter-agnostic structures of control tasks
and excel in few-shot learning, suggesting its potential for general control
applications as a foundational controller. In Section V, we discuss various
design choices and list several future research directions.

###  I-A Related Literature

RL methods have surpassed human performance in many decision-making tasks with
full-state observability [30, 6, 31]. However, real-world control tasks often
involve noisy, lower-dimensional, and delayed observations, requiring policies
to infer or reconstruct missing state information [25]. Control tasks also
require policies to stabilize the closed-loop system over an extended period,
in addition to maneuvering the intense dynamics of aerospace systems [9] or
turbulent flows [32] to avoid catastrophic failures. Lastly, the scalability
issue arises in high-dimensional control problems. Motivated by these unique
challenges, controlgym was introduced in [33] to benchmark RL-based
controllers. Another line of research also studies the theoretical foundation
of RL for learning control policies [7, 34, 35, 36, 37, 8].

Motivated by LLMs’ success, the Transformer architecture has been widely
applied and evaluated in fully observable decision-making and control tasks
[22, 23, 38, 39, 40, 41, 42, 43, 44], particularly on D4RL [45], Meta-World
[46], and DM_Control [47] benchmarks. Specifically, [22] introduced DT for
modeling RL as a sequence prediction problem. [38] applied DT to learn a suite
of Atari games simultaneously and achieved human-level performances. [39, 42]
generalized DT to new tasks by prompting demonstration trajectories and adding
task-specific adaptation modules, respectively. [41] improved learning from
sub-optimal demonstration trajectories by incorporating Q-values into DT’s
return-to-go input modality. [40] introduced an online fine-tuning step to
improve learning from sub-optimal demonstrators. [43] refined the DT
architecture and proposed co-training of language and decision-making tasks.
[44] investigated generalizations to new tasks without forgetting the
previously learned skills. Besides the DT literature, [48] leveraged a
Transformer architecture to predict Q-values in a Deep-Q-Network algorithm for
partially observable RL problems. In contrast, our work builds upon the DT
literature, especially the most recent developments in [43], to directly
predict actions from history in partially observable control.

In parallel to our investigations, a significant body of literature have
leveraged Transformer models to auto-regressively generate control inputs in
robotic systems and embodied agents [49, 50, 51, 52, 53, 54, 55, 56, 57, 58].
For comprehensive reviews on this topic, we refer the readers to [59, 60].
These studies have focused on tokenizing multimodal inputs, including vision,
language, sensor data, and robotic actions, and utilizing the Transformer
backbone to train visual-language-sensory-motor policies. The overarching goal
is to develop robotic foundation models that integrate traditional perception,
planning, and control stacks into a unified framework. In contrast, our work
provides a control-centric perspective, aiming to thoroughly explore both the
potential and the challenges of applying Transformers solely to control
problems.

##  II Preliminaries

Figure 1: Illustration of the DT Architecture. Left: An offline control
dataset sampled from some unknown behavior policies. Middle: DT predicts an
optimal action $a_{t}$ autoregressively based on an input sequence of ‘reward-
to-go’s, observations and actions. The pre-trained language weights of DT are
kept frozen while we employ LoRA for control training. Right: DT quickly
generalizes to new control tasks after seeing minimal offline demonstrations.

We consider the discrete-time nonlinear system dynamics

$\displaystyle s_{t+1}=f(s_{t},a_{t};w_{t}),\quad o_{t}=g(s_{t};v_{t}),$
(II.1)

where $f:\mathbb{R}^{n_{s}}\times\mathbb{R}^{n_{a}}\to\mathbb{R}^{n_{s}}$ and
$g:\mathbb{R}^{n_{s}}\to\mathbb{R}^{n_{o}}$ are time-invariant mappings that
define the state transition and observation process, respectively. We use
$s_{t}$ to denote the state, $a_{t}$ to denote the action/control input,
$o_{t}$ to denote the observation, and $w_{t},v_{t}$ to denote noises and
disturbances. We define the one-step reward to be the negative quadratic
tracking cost

$\displaystyle
r_{t}(s_{t},a_{t})=-(s_{t}-s_{r})^{\top}Q_{1}(s_{t}-s_{r})-a_{t}^{\top}Q_{2}a_{t}-2(s_{t}-s_{r})^{\top}Q_{3}a_{t},$

where $Q_{1}\geq 0$ and $Q_{2}>0$ are symmetric weighting matrices, $Q_{3}$ is
another weighting matrix in $\mathbb{R}^{n_{s}\times n_{a}}$, and
$s_{r}\in\mathbb{R}^{n_{s}}$ is the target state. We additionally let
$r_{t}:=r_{t}(s_{t},a_{t})$. The objective is to compute a policy $\pi$ that
maps the available information up to and including step $t$, i.e.,
$\\{o_{0},a_{0},r_{0},\cdots,o_{t}\\}$, to the current action $a_{t}$, so as
to maximize the expected cumulative reward
$\mathbb{E}\sum_{t=0}^{T}\gamma^{t}r_{t}$. Here, $\gamma\in[0,1)$ is the
discount factor and $T$ is a possibly infinite problem horizon length111Our
setting can be equivalently viewed as a partially observable Markov decision
process (POMDP) [61], denoted by the tuple $\langle
S,A,T,R,\Omega,O,\gamma\rangle$. The $S$, $A$, and $\Omega$ in POMDP
correspond to the continuous spaces $\mathbb{R}^{n_{s}}$,
$\mathbb{R}^{n_{a}}$, and $\mathbb{R}^{n_{o}}$ in our setting. The reward
function $R$ corresponds to our $r_{t}$. The transition and observation models
$T$ and $O$ correspond to $f$ and $g$ in (II.1)..

###  II-A Learning a Single Control Task

In decision-making and control tasks, model-free RL can be generally
categorized into online and offline methods. Online RL directly interacts with
the environment, utilizing real-time feedback to refine the control policy.
Conversely, offline RL learns a control policy from a pre-collected dataset of
system trajectories that are sampled using some unknown behavior policies
[62]. In this work, we study the offline RL setting for control. Suppose we
have access to $\mathcal{D}$, a set of system trajectories generated by some
unknown behavior policies. Each trajectory has the form of

$\displaystyle\\{o_{0},a_{0},r_{0},\cdots,o_{T},a_{T},r_{T}\\},\quad
T\in\mathbb{Z}^{+}.$ (II.2)

The objective is to learn a control policy solely from the dataset
$\mathcal{D}$ and achieve a matching or higher cumulative reward than the
unknown behavior policy.

Compared with standard offline RL [62, 29], Transformer’s superior
representation ability enables compressing history into an “approximate
information state”, serving as a proxy for predicting the true system
evolution [24].

###  II-B Control of Multiple Systems and Meta-RL

In simultaneously controlling multiple systems or a system with varying
parameters, the controller must quickly adapt to unseen scenarios. Adaptive
control methods are widely used for this purpose [4], notably in aerospace
control systems [9]. Alternatively, $H_{\infty}$ control concerns a continuum
of systems (slightly) perturbed from the nominal dynamics, enabling a
controller based on the nominal dynamics to maintain performance in the
perturbed environment without modification, i.e., a zero-shot approach [2, 3].
However, not having a learning component, $H_{\infty}$ control design cannot
cope with substantial perturbations and may compromise performance excessively
for robustness.

Recently, meta-RL has been proposed to train on multiple tasks and then adapt
rapidly to unseen tasks with minimal iterations or demonstrations,
respectively, in online and offline contexts [10, 11, 12]. Its application to
continuous control tasks has been studied in [13, 14, 15, 16]. In the offline
setting, the goal of meta-RL is to learn a policy from $\mathcal{D}$ that can
act as an effective initial point for rapid adaptation to similar new tasks
drawn from the same distribution.

###  II-C Decision Transformer as a Foundation Model for Control

DT models control tasks as sequence prediction problems [22]. At time $t$, DT
predicts an optimal action $a_{t}$ autoregressively based on history
$\\{\hat{R}_{t-K+1},o_{t-K+1},a_{t-K+1},\cdots,\hat{R}_{t},o_{t}\\}$ of
context length $K$. Here, $\hat{R}_{t}:=\sum_{\tau=t}^{T}r_{\tau}$ represents
the reward-to-go. We utilize three multi-layer perceptrons (MLPs) for
embedding raw inputs (i.e., rewards-to-go, observations, and actions)
alongside an additional MLP for generating actions [43], as depicted in Figure
1.

We adopt the pre-trained GPT-2 language model with $117$ millions of
parameters [17] to initialize DT. Subsequently, we train DT on the control
dataset $\mathcal{D}$ using LoRA [28]. LoRA keeps the pre-trained language
weights frozen and adjusts DT by training rank decomposition matrices,
reducing the number of trainable parameters by orders of magnitudes.

After training, we can apply DT to new, unseen control tasks in a zero-shot
approach. Alternatively, we can further adapt DT using a few demonstration
trajectories of the new task. Hereafter, we refer to this approach as $k$-shot
adaptation. The required number of demonstrations, $k$, is typically much
lower than that of offline RL from scratch thanks to the pre-trained language
weights and control training.

In the inference phase, we fix the DT weights and use them to directly
interact with the control environment. Unlike the training stage, access to
the reward-to-go from a dataset is not available. Consequently, we substitute
DT’s input modality $\hat{R}_{t}$ with a prompt of the desired target return,
i.e., the discounted cumulative rewards [22]. This target return may be the
maximal achievable return for this task or a specific performance threshold.
Based on the desired target return and observations, DT predicts an action to
be executed by the actuators. Then, upon receiving the one-stage reward, we
decrease the desired target return by the same amount and use it as the next
prompt input. This process is repeated until the end of the episode.

##  III Learning a Single Control Task

As a proof of concept, we first train DT using LoRA to learn a single control
task from offline demonstrations. Our experiments include five controlgym
tasks [33]: the helicopter and aircraft examples (he1) and (ac4), a high-
dimensional cable-mass system (cm3), and PDE control problems (CDR) and
(Burgers). The detailed settings are listed in Appendix -A.

In each task, we generate the “expert dataset” using the converged Proximal
Policy Optimization (PPO) policy [63] while sampling the “medium dataset”
using an early-stopped PPO policy. We then normalize the cumulative rewards to
the range of $[0,1]$, where the expert behavior scores $1$ and the medium
behavior scores $0$. We refer to Section III-B for ablation studies in dataset
generation.

###  III-A Single-Task Experimental Results

TABLE I: Learning a Single Control Task from Offline Demonstrations

Dataset | Task | PPO | CQL | BC | DT | DT w/ co-training
---|---|---|---|---|---|---
| he1 | $1.000$ | $1.013\pm 0.074$ | $1.025\pm 0.060$ | $\mathbf{1.044\pm 0.040}$ | $1.042\pm 0.036$
| ac4 | $1.000$ | $1.209\pm 0.240$ | $0.692\pm 0.695$ | $\mathbf{1.410\pm 0.086}$ | $1.275\pm 0.087$
Expert | cm3 | $1.000$ | $1.146\pm 0.148$ | $0.895\pm 0.290$ | $\mathbf{1.243\pm 0.235}$ | $0.997\pm 0.224$
| CDR | $1.000$ | $\mathbf{1.017\pm 0.042}$ | $-0.016\pm 0.571$ | $0.990\pm 0.018$ | $0.997\pm 0.017$
| Burgers | $1.000$ | $1.171\pm 0.220$ | $1.070\pm 0.312$ | $1.210\pm 0.167$ | $\mathbf{1.319\pm 0.076}$
| he1 | $1.000$ | $0.339\pm 0.297$ | $-0.724\pm 0.348$ | $0.527\pm 0.143$ | $\mathbf{0.590\pm 0.124}$
| ac4 | $1.000$ | $0.127\pm 0.240$ | $-0.309\pm 0.677$ | $0.547\pm 0.210$ | $\mathbf{0.583\pm 0.224}$
Medium | cm3 | $1.000$ | $0.050\pm 0.334$ | $0.376\pm 0.483$ | $0.499\pm 0.189$ | $\mathbf{0.609\pm 0.203}$
| CDR | $1.000$ | $\mathbf{0.037\pm 0.078}$ | $-0.440\pm 0.364$ | $-0.049\pm 0.097$ | $-0.026\pm 0.079$
| Burgers | $1.000$ | $0.225\pm 0.178$ | $0.234\pm 0.171$ | $0.371\pm 0.086$ | $\mathbf{0.501\pm 0.151}$

Normalized rewards in five control tasks, where we compare CQL, BC, DT, and DT
with co-training.

We evaluate DT’s performance and compare DT against the unknown PPO behavior
policies, Conservative Q-Learning (CQL), and Behavior Cloning (BC). Training
on expert datasets, Table I indicates that DT consistently matches or
surpasses expert demonstrators whose scores are normalized as $1$.
Furthermore, DT matches or exceeds the performance of state-of-the-art
baselines across five experiments. Notably, in the aircraft example ac4, DT
significantly outperforms both the demonstrator and other baselines,
highlighting its capability in maneuvering the intense dynamics of aircraft
flight control.

When trained on medium datasets (whose behavior policy score is normalized to
$0$), the performance advantage of DT and its variant with language co-
training over demonstrators and other baselines becomes even more substantial.
Specifically, DT consistently achieves normalized scores between $0.5$ and
$0.6$ in four out of the five experiments (he1, ac4, cm3, and Burgers). DT
matches the unknown early-stopped PPO behavior policy in CDR. In contrast, the
scores for CQL and BC vary widely across different experiments, and BC could
fail to match medium demonstrators in the he1, ac4, and CDR environments.

###  III-B Ablation Studies

We conduct ablation studies on dataset generation and training DT, and list
here some interesting observations.

Non-Stationary Behavior Policy: The optimal policy for finite-horizon control
problems, such as our $50$-step linear ones, is non-stationary [2]. Therefore,
it is logical to use, e.g., the finite-horizon non-stationary linear-quadratic
Gaussian (LQG) policy, to sample the expert dataset for these linear control
tasks. However, DT randomly segments trajectories in training without encoding
their absolute time steps, which reveals (to the policy) only the relative
temporal information within each segment. This issue also arises in the fully
observable (state-feedback) setting. We conjecture that adapting the DT
architecture is needed for effectively learning from datasets generated by
dynamic LQG control policies with an intricate in-loop state estimator.

On the Context Length $K$: To avoid the above non-stationarity issue in the
$50$-step linear control tasks, one can apply the infinite-horizon LQG policy,
which is sub-optimal but very cheap to compute. In comparison, the PPO policy
we used in the expert dataset, which is much more expensive to obtain, is also
slightly sub-optimal. However, the infinite-horizon LQG policy is not a
desired demonstrator to DT since the actions it generates implicitly depend on
all previous observations through the evolution of a state estimator. Since
the Transformer policy is high-dimensional, it is capable of fitting
demonstration trajectories generated by any lower-dimensional stationary
behavior policies, if their actions depend only on history with a context
length of $K$ or less. For instance, PPO actions rely on immediate
observations (context length of $0$), whereas LQG actions could depend on a
much longer history sequence $T>>K$. This is not an issue in the fully
observable Markovian setting, because current-state-feedback stationary
policies such as infinite-horizon LQR are good demonstrators.

Language Co-Training: We study the effects of adding an auxiliary language
prediction loss to DT in our control tasks, which has been advocated in [43].
Table I reports mixed outcomes from this additional language co-training. It
generally fails to improve, and may even degrade, DT’s efficacy when training
on expert datasets, possibly because the auxiliary language prediction loss
distracts DT from its control objective. In contrast, on medium datasets, co-
training can substantially enhance DT’s control performances, as a by-product
of its original motivation of preserving the pre-trained language capabilities
(cf., Section 5.5 of [43]). Based on this observation and that we only train
DT on expert datasets in multi-task experiments, hereafter we will primarily
focus on analyzing DT without co-training.

##  IV Generalizing to Unseen Control Tasks

We examine DT’s ability to generalize across new control tasks. In this multi-
task framework, we construct the training tasks by randomly perturbing the
nominal system matrices in the single-task he1, ac4, and cm3 experiments, and
randomly sample physical parameters of the CDR and Burgers tasks. Then, we use
the converged PPO policy to generate the dataset. For the evaluation phase, we
differentiate between in-distribution and out-of-distribution tests. In-
distribution tests adhere to the training dataset’s task distribution, whereas
out-of-distribution tests involve either introducing larger perturbations or
selecting physical parameters that deviate significantly from the training
task distribution. We evaluate two versions of DT in both test scenarios:
zero-shot and $10$-shot, where “shot” refers to the number of demonstration
trajectories used for adaptation. Details on the configuration of these multi-
task experiments can be found in Appendix -A. We provide two additional
experiments in Subsection IV-B.

TABLE II: Few-Shot Generalizations to New Tasks

| Task | Prompt DT | DT (0-shot) | DT (10-shot)
---|---|---|---|---
| he1 | $-0.384\pm 0.077$ | $0.860\pm 0.107$ | $\mathbf{0.979\pm 0.134}$
| ac4 | $\mathbf{1.012\pm 0.063}$ | $0.679\pm 0.228$ | $0.651\pm 0.140$
In. | cm3 | $1.151\pm 0.176$ | $1.120\pm 0.067$ | $\mathbf{1.255\pm 0.082}$
| CDR | $\mathbf{1.144\pm 0.058}$ | $0.403\pm 0.161$ | $1.013\pm 0.004$
| Burgers | $1.154\pm 0.071$ | $0.663\pm 0.106$ | $\mathbf{1.203\pm 0.049}$
| he1 | $-0.010\pm 0.013$ | $0.728\pm 0.137$ | $\mathbf{1.017\pm 0.159}$
| ac4 | $-0.585\pm 0.028$ | $0.626\pm 0.146$ | $\mathbf{1.129\pm 0.162}$
Out. | cm3 | $-1.000\pm 0.000$ | $1.270\pm 0.338$ | $\mathbf{1.455\pm 0.347}$
| CDR | $-0.466\pm 0.050$ | $0.597\pm 0.239$ | $\mathbf{1.010\pm 0.006}$
| Burgers | $0.168\pm 0.129$ | $0.626\pm 0.298$ | $\mathbf{1.071\pm 0.209}$

Normalized rewards in new tasks, averaging over $9$ tests for in-distribution
(In.) and out-of-distribution (Out.) parameters. The normalization process is
identical to that of the single-task experiments: the range is $[0,1]$ with
the expert behavior policy scores $1$ and the medium behavior policy scores
$0$.

###  IV-A Multi-Task Experimental Results

We train DT on $30$ control tasks, and evaluate its generalization
capabilities to $9$ in-distribution tests and $9$ out-of-distribution tests;
see Appendix -A for details. The results of the multi-task experiments are
listed in Table II, where we compare the normalized scores of zero- and
$10$-shot DT with those of Prompt-DT (PDT) [39]. An immediate observation from
Table II is that the performance of PDT could deteriorate significantly in
out-of-distribution tests. In contrast, DT ($10$-shot) consistently yields a
strong performance in all tests. This observation demonstrates DT’s rapid
adaptation ability to surpass “expert” performance after seeing a minimal $10$
demonstrations. Moreover, DT zero-shot achieves reasonable performance across
different tasks, which indicates DT’s capability to capture the parameter-
agnostic structure of the underlying control tasks from the training set. Our
experimental results demonstrate the power of the Transformer architecture in
zero-shot generalization and few-shot adaptation to new tasks.

###  IV-B Additional Experiments

Cross-Environment Generalization: As a preliminary study, we generalize the DT
trained on the Burgers dataset to the CDR tasks, facilitated by the shared
dimensionalities of the two environments. As shown in Table III, DT zero-shot
is ineffective since it has not seen any CDR trajectories from training or
adaptation. However, with as few as 10 demonstration trajectories, DT quickly
adapts to CDR tasks, achieving cumulative rewards comparable to those of the
DT trained exclusively on CDR (cf., Table II). This preliminary study
indicates that DT might be capable of generalizing to different environments
in the few-shot setting.

TABLE III: Cross-Environment Generalization | Task | Burgers-DT (0-shot) | Burgers-DT (10-shot)
---|---|---|---
In. | CDR | $-1.000\pm 0.000$ | $1.011\pm 0.007$
Out. | CDR | $-0.715\pm 0.016$ | $1.006\pm 0.004$

Comparing to Model-Based $H_{2}/H_{\infty}$ Control: We compare DT’s
performance against that of the output-feedback $H_{2}/H_{\infty}$ controller
in three linear control tasks he1, ac4, and cm3. The $H_{2}/H_{\infty}$
controller is computed based on the unperturbed system model used in the
single task experiments, as detailed in Chapter 6 of [2]. We select the
robustness parameter of the $H_{2}/H_{\infty}$ controller from the set
$\\{0.5,1,2,3,4,5,10,20,50,100,200,500,1000\\}$ that yields the highest
average score in in-distribution tests. This optimized $H_{2}/H_{\infty}$
controller is then applied to out-of-distribution tests. We note that the
$H_{2}/H_{\infty}$ controller has access to the entire history, while DT is
limited to accessing the $K$-step history. To mitigate the impact of some
deficient cases, scores of the $H_{2}/H_{\infty}$ controller are capped within
the range of $[-1,2]$ before taking averages. Our comparison favors the
$H_{2}/H_{\infty}$ controller in every possible aspect.

TABLE IV: Comparison with Model-Based H2/H∞ Controller | Task | H2/H∞ (0-shot) | DT (0-shot) | DT (10-shot)
---|---|---|---|---
In. | he1 | $\mathbf{1.472\pm 0.358}$ | $0.860\pm 0.107$ | $0.979\pm 0.134$
Out. | he1 | $0.665\pm 1.219$ | $0.728\pm 0.137$ | $\mathbf{1.017\pm 0.159}$
In. | ac4 | $-0.552\pm 0.738$ | $\mathbf{0.679\pm 0.228}$ | $0.651\pm 0.140$
Out. | ac4 | $-0.297\pm 1.232$ | $0.626\pm 0.146$ | $\mathbf{1.129\pm 0.162}$
In. | cm3 | $\mathbf{1.310\pm 0.279}$ | $1.120\pm 0.067$ | $1.255\pm 0.082$
Out. | cm3 | $0.522\pm 1.359$ | $1.270\pm 0.338$ | $\mathbf{1.455\pm 0.347}$

We now discuss the implications of the results in Table IV. First, as
perturbation size grows, the performance of the $H_{2}/H_{\infty}$ controller
deteriorates, aligning with our expectation from the robust control theory.
Conversely, DT’s performance remains unaffected or becomes even better with
large perturbations in both zero-shot and $10$-shot scenarios. Therefore, we
hypothesize that further growing the perturbation size would destroy the
$H_{2}/H_{\infty}$ performance, whereas DT can continue to generalize.
Moreover, in out-of-distribution tests, applying DT zero-shot achieves higher
scores across all tasks than the $H_{2}/H_{\infty}$ controller. This
observation again verifies DT’s capability to capture the parameter-agnostic
structure underlying each task environment.

##  V Discussion and Conclusion

In contrast to D4RL [45], controlgym environments [33] do not have a publicly
available offline dataset. Therefore, we have generated our own dataset for
all the experiments. We normalized the cumulative rewards to $[0,1]$ with
respect to the early-stopped and the converged PPO behavior policies,
respectively, due to the varying reward scales across tasks. For the “expert”
behavior policy normalized to $1$, we chose the PPO policy due to the absence
of analytical solutions in some environments. Despite being termed “expert”,
the converged PPO policy’s performance is suboptimal as it is static output-
feedback, discarding the past history crucial in partially-observable
decision-making [2, 24]. With access to past history, as in DT, normalized
rewards greater than $1$ are achievable. For the “medium” behavior policy
normalized to $0$, we chose the early-stopped PPO policy over a random control
policy to avoid incurring near-negative infinity rewards in our control
settings, which is in stark contrast to standard RL tasks. Overall, lack of a
high-quality offline dataset hinders evaluating DT in other more challenging
controlgym environments and extending problem horizons. This is due to the
difficulties in obtaining “good” behavior policies. In the future, we aim to
sample a high-quality offline dataset with a diverse array of behavior
policies to enhance understanding of DT’s efficacy.

In summary, we have investigated DT as a candidate foundational controller for
general partially observable settings. By framing the control task as
predicting the current optimal action based on a finite-window truncation of
past observations, actions, and rewards, DT eliminates the need for a separate
estimator design. Utilizing pre-trained weights from language tasks, we
trained DT for control applications, demonstrating its ability to generalize
to unseen tasks with minimal demonstrations effectively. Our findings open
several directions for future research, including: 1) extending DT’s
generalization capabilities to different environments with varying state and
action space dimensions; 2) adapting DT for longer or infinite problem
horizons; 3) enhancing DT’s inference speed to facilitate high-frequency real-
time control; 4) refining DT’s architecture to learn efficiently from highly
non-stationary demonstration policies; and 5) embedding control-specific
insights into the DT architecture.

## Acknowledgment

XZ and TB were supported in part by the US Army Research Office (ARO) Grant
24-1-0085 and in part by the ARO MURI Grant AG285. WM, HQ, and TB were
supported in part by the National Science Foundation (NSF) Grant No. CCF
20-29049 and in part by the IBM-ILLINOIS Discovery Accelerator Institute
(IIDAI).

## References

  * [1] B. D. Anderson and J. B. Moore.  Optimal Filtering.  Prentice-Hall, 1979.
  * [2] T. Başar and P. Bernhard.  H-infinity Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach.  Birkhäuser, Boston., 1995.
  * [3] K. Zhou, J. C. Doyle, and K. Glover.  Robust and Optimal Control, volume 40.  Prentice Hall New Jersey, 1996.
  * [4] K. J. Åström and B. Wittenmark.  Adaptive Control.  Courier Corporation, 2008.
  * [5] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation.  arXiv preprint arXiv:1506.02438, 2015.
  * [6] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [7] B. Recht.  A tour of reinforcement learning: The view from continuous control.  Annual Review of Control, Robotics, and Autonomous Systems, 2:253–279, 2019.
  * [8] B. Hu, K. Zhang, N. Li, M. Mesbahi, M. Fazel, and T. Başar.  Toward a theoretical foundation of policy optimization for learning control policies.  Annual Review of Control, Robotics, and Autonomous Systems, 6:123–158, 2023.
  * [9] E. Lavretsky and K. Wise.  Robust and Adaptive Control: With Aerospace Applications.  Springer, 2012.
  * [10] C. Finn, P. Abbeel, and S. Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135, 2017.
  * [11] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine.  Online meta-learning.  In International Conference on Machine Learning, pages 1920–1930, 2019.
  * [12] E. Mitchell, R. Rafailov, X. B. Peng, S. Levine, and C. Finn.  Offline meta-reinforcement learning with advantage weighting.  In International Conference on Machine Learning, pages 7780–7791, 2021.
  * [13] G. Shi, K. Azizzadenesheli, M. O’Connell, S.-J. Chung, and Y. Yue.  Meta-adaptive nonlinear control: Theory and algorithms.  Advances in Neural Information Processing Systems, 34:10013–10025, 2021.
  * [14] S. M. Richards, N. Azizan, J.-J. Slotine, and M. Pavone.  Control-oriented meta-learning.  The International Journal of Robotics Research, 42(10):777–797, 2023.
  * [15] N. Musavi and G. E. Dullerud.  Convergence of gradient-based MAML in LQR.  In IEEE Conference on Decision and Control, pages 7362–7366. IEEE, 2023.
  * [16] L. F. Toso, D. Zhan, J. Anderson, and H. Wang.  Meta-learning linear quadratic regulators: A policy gradient maml approach for the model-free lqr.  arXiv preprint arXiv:2401.14534, 2024.
  * [17] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8):9, 2019.
  * [18] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  * [19] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  GPT-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [20] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [22] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  Advances in Neural Information Processing Systems, 34:15084–15097, 2021.
  * [23] M. Janner, Q. Li, and S. Levine.  Offline reinforcement learning as one big sequence modeling problem.  Advances in Neural Information Processing Systems, 34:1273–1286, 2021.
  * [24] J. Subramanian, A. Sinha, R. Seraj, and A. Mahajan.  Approximate information state for approximate planning and reinforcement learning in partially observed systems.  Journal of Machine Learning Research, 23(12):1–83, 2022.
  * [25] K. J. Astrom.  Introduction to Stochastic Control Theory.  Elsevier, 1971.
  * [26] X. Zhang, B. Hu, and T. Başar.  Learning the Kalman filter with fine-grained sample complexity.  In American Control Conference, pages 4549–4554, 2023.
  * [27] X. Zhang, S. Mowlavi, M. Benosman, and T. Başar.  Global convergence of receding-horizon policy search in learning estimator designs.  arXiv preprint arXiv:2309.04831, 2023.
  * [28] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [29] A. Kumar, A. Zhou, G. Tucker, and S. Levine.  Conservative Q-learning for offline reinforcement learning.  Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  * [30] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [31] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al.  Mastering the game of Go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [32] S. L. Brunton and B. R. Noack.  Closed-loop turbulence control: Progress and challenges.  Applied Mechanics Reviews, 67(5):050801, 2015.
  * [33] X. Zhang, W. Mao, S. Mowlavi, M. Benosman, and T. Başar.  Controlgym: Large-scale safety-critical control environments for benchmarking reinforcement learning algorithms.  arXiv preprint arXiv:2311.18736, 2023.
  * [34] M. Fazel, R. Ge, S. M. Kakade, and M. Mesbahi.  Global convergence of policy gradient methods for the linear quadratic regulator.  In International Conference on Machine Learning, pages 1467–1476, 2018.
  * [35] K. Zhang, B. Hu, and T. Başar.  Policy optimization for $\mathcal{H}_{2}$ linear control with $\mathcal{H}_{\infty}$ robustness guarantee: Implicit regularization and global convergence.  SIAM Journal on Control and Optimization, 59(6):4081–4109, 2021\.
  * [36] K. Zhang, X. Zhang, B. Hu, and T. Başar.  Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity.  Advances in Neural Information Processing Systems, 34:2949–2964, 2021.
  * [37] X. Zhang and T. Başar.  Revisiting LQR control from the perspective of receding-horizon policy gradient.  IEEE Control Systems Letters, 7:1664–1669, 2023.
  * [38] K.-H. Lee, O. Nachum, M. S. Yang, L. Lee, D. Freeman, S. Guadarrama, I. Fischer, W. Xu, E. Jang, H. Michalewski, et al.  Multi-game decision transformers.  Advances in Neural Information Processing Systems, 35:27921–27936, 2022.
  * [39] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan.  Prompting decision transformer for few-shot policy generalization.  In International Conference on Machine Learning, pages 24631–24645, 2022.
  * [40] Q. Zheng, A. Zhang, and A. Grover.  Online decision transformer.  In International Conference on Machine Learning, pages 27042–27059, 2022.
  * [41] T. Yamagata, A. Khalil, and R. Santos-Rodriguez.  Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl.  In International Conference on Machine Learning, pages 38989–39007, 2023.
  * [42] M. Xu, Y. Lu, Y. Shen, S. Zhang, D. Zhao, and C. Gan.  Hyper-decision transformer for efficient online policy adaptation.  arXiv preprint arXiv:2304.08487, 2023.
  * [43] R. Shi, Y. Liu, Y. Ze, S. S. Du, and H. Xu.  Unleashing the power of pre-trained language models for offline reinforcement learning.  arXiv preprint arXiv:2310.20587, 2023.
  * [44] T. Schmied, M. Hofmarcher, F. Paischer, R. Pascanu, and S. Hochreiter.  Learning to modulate pre-trained models in rl.  Advances in Neural Information Processing Systems, 36, 2024.
  * [45] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4RL: Datasets for deep data-driven reinforcement learning.  arXiv preprint arXiv:2004.07219, 2020.
  * [46] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In Conference on Robot Learning, pages 1094–1100. PMLR, 2020.
  * [47] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al.  DeepMind control suite.  arXiv preprint arXiv:1801.00690, 2018.
  * [48] K. Esslinger, R. Platt, and C. Amato.  Deep transformer Q-networks for partially observable reinforcement learning.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [49] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al.  A generalist agent.  arXiv preprint arXiv:2205.06175, 2022.
  * [50] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al.  RT-1: Robotics transformer for real-world control at scale.  arXiv preprint arXiv:2212.06817, 2022.
  * [51] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan.  VIMA: General robot manipulation with multimodal prompts.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [52] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei.  MetaMorph: Learning universal controllers with transformers.  arXiv preprint arXiv:2203.11931, 2022.
  * [53] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al.  RT-2: Vision-language-action models transfer web knowledge to robotic control.  arXiv preprint arXiv:2307.15818, 2023.
  * [54] K. Bousmalis, G. Vezzani, D. Rao, C. M. Devin, A. X. Lee, M. B. Villalonga, T. Davchev, Y. Zhou, A. Gupta, A. Raju, et al.  RoboCat: A self-improving generalist agent for robotic manipulation.  Transactions on Machine Learning Research, 2023.
  * [55] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn.  Learning fine-grained bimanual manipulation with low-cost hardware.  arXiv preprint arXiv:2304.13705, 2023.
  * [56] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, et al.  Octo: An open-source generalist robot policy, 2023.
  * [57] I. Radosavovic, B. Shi, L. Fu, K. Goldberg, T. Darrell, and J. Malik.  Robot learning with sensorimotor pre-training.  In Conference on Robot Learning, pages 683–693, 2023.
  * [58] I. Radosavovic, B. Zhang, B. Shi, J. Rajasegaran, S. Kamat, T. Darrell, K. Sreenath, and J. Malik.  Humanoid locomotion as next token prediction.  arXiv preprint arXiv:2402.19469, 2024.
  * [59] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, A. Kapoor, K. Hausman, et al.  Foundation models in robotics: Applications, challenges, and the future.  arXiv preprint arXiv:2312.07843, 2023.
  * [60] K. Kawaharazuka, T. Matsushima, A. Gambardella, J. Guo, C. Paxton, and A. Zeng.  Real-world robot applications of foundation models: A review.  arXiv preprint arXiv:2402.05741, 2024.
  * [61] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra.  Planning and acting in partially observable stochastic domains.  Artificial Intelligence, 101(1-2):99–134, 1998.
  * [62] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643, 2020.
  * [63] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.

###  -A Experiment Settings

Our experiments include five control tasks from the controlgym library [33],
where their corresponding dimensions $n_{s},n_{a}$, and $n_{o}$ and horizon
lengths are listed below.

Task | $n_{s}$ | $n_{a}$ | $n_{o}$ | n_steps | sample_time
---|---|---|---|---|---
he1 | $4$ | $2$ | $1$ | $50$ | $0.05$
ac4 | $9$ | $1$ | $2$ | $50$ | $0.05$
cm3 | $120$ | $1$ | $2$ | $50$ | $0.25$
CDR | $64$ | $5$ | $10$ | $100$ | $0.1$
Burgers | $64$ | $5$ | $10$ | $100$ | $0.05$

We now describe the parameters of these environments. In he1, ac4, and cm3, we
set the noise_cov parameter to $0.01$, which determines the intensity of the
random noise in both the system dynamics and the measurement process. In
generating the offline dataset $\mathcal{D}$ and also conducting the
experiments on in-distribution generalizations, we first randomly sample
vectors $\Delta A$ and $\Delta B_{2}$ of appropriate dimensions from a zero-
mean Gaussian distribution. Then, we perturb the system matrices $A$ and
$B_{2}$ in he1, ac4, and cm3 by normalizing $[\Delta A,\Delta B_{2}]$ to the
size of $\big{\\{}[0.05,0.05],[0.1,0.1],[0.1,0.1]\big{\\}}$, respectively. In
generating out-of-distribution test cases, we set the perturbation sizes
$[\Delta A,\Delta B_{2}]$ to
$\big{\\{}[0.15,0.15],[0.2,0.2],[0.15,0.15]\big{\\}}$ for tasks $\\{$he1, ac4,
cm3$\\}$, respectively.

Figure -A.1: Left: PPO policy in Burgers with $\nu=10^{-2}$ and $\phi=0.125$.
Right: We sample the Burgers parameters uniformly from the shaded region for
generating the dataset and in-distribution tests. The out-of-distribution
tests are manually selected.

In CDR and Burgers, we perturb the system dynamics by varying the physical
parameters of their governing PDE. For Burgers, we create training and in-
distribution testing sets by uniformly sampling the diffusivity_constant
($\nu$) within a logarithmic scale from $10^{-3}$ and $10^{-1}$, and also
uniformly sampling the control_sup_width ($\phi$) between $0.09L$ and $0.16L$
(with $L=1$ denoting domain length). We deterministically generate $9$ out-of-
distribution test cases as all $(\nu,\phi)$ pairs with
$\nu\in\\{10^{-3},5.5\times 10^{-4},10^{-4}\\}$ and
$\phi\in\\{0.09,0.08,0.07\\}$, presenting more challenging control tasks due
to shock-like behavior from lower $\nu$ and reduced control authority with
smaller $\phi$. For all cases, we set process_noise_cov and sensor_noise_cov
parameters to $0$ and $0.25$, respectively. We choose the target state to the
discretization of the continuous field $-0.1\cdot\cos(2\pi x/L)$ at $n_{s}$
equally-spaced points in the spatial domain, where $x$ represents the domain
coordinate [33]. Figure -A.1 illustrates the converged PPO policy’s tracking
behavior in a single Burgers experiment and the task generation process for
multi-task experiments.

Figure -A.2: PPO policy in CDR with $\nu=10^{-2}$, $c=0.1$, $\zeta=0$, and
$\phi=0.1$.

Similar to Burgers, we create the CDR training and in-distribution testing
sets by uniformly sampling the diffusivity_constant ($\nu$) within a
logarithmic scale from $10^{-3}$ and $10^{-1}$, the convective_velocity ($c$)
from $0$ to $0.2$, the reaction_constant ($\zeta$) from $-0.1$ to $0.1$, and
the control_sup_width ($\phi$) from $0.08L$ and $0.12L$ (with $L=1$). For the
$9$ out-of-distribution tests, we choose $\nu,c,\zeta,\phi$ deterministically
to be

$\displaystyle(\nu,c,\zeta,\phi)\in\Big{\\{}(5\times 10^{-4},0.25,0.15,0.1),$
$\displaystyle\hskip 18.00003pt(5\times 10^{-4},0.25,0.2,0.1),(5\times
10^{-4},0.3,0.15,0.1),$ $\displaystyle\hskip 18.00003pt(5\times
10^{-4},0.3,0.2,0.1),(1\times 10^{-4},0.25,0.15,0.1),$ $\displaystyle\hskip
18.00003pt(1\times 10^{-4},0.25,0.2,0.1),(1\times 10^{-4},0.3,0.15,0.1),$
$\displaystyle\hskip 18.00003pt(1\times 10^{-4},0.3,0.2,0.1),(1\times
10^{-4},0.3,0.2,0.08)\Big{\\}}.$

Compared to the training and in-distribution testing cases, these out-of-
distribution parameters result in more challenging control tasks (cf., Section
3.1 of [33]). For all cases, we choose the process_noise_cov and the
sensor_noise_cov parameters to $0$ and $0.25$, respectively. We choose the
target state to the discretization of the continuous field
$0.5-0.5\cdot\mathrm{sech}(20x-10L)$ at $n_{s}$ equally-spaced points in the
spatial domain. Figure -A.2 shows the converged PPO policy’s tracking
performance in a CDR experiment with parameters being the mean of the training
distribution.

###  -B DT Parameters

We have adopted the GPT-2 architecture with 117 millions of
parameters222https://huggingface.co/openai-community/gpt2. Specifically, it
includes 12 Transformer layers and operates with a hidden dimension of 768. We
also use 3 MLP layers, a LoRA rank of 32, and is trained with a batch size of
64. Moreover, we define a dropout rate of 0.1 to mitigate overfitting. All
experiments are run on a single NVIDIA GeForce RTX 3090 GPU with 24 GB Memory.

Moreover, we employ the learning rate of $10^{-4}$, the weight decay rate of
$10^{-5}$, the context length of $20$, and the ‘return-to-go’s to be
$(-50,-10)$ for he1, $(-0.3,-0.1)$ for ac4, $(-10,-5)$ for cm3, $(-1500,-300)$
for CDR, and $(-150,-110)$ for Burgers, respectively.