gpzparam_short = gpzparam.replace(” ”, ””).lower()

XCMD(”poise –create dosy {} 20 80 50 2”

” dosy poise_1d”.format(gpzparam_short))

XCMD(”poise –create dosy_aux {} 79 81 80 0.1”

” dosy_aux poise_1d”.format(gpzparam_short))

# Get the expno of the 2D dataset, and store the parameter set. This is

# important because we want the optimisation to inherit most parameters,

# e.g. D1, SW, etc. from the parent experiment (which has presumably been

# set up by the user).

original_2d_expno = int(CURDATA()[1])

original_2d_dataset = CURDATA()

XCMD(”wpar dosytemp all”)

# Set up the reference experiment.

reference_expno = 99998

reference_dataset = list(original_2d_dataset)

reference_dataset[1] = str(reference_expno)

NEWDATASET(reference_dataset, None, ”dosytemp”)

RE(reference_dataset)

PUTPAR(”PULPROG”, pp1d)

PUTPAR(”PARMODE”, ”1D”)

PUTPAR(”PPARMOD”, ”1D”)

PUTPAR(gpzparam, ”10”)

XCMD(”sendgui browse_update_tree”)

# Acquire reference dataset.

msg_nonmodal(”dosy_opt: optimising diffusion delay…”)

XCMD(”poise_1d”, wait=WAIT_TILL_DONE)

# Set up optimisation expno.

optimisation_expno = 99999

optimisation_dataset = list(original_2d_dataset) # make a copy

optimisation_dataset[1] = str(optimisation_expno)

NEWDATASET(optimisation_dataset, None, ”dosytemp”)

RE(optimisation_dataset)

PUTPAR(”PULPROG”, pp1d)

PUTPAR(”PARMODE”, ”1D”)

PUTPAR(”PPARMOD”, ”1D”)

PUTPAR(gpzparam, ”80”)

XCMD(”sendgui browse_update_tree”)

# Optimise the diffusion delay first.

while True:

# Here we use the ‘–maxfev 1‘ trick to evaluate a cost function at 80%

# gradient. The cost function is stored as the ‘TI‘ parameter after the

# experiment has been recorded.

RE(optimisation_dataset)

PUTPAR(”TI”, ” ”) # has to be one space, not an empty string (try it)

XCMD(”poise dosy_aux -a nm –maxfev 1 -q”, wait=WAIT_TILL_DONE)

# If TI isn’t a valid float, something went wrong with POISE.

try:

cf = float(GETPAR(”TI”))

except ValueError:

EXIT()

# The dosy_aux cost function is (opt. intensity/ref. intensity) minus

# 0.25. If the opt. spectrum has been attenuated too little, then this

# value will be positive.

if cf > 0:

# Increase delay, i.e. big Delta, by 100 ms

current_delay = float(GETPAR(dparam))

new_delay = current_delay + 0.1

PUTPAR(dparam, str(new_delay))

# Set it in the reference spectrum too

RE(reference_dataset)

PUTPAR(dparam, str(new_delay))

# Re-acquire the reference spectrum.

XCMD(”xau poise_1d”, wait=WAIT_TILL_DONE)

# If the cost function is negative, then that means it’s sufficiently

# attenuated, and we can go on to the actual optimisation.

else:

break

# At this point, we know that 80% gradient strength provides at least 75%

# attenuation. So we can run an optimisation to find the actual value that

# does provide 75% attenuation (which is somewhere between 10% and 80%).

RE(optimisation_dataset)

best_delay = GETPAR(dparam)

msg_nonmodal(”dosy_opt: diffusion delay set to {} seconds.\n”

” now optimising gradient amplitude…”.format(best_delay))

PUTPAR(”TI”, ” ”)

XCMD(”poise dosy -a bobyqa -q”, wait=WAIT_TILL_DONE)

# Again, check to make sure that TI is a valid float.

try:

float(GETPAR(”TI”))

except ValueError:

EXIT()

best_gpz = GETPAR(gpzparam)

# Jump back to original 2D expt. Set dparam, but not gpzparam as that

# should always be 100% (the actual gradient amplitude is controlled by

# Difframp, which will be generated by the ‘dosy‘ AU programme).

RE(original_2d_dataset)

PUTPAR(dparam, best_delay)

PUTPAR(gpzparam, ”100”)

# Generate Difframp, etc. and then run the AU programme.

msg_nonmodal(”dosy_opt: maximum gradient set to {}%.\n”

” starting 2D experiment with optimised”

” parameters…”.format(best_gpz))

XCMD(”xau dosy 10 {} 16 l y”.format(best_gpz))

def msg_nonmodal(s):

”””

Show a message box that does not force the user to click OK before the

programme continues. See also the same function in poise.py (the frontend

script).

”””

dialog = mfw.BInfo()

dialog.setMessage(s)

dialog.setTitle(”dosy_opt”)

dialog.setBlocking(0) # in ordinary MSG() this is 1

dialog.show()

if __name__ == ”__main__”:

main()

###  S12.3 Sequential optimisation details

The second part of the optimisation ($G_{\mathrm{max}}$ optimisation) is the
only place where using different algorithms can have an effect. In order to
evaluate the effect of using different algorithms, we ran the second
optimisation using each of the algorithms, using the diffusion delay
$\Delta=$100\text{\,}\mathrm{ms}$$ obtained from the first step. The results
are summarised in Table S24.

Entry | Algorithm | Optimum found (%) | $n_{\text{fev}}$ | Time taken ($\mathrm{s}$)
---|---|---|---|---
1 | Nelder–Mead | 75.00 | 9 | 195–197
2 | MDS | 75.00 | 9 | 196–197
3 | BOBYQA | 75.00–75.60 | 8–9 | 175–197
Table S24:  Optimisations of maximum gradient amplitude ($G_{\mathrm{max}}$)
for diffusion NMR spectroscopy. The POISE routine used here is: . The pulse
programme used was doneshot_1d_jy (Section S12.5). Key acquisition parameters:
NS=4, DS=2, D1=1.5, AQ=1.14. $n_{\text{fev}}$ refers to the number of function
evaluations, i.e. the number of spectra acquired in the course of the
optimisation. The results are aggregated from five separate runs for each
optimisation algorithm.

###  S12.4 Simultaneous optimisation strategy

We also tried directly performing a simultaneous two-parameter optimisation of
both the gradient amplitude and the diffusion delay, instead of the entire
procedure above. The POISE routine and its associated cost function are called
dosy_2p. The routine is:

⬇

{”name”: ”dosy_2p”, ”pars”: [”gpz1”, ”d20”], ”lb”: [20.0, 0.05], ”ub”: [80.0,
0.4], ”init”: [50.0, 0.05], ”tol”: [2.0, 0.1], ”cf”: ”dosy_2p”, ”au”:
”poise_2pdosy”}

In order to bias the optimisation towards smaller values of $\Delta$ (which is
desirable as it minimises $T_{1}$ losses), we added a term to the cost
function that is proportional to $\Delta$. This cost function can be found in
.../py/user/poise_backend/costfunctions.py as well, although it is commented
out by default.

$f=|f_{\mathrm{att}}|+(\Delta/\mathrm{s})$ (8)

Unfortunately, although this optimisation did work, it was far slower: for
example, using the Nelder–Mead algorithm, this took 36 function evaluations
and 1572 seconds (over 26 minutes). The optimum converged to was
$G_{\mathrm{max}}=79.3\%$ and $\Delta=$91.6\text{\,}\mathrm{ms}$$, which is
very similar to that found above. Part of the reason why this is so slow is
that every time the diffusion delay $\Delta$ is changed, the reference
spectrum must be reacquired: this means that every function evaluation in this
two-parameter optimisation takes twice as long as every function evaluation in
the individual optimisation of $G_{\mathrm{max}}$.

###  S12.5 Oneshot DOSY pulse programmes

####  S12.5.1 2D Oneshot

Some modifications from the published version were made so that it could be
run on our spectrometers.

⬇

;doneshot_2d_jy

;

;download from: https://git.io/JUHOX

;

;Jonathan Yong - University of Oxford - 26 Aug 2020

;minor modifications to original oneshot DOSY sequence to make it play better
with TopSpin.

;

;modified from:

;Ralph Adams, Juan Aguilar, Robert Evans, Mathias Nilsson and Gareth Morris

;University of Manchester

;Release 1.0c (27Mar2012)

;Source citation:

;M.D. Pelta, G.A. Morris, M.J. Stchedroff, S.J. Hammond, Magn. Reson. Chem. 40
(2002) S147-S152.

;

;Other relevant papers that could be of use include:

;A. Botana, J.A. Aguilar, M. Nilsson, G.A. Morris, J. Magn. Reson. 208 (2011)
270-278.

;

;2D Doneshot DOSY pulse sequence

;$CLASS=HighRes

;$DIM=2D

;$TYPE=

;$SUBTYPE=

;$COMMENT=

#include <Avance.incl>

#include <Grad.incl>

#include <Delay.incl>

define list<gradient> diff=<Difframp>

;JY - alpha is hardcoded as 0.2

”cnst14=0.2”

”cnst17=(2*p1+d16+p30)*0.000001”; Dtau

;Assuming half-sine gradient pulses [most common on Bruker systems]

”cnst18=0.000001*p30*2*0.000001*p30*2*(d20 -
(2*0.000001*p30*(5-3*cnst14*cnst14)/16) - (cnst17*(1-cnst14*cnst14)/2) )” ;
Dosytimecubed

”cnst15=1+cnst14” ; 1 + alpha

”cnst16=1-cnst14” ; 1 - alpha

”p2=p1*2”

”DELTA1=d20-4.0*p1-4.0*p30-5.0*d16-p19”

”acqt0 = -p1*2/3.1416”

1 ze

2 d1

3 50u UNBLKGRAD

p19:gp2*-1 ;Spoiler gradient balancing pulse

d16

p1 ph1 ;1st 90

p30:gp1*diff*cnst16 ;1 - alpha

d16

p2 ph2 ;First 180

p30:gp1*diff*cnst15*-1 ;1 + alpha

d16

p1 ph3 ;2nd 90

p30:gp1*diff*cnst14*2 ;Lock refocusing pulse pulse

d16

p19:gp2 ;Spoiler gradient balancing pulse

d16

DELTA1

p30:gp1*diff*cnst14*2 ;Lock refocusing pulse pulse

d16

p1 ph4 ;3rd 90

p30:gp1*diff*cnst16 ;1 - alpha

d16

p2 ph5

p30:gp1*diff*cnst15*-1 ;1 + alpha

d16

4u BLKGRAD

go=2 ph31

d1 mc #0 to 2 F1QF(calgrad(diff))

exit

ph1 = 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2
1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0
2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2
0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0
2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3
0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1
3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3

ph2 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

ph3 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

ph4 = 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2
0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3
3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2
0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3
3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1
0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2
2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1

ph5 =0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

ph31 = 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 2 0 0 2
3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3
1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0
0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1
3 3 1 2 0 0 2 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0
2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 0 2 2 0 1 3 3
1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2

;pl1 : f1 channel - power level for pulse (default)

;p1 : f1 channel - 90 degree high power pulse

;p19 : gradient pulse 2 (spoil gradient)

;p30 : gradient pulse (little DELTA*0.5)

;d1 : relaxation delay; 1-5 * T1

;d16 : delay for gradient recovery

;d20 : diffusion time (big DELTA)

;gpnam1: SMSQ10.100

;gpnam2: SMSQ10.100

;gpz1 : diffusion encoding gradient (100%)

;gpz2 : Oneshot CTP gradients

;cnst14 : alpha, typically 0.2

;cnst15 : 1+alpha

;cnst16 : 1-alpha

;cnst17 : Dtau

;cnst18 : dosytimecubed

;NS : 1 * n

;DS : 1 * m

;td1: number of experiments

;FnMODE: QF

;use AU-program dosy to calculate gradient ramp-file Difframp

; use xf2 and DOSY processing

; use ”setdiffparmUoM” if ’setdiffparm’ does not work

; or use ”setdiffparm STEbp” but this gives slighly distorted diffusion
coefficients

####  S12.5.2 1D Oneshot

⬇

; doneshot_1d_jy

;

; download from: https://git.io/JUHOP

;

; 1D version of Doneshot

; Jonathan Yong - University of Oxford - 26 Aug 2020

; modified from

; Doneshot

; DOSY pulse sequence

; Ralph Adams, Juan Aguilar, Robert Evans, Mathias Nilsson and Gareth Morris

; University of Manchester

; Release 1.0c (27Mar2012)

;

; Source citation:

; M.D. Pelta, G.A. Morris, M.J. Stchedroff, S.J. Hammond, Magn. Reson. Chem.
40 (2002) S147-S152.

;

; Other relevant papers that could be of use include:

; A. Botana, J.A. Aguilar, M. Nilsson, G.A. Morris, J. Magn. Reson. 208 (2011)
270-278.

;

;$CLASS=HighRes

;$DIM=1D

;$TYPE=

;$SUBTYPE=

;$COMMENT=

#include <Avance.incl>

#include <Grad.incl>

#include <Delay.incl>

”cnst17=(2*p1+d16+p30)*0.000001” ; Dtau

;JY - alpha is hardcoded as 0.2

”cnst14=0.2”

;Assuming half-sine gradient pulses [most common on Bruker systems]

”cnst18=0.000001*p30*2*0.000001*p30*2*(d20 -
(2*0.000001*p30*(5-3*cnst14*cnst14)/16) - (cnst17*(1-cnst14*cnst14)/2) )” ;
Dosytimecubed

”cnst15=1+cnst14” ; 1 + alpha

”cnst16=1-cnst14” ; 1 - alpha

”p2=p1*2”

”DELTA1=d20-4.0*p1-4.0*p30-5.0*d16-p19”

;The below line should be commented out for Topspin < 2.0

”acqt0 = -p1*2/3.1416”

1 ze

2 d1

3 50u UNBLKGRAD

p19:gp2*-1 ;Spoiler gradient balancing pulse

d16

p1 ph1 ;1st 90

p30:gp1*cnst16 ;1 - alpha

d16

p2 ph2 ;First 180

p30:gp1*cnst15*-1 ;1 + alpha

d16

p1 ph3 ; 2nd 90

p30:gp1*cnst14*2 ;Lock refocusing pulse pulse

d16

p19:gp2 ;Spoiler gradient balancing pulse

d16

DELTA1

p30:gp1*cnst14*2 ;Lock refocusing pulse pulse

d16

p1 ph4 ; 3rd 90

p30:gp1*cnst16 ; 1 - alpha

d16

p2 ph5

p30:gp1*cnst15*-1 ;1 + alpha

d16

4u BLKGRAD

go=2 ph31

d1 mc #0 to 2 F0(zd)

exit

ph1 = 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2
1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0
2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2
0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0
2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3
0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1
3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3 0 2 0 2 1 3 1 3

ph2 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

ph3 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

ph4 = 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2
0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3
3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2
0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3
3 1 1 3 3 1 1 0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1
0 0 2 2 0 0 2 2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1 0 0 2 2 0 0 2
2 1 1 3 3 1 1 3 3 2 2 0 0 2 2 0 0 3 3 1 1 3 3 1 1

ph5 =0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

ph31 = 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 2 0 0 2
3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3
1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0
0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1
3 3 1 2 0 0 2 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0
2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2 0 2 2 0 1 3 3 1 3 1 1 3 0 2 2 0 0 2 2 0 1 3 3
1 3 1 1 3 0 2 2 0 2 0 0 2 3 1 1 3 1 3 3 1 2 0 0 2

;pl1: f1 channel - power level for pulse (default)

;p1 : f1 channel - 90 degree high power pulse

;p19: gradient pulse 2 (spoil gradient)

;p30: gradient pulse (little DELTA*0.5)

;d1 : relaxation delay; 1-5 * T1

;d16: delay for gradient recovery

;d20: diffusion time (big DELTA)

;cnst14: alpha, typically 0.2

;cnst15: 1+alpha

;cnst16: 1-alpha

;cnst17: Dtau

;cnst18: dosytimecubed

;ns: 1*n

;ds: 1*m

;td1: number of experiments

;gpnam1: SMSQ10.100

;gpnam2: SMSQ10.100

;gpz1: DOSY gradient strength. DO NOT EXCEED 80

;gpz2: Oneshot CTP gradients

##  S13 Documentation

Finally, the current version of the POISE documentation is attached here. This
is meant to act as a user guide, with specific instructions on how to set up
and run optimisations.

Please note that the latest version of the documentation can always be found
online at

⬇

https://foroozandehgroup.github.io/nmrpoise

and a PDF version of this can be downloaded at

⬇

https://foroozandehgroup.github.io/nmrpoise/poise.pdf

See pages - of ./files/docu.pdf


# AA-TransUNet: Attention Augmented TransUNet For Nowcasting Tasks

Yimin Yang and Siamak Mehrkanoon *corresponding author. Department of Data
Science and Knowledge Engineering
Maastricht University
Maastricht, The Netherlands
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Data driven modeling based approaches have recently gained a lot of attention
in many challenging meteorological applications including weather element
forecasting. This paper introduces a novel data-driven predictive model based
on TransUNet for precipitation nowcasting task. The TransUNet model which
combines the Transformer and U-Net models has been previously successfully
applied in medical segmentation tasks. Here, TransUNet is used as a core model
and is further equipped with Convolutional Block Attention Modules (CBAM) and
Depthwise-separable Convolution (DSC). The proposed Attention Augmented
TransUNet (AA-TransUNet) model is evaluated on two distinct datasets: the
Dutch precipitation map dataset and the French cloud cover dataset. The
obtained results show that the proposed model outperforms other examined
models on both tested datasets. Furthermore, the uncertainty analysis of the
proposed AA-TransUNet is provided to give additional insights on its
predictions.

###### Index Terms:

UNet, Transformer, Precipitation Nowcasting, Cloud Cover Nowcasting, Deep
Learning

##  I Introduction

Accurate prediction of weather conditions impacts human life and it has a
direct effect on many sectors including for instance economic, agriculture,
business, transport and logistics among others. In particular, precipitation
nowcasting, i.e. short-term forecasts of up to 6 hours, is becoming an
increasingly popular research topic due to its high impact in supporting the
socioeconomic needs of the above-mentioned industrial sectors [1]. In
addition, precipitation nowcasting plays an important role in water-related
risk management. Often warning systems also rely on nowcasts in order to
generate warnings [2].

Traditional nowcasting methods are based on Numerical weather prediction (NWP)
[3]. However, their spatial and temporal resolution is often too low for
precipitation nowcasting. In addition, NWP based models demand a substantial
amount of computing power as they generate predictions by simulating the
underlying physics of the atmosphere and ocean [4, 5]. Therefore, recent years
have witnessed the development of new data driven models to overcome the
shortcomings of the NWP based models. They often do not rely on the underlying
physics of the atmosphere and ocean but instead concentrate solely on learning
from the historical data [6]. In particular, models with deep neural network
architecture have shown to outperform traditional data driven models in
several application domains [7, 8, 9, 4, 10, 11, 12, 13, 6, 14, 15].

Convolutional Neural Networks (CNN) based models such as AlexNet [16], ResNet
[17], Convolutional Autoencoders (CAEs) [18] and their variants are among the
most used neural networks architectures in computer vision tasks. Following
the idea of CAEs, the authors in [19] introduced the well-known UNet model,
which uses the classic encoder (down-sampling) and decoder (up-sampling) with
skip connection structure [19]. UNet was initially used for medical image
segmentation, but it gradually began to shine in many other computer vision
tasks. Inspired by the success of UNet model, several other architectures have
been developed in the literature which continues to adhere to the core concept
of UNet, albeit incorporating new modules or other techniques.

Another impactful model is the Transformer architecture [20] which has
revolutionized the field of Natural Language Processing (NLP). This sequence
to sequence architecture which is equipped with attention mechanism is
designed to handle sequential input data. Compared to traditional models used
for sequence-to-sequence learning, Transformer has produced better results
[20]. With the transformer model dominating the domain of NLP, many
researchers began to investigate its applicability in computer vision tasks.
For instance, Google researchers released the Vision Transformer model [21]
which divides the image into small blocks of fixed size, linearly embeds each
small block, adds position embedding and feeds the resulting vector sequence
into a standard Transformer encoder.

Although the Transformer based model has great potential in the computer
vision field, it also faces some significant challenges such as the high
demand for data [22]. Likewise, the UNet structure that has consistently held
a dominant position in the field of image segmentation also poses some
limitations such as the inability to capture the long-term dependence [23].
Consequently, combining the transformer and the UNet model has become a new
research direction. Based on this, the authors in [23] proposed the TransUNet,
which is an image segmentation model that fully utilizes the advantages of
UNet and Transformer. The TransUNet model has been shown to perform well in
medical image segmentation, but it has not been widely used in other fields.

In this paper, we propose a new model called Attention Augmented TransUNet
(AA-TransUNet). Here, the TransUNet is used as core model and it is further
equipped with Convolutional Block Attention Module(CBAM) [24] and Depthwise-
separable Convolutions (DSC) [25]. Thanks to the incorporation of these new
elements in the core TransUNet model, the number of trainable parameters is
significantly reduced without degrading the accuracy of the model. We seek the
application of the proposed AA-TransUNet on precipitation nowcasting tasks. In
particular, here we use the same datasets as those used in [5], i.e
precipitation map dataset and cloud cover dataset. As in [5], in the case of
the precipitation map dataset, the model input consists of 60 minutes of
precipitation maps collected from the cartographic radar and the model output
is another sequence of precipitation maps predicted for the next 30 minutes.
We also run related experiments on the cloud cover dataset to demonstrate the
applicability of the proposed model.

This paper is organized as follows. A brief overview of the related research
works is given in Section II. Section III introduces the proposed AA-TransUNet
model. The experimental settings and description of the used datasets are
given in Section IV. The obtained results are discussed in Section V and the
conclusion is drawn in Section VI.

##  II Related works

The data driven based models have gained a lot of attentions for weather
elements forecasting such as temperature, wind speed, and precipitation among
others. Weather elements forecasting can be cast into sequence prediction
problem and hence it is closely linked to Recurrent Neural Networks(RNN)
models such as Long Short-term Memory (LSTM) [26]. In addition due to the
presence of spatial structure of the weather data, it is also linked to
convolutional Neural Networks (CNN). Therefore, recent years have witnessed
the emergence of powerful deep learning architectures based on LSTM, CNN, and
UNet for addressing challenging tasks in weather elements forecasting.

For instance, the authors in [27] proposed the Convolutional LSTM (ConvLSTM)
model that better captures the underlying spatiotemporal correlations of the
data and consistently outperforms normal LSTM. Continuing in the ConvLSTM
research direction, the authors in [28] proposed Convcast, an embedded
convolutional LSTM-based architecture. The author in [4] introduced different
CNN architectures including 1-D, 2-D and 3-D convolutions to accurately
predict wind speed in the next 6 to 12 hours. The UNet architecture is used
for precipitation nowcasting in [29]. The SmaAt-UNet model, an extension of
UNet model, which greatly reduces the UNet parameters without compromising its
performance is introduced in [5]. The authors in [6], introduced Broad-UNet by
equipping the UNet model with asymmetric parallel convolutions as well as
Atrous Spatial Pyramid Pooling (ASPP) module.

Despite its many advantages, UNet can still result in limited abilities when
modeling long-range dependencies due to the intrinsic locality of the
convolution based operations. On the other hand, Transformer model, which is
initially developed for sequence-to-sequence prediction and uses the global
self-attention mechanisms [20], has also limitations in localization abilities
due to insufficient low-level details [23]. Subsequently, the authors in [23]
proposed TransUNet model that has the best of two UNet and Transformer models
and they showed its successful application in medical image segmentation
tasks.

In this paper, we use TransUNet as the core model and extend it by
incorporating Convolutional Block Attention Modules (CBAM) and employing the
Depthwise-separable Convolutions (DSC) in the core model. CBAM is a
lightweight and general attention module for feed-forward Convolutional Neural
Networks. Given an intermediate feature map, CBAM infers attention maps in two
dimensions (channel attention module and spatial attention module)
sequentially [24]. Depthwise (DW) convolution and Pointwise (PW) convolution
are collectively called Depthwise-separable Convolution. This structure is
similar to regular convolution operations and can be used to extract features,
although its parameter sizes are substantially smaller than standard
convolution operations [25].

##  III Method

This section introduces the proposed AA-TransUNet model which uses the
TransUNet [23] as the core model and extends it to reduce its parameters and
improve its forecasting performance. We then investigate the application of
the proposed model in precipitation nowcasting tasks.

Figure 1: The architecture of AA-TransUNet. (a) depicts the deconstruction
schematic of Transformer Layer. (b) depicts the proposed model. Each bar in
the figure represents a multi-channel feature map. The numbers below each bar
display the number of channels and the numbers next to the bar represent the
resolution. The input is the ground truth of an exemplary precipitation map,
whereas the output is the corresponding prediction map produced by AA-
TransUNet.

###  III-A Proposed Model

The present study builds upon and extends the work of [23], who introduced
TransUNet. The original TransUNet is an encoder-decoder architecture where
CNN-Transformer is used as the encoder, while the original UNet decoder is
utilized as the decoder. This model has been successfully applied in medical
image segmentation tasks. Here, we propose a new encoder-decoder architecture
called AA-TransUNet which equips the classical TransUNet with two key
elements: Convolutional Block Attention Modules (CBAM) and Depthwise-separable
Convolutions (DSC). The model is shown in Fig. 1 and is used for precipitation
forecasting tasks. We integrate CBAM, which consists of a Channel Attention
Module (CAM) and a Spatial Attention Module (SAM), to both encoder and decoder
path of the TransUNet model. In particular, in the encoder path, CBAM is
incorporated into the CNN part of the hybrid CNN-Transformer layer, and in the
decoder path it is placed after all convolution layers (see Fig. 1). This
allows the model to perform both channel-wise and spatial-wise attention and
therefore can better explore the inter-channel as well as inter-spatial
relationship of the features. In addition, the original convolutions of the
decoder of the TransUNet is replaced by DSC aiming to reduce their parameter
size. In Table I, we compare the number of parameters of the decoder of
TransUNet and AA-TransUNet models. Despite the added CBAM layers, the number
of parameters of AA-TransUNet decoder are still 68.94% lower than that of
TransUNet. It should be noted that here we only compare the number of
parameters of the decoder. As CBAM, which is a lightweight and general module
[24], is only added to the encoder, therefore the number of parameters of the
encoder path of the AA-TransUNet is relatively the same as that of TransUNet.

Table I: Comparison of the number parameters of the decoder of the two models. Model | CBAM layers | Conv layers | Total Parameters
---|---|---|---
TransUNet | 0 |  $\sim$2.93M |  $\sim$2.93M
AA-TransUNet |  $\sim$0.04M |  $\sim$0.87M |  $\sim$0.91M

Despite [5], here the input to the encoders is the image feature with the
attention mechanism applied. In addition, as oppose to [5], CBAM is used in
each decoder level (see purple arrows in the decode path of Fig. 1) to
identify important features across channels and spatial dimensions of an image
[5, 24].

Similar to TransUNet [23], the encoder of the AA-TransUNet model is a
Transformer based model, which is typically used to process sequence data.
Therefore, when using it as an encoder, the image sequentialization step needs
to be taken first to covert 3D images to 2D patches [23]. The obtained patches
are then fed into the Transformer layer, which consists of Multihead Self-
Attention (MSA) layers [20] and Multi-Layer Perceptron (MLP) blocks (each MSA
and MLP pass through the layer normalization gate). At the same time,
following [23], instead of using a pure Transformer as an encoder, we employ
CNN-Transformer to make use of the intermediate high-resolution CNN feature
maps in the decoding path and improve the model performance.

An architecture similar to UNet decoder is used for the proposed AA-TransUNet
decoder. As shown in Fig. 1, four operations are used in the AA-TransUNet
decoder: upsampling (green arrows), feature concatenation (dotted rectangles),
double convolution (cyan arrows) and finally the CBAM (purple arrows). The
last layer of the model is a 1 x 1 convolution (orange arrow), which produces
a single feature map that represents the prediction results.

###  III-B Training

We have compared our proposed AA-TransUNet model with four other models, i.e.
persistence method, UNet [19], SmaAt-UNet [5] and TransUNet [23] models on two
datasets: precipitation maps as well as cloud cover dataset. The persistence
method, a commonly used approach in nowcasting tasks, refers to the model that
uses the last input image of a sequence as the prediction image. The other
four examined UNet based models are trained under the same settings. The
maximum number of epochs for precipitation maps dataset and cloud cover
dataset is set to 200 and 100 respectively. Following the lines of [5], an
early stopping criterion method is adopted to improve training efficiency and
avoid overfitting. In this case, the model will cease training if the
validation set does not increase in the last 20 epochs. For initializing the
learning rate a range of values, i.e 0.01, 0.005, 0.001, 0.0005 and 0.0001
have been tested and we empirically found that 0.001 was the optimal choice to
be set as the initial learning rate. We employ a learning rate scheduler that
reduced the learning rate to a tenth of the previous learning rate when the
validation loss does not increase for four epochs. The Adam optimizer is used
and the batch size is set to 6. The Mean Squared Error (MSE) is used as loss
function. Different number of Transformer layers, i.e. 1, 3, 6, 12 and 18
layers, have been examined. However, in all the conducted experiments, the
number of Transformer layer in both TransUNet and AA-TransUNet is set to 1 as
it was empirically found to be the optimal choice. All the compared models are
trained using Pytorch on NVIDIA Tesla P100 with 16GB of RAM. The Pytorch
implementation of the models is available at GitHub
111https://github.com/YangYimin98/AA-TransUNet.

###  III-C Model Evaluation

The results of five metrics including Mean Squared Error (MSE), Accuracy,
Precision, Recall and F1-score are reported to evaluate the performance of the
models. Similar to [5], in order to calculate true positives (TP), false
positives (FP), true negatives (TN) and false negatives (FN), each pixel of
the model output and the target image are converted to a boolean mask by
comparing with a fixed threshold. The threshold value is set to 0.5 in all
test cases.

##  IV Experiments

In this section, the details of the conducted experiments on these two
datasets, i.e. precipitation maps and cloud cover datasets are given. We use
the same datasets as those used in [5].

###  IV-A Precipitation maps nowcasting

The data is from the Royal Netherlands Meteorological Institute (Koninklijk
Nederlands Meteorologisch Instituut, KNMI) and it has previously been used in
[5]. It covers the precipitation information in the Kingdom of the Netherlands
and its neighboring countries from 2016 to 2019. The data was collected using
two C-band Doppler weather radar stations located in De Bilt and Den Helder.
The satellite collects precipitation data every five minutes and stores it in
the image format. The total number of precipitation maps is 420,000. Each raw
image is 756 x 700 in size and the value of each pixel represents the total
amount of precipitation collected per square kilometer in the past five
minutes and is stored in an integer format. The details of the dataset are
given in [5] and the dataset can be found at GitHub
222https://github.com/HansBambel/SmaAt-UNet.

The same pre-processing steps as those used in [5] are also applied here. All
the images in the precipitation dataset are cropped to 288 x 288 so that the
model can focus more on the original image containing the rain pixels.
Moreover, there is a high imbalance between pixels with and without rain, with
many images lacking raining pixels. Therefore, as in [5], we created two
datasets, i.e. NL-50 and NL-20, by filtering the original precipitation
dataset and choosing only the images with at least 50% and 20% of pixels
containing any amount of rain respectively. The information of the datasets
before and after cropping can be found in Table II. Fig. 2 shows an example of
NL-50 dataset over 30 minutes.

As a pre-processing step, we normalize both NL-50 and NL-20 datasets by
dividing them by the highest value of the NL-50 training set. Each data sample
is a sequence of 18 images. In order to make a fair comparison, similar to
[5], we use the first 60 minutes data (first 12 images) as the model input and
the last 30 minutes data as the target image for comparison with the model
predictions. Additionally, the dataset from 2016 to 2018 is used as training
set and the dataset of 2019 is used as test set. 10% of the training set is
randomly selected as a validation set to hint when to stop training.

The training set of NL-50 is used to train the models. The validation set of
NL-50 is used for model selection. Once all the models are trained, they are
tested on both test set of NL-50 as well as NL-20 datasets. In this way, the
trained models are tested on different conditions. In particular, by testing
the models on NL-20 test set, the generalization ability of the models are
examined.

Table II: The size of three precipitation datasets. Only NL-50 is used for training the models. The trained models are tested on both NL-50 and NL-20 test sets. Dataset | Rain | Training Set | Validation Set | Test Set
---|---|---|---|---
Index | Pixels | (2016-2018) | (%10 Training) | (2019)
Original | No Extraction | 314940 | 31494 | 105003
NL-20 | 20% pixels | 31674 | 3168 | 11276
NL-50 | 50% pixels | 5734 | 573 | 1557
Figure 2: Example of NL-50 dataset over 30 minutes.

###  IV-B Cloud cover nowcasting

The cloud cover data is introduced in [18] and previously used in [5]. The
cloud cover dataset is classified into 16 classes according to the height and
type of the cloud. This classification is computed with various visible and
infrared channel images shot by Meteosat Second Generation (MSG), a
geostationary satellite at longitude 0 degrees from 2017 and 2018 in France.
The data collected by the satellite every 15 minutes contains images of 3712 x
3712 size.

In the cloud cover image, each pixel might have 15 different values. The
details of the cloud cover states are reported in Table III. The data is
binarized depending on whether there is a cloud cover in the image. More
precisely, as in [5], values of 1 to 4 is converted to 0 (no cloud) and values
of 5 to 15 is converted to 1 (with cloud). Subsequently, the images are
cropped into the size of 256 x 256 based on the French boundaries [6].

Table III: The details of 15 possible values of every pixel, and the values after being binarized. Values | Cloud Cover States | Binarized
---|---|---
1 | Cloud-free land | 0
2 | Cloud-free sea | 0
3 | Snow over land | 0
4 | Sea ice | 0
5 | Very low clouds | 1
6 | Low clouds | 1
7 | Mid-level clouds | 1
8 | High opaque clouds | 1
9 | Very high opaque clouds | 1
10 | Fractional clouds | 1
11 | High semitransparent thin clouds | 1
12 | High semitransparent meanly thick clouds | 1
13 | High semitransparent thick clouds | 1
14 | High semitransparent above low or medium clouds | 1
15 | High semitransparent above snow/ice | 1

Fig. 3, shows a cloud cover data sample. Following the lines of [5], we
predict the cloud cover state for the last 90 minutes (last 6 images) based on
the first 60 minutes (first 4 images). Similar to [5], we use the data from
2017 and the first semester of 2018 as the training set and randomly select
$10\%$ of them as the validation set. The remaining part of the dataset in
2018 is then used as a test set. Unlike the previous precipitation dataset
processing method, the cloud cover dataset is not normalized since it only
contains images with binary values of 0 and 1.

Figure 3: An example of cloud cover data samples collected every 15 minutes.
First 4 images are used as input and the remaining 6 images as output.

##  V Results & Discussion

The obtained results of the examined models on the test set of three datasets,
i.e. NL-50, NL-20 and cloud cover as well as their corresponding analysis are
given in this section.

###  V-A Evaluation of the precipitation maps nowcasting

Table IV shows the obtained results on the NL-50 test set. From Table IV, it
can be observed that all models outperform the persistent baseline model. It
should be noted that as the weather data does not vary substantially in a
short period of time (input is only 1 hour for both precipitation forecasting
and cloud cover forecasting), the persistence model can already offer good
prediction results. However, Table IV shows that on the NL-50 test set, even
the worst-performing UNet based model, i.e. TransUNet, has an MSE value less
than half that of the persistence model.

The proposed AA-TransUNet model obtained the lowest MSE value compared to all
the other examined models. The obtained MSE of TransUNet model is 0.0124 which
is close to that of UNet and SmaAt-UNet. However, when Convolutional Block
Attention Modules (CBAM) and Depthwise-separable Convolutions (DSC) are
incorporated, the MSE score reduces to 0.0110, indicating that the proposed
model improves over classical TransUNet by 11%. In addition, note that after
replacing the ordinary convolution by DSC in TransUNet decoder, the number of
parameters of the decoder of TransUNet is greatly reduced. Moreover, besides
the obtained MSE values, the proposed model also performs well on the other
metrics.

Table IV: The obtained results of different metrics on NL-50 test set. The best results are presented in bold. Additionally, ↑ indicates that the higher the value of metrics, the better the performance, and ↓ indicates that the lower the value, the better the performance. Model | MSE↓ | Accuracy↑ | Precision↑ | Recall↑ | F1↑
---|---|---|---|---|---
Persistence | 0.0248 | 0.756 | 0.678 | 0.643 | 0.660
UNet | 0.0122 | 0.836 | 0.740 | 0.855 | 0.794
SmaAt-UNet | 0.0122 | 0.829 | 0.730 | 0.850 | 0.786
TransUNet | 0.0124 | 0.831 | 0.741 | 0.832 | 0.784
AA-TransUNet | 0.0110 | 0.837 | 0.758 | 0.816 | 0.790

An example of the ground truth sample images of NL-50 and NL-20 as well as the
AA-TransUNet predictions are shown in Fig. 4. As it can be seen the sample
image of NL-50 has significantly more rain pixels than that of NL-20. By
comparing the ground truth and the model prediction, one can observe that our
proposed model can accurately predict the concentrated locations of the rain
pixels. At the same time, the prediction results of the model are a little
blurry when compared to the ground truth. This is due to the fact that we use
MSE as the loss function for training the model, and MSE loss tends to perform
gradient descent in the direction of producing blurry images [30].

Figure 4: Example of precipitation nowcasting of AA-TransUNet model. (a) and
(b): The ground truth and the AA-TransUNet prediction for NL-50 dataset,
respectively. (c) and (d): The ground truth and the AA-TransUNet prediction
for NL-20 dataset, respectively.

We also test the models on the NL-20 test set. Table V shows that our proposed
AA-TransUNet model outperforms the other models on NL-20, despite the fact
that the density of the rain pixels on NL-20 is much lower than that of NL-50.

Table V: The obtained results of different metrics on NL-20 test set. The best results are presented in bold. Additionally, ↑ indicates that the higher the value of metrics, the better the performance of the model, and ↓ indicates the lower the values, the better the performance. Model | MSE↓ | Accuracy↑ | Precision↑ | Recall↑ | F1↑
---|---|---|---|---|---
Persistence | 0.0227 | 0.827 | 0.559 | 0.543 | 0.551
UNet | 0.0111 | 0.880 | 0.666 | 0.782 | 0.719
SmaAt-UNet | 0.0111 | 0.867 | 0.626 | 0.801 | 0.703
TransUNet | 0.0112 | 0.872 | 0.637 | 0.805 | 0.705
AA-TransUNet | 0.0106 | 0.880 | 0.671 | 0.760 | 0.711

Table V shows that the prediction results of all the models greatly exceed the
results of the persistence model. Furthermore, the performance of AA-
TransUNet, which obtains the lowest MSE value also achieves 5% improvement
compared to classical TransUNet.

Additionally, We also give a comparison of the prediction map on NL-20 and the
ground truth map. One can observe from Fig. 4(c, d) that despite the fact that
the density of the ground truth rain pixels is very low, our model still
predicts almost all of the contours of rain pixels.

Next, we analyze the influence of different number of Transformer layers on
the performance of the AA-TransUNet model for the NL-50 test set. As it can be
seen from Fig. 1, each Transformer layer consists of Multihead Self-Attention
(MSA) and Multi-Layer Perceptron (MLP) blocks. We have examined the cases
where there are 1, 3, 6, 12 and 18 blocks (corresponding to 1, 3, 6, 12 and 18
Transformer layers). According to Fig. 5, the model with one Transformer layer
(one MSA and one MLP block) performs the best on NL-50 validation set. This
observation is in agreement with the findings of [31, 32, 33], i.e. when the
dataset size is small, using Transformer with fewer number of blocks
(Transformer layers) can result in better performance.

Figure 5: The obtained MSE, Accuracy, Precision, Recall and F1-Score for NL-50
Validation Set using AA-TransUNet model with different Transformer layers.

###  V-B Evaluation of the cloud cover nowcasting

The above-mentioned models are trained and tested on the cloud cover dataset
and the obtained results are tabulated in Table VI. From Table VI, one can
observe the same performance pattern as that of precipitation maps dataset.
All the models have achieved results far exceeding the persistence model. At
the same time, AA-TransUNet has the lowest MSE value and the highest value in
accuracy and precision among all models. It is worth mentioning that the MSE
value of all the models on the cloud cover data set is higher than the value
previously obtained on the precipitation dataset since the cloud cover dataset
only contains binary values. Therefore, the value of the actual label is
sometimes very different from the predicted value, which consequently results
in higher MSE values.

Table VI: The obtained results of different metrics for cloud cover test set. The best results is presented in bold. Additionally, ↑ indicates that the higher the value of metrics, the better the performance of the model, and ↓ indicates the lower the value, the better the performance. Model | MSE↓ | Accuracy↑ | Precision↑ | Recall↑ | F1↑
---|---|---|---|---|---
Persistence | 0.1491 | 0.851 | 0.849 | 0.849 | 0.849
UNet | 0.0785 | 0.890 | 0.895 | 0.919 | 0.907
SmaAt-UNet | 0.0794 | 0.889 | 0.892 | 0.921 | 0.906
TransUNet | 0.0805 | 0.887 | 0.894 | 0.914 | 0.904
AA-TransUNet | 0.0784 | 0.891 | 0.900 | 0.913 | 0.906

An example of the AA-TransUNet prediction and the ground truth image of the
cloud cover data is shown in Fig. 6. The model has accurately identified all
the appearances of the cloud cover on the 256 x 256 size image.

Figure 6: Examples of cloud cover nowcasting using AA-TransUNet model. The
first and second row correspond to 15mins and 90mins ahead predictions
respectively. The four columns represent the Ground Truth, AA-TransUnet, TTD
uncertainty map and TTD prediction respectively.

###  V-C Uncertainty analysis

We quantify the uncertainty by means of Epistemic uncertainty which measures
the uncertainty of the model predictions. In theory, Epistemic uncertainty can
be obtained using Bayesian Neural Networks [34]. However, as in [34, 35], here
we use test time dropout (TTD) to approximate the Bayesian inference. In
particular, at test time, a posterior distribution is generated by running the
model for $k$ (in our case we set $k$=20) epochs for each target precipitation
map. The uncertainty map of the model is then measured by taking the variance
of the obtained $k$ precipitation maps (posterior sampled distribution) [34].
In addition, we also take the mean of the posterior sampled distribution that
can be considered as model prediction and is referred to as TTD prediction.
The obtained TTD uncertainty map and TTD prediction for could cover as well as
NL-20 datasets are shown in Fig. 6 and Fig. 7. Furthermore, Fig. 8 shows that
as the number of step ahead prediction increases, the uncertainty of the
proposed AA-TransUNet model also increases.

Figure 7: Examples of precipitation nowcasting on NL-20 using AA-TransUNet
model. The first and second row correspond to 5mins and 30mins ahead
predictions respectively. The four columns represent the Ground Truth, AA-
TransUnet, TTD uncertainty map and TTD prediction respectively. Figure 8: The
uncertainty quantification results of all the three datasets, i.e. NL-50,
NL-20 and cloud cover dataset. The uncertainty value is obtained by taking the
log10 of the mean of the uncertainty maps.

##  VI Conclusion

In this paper a novel Attention Augmented TransUNet model (AA-TransUNet) is
proposed for precipitation nowcasting tasks. We show that incorporating the
Convolutional Block Attention Modules (CBAM) and Depthwise-separable
Convolution(DSC) into the classical TransUNet gives rise to better performance
while significantly reducing the number of parameters in the decoder part of
the model. The applicability of the proposed model is shown on two nowcasting
tasks, i.e. precipitation as well as cloud cover nowcasting. The experimental
results demonstrate that the AA-TransUNet model outperforms the original
TransUNet model and other examined models. The implementation of our proposed
model including the trained models is available at Github:
https://github.com/YangYimin98/AA-TransUNet.

## References

  * [1] T. Hall, H. E. Brooks, and C. A. Doswell III, “Precipitation forecasting using a neural network,” _Weather and forecasting_ , vol. 14, no. 3, pp. 338–345, 1999.
  * [2] D. De Luca, “Rainfall nowcasting models for early warning systems; wong, tsw, ed,” 2013.
  * [3] P. Bauer, A. Thorpe, and G. Brunet, “The quiet revolution of numerical weather prediction,” _Nature_ , vol. 525, no. 7567, pp. 47–55, 2015.
  * [4] S. Mehrkanoon, “Deep shared representation learning for weather elements forecasting,” _Knowledge-Based Systems_ , vol. 179, pp. 120–128, 2019.
  * [5] K. Trebing, T. Stańczyk, and S. Mehrkanoon, “SmaAt-UNet: Precipitation nowcasting using a small attention-unet architecture,” _Pattern Recognition Letters_ , vol. 145, pp. 178–186, 2021.
  * [6] J. G. Fernandez and S. Mehrkanoon, “Broad-UNet: Multi-scale feature learning for nowcasting tasks,” _arXiv preprint arXiv:2102.06442_ , 2021.
  * [7] B. Yegnanarayana, _Artificial neural networks_. PHI Learning Pvt. Ltd., 2009.
  * [8] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [9] S. Scher, “Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning,” _Geophysical Research Letters_ , vol. 45, no. 22, pp. 12–616, 2018.
  * [10] K. Trebing and S. Mehrkanoon, “Wind speed prediction using multidimensional convolutional neural networks,” in _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2020, pp. 713–720.
  * [11] T. Stańczyk and S. Mehrkanoon, “Deep graph convolutional networks for wind speed prediction,” _arXiv preprint arXiv:2101.10041_ , 2021.
  * [12] D. Aykas and S. Mehrkanoon, “Multistream graph attention networks for wind speed forecasting,” _arXiv preprint arXiv:2108.07063_ , 2021.
  * [13] I. A. Abdellaoui and S. Mehrkanoon, “Symbolic regression for scientific discovery: an application to wind speed forecasting,” _arXiv preprint arXiv:2102.10570_ , 2021.
  * [14] O. Bilgin, P. Mąka, T. Vergutz, and S. Mehrkanoon, “TENT: Tensorized Encoder Transformer for temperature forecasting,” _arXiv preprint arXiv:2106.14742_ , 2021.
  * [15] J. G. Fernández, I. A. Abdellaoui, and S. Mehrkanoon, “Deep coastal sea elements forecasting using U-Net based models,” _arXiv preprint arXiv:2011.03303_ , 2020.
  * [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [18] L. Berthomier, B. Pradel, and L. Perez, “Cloud cover nowcasting with deep learning,” in _2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 2020, pp. 1–6.
  * [19] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [21] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [22] R. Nogueira, Z. Jiang, and J. Lin, “Investigating the limitations of transformers with simple arithmetic tasks,” _arXiv preprint arXiv:2102.13019_ , 2021.
  * [23] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, “Transunet: Transformers make strong encoders for medical image segmentation,” _arXiv preprint arXiv:2102.04306_ , 2021.
  * [24] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [25] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [26] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [27] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Advances in neural information processing systems_ , 2015, pp. 802–810.
  * [28] A. Kumar, T. Islam, Y. Sekimoto, C. Mattmann, and B. Wilson, “Convcast: An embedded convolutional lstm based architecture for precipitation nowcasting using satellite data,” _Plos one_ , vol. 15, no. 3, p. e0230114, 2020.
  * [29] S. Agrawal, L. Barrington, C. Bromberg, J. Burge, C. Gazen, and J. Hickey, “Machine learning for precipitation nowcasting from radar images,” _arXiv preprint arXiv:1912.12132_ , 2019.
  * [30] E. Denton and R. Fergus, “Stochastic video generation with a learned prior,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1174–1183.
  * [31] S. H. Lee, S. Lee, and B. C. Song, “Vision transformer for small-size datasets,” _arXiv preprint arXiv:2112.13492_ , 2021.
  * [32] B. Neyshabur, “Towards learning convolutions from scratch,” _arXiv preprint arXiv:2007.13657_ , 2020.
  * [33] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng, “Deepvit: Towards deeper vision transformer,” _arXiv preprint arXiv:2103.11886_ , 2021.
  * [34] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [35] P. Natekar, A. Kori, and G. Krishnamurthi, “Demystifying brain tumor segmentation networks: interpretability and uncertainty analysis,” _Frontiers in computational neuroscience_ , vol. 14, p. 6, 2020.


# Fine-Tuning the Retrieval Mechanism
for Tabular Deep Learning

Felix den Breejen
KAIST AI
<EMAIL_ADDRESS>
&Sangmin Bae
KAIST AI
<EMAIL_ADDRESS>
&Stephen Cha
KAIST AI
<EMAIL_ADDRESS>

&Tae-Young Kim
KT
<EMAIL_ADDRESS>
&Seoung Hyun Koh
KT
<EMAIL_ADDRESS>
&Se-Young Yun†
KAIST AI
<EMAIL_ADDRESS>

###### Abstract

While interests in tabular deep learning has significantly grown, conventional
tree-based models still outperform deep learning methods. To narrow this
performance gap, we explore the innovative retrieval mechanism, a methodology
that allows neural networks to refer to other data points while making
predictions. Our experiments reveal that retrieval-based training, especially
when fine-tuning the pretrained TabPFN model, notably surpasses existing
methods. Moreover, the extensive pretraining plays a crucial role to enhance
the performance of the model. These insights imply that blending the retrieval
mechanism with pretraining and transfer learning schemes offers considerable
potential for advancing the field of tabular deep learning.

##  1 Introduction

Tabular data has been widely utilized in the industry, leading to an increased
interest in the research field of deep learning for tabular data. However,
even with the significant advancements of deep learning in vision and language
domains, previous studies [12, 17, 25] have confirmed that tree-based methods,
such as Random Forests [3] and XGBoost [5], still prevail over deep learning
techniques in tabular domains. In order to narrow the performance gap, recent
research has introduced architectures that leverage the decision tree
algorithm [22, 15] or the self-attention mechanism [1, 14, 11, 26]. Moreover,
several works have focused on transfer learning to exploit the benefits of
deep neural networks by pretraining on extensive datasets [28, 2, 13, 29].

Among various methodologies, a retrieval mechanism, which retrieves data
points as the reference for prediction, has opened up new avenues in the field
of tabular deep learning [16, 26, 23, 13, 10]. These neural networks make
predictions by learning relationships between features and labels but also by
referring to other samples, which resembles models like decision trees and
nearest neighbors [7]. One specific application involves using input that
contains a support set,
${\mathcal{D}}=({\bm{X}}_{\mathit{support}},{\bm{y}}_{\mathit{support}},{\bm{X}}_{\mathit{query}})$,
to predict the label of a query sample.

Specifically, TabPFN [13] builds on prior-data fitted networks [18] to
pretrain on synthetically generated tabular prior datasets. Their goal is to
approximate the posterior predictive distribution of a test sample, which is
conditioned on the set of training samples (see Figure 1). While the trained
models exhibit strong performance on unseen real-world tabular datasets in a
single forward pass, this in-context learning [4] approach could have led to
overlooking the further potential of retrieval-based training, potentially
constraining the ability to transfer knowledge gained during pretraining.

In this paper, our comprehensive experiments, grounded on TabPFN and tabular
benchmarks [12], reveal crucial insights emphasizing the significance of
retrieval-based training. We demonstrate that the retrieval mechanism
outperforms previous methods by further fine-tuning the pretrained TabPFN
model. Moreover, a larger number of retrieved data points contributes to
higher performance gains by offering enhanced references for prediction. We
also find that pretraining on extensive datasets plays an important role when
comparing to models trained from scratch. Consequently, we believe that
integrating the retrieval mechanism with pretraining and transfer learning
schemes will significantly influence the future landscape of tabular deep
learning.

Figure 1: The overview of TabPFN training framework. Synthetically generated
datasets are randomly divided into support and query sets during the
pretraining phase, while train and test sets of tabular benchmarks are
utilized during the evaluation.

##  2 Related Work

###  2.1 Retrieval Mechanism in Tabular Data

Recent retrieval mechanism that implicitly or explicitly retrieves data points
as the reference for prediction have shown significant improvements for
tabular datasets. Non-Parameteric Transformers [16] take the entire dataset as
input and learn relationships through self-attention between datapoints.
Similarly, intersample attention from SAINT [26] associates the target row
with other rows in the table. On the other hand, there are explicit retrieval-
based models, like RIM [23] and TabR [10], that initially retrieve relevant
instances using search engine techniques and a retrieval module, respectively.
Subsequently, they utilize the features and labels of the selected rows to
make the final prediction for the target row. Our work is primarily inspired
by the TabPFN method [13], which let the token representations of test samples
attend feature and label representations of training sets. While they only
demonstrated the zero-shot evaluation performance, we fine-tuned pretrained
models on actual tabular benchmarks, surpassing the previous deep learning
approaches for tabular data.

###  2.2 Tabular Transfer Learning

Leveraging transfer learning methods from computer vision [6] and natural
language processing [8] is a promising approach for tabular deep learning.
Striving for generalization to unseen tables, various works in the tabular
deep learning literature have attempted to pretrain on a large collection of
different tables to exploit cross-table information [29, 27], harness biases
from LLMs [19, 27] and adopt self/semi-supervised learning [20, 19]. XTab [29]
processes entire tables using data featurizers and uses transformers for
knowledge transfer. CT-BERT [27], similar to XTab, employs a pretrained BERT
for natural language and translates tables into "[column name] is [value]"
strings. STUNT [20] uses few-shot semi-supervised learning on self-generated
tasks from unlabeled data, emphasizing column correlations. SPROUT [19] learns
from unlabeled tables through LLMs, focusing on columns correlated with target
labels, and applies in-context learning on generated prompts. The retrieval
mechanism as used in this paper, can be seen as an alternative way of enabling
transfer learning.

##  3 Methodology

TabPFN is a transformer-based architecture designed for zero-shot inference on
small classification tasks. It is built on a basic transformer architecture
that takes
$({\bm{X}}_{\mathit{support}},{\bm{y}}_{\mathit{support}},{\bm{X}}_{\mathit{query}})$
as input. Each observation in the dataset is tokenized during the embedding
process. For observation ${\bm{x}}_{i}\in{\mathbb{R}}^{d_{f}}$ with
corresponding class label $y_{i}\in{\mathbb{Z}}$ and $d_{f}$ features, the
embedding is calculated as follows:

$\displaystyle{\bm{z}}^{\mathit{support}}_{i}$
$\displaystyle={\bm{W}}_{x}{\bm{x}}_{i}+{\bm{w}}_{y}y_{i}$ (1)
$\displaystyle{\bm{z}}^{\mathit{query}}_{i}$
$\displaystyle={\bm{W}}_{x}{\bm{x}}_{i}$ (2)

with weight matrices ${\bm{W}}_{x}\in{\mathbb{R}}^{d\times d_{f}}$ and
${\bm{w}}_{y}\in{\mathbb{R}}^{d\times 1}$ for hidden dimension $d$. These
embedded tokens ${\bm{z}}_{i}$ are then processed through a basic transformer
where attention is applied between the observations, with the exception of
attention between pairs of inference observations. In the final layer, tokens
corresponding to the inference observations are passed through a
classification head. Figure 1 illustrates the overview of the TabPFN
architecture.

The features ${\bm{x}}_{i}$ are all variable-wise quantile transformed and
scaled by $\nicefrac{{d_{f}}}{{d_{f}^{i}}}$, where $d_{f}^{i}$ is the
effective number of features of observation $i$. The target $y_{i}$ is kept as
an unnormalized integer representing the class index. Categorical features are
not one-hot encoded before the quantile transformation. Under the pretraining
settings outlined in the TabPFN paper, the model accomodates up to $d_{f}=100$
features per observation and a maximum of 10 classes.

For fine-tuning TabPFN, we use the checkpoint from the pretrained TabPFN
model. Given a real-life test dataset
${\mathcal{D}}=({\bm{X}}_{\mathit{train}},{\bm{y}}_{\mathit{train}},{\bm{X}}_{\mathit{test}},{\bm{y}}_{\mathit{test}})$,
we create a distinct randomized 80% split of ${\bm{X}}_{\mathit{train}}$ and
${\bm{y}}_{\mathit{train}}$ to make a new training dataset
${\mathcal{D}}_{\mathit{train}}=({\bm{X}}_{\mathit{train}}^{\mathit{support}},{\bm{y}}_{\mathit{train}}^{\mathit{support}},{\bm{X}}_{\mathit{train}}^{\mathit{query}},{\bm{y}}_{\mathit{train}}^{\mathit{query}}$)
at every training step. We fine-tune on these splits and evaluate by
predicting on
$({\bm{X}}_{\mathit{train}},{\bm{y}}_{\mathit{train}},{\bm{X}}_{\mathit{test}})$.

##  4 Experiments

We evaluate our fine-tuned TabPFN on tabular benchmarks [12] consisting of
approximately 50 datasets categorized based on feature type (numerical or
numerical and categorical) and tasks (classification or regression). We focus
on the medium-size benchmarks containing about 10,000 observations and less
than a hundred features. To ensure consistency to benchmarks, we follow the
same procedures for data splitting, optimizer selection, learning rate
scheduling, early stopping, and accuracy or $R^{2}$ score normalization.

The hyperparameters are as follows. We use a learning rate of $1.0\times
10^{-5}$ and no weight decay for fine-tuning. When training from scratch, we
use a learning rate of $1.0\times 10^{-4}$ and weight decay of $1.0\times
10^{-5}$. We use a support set size of 10,000 or 1,000. For a support set size
of 10,000, the entire training dataset is processed through the transformer,
while for 1,000, we randomly sample 1,000 samples from the training set and
ensemble 10 times for each test observation. For all TabPFN variants, we run
default settings without random search, as the search showed little benefit.

###  4.1 Main Results

We compare our finetuned TabPFN with three tree-based methods (XGBoost [5],
Random Forest [3], and GradientBoostingTree [9]) and four neural network-based
methods (MLP, Resnet, SAINT [26], and FT-Transformer [11]). Since TabPFN
pretraining relies on synthetically generated data, we consider this a fair
comparison in the quantity of real-world data used.

Figure 2: Comparison between fine-tuning TabPFN and baselines using the full
10,000 samples.

Figure 2 presents the overall comparison of classification accuracy. Here, we
fine-tuned TabPFN with 10,000 number of retrieved samples. Utilizing default
settings for fine-tuning TabPFN outperforms all neural network-based tabular
data methods, even after extensive random hyperparameter searches for
baselines. Furthermore, it also outperforms the tree-based methods under
default settings, although tree-based methods tend to perform better on
average after about 10 random searches. Additional dataset-specific results
are available in Appendix B.

###  4.2 Ablation Study

Table 1 displays results for various TabPFN variations on mixed and numerical
classification tasks, referring to the settings in Figure 2. There is a
significant gap in normalized accuracy between fine-tuning on 10k samples and
learning from scratch, with Zeroshot performance falling in between. We
believe that these results imply the importance of fine-tuning phase after
pretraining.

Table 1: Results on TabPFN variants for classification accuracy and regression $R^{2}$ score. The pretraining and fine-tuning are conducted on synthetic datasets and actual tabular datasets, respectively. ICL denotes the in-context learning, Scratch denotes training from scratch. |  |  |  | classification | regression
---|---|---|---|---|---
method | pretrain | fine-tune | # retrieval | mixed | numerical | mixed | numerical
Scratch | ✗ | ✗ | 10k | 0.5008 | 0.6057 | 0.2199 | 0.2906
ICL | ✓ | ✗ | 1k | 0.4451 | 0.5421 | 0.0000 | 0.0000
ICL | ✓ | ✗ | 10k | 0.5989 | 0.6609 | 0.0000 | 0.0000
Fine-tune | ✓ | ✓ | 1k | 0.6221 | 0.7073 | 0.0677 | 0.0877
Fine-tune | ✓ | ✓ | 10k | 0.7598 | 0.8057 | 0.0353 | 0.0913

We also examine the size of the support set. TabPFN is pretrained on datasets
smaller than 1,000 samples. The classification ICL and Fine-tuning results
indicate that processing 10,000 samples is preferable to ensembling
1,000-sample segments. For large datasets with over one million samples, the
best approach appears to be utilizing as many observations as the GPU memory
allows.

In regression tasks, fine-tuned TabPFN underperforms, as the expected $R^{2}$
should be around 0.8 (see Appendix A). This results comes as no surprise, as
TabPFN is not pretrained for regression. The model’s embedding expects integer
class labels, but we provide quantile-transformed regression targets. This
suggests the need for pretraining on regression or exploring alternative
methods.

##  5 Conclusion

Our findings suggest that fine-tuning tabular data transformers pretrained
using a retrieval mechanism is a compelling avenue for improving neural
network based tabular data prediction. While research was predominantly
centered on training neural networks from scratch, our results suggest that
adopting the transfer learning paradigm holds significant potential.
Pretrained transformers, unlike small tabular data architectures, can scale
effectively in the size of the network. We imagine that large-scale companies
could take this concept by developing a billion parameter transformer fine-
tunable on downstream datasets. Such an approach, we believe, has the
potential to decisively surpass the performance of all tree-based methods.

Our analysis identifies several areas for future research, with one pressing
concern the scalability in the number of observations. Models like GPT-4 [21]
are limited by a context length of 32k tokens and CodeLlama [24] by context
length of 100k, but real-world tabular datasets can have millions of
observations. Further research areas include refining the architecture to
tailor it specifically to tabular data, exploring the limitations of synthetic
data, and enabling regression within the model. In essence, our study opens
doors to a range of exciting possibilities for enhancing tabular data
prediction through transfer learning, paving the way for general tabular data
solver.

## Acknowledgements

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School
Program(KAIST)). This work was also supported by the “KT-KAIST Open R&D”
project funded by KT Corporation.

## References

  * [1] Sercan Ö Arik and Tomas Pfister.  Tabnet: Attentive interpretable tabular learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679–6687, 2021.  Issue: 8.
  * [2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler.  SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption, March 2022.  arXiv:2106.15147 [cs].
  * [3] Leo Breiman.  Random forests.  Machine learning, 45:5–32, 2001.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [5] Tianqi Chen and Carlos Guestrin.  XGBoost: A Scalable Tree Boosting System.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, August 2016.  arXiv:1603.02754 [cs].
  * [6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [7] Thomas Cover and Peter Hart.  Nearest neighbor pattern classification.  IEEE transactions on information theory, 13(1):21–27, 1967.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Jerome H. Friedman.  Greedy Function Approximation: A Gradient Boosting Machine.  The Annals of Statistics, 29(5):1189–1232, 2001.  Publisher: Institute of Mathematical Statistics.
  * [10] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko.  Tabr: Unlocking the power of retrieval-augmented tabular deep learning.  arXiv preprint arXiv:2307.14338, 2023.
  * [11] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  Advances in Neural Information Processing Systems, 34:18932–18943, 2021.
  * [12] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux.  Why do tree-based models still outperform deep learning on tabular data?  In Advances in Neural Information Processing Systems (NeurIPS). arXiv, July 2022.  arXiv:2207.08815 [cs, stat].
  * [13] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter.  TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.  In International Conference on Learning Representations (ICLR). arXiv, October 2022.  arXiv:2207.01848 [cs, stat].
  * [14] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin.  Tabtransformer: Tabular data modeling using contextual embeddings.  arXiv preprint arXiv:2012.06678, 2020.
  * [15] Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie-Yan Liu.  Deepgbm: A deep learning framework distilled by gbdt for online prediction tasks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 384–394, 2019.
  * [16] Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal.  Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning.  In Advances in Neural Information Processing Systems, volume 34, pages 28742–28756. Curran Associates, Inc., 2021.
  * [17] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White.  When Do Neural Nets Outperform Boosted Trees on Tabular Data?, May 2023.  arXiv:2305.02997 [cs, stat].
  * [18] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter.  Transformers can do bayesian inference.  arXiv preprint arXiv:2112.10510, 2021.
  * [19] Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, and Jinwoo Shin.  Semi-supervised tabular classification via in-context learning of large language models.  In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
  * [20] Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin.  STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables.  In International Conference on Learning Representations (ICLR). arXiv, March 2023.  arXiv:2303.00918 [cs].
  * [21] OpenAI.  Gpt-4 technical report, 2023.
  * [22] Sergei Popov, Stanislav Morozov, and Artem Babenko.  Neural oblivious decision ensembles for deep learning on tabular data.  arXiv preprint arXiv:1909.06312, 2019.
  * [23] Jiarui Qin, Weinan Zhang, Rong Su, Zhirong Liu, Weiwen Liu, Ruiming Tang, Xiuqiang He, and Yong Yu.  Retrieval & interaction machine for tabular data prediction.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1379–1389, 2021.
  * [24] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code, 2023.
  * [25] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  Information Fusion, 81:84–90, May 2022.
  * [26] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein.  SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training.  In NeurIPS Workshop: Table Representation Learning. arXiv, June 2021.  arXiv:2106.01342 [cs, stat].
  * [27] Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao.  Ct-bert: Learning better tabular representations through cross-table pre-training.  arXiv preprint arXiv:2307.04308, 2023.
  * [28] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar.  VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain.  In Advances in Neural Information Processing Systems, volume 33, pages 11033–11043. Curran Associates, Inc., 2020.
  * [29] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran.  XTab: Cross-table Pretraining for Tabular Transformers.  In International Conference on Learning Representations (ICLR), June 2023.

##  Appendix A Normalized Graphs for Regression from Scratch

##  Appendix B Dataset Graphs