BC | 35.49 | 0.057 | 21.48 | 20.08 | 0.0556 | 35.48 | 35.14 | 0.0594 | 24.27
BPPO | 104.28 | 0.0746 | -29.67 | 54.83 | 0.0518 | -3.02 | 16.97 | 0.0598 | 42.83
DiffPhyCon-lite | -0.35 | 0.0782 | 78.56 | 2.92 | 0.0779 | 74.97 | -10.89 | 0.0760 | 86.93
DiffPhyCon | 173.27 | 0.1636 | -9.66 | 150.21 | 0.1269 | -23.32 | 101.25 | 0.1025 | 1.21

Figure 13: More examples of 2D simulation visualizations controlled by our
method.

###  H.7 Myopic failure modes of SAC

To further confirm the advantage of diffusion models over the baseline SAC in
alleviating the myopic failure modes, we perform additional 2D experiments.
Specifically, when solving the jellyfish movement problem using SAC, we
incorporated a constraint to achieve periodic motion of the jellyfish in the
reward function, namely, we added a hyperparameter as the weight of the term
$d(w_{T},w_{0})$ in Eq. (19). The challenge caused by the periodicity
condition is that the average speed and $R(\mathbf{w})$ are calculated over
the whole horizon while the periodicity condition is only evaluated at the
final time step of the horizon. Thus, when the control objective consists of
all three terms, it requires global optimization over the whole horizon to
balance different terms. In Table 18, we present the original results
(weight=1000, corresponding to Table 3) and the results after reducing the
weighting of this term (weight=100). It can be observed that while the speed
of the jellyfish increases (though still not reaching the results of
DiffPhyCon), the $R(\mathbf{w})$ and constraint term for periodicity sharply
rise. From these results and the additional results in Table 16 and Table 17
where we test the performance of SAC under different weights $\zeta$ of
$R(\mathbf{w})$, we conclude that SAC struggles to provide satisfactory
policies that simultaneously satisfy multiple conflicting constraints with
different time coverage, resulting in a myopic failure model. The reason may
be that it is difficult to estimate the effect of each term in the control
objective in each time step in the iterative planning fashion [1], which this
issue could be well addressed by our diffusion models with an inherent nature
of global optimization.

Table 18: Results of myopic failure modes of SAC on 2D jellyfish movement control. $\lambda_{0}$ | Average speed ($\bar{v}$) | $R(\mathbf{w})$ | objective $\mathcal{J}$ | periodicity error
---|---|---|---|---
SAC (weight=100) | 59.82 | 0.0263 | -33.48 | 0.35187
SAC (weight=1000) | -166.96 | 0.0112 | 178.14 | 0.02299
DiffPhyCon | 279.87 | 0.2058 | -74.11 | 0.01157

##  Appendix I Efficiency Comparison

We test the inference (control) efficiency of DiffPhyCon and baselines on both
1D Burgers’ equation and 2D jellyfish movement tasks on a single Tesla-V100
GPU with 8 CPUs. On the 1D Burgers’ equation task, we test the efficiency of
partial observation and full control; on the 2D jellyfish movement task, we
test the efficiency of full observation and full control. The average
inference times over all test samples are listed in Table 19 and Table 20,
respectively. These results reveal that although DiffPhyCon is not as
efficient as reinforcement learning methods like SAC, BC, and BPPO, it has
competitive efficiency compared to SL, and MPC. In particular, by using the
fast sampling method DDIM [62], DiffPhyCon could be accelerated significantly
by reducing the number of samplings.

Table 19: Efficiency comparison on 1D Burgers’ equation on a Tesla-V100 GPU with 8 CPUs. Methods | Time (s)
---|---
DiffPhyCon-lite | 21.13
DiffPhyCon | 58.97
DiffPhyCon-DDIM (8 sampling steps) | 0.53
PID | 3.61
SL | 74.85
SAC | 0.11
BC | 1.22
BPPO | 0.82
Table 20: Efficiency comparison on 2D jellyfish movement on a Tesla-V100 GPU with 8 CPUs. Methods | Time (s)
---|---
DiffPhyCon | 252.2
DiffPhyCon-DDIM (8 sampling steps) | 12.6
MPC | 1401.7
SL | 133.5
SAC | 0.2
BC | 0.99
BPPO | 1.08

##  Appendix J Effect of Hyperparameters

###  J.1 Effect of $\gamma$

Table 21: Results of different $\gamma$ in DiffPhyCon on 2D jellyfish movement control task. $\gamma_{1}$ | $\xi$ | Average speed ($\bar{v}$) | $R(\mathbf{w})$ | objective $\mathcal{J}$
---|---|---|---|---
0.6 | 0.4 | 410.6 | 0.2581 | -152.51
0.7 | 0.3 | 279.87 | 0.2058 | -74.11
0.8 | 0.2 | 197.18 | 0.1312 | -65.99
0.9 | 0.1 | 76.97 | 0.0741 | -2.84
1.0 | 0 | 95.04 | 0.0746 | -20.47
1.1 | -0.1 | 81.41 | 0.0742 | -7.21
1.2 | -0.2 | 84.56 | 0.0736 | -10.93
1.3 | -0.3 | 65.12 | 0.0725 | 7.38
1.4 | -0.4 | 65.02 | 0.0734 | 8.43
1.5 | -0.5 | 64.07 | 0.0752 | 11.1
Figure 14: Results of different $\gamma$ in DiffPhyCon on 2D jellyfish
movement control task.

1D Burgers’ Equation control. In the 1D Burgers control task, the prior
reweighting is intrinsically unnecessary. The evaluated control target follows
the same distribution as the training set, and the samples in the training set
are defined to be the optimal control. Therefore, the generated control
sequences following $p(w|u_{0},u_{T})$ are already the (near) optimal
distribution which alleviates the need for prior reweighting.

We tested different scheduling of the prior reweighting $\gamma$ and found
their performance similar. Finally, we use the same cosine schedule as that in
the DDPM noise for clarity. Our experiment results also revealed that the
performance of DiffPhyCon in 1D Burgers equation control is insensitive to the
prior reweighting intensity $\gamma$ as we reported in Table 1. Besides, we
conducted a more comprehensive experiment on the prior reweighting intensity
$\gamma$ in all three settings (FO-PC; PO-FC; PO-PC) as in Table 22, 23, and
24. The results show that prior reweighting does not significantly impact the
performance of DiffPhyCon in 1D Burgers control task, where State MSE denotes
the deviation between generated $u$ and the ground truth $u_{\text{gt}}$ given
by the numerical simulator based on generated $f$.

Table 22: Results of different $\gamma$ in DiffPhyCon on FO-PC 1D Burgers equation control task. $\gamma$ | 0.0 | 0.3 | 0.5 | 0.7 | 1.0
---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.00038 | 0.00037 | 0.00037 | 0.00037 | 0.00037
State MSE | 0.00063 | 0.00043 | 0.00034 | 0.00028 | 0.00025
Table 23: Results of different $\gamma$ in DiffPhyCon on PO-FC 1D Burgers equation control task. $\gamma$ | 0.0 | 0.3 | 0.5 | 0.7 | 1.0
---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.01159 | 0.01155 | 0.01152 | 0.01148 | 0.01139
State MSE | 0.02643 | 0.02642 | 0.02642 | 0.02641 | 0.02639
Table 24: Results of different $\gamma$ in DiffPhyCon on PO-PC 1D Burgers equation control task. $\gamma$ | 0.0 | 0.3 | 0.5 | 0.7 | 1.0
---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.00493 | 0.00493 | 0.00493 | 0.00493 | 0.00494
State MSE | 0.00675 | 0.00667 | 0.00665 | 0.00664 | 0.00663

2D jellyfish movement control. Performance of DiffPhyCon is determined by the
hyperparameter $\gamma$. Since diffusion models denoise gradually, we use a
varying sequence of $\gamma=\\{\gamma_{k}\\}_{k=1}^{K}$ to subtract prior in
DiffPhyCon. Specifically, the schedule of $\gamma$ is set as
$\gamma_{k}=1-\xi\cdot\beta_{K-k},k=1,\cdots,K$, where $\xi$ is a fixed
coefficient to control the scale of $\gamma$ and
$\beta=\\{\beta_{k}\\}_{k=0}^{K}$ is the schedule of variances of noise in
DDPM [18]. In our implementation, we use the sigmoid schedule of $\beta$ [24].
The total number of inference steps is $K=1000$. Thus we only need to tune
$\xi$ to examine the effect of $\gamma$. When $\xi<0$, DiffPhyCon is prone to
restrict $\mathbf{w}$ within its prior distribution of training dataset in
inference. When $\xi>0$, DiffPhyCon is more likely to generate new kinds of
$\mathbf{w}$ beyond training ones. When $\xi=0$, DiffPhyCon degenerates to
DiffPhyCon-lite. In 2D experiments, We set default $\xi=0.3$ and the
corresponding $\gamma_{1}=0.7$ as we empirically find this value performs well
and steadily. We present the performance of DiffPhyCon on the 2D jellyfish
movement control task under different $\gamma$ in Figure 14 and Table 21. We
can observe that the performance increases along with decreasing of
$\gamma_{1}$. When $\gamma_{1}<0.6$, invalid generated control sequences
emerge because the prior is largely overlooked. Thus the valid interval for
$\gamma$ of prior reweighting on this task is $[0.6,1.0]$. It is interesting
to find that when $\gamma_{1}>1$, the performance decreases. This may be
caused by the strict constraint of the prior distribution of
$p(\mathbf{w},\mathbf{c})$, which results in generating control sequences
similar to those from training datasets and thus not good.

###  J.2 Effect of $\lambda$

1D Burgers’ Equation control. In 1D Burgers control, DiffPhyCon uses guidance
conditioning since it empirically performs better than explicit guidance as
shown in Table 25. Therefore, our model does not use $\lambda$ to generate the
optimal control. However, when we consider the energy cost as in Figure 3, we
use explicit guidance to control the magnitude of control signals as it is a
more natural way to incorporate the objective
$\lambda\mathcal{J}_{\text{energy}}$ into the overall diffusion objective
$E_{\theta}^{(\gamma)}(u,w,c)+\lambda\mathcal{J}_{\text{energy}}$ where the
condition $c$ is $u_{T}=u_{d}$.

The guidance intensity uses a cosine schedule [44] where
$\lambda=\lambda_{0}\beta_{k},~{}k=1,2,...,K$ where $\beta_{k}$ starts from
$\beta_{1}=0.001$ to $\beta_{K}=1$. The schedule decreases as the diffusion
step decreases during inference, where the gradually decreasing guidance
intensity provides strong guides in the initial stage while reducing the
impact on the near-clean samples.

In such a setting, varying $\lambda$ changes how the generated samples are
shifted towards lower energy regions, but an excessively large $\lambda$
causes the sample to deviate from the physically feasible region. Results in
Table 26, 27, and 28 show that the behavior of DiffPhyCon under different
$\lambda$ is expected. Note that “physically feasible” samples here are
samples with lower $\mathcal{J}_{\text{actual}}$.

Table 25: Results of two different guidance type of DiffPhyCon in 1D Burgers equation control task. Guidance Type | FO-PC | PO-FC | PO-PC
---|---|---|---
Explicit Guidance | 0.02789 | 0.03257 | 0.05584
Conditioning Guidance | 0.00037 | 0.01139 | 0.00494
Table 26: Results of different $\lambda$ for $\mathcal{J}_{\text{energy}}$ of
DiffPhyCon in FO-PC 1D Burgers equation control task.

$\lambda$ | 0 | 10 | 100 | 500 | 1000 | 10000 | 100000 | 1000000
---|---|---|---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.00037 | 0.00037 | 0.00037 | 0.00036 | 0.00037 | 0.00147 | 0.02202 | 0.06444
State MSE | 0.00025 | 0.00025 | 0.00024 | 0.00024 | 0.00024 | 0.00083 | 0.00802 | 0.01838
$\mathcal{J}_{\text{energy}}$ | 1320.98206 | 1309.31104 | 1237.06958 | 1061.67761 | 946.07617 | 584.32098 | 225.50281 | 38.25005

Table 27: Results of different $\lambda$ for $\mathcal{J}_{\text{energy}}$ of
DiffPhyCon in PO-FC 1D Burgers equation control task.

$\lambda$ | 0 | 10 | 100 | 500 | 1000 | 10000 | 100000 | 1000000
---|---|---|---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.01139 | 0.01152 | 0.01252 | 0.01906 | 0.02100 | 0.04411 | 0.08472 | 0.11934
State MSE | 0.02639 | 0.02641 | 0.02661 | 0.02919 | 0.02990 | 0.03857 | 0.05472 | 0.06471
$\mathcal{J}_{\text{energy}}$ | 862.98633 | 860.05164 | 839.11932 | 770.45819 | 704.68665 | 311.21915 | 69.87619 | 8.93373

Table 28: Results of different $\lambda$ for $\mathcal{J}_{\text{energy}}$ of
DiffPhyCon in PO-PC 1D Burgers equation control task.

$\lambda$ | 0 | 10 | 100 | 500 | 1000 | 10000 | 100000 | 1000000
---|---|---|---|---|---|---|---|---
$\mathcal{J}_{\text{actual}}$ | 0.00494 | 0.00500 | 0.00549 | 0.00794 | 0.01426 | 0.02411 | 0.03979 | 0.08153
State MSE | 0.00663 | 0.00676 | 0.00768 | 0.01048 | 0.01525 | 0.02766 | 0.03635 | 0.05459
$\mathcal{J}_{\text{energy}}$ | 1314.90698 | 1297.32849 | 1192.89026 | 945.77179 | 793.28442 | 331.45758 | 108.39325 | 16.13734

2D jellyfish movement control. In our jellyfish movement control task, we use
explicit guidance with $\lambda$ involved. Since diffusion models denoise
gradually, similar to the hyperparameter $\gamma$, we use a varying sequence
of $\lambda=$ $\lambda_{k}$$=\lambda_{0}\times$ $\beta_{k}$, where $\beta_{k}$
is the schedule of variances of noise in DDPM [2] increasing monotonically
from $\beta_{1}=0.0003$ to $\beta_{K}=1$. The motivation of this schedule is
that the confidence of gradient estimation becomes stronger along with the
denoising process from $k=K$ to $k=1$. The results are shown in the following
table. We can observe that as $\lambda_{0}$ (which determines the $\lambda$
schedule) increases from 0 to 0.6, the performance of DiffPhyCon improves
consistently. Compared to Table 3, DiffPhyCon always outperforms baselines for
a wide range of $0.1\leq\lambda_{0}\leq 0.5$. When $\lambda_{0}\geq 0.6$, the
results are no longer valid since the generated control sequences may be
unfeasible. This reflects that $\lambda$ should not be overly large to make it
safe, which is a consistent conclusion with our previous analysis.

Table 29: Results of different $\lambda$ on 2D jellyfish movement control. $\lambda_{0}$ | Average speed ($\bar{v}$) | $R(\mathbf{w})$ | objective $\mathcal{J}$
---|---|---|---
0.6 | - | - | -
0.5 | 362.82 | 0.192 | -170.79
0.4 | 322.04 | 0.182 | -140.04
0.3 | 279.87 | 0.206 | -74.11
0.2 | 192.77 | 0.114 | -78.57
0.1 | 103.89 | 0.091 | -12.32
0 | -86.69 | 0.042 | 128.67

##  Appendix K Extensions to Finer-grained Jellyfish Movement Control Task

To test the scalability of DiffConPDE in handling high-dimensional and complex
dynamic systems, we extend the jellyfish movement control experiment from
rigid boundaries of wings to soft ones.

###  K.1 Experimental Setting

Similar to previous setups in Appendix E, this extension aims to control the
movement of a flapping jellyfish with two wings in a 2D fluid field where
fluid flows with constant initial speed. However, in this task, we parametrize
the jellyfish’s boundary as the coordinate change ($\Delta x$, $\Delta y$) for
each cell within the boundary, which serves as the control sequence
$\mathbf{w}$. Undoubtedly, this is a high-dimensional and complex control
task. Firstly, the control sequence $\mathbf{w}$ is elevated to three
dimensions. In addition to ensuring optimization towards the control
objective, maintaining consistency in the movement of different cells within
the jellyfish boundary is also crucial. Bearing this in mind, we have chosen
to employ DiffConPDE-lite for this experimental setup to minimize potential
disruptions to the jellyfish boundary. Regarding the PDE state, the experiment
adopts the full observation setup. All other settings remain the same with
Appendix E.

###  K.2 Data Generation

In this extension, we leverage data previously generated under the vanilla
setting. Additionally, we utilize opening angles $\Theta$, The coordinates of
the rotation center $\mathbf{h}$, and the boundary mask to generate the
required control sequence $\mathbf{w}$. We represent $\mathbf{w}$ as a tensor
of shape $[\tilde{T},2,64,64]$. Each cell has two features representing the
changes in the two coordinates. For each trajectory, $\mathbf{w}$ is generated
as follows: Firstly, for the initial $\mathbf{w}_{0}$ and final
$\mathbf{w}_{T}$, we set all elements in the tensor to $0$. Then, for each
$w_{[1,T-1]}$ corresponding to the cells within the boundary, we first obtain
the relative coordinates $c=(x,y)$ of the cell with respect to the rotation
center using the boundary mask and the coordinates of the rotation center
$\mathbf{h}$. Afterwards, we construct the rotation matrix $M_{t}(\Theta_{t})$
at different time steps using the opening angle $\Theta_{[1,T-1]}$:

$M_{t}(\Theta_{t})=\left[\begin{array}[]{cc}\cos(\Theta_{t})&-\sin(\Theta_{t})\\\
\sin(\Theta_{t})&\cos(\Theta_{t})\end{array}\right].$

The coordinate change $w_{t}$ can be obtained using the following formula:

$w_{t}=cM_{t}-c$

Additionally, for cells that are not within the boundary, the corresponding
$w_{[1,T-1]}$ is also directly set to $(0,0)$. Therefore, all the data
involved in this experiment are as follows:

  * •

Coordinate change of cells $\mathbf{w}$: shape $[\tilde{T},2,64,64]$. For each
step, we save the coordinate change of each cell within the boundary.

  * •

PDE states $u$: shape $[\tilde{T},3,64,64]$. For each step, we save the states
of the fluid field consisting of velocity in $x$ and $y$ directions and
pressure. To save space, we downsample the resolution from $128\times 128$ to
$64\times 64$.

    * –

velocity: $[\tilde{T},2,64,64]$.

    * –

pressure: $[\tilde{T},1,64,64]$.

  * •

opening angels $\Theta$: shape $[\tilde{T}]$. For each step, we save the
opening angle in radians.

  * •

boundary mask and offsets $b$: $[\tilde{T},3,64,64]$. For each step, we save
the mask of merged wings with half coordinates of boundary points and offsets
in both $\mathbf{x}$ and $y$ directions. The resolution is $64\times 64$,
compatible with that of the states.

    * –

mask: $[\tilde{T},1,64,64]$.

    * –

offsets: $[\tilde{T},2,64,64]$.

  * •

force: shape $[\tilde{T},2]$. For each step, the simulator outputs the
horizontal and vertical force from the fluid to the jellyfish. The horizontal
force is regarded as a thrust to jellyfish if positive and a drag otherwise.

###  K.3 Model

Similar to the vanilla setting, this experiment also employs U-net as the
backbone for the diffusion model. The U-net architecture remains consistent.
The input of the U-net includes PDE state $\mathbf{u}([T,3,64,64])$, control
sequence $\mathbf{w}([T,2,64,64])$, initial boundary mask and offset
$([1,3,64,64])$. It is worth noting that to align the initial mask and offset
with others in terms of shape, we expand them along the time dimension,
resulting in the final shapes of $[T,3,64,64]$. Therefore, the shape of the
input to the model is $[T,8,64,64]$, and the output of the model includes both
noise of the predicted state and the control sequence, with a shape of
$[T,5,64,64]$.

###  K.4 Training, Inference, and Evaluation

##### Training.

During the training phase, the diffusion-flow task adopts a similar setup,
utilizing the Mean Squared Error (MSE) between the model predictions and
Gaussian noise as the loss function (Eq.4). The batch size is chosen as 16,
and the training involves 180,000 iterations. The learning rate starts at
$1\times 10^{-4}$ and increases by a factor of 0.1 at the 50,000th and
150,000th iterations. More training details are presented in Table 30.

Table 30: Hyperparameters of network architecture and training for the Finer-grained Jellyfish Boundary Control Task. Hyperparameter name | full observation
---|---
Batch size | 16
Optimizer | Adam
Learning rate | 0.001
Loss function | MSE

##### Inference.

The pipeline of inference is shown in Figure 15. Both diffused variables
$\mathbf{u}_{[0,T]}$ and $\mathbf{w}_{[0,T]}$ are initialized from Gaussian
prior and gradually denoised from denoising step $k=1000$ to $k=0$ based on
denoising networks and guidance. Similar to the vanilla setting, due to the
introduction of the initial boundary mask and offset as additional inputs, the
number of channels in the model’s input and output are inconsistent. For
guidance, we fix $\lambda=5\times 10^{-5}$ and we also utilized a surrogate
model to approximate the force of fluid on the jellyfish. In the $k$-th
denoising step, the inputs of this model include the noise-free state
$\hat{\mathbf{u}}_{[0,T],k}$, $\hat{\mathbf{w}}_{[0,T],k}$ estimated from
$\mathbf{u}_{[0,T],k}$,$\mathbf{w}_{[0,T],k}$ by Eq. (6), initial boundary
mask and offset. The model output is the force. The remaining aspects such as
model architecture, training details, etc., remain the same. As for the
regularization term $R(\mathbf{w})$, we approximate the opening angle of the
jellyfish’s wings at time $t$ using the change of the coordinate $(\Delta
x_{t}$, $\Delta y_{t})$ and the initial coordinates $(x,y)$ of each cell to
achieve regularization. Firstly, we utilize the following formula to
approximate the angle change of each cell:

$\hat{\theta_{t}}=\sqrt{(\Delta x_{t}^{2}+\Delta y_{t}^{2})/(x^{2}+y^{2})}$

Then, we calculate the average value of the angles of different cells, which
serves as an approximation of the angle between the two wings of the jellyfish
at time $t$, denoted as $\hat{\Theta}_{t}$. It is evident that optimizing this
term intuitively minimizes the coordinate changes of each cell, aligning with
the objective of regularization. The control objective $\mathcal{J}$ in Eq.
(19) is computed as a summation of force and $R(\hat{\mathbf{w}}_{[0,T]})$,
and we fix $\zeta=1000$. Then the gradients of the objective $\mathcal{J}$ in
terms of $\hat{\mathbf{u}}_{[0,T]}$ and $\hat{\mathbf{w}}_{[0,T]}$ are
computed and used in guidance. Because this experiment selects DiffConPDE-
lite, all these gradients are directly subtracted from
$[\mathbf{u}_{[0,T],k},\mathbf{w}_{[0,T],k}]$ to generate
$[\mathbf{u}_{[0,T],k-1},\mathbf{w}_{[0,T],k-1}]$.

Figure 15: Inference of our finer-grained jellyfish boundary control task.

##### Evaluation

The inference outputs coordinate change for each cell of $T=20$ steps for 50
testing samples. To align with the evaluation metrics of the vanilla setting,
we first need to calculate the opening angle $\Theta_{t}$ using the coordinate
change of the cells at time $t$. Firstly, at time $t\in[1,T-1]$, for each
cell, we obtain the rotated coordinates $(x^{\prime}_{t},y^{\prime}_{t})$
using the initial coordinates $(x,y)$ and the coordinate change $(\Delta
x_{t},\Delta y_{t})$. Then, combined with the coordinates of the rotation
center $\mathbf{h}$, we can respectively obtain the radial distance of the
cell relative to the rotation center before and after rotation, denoted as $r$
and $r^{\prime}$. Then, according to the law of cosines, we can obtain the
rotation angle $\delta\theta$ of the cell at time $t$ compared to the initial
time:

$|\Delta\theta_{t}|=\frac{{r^{\prime}_{t}}^{2}+r_{t}^{2}-(\delta
x_{t}^{2}+\delta y_{t}^{2})}{2r^{\prime}_{t}r_{t}}.$

Afterward, we take the average of the angle changes of all cells within the
boundary as the angle change of the jellyfish’s two wings at time $t$, denoted
as $\Delta\Theta_{t}$. Finally, based on the average value of the flow in the
$x$-direction at time $t$, the opening or closing status of the jellyfish
relative to the initial moment is determined. If it is negative, it indicates
opening; otherwise, it suggests closing. This process allows us to obtain the
specific angle change for each time step compared to the first moment:

$\displaystyle\Theta_{t}=\begin{cases}\Theta_{0}+|\Delta\Theta_{t}|&\Delta\bar{x}_{t}<0\\\
\Theta_{0}-|\Delta\Theta_{t}|&\Delta\bar{x}_{t}>0\\\
\Theta_{0}&\Delta\bar{x}_{t}=0.\end{cases}$

After obtaining the theta sequence, we can follow the vanilla evaluation
procedure.

###  K.5 Results

We present the results in Table 31. From the table, it can be observed that
our method optimized the objective $\mathcal{J}$ to 84.31 for this task.
Compared to the results based on the original setting in Section 4.2, our
method is still competitive with baselines (shown in Table 3) in this more
challenging setting. An example of optimized soft boundary shapes is
illustrated in Figure 16. These results demonstrate the good scalability of
our method when facing higher-dimensional and complex control tasks.

Table 31: Performance of DiffPhyCon-lite on the finer-grained jellyfish boundary control task. Average speed ($\bar{v}$) | $R(\mathbf{w})$ | objective ($\mathcal{J}$)
---|---|---
-33.91 | 0.0504 | 84.31
Figure 16: Inference results of soft boundaries on the finer-grained jellyfish
control task.

##  Appendix L Future Work

Moving forward, we plan to expedite the inference process by drawing insights
from relevant prior works, such as [57, 40, 32]. Additionally, we aim to build
upon our existing trajectory knowledge to generalize future trajectories.
Furthermore, we intend to incorporate solver interactions into our training
methodology, enhancing the model’s adaptability and effectiveness.

##  Appendix M Social Impact Statements

The method we propose offers a means to actively interact with the physical
world and achieve specific control objectives. This approach presents
significant opportunities for various domains, including fluid control,
robotic control, controllable nuclear fusion, and more. However, it is
imperative to remain vigilant about potential negative consequences that may
arise from this technology to prevent its application to unethical or illegal
control issues.


# XAI meets LLMs: A Survey of the Relation between Explainable AI and Large
Language Models

Erik Cambria
School of Computer Science and Engineering
Nanyang Technological University, Singapore
<EMAIL_ADDRESS>
Lorenzo Malandri
Dept. of Statistics and Quantitative Methods
University of Milano-Bicocca, Milan, Italy
<EMAIL_ADDRESS>
Fabio Mercorio
Dept. of Statistics and Quantitative Methods
University of Milano-Bicocca, Milan, Italy
<EMAIL_ADDRESS>
Navid Nobani
Dept. of Statistics and Quantitative Methods
University of Milano-Bicocca, Milan, Italy
<EMAIL_ADDRESS>
Andrea Seveso
Dept. of Statistics and Quantitative Methods
University of Milano-Bicocca, Milan, Italy
<EMAIL_ADDRESS>

###### Abstract

In this survey, we address the key challenges in Large Language Models (LLM)
research, focusing on the importance of interpretability. Driven by increasing
interest from AI and business sectors, we highlight the need for transparency
in LLMs. We examine the dual paths in current LLM research and eXplainable
Artificial Intelligence (XAI): enhancing performance through XAI and the
emerging focus on model interpretability. Our paper advocates for a balanced
approach that values interpretability equally with functional advancements.
Recognizing the rapid development in LLM research, our survey includes both
peer-reviewed and preprint (arXiv) papers, offering a comprehensive overview
of XAI’s role in LLM research. We conclude by urging the research community to
advance both LLM and XAI fields together.

Explainable Artificial Intelligence, Interpretable Machine Learning, Large
Language Models, Natural Language Processing

##  1 Introduction

The emergence of LLMs has significantly impacted Artificial Intelligence (AI),
given their excellence in several Natural Language Processing (NLP)
applications. Their versatility reduces the need for handcrafted features,
enabling applications across various domains. Their heightened creativity in
content generation and contextual understanding contributes to advancements in
creative writing and conversational AI. Additionally, extensive pre-training
on large amounts of data enables LLMs to exhibit strong generalisation
capacities without further domain-specific data from the user Zhao et al.
(2023a); Amin et al. (2023). For those reasons, LLMs are swiftly becoming
mainstream tools, deeply integrated into many industry sectors, such as
medicine (see, e.g., Thirunavukarasu et al. (2023)) and finance (see, e.g., Wu
et al. (2023a)), to name a few.

However, their emergence also raises ethical concerns, necessitating ongoing
efforts to address issues related to bias, misinformation, and responsible AI
deployment. LLMs are a notoriously complex “black-box” system. Their inner
workings are opaque, and their intricate complexity makes their interpretation
challenging Kaadoud et al. (2021); Cambria et al. (2023a). Such opaqueness can
lead to the production of inappropriate content or misleading outputs
Weidinger et al. (2021). Finally, lacking visibility on their training data
can further hinder trust and accountability in critical applications Liu
(2023).

In this context, XAI is a crucial bridge between complex LLM-based systems and
human understanding of their behaviour. Developing XAI frameworks for LLMs is
essential for building user trust, ensuring accountability and fostering a
responsible and ethical use of those models.

In this article, we review and categorise current XAI for LLMs in a structured
manner. Emphasising the importance of clear and truthful explanations, as
suggested by Sevastjanova and El-Assady (2022), this survey aims to guide
future research towards enhancing LLMs’ explainability and trustworthiness in
practical applications.

###  1.1 Contribution

The contribution of our work is threefold:

  1. 1.

We introduce a novel categorisation framework for assessing the body of
research concerning the explainability of LLMs. The framework provides a clear
and organised overview of the state of the art.

  2. 2.

We conduct a comprehensive survey of peer-reviewed and preprint papers based
on ArXiv and DBLP databases, going beyond using common research tools.

  3. 3.

We critically assess current practices, identifying research gaps and issues
and articulating potential future research trajectories.

###  1.2 Research questions

In this survey, we explore the coexistence of XAI methods with LLMs and how
these two fields are merged.Specifically, our investigation revolves around
these key questions:

  1. Q1

How are XAI techniques currently being integrated with LLMs?

  2. Q2

What are the emerging trends in converging LLMs with XAI methodologies?

  3. Q3

What are the gaps in the current related literature, and what areas require
further research?

##  2 The Need for Explanations in LLMs

In XAI field, the intersection with LLMs presents unique challenges and
opportunities. This survey paper aims to dissect these challenges, extending
the dialogue beyond the conventional understanding of XAI’s objective, which
is to illuminate the inner mechanisms of opaque models for various
stakeholders while avoiding the introduction of new uncertainties (See e.g.,
Cambria et al. (2023b); Burkart and Huber (2021)).

Despite their advancements, LLMs struggle with complexity and opacity, raising
design, deployment and interpretation issues. Inspired by Weidinger et al.
(2021), this paper categorises LLM challenges into user-visible and invisible
ones.

##### Visible User Challenges

Directly perceivable challenges for users without specialised tools.

##### Trust and Transparency

Trust issues arise in crucial domains, e.g., healthcare Mercorio et al.
(2020); Gozzi et al. (2022); Alimonda et al. (2022) or finance Xing et al.
(2020); Castelnovo et al. (2023); Yeo et al. (2023), due to the opacity of
black-box models, including LLMs. XAI must offer transparent, ethically
aligned explanations for wider acceptance, especially under stringent
regulations that mandate explainability (e.g., EU’s GDPR Novelli et al.
(2024)). This impacts regulatory compliance and public credibility, with
examples in European skill intelligence projects requiring XAI for decision
explanations Malandri et al. (2022a, 2024, b, c).

##### Misuse and Critical Thinking Impacts

LLMs’ versatility risks misuse, such as content creation for harmful purposes
and evading moderation Shen et al. (2023). Over-reliance on LLMs may also
erode critical thinking and independent analysis, as seen in educational
contexts (see, e.g. Abd-Alrazaq et al. (2023)).

##### Invisible User Challenges

Challenges requiring deeper model understanding.

##### Ethical and Privacy Concerns

Ethical dilemmas from LLM use, such as fairness and hate speech issues, and
privacy risks like sensitive data exposure, require proactive measures and
ethical guidelines Weidinger et al. (2021); Yan et al. (2023); Salimi and
Saheb (2023).

##### Inaccuracies and Hallucinations

LLMs can generate false information, posing risks in various sectors like
education, journalism, and healthcare. Addressing these issues involves
improving LLM accuracy, educating users, and developing fact-checking systems
Rawte et al. (2023); Azaria and Mitchell (2023).

##  3 Methodology

Systematic Mapping Studies (SMSs) are comprehensive surveys that categorise
and summarise a range of published works in a specific research area,
identifying literature gaps, trends, and future research needs. They are
especially useful in large or under-explored fields where a detailed
Systematic Literature Review (SLR) may not be feasible.

SMS and SLR follow a three-phase method (planning, conducting, reporting) but
differ in their approach, as SMSs address broader questions, cover a wider
range of publications with a less detailed review, and aim to provide an
overview of the research field. In contrast, SLRs focus on specific questions,
thoroughly review fewer publications, and strive for precise, evidence-based
outcomes Barn et al. (2017).

Following Martínez-Gárate et al. (2023), we designed our SMS for XAI and LLMs,
including peer-reviewed and preprint papers. The latter choice is because we
believe in rapidly evolving fields like computer science, including preprints
offering access to the latest research, essential for a comprehensive review
Oikonomidi et al. (2020).

We followed these steps to structure our SMS: Section 1.2 proposes and defines
the research questions, Section 3.1 describes how the paper retrieval has been
performed; Section 3.2 describes the paper selection process based on the
defined criteria; Section 3.3 explains who we dealt with false positive
results and finally in Section 4 we describe the obtained results.

###  3.1 Paper retrieval

##### Overview

Instead of utilising common scientific search engines such as Google Scholar,
we employed a custom search methodology described in the following part. By
scrutinising the titles and abstracts of the obtained papers, we conducted
targeted searches using a predefined set of keywords pertinent to LLMs and
XAI. This manual and deliberate search strategy was chosen to minimise the
risk of overlooking relevant studies that automated search algorithms might
miss and ensure our SMS dataset’s accuracy and relevance. Through this
rigorous process, we constructed a well-defined corpus of literature poised
for in-depth analysis and review. Figure 1 provides an overview of this
process.

##### Peer-reviewed papers

We initiated this step by identifying top-tier Q1 journals within the
“Artificial Intelligence” category of 2022 (last year available at the start
of the study), providing us with 58 journals from which to draw relevant
publications.

Subsequently, we utilised the XML dump111https://dblp.org/xml/dblp.xml.gz from
dblp computer science bibliography to get the titles of all papers published
in the identified Q1 journals, except ten journals not covered by dblp. Once
we gathered these paper titles, we proceeded to find their abstract. To do so,
we initially used the last available citation network of
AMiner222https://originalfileserver.aminer.cn/misc/dblp_v14.tar.gz but given
that this dump lacks the majority of 2023 publications, we leveraged Scopus
API, a detailed database of scientific abstracts and citations, to retrieve
the missing abstracts corresponding to the amassed titles.

Figure 1: The process used for getting the papers related to our keywords,
including the definition of research questions, paper retrieval, paper
selection, elimination of false positives and classifying papers in the pre-
defined categories.

##### Pre-print papers

We scraped all computer science papers presented in the Arxiv database from
2010 until October 2023, resulting in 548,711 papers. Consequently, we used
the Arxiv API to get the abstracts of these papers.

###  3.2 Paper selection

We employed a comprehensive set of keywords to filter the collected papers for
relevance to LLMs and XAI. The search terms were carefully chosen to encompass
the various terminologies and phrases commonly associated with each field.333
The keywords for XAI included: [’xai’, ’explain’, ’explanation’, ’interpret’,
’black box’, ’black-box’, ’blackbox’, ’transparent model understanding’,
’feature importance’, ’accountable ai’, ’ethical ai’, ’trustworthy ai’,
’fairness’, ’ai justification’, ’causal inference’, ’ai audit’] While for
LLMs, the keywords are; [’llm’, ’large language model’, ’gpt-3’, ’gpt-2’,
’gpt3’, ’gpt2’, ’bert’, ’language model pre-training’, ’fine-tuning language
models’, ’generative pre-trained transformer’, ’llama’, ’ bard’, ’roberta’, ’
T5’, ’xlnet’, ’megatron’, ’electra’, ’deberta’, ’ ernie’, ’ albert’, ’ bart’,
’blenderbot’, ’open pre-trained transformer’, ’mt-nlg’, ’turing-nlg’,
’pegasus’, ’gpt-3.5’, ’gpt-4’, ’gpt3.5’, ’gpt4’, ’ cohere’, ’claude’,
’jurassic-1’, ’openllama’, ’falcon’, ’dolly’, ’mpt’, ’guanaco’, ’bloom’, ’
alpaca’, ’openchatkit’, ’gpt4all’, ’flan-t5’, ’orca’]

In our search, we applied a logical OR operator within the members of each
list to capture any of the terms within a single category, and an AND operator
was used between the two lists to ensure that only papers containing terms
from both categories were retrieved for our analysis.

###  3.3 Dealing with false positives

Upon completion of the initial retrieval phase, we identified a total of 1,030
manuscripts. Since some research keywords possess a broad meaning, for
instance the words ’explain’ and ’interpret’ can be used in contexts different
from the one of XAI, we retrieved few false positive papers, i.e., papers not
dealing with both XAI and LLMs. We excluded the false positives—publications
that address only XAI or LLMs independently or none of them. To do so, we
manually analysed the title and abstract of each paper. This meticulous
vetting process resulted in 233 papers relevant to XAI and LLMs.

Given that including all these papers in our survey was not feasible, we have
selected the most relevant ones, based on their average number of citations
per year. The whole research process resulted in 35 articles selected.

##  4 Retrieval Results

We divide papers into two macro-categories of Applicaiton papers, i.e., papers
that somehow generated explanations, either towards explainability or to use
them as a feature for another task, and Discussion papers, i.e., papers that
do not engage with explanation generation but address an issue or research gap
regarding the explainable LLM models.

###  4.1 Application Papers

The first macro-category includes papers using LLMs in a methodology, tool, or
task. Based on how LLMs are used, we further divide this category into two
sub-categories as follows: ”To explain”, i.e., papers which try to explain how
LLMs work and provide an insight into the opaque nature of these models. The
second sub-category of papers called ”As feature”, uses the explanations and
features generated by LLMs to improve the results of various tasks. The
following parts discuss these sub-categories:

Paper and Tool |  Star |  Fork |  Update |  Target |  Agnostic | Goal
---|---|---|---|---|---|---
Vig (2019) BertViz |  6.1k |  734 |  08/23 |  Transformers |  ✓ |  C E IMP INT R
Swamy et al. (2021) Experiments |  19 |  2 |  05/22 |  BERT-based LM |  ✗ |  C E IMP INT R
Wu et al. (2021) Polyjuice |  90 |  16 |  08/22 |  - |  ✓ |  C E IMP INT R
Wang et al. (2022) TransformerLens |  48 |  161 |  01/23 |  GPT2-small |  ✗ |  C E IMP INT R
Menon and Vondrick (2022) - |  - |  - |  - |  Vision-LM |  ✓ |  C E IMP INT R
Gao et al. (2023a) Experiments |  17 |  0 |  10/23 |  ChatGPT |  ✗ |  C E IMP INT R
Pan et al. (2023) - |  - |  - |  - |  LLMs |  ✓ |  C E IMP INT R
Conmy et al. (2023) ACDC |  105 |  23 |  11/23 |  Transformers |  ✓ |  C E IMP INT R
He et al. (2022) RR |  38 |  2 |  02/23 |  LLMs |  ✓ |  C E IMP INT R
Yoran et al. (2023) MCR |  71 |  9 |  01/24 |  LLMs |  ✓ |  C E IMP INT R
Sarti et al. (2023) Inseq |  250 |  26 |  01/24 |  SeqGen models |  ✓ |  C E IMP INT R
Wu et al. (2023b) Boundless DAS |  0 |  17 |  01/24 |  LLMs |  ✓ |  C E IMP INT R
Li et al. (2023) XICL |  1 |  3 |  11/23 |  LLMs |  ✓ |  C E IMP INT R
Chen et al. (2023) LMExplainer |  - |  - |  - |  LLMs |  ✓ |  C E IMP INT R
Gao et al. (2023b) Chat-REC |  - |  - |  - |  Rec. systems |  ✗ |  C E IMP INT R
Zhang et al. (2022) DSRLM |  9 |  1 |  07/23 |  LLMs |  ✓ |  C E IMP INT R
Singh et al. (2023) SASC |  61 |  14 |  01/24 |  LLMs |  ✓ |  C E IMP INT R
Li et al. (2022) - |  - |  - |  - |  LLMs |  ✓ |  C E IMP INT R
Ye and Durrett (2022) TextualExplInContext |  11 |  2 |  02/23 |  LLMs |  ✓ |  C E IMP INT R
Turpin et al. (2023) Experiments |  25 |  9 |  03/23 |  LLMs |  ✓ |  C E IMP INT R
Kang et al. (2023) AutoSD |  - |  - |  - |  Debugging models |  ✗ |  C E IMP INT R
Krishna et al. (2023) AMPLIFY |  - |  - |  - |  LLMs |  ✓ |  C E IMP INT R
Yang et al. (2023) Labo |  51 |  4 |  12/23 |  CBM |  ✗ |  C E IMP INT R
Bitton-Guetta et al. (2023) WHOOPS! |  - |  - |  - |  LLMs |  ✓ |  C E IMP INT R
Shi et al. (2023) Chatgraph |  2 |  0 |  07/23 |  LLMs |  ✓ |  C E IMP INT R

Table 1: Synthesis of recent application papers, summarising engagement
indicators as of January 2024, update timelines, model specificity, and the
overarching aims of each study. In the first section of the table, To Explain
papers are listed, and As Feature works in the second. Stars, forks, and last
updates are not reported (-) for papers lacking associated repositories.
Target is the specific focus of the study, such as a particular type of
language model. Agnostic indicates whether the study is model-agnostic or not.
The goal represents the primary objective of each study: comparison of models
(C), explanation (E), improvement (IMP), interpretability (INT), and reasoning
(R).

####  4.1.1 To Explain

Most papers, i.e., 17 out of 35, fit into this sub-category, with most
addressing the need for more interpretable and transparent LLMs.

For instance, Vig (2019) introduces a visualisation tool for understanding the
attention mechanism in Transformer models like BERT and GPT-2. Their proposed
tool provides insights at multiple scales, from individual neurons to whole
model layers, helping to detect model bias, locate relevant attention heads,
and link neurons to model behaviour.

Swamy et al. (2021) presents a methodology for interpreting the knowledge
acquisition and linguistic skills of BERT-based language models by extracting
knowledge graphs from these models at different stages of their training.
Knowledge graphs are often used for explainable extrapolation reasoning Lin et
al. (2023).

Wu et al. (2021) propose Polyjuice, a general-purpose counterfactual
generator. This tool generates diverse, realistic counterfactuals by fine-
tuning GPT-2 on multiple datasets, allowing for controlled perturbations
regarding type and location.

Wang et al. (2022) investigates the mechanistic interpretability of GPT-2
small, particularly its ability to identify indirect objects in sentences. The
study involves circuit analysis and reverse engineering of the model’s
computational graph, identifying specific attention heads and their roles in
this task.

Menon and Vondrick (2022) introduce a novel approach for visual classification
using descriptions generated by LLMs. This method, which they term
“classification by description,” involves using LLMs like GPT-3 to generate
descriptive features of visual categories. These features are then used to
classify images more accurately while providing more transparent results than
traditional methods that rely solely on category names.

Gao et al. (2023a) examines ChatGPT’s capabilities in causal reasoning using
tasks like Event Causality Identification (ECI), Causal Discovery (CD), and
Causal Explanation Generation (CEG). The authors claim that while ChatGPT is
effective as a causal explainer, it struggles with causal reasoning and often
exhibits causal hallucinations. The study also investigates the impact of In-
Context Learning (ICL) and Chain-of-Thought (CoT) techniques, concluding that
ChatGPT’s causal reasoning ability is highly sensitive to the structure and
wording of prompts.

Pan et al. (2023) is a framework that aims to enhance LLMs with explicit,
structured knowledge from KGs, addressing issues like hallucinations and lack
of interpretability. The paper outlines three main approaches: KG-enhanced
LLMs, LLM-augmented KGs, and synergised LLMs with KGs. This unification
improves the performance and explainability of AI systems in various
applications.

Conmy et al. (2023) focuses on automating a part of the mechanistic
interpretability workflow in neural networks. Using algorithms like Automatic
Circuit Discovery (ACDC), the authors automate the identification of sub-
graphs in neural models that correspond to specific behaviours or
functionalities.

He et al. (2022) presents a novel post-processing approach for LLMs that
leverages external knowledge to enhance the faithfulness of explanations and
improve overall performance. This approach, called Rethinking with Retrieval,
uses CoT prompting to generate reasoning paths refined with relevant external
knowledge. The authors claim that their method significantly improves the
performance of LLMs on complex reasoning tasks by producing more accurate and
reliable explanations.

Multi-Chain Reasoning (MCR) introduced by Yoran et al. (2023) improves
question-answering in LLMs by prompting them to meta-reason over multiple
reasoning chains. This approach helps select relevant facts, mix information
from different chains, and generate better explanations for the answers. The
paper demonstrates MCR’s superior performance over previous methods,
especially in multi-hop question-answering.

Inseq Sarti et al. (2023) is a Python library that facilitates
interpretability analyses of sequence generation models. The toolkit focuses
on extracting model internals and feature importance scores, particularly for
transformer architectures. It centralises access to various feature
attribution methods, intuitively representable with visualisations such as
heatmaps Aminimehr et al. (2023), promoting fair and reproducible evaluations
of sequence generation models.

Boundless Distributed Alignment Search (Boundless DAS) introduced by Wu et al.
(2023b) is a method for identifying interpretable causal structures in LLMs.
In their paper, the authors demonstrate that the Alpaca model, a 7B parameter
LLM, solves numerical reasoning problems by implementing simple algorithms
with interpretable boolean variables.

Li et al. (2023) investigate how various demonstrations influence ICL in LLMs
by exploring the impact of contrastive input-label demonstration pairs,
including label flipping, input perturbation, and adding complementary
explanations. The study employs saliency maps to qualitatively and
quantitatively analyse how these demonstrations affect the predictions of
LLMs.

LMExplainer Chen et al. (2023) is a method for interpreting the decision-
making processes of LMs. This approach combines a knowledge graph and a graph
attention neural network to explain the reasoning behind an LM’s predictions.

Gao et al. (2023b) propose a novel recommendation system framework, Chat-REC,
which integrates LLMs for generating more interactive and explainable
recommendations. The system converts user-profiles and interaction histories
into prompts for LLMs, enhancing the recommendation process with the ICL
capabilities of LLMs.

DSR-LM proposed by Zhang et al. (2022) is a framework combining differentiable
symbolic reasoning with pre-trained language models. The authors claim their
framework improves logical reasoning in language models through a symbolic
module that performs deductive reasoning, enhancing accuracy on deductive
reasoning tasks.

####  4.1.2 As Feature

Papers in this sub-category do not directly aim to provide more transparent
models or explain LLM-based models. Instead, they use LLMs to generate
reasoning and descriptions, which are used as input to a secondary task.

For instance, Li et al. (2022) explore how LLMs’ explanations can enhance the
reasoning capabilities of smaller language models (SLMs). They introduce a
multi-task learning framework where SLMs are trained with explanations from
LLMs, leading to improved performance in reasoning tasks.

Ye and Durrett (2022) evaluates the reliability of explanations generated by
LLMs in few-shot learning scenarios. The authors claim that LLM explanations
often do not significantly improve learning performance and can be factually
unreliable by highlighting the potential misalignment between LLM reasoning
and factual correctness in their explanations.

Turpin et al. (2023) investigates the reliability of CoT reasoning. The
authors claim that while CoT can improve task performance, it can also
systematically misrepresent the true reason behind a model’s prediction. They
demonstrate this through experiments showing how biasing features in model
inputs, such as reordering multiple-choice options, can heavily influence CoT
explanations without being acknowledged in the explanation itself.

Kang et al. (2023) introduce an approach for automating the debugging process
called Automated Scientific Debugging (AutoSD). This approach leverages LLMs
to generate hypotheses about bugs in code and uses debuggers to interact with
the buggy code. This approach leads to automated conclusions and patch
generation and provides clear explanations for the debugging decisions,
potentially leading to more efficient and accurate decisions by developers.

Krishna et al. (2023) present a framework called Amplifying Model Performance
by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), aiming
to improve the performance of LLMs on complex reasoning and language
understanding tasks by automating the generation of rationales. It leverages
post hoc explanation methods, which output attribution scores indicating the
influence of each input feature on model predictions, to construct natural
language rationales. These rationales provide corrective signals to LLMs.

Yang et al. (2023) introduces Language Guided Bottlenecks (LaBo), a method for
constructing high-performance Concept Bottleneck Models (CBMs) without manual
specification of concepts. LaBo leverages GPT-3 to generate factual sentences
about categories, forming candidate concepts for CBMs. These concepts are then
aligned with images using CLIP Radford et al. (2021) to form a bottleneck
layer. The method efficiently searches for bottlenecks using a submodular
utility, focusing on discriminative and diverse information. The authors claim
their method outperforms black box linear probes in few-shot classification
tasks across 11 diverse datasets, showing comparable or better performance
with more data.

Bitton-Guetta et al. (2023) introduces WHOOPS!, a new dataset and benchmark
designed to test AI models’ visual commonsense reasoning abilities. The
dataset comprises images intentionally defying commonsense, created using
image generation tools like Midjourney. The paper assesses AI models on tasks
such as image captioning, cross-modal matching, visual question answering, and
the challenging task of explanation generation, where models must identify and
explain the unusualness of an image. Results show that even advanced models
like GPT3 and BLIP2 struggle with these tasks, highlighting a gap in AI’s
visual commonsense reasoning compared to human performance.

###  4.2 Discussion Papers

Unlike the Application papers, this category includes papers that target the
argument of XAI through LLMs and vice versa but do not necessarily provide any
specific methodology, framework or application. This category, in turn, is
divided into two subcategories of Issues, or works which mention a concern and
Benchmark and Metrics, which mainly focus on evaluation and assessment of XAI
methods in LLM field.

####  4.2.1 Issues

Bowman (2023) critically examines LLMs, highlighting their unpredictability
and the emergent nature of their capabilities with scaling. They underscore
the challenges in steering and interpreting LLMs and the necessity for a
nuanced understanding of their limitations and potential.

Liu et al. (2023) offers a survey and set of guidelines for assessing the
alignment of LLMs with human values and intentions. They categorise and detail
aspects of LLM trustworthiness, including reliability, safety, fairness,
resistance to misuse, explainability, adherence to social norms, and
robustness.

Liao and Vaughan (2023) emphasise the need for transparency in LLMs from a
human-centred perspective. The authors discuss the unique challenges of
achieving transparency with LLMs, differentiating them from smaller, more
specialised models. The paper proposes a roadmap for research, emphasising the
importance of understanding and addressing the transparency needs of diverse
stakeholders in the LLM ecosystem. It advocates for developing and designing
transparency approaches that consider these stakeholder needs, the novel
applications of LLMs, and their various usage patterns and associated
challenges.

Lastly, Xie et al. (2023) highlights the limitations of ChatGPT in
explainability and stability in the context of financial market analysis
through a zero-shot analysis. The authors suggest the need for more
specialised training or fine-tuning.

####  4.2.2 Benchmark and Metrics

Lu et al. (2022) introduce SCIENCEQA, a new dataset for multimodal science
question answering. This dataset includes around 21k questions with diverse
science topics and annotations, featuring lectures and explanations to aid in
understanding the reasoning process. The authors demonstrate how language
models, particularly LLMs, can be trained to generate these lectures and
explanations as part of a CoT process, enhancing their reasoning capabilities.
The study shows that CoT improves question-answering performance and provides
insights into the potential of LLMs to mimic human-like multi-step reasoning
in complex, multimodal domains.

Golovneva et al. (2022) introduce ROSCOE, a set of metrics designed to
evaluate the step-by-step reasoning of language models, especially in
scenarios without a golden reference. This work includes a taxonomy of
reasoning errors and a comprehensive evaluation of ROSCOE against baseline
metrics across various reasoning tasks. The authors demonstrate ROSCOE’s
effectiveness in assessing semantic consistency, logicality, informativeness,
fluency, and factuality in model-generated rationales.

Zhao et al. (2023b) presents a comprehensive survey on explainability
techniques for LLMs, focusing on Transformer-based models. It categorises
these techniques based on traditional fine-tuning and prompting paradigms,
detailing methods for generating local and global explanations. The paper
addresses the challenges and potential directions for future research in
explainability, highlighting LLMs’ unique complexities and capabilities
compared to conventional deep-learning models. Nevertheless, the survey mainly
focuses on XAI in general and has minimal coverage of the relationship between
XAI and LLMs.

##  5 Discussion

Our analysis indicates that a limited number of the reviewed publications
directly tackle the challenges highlighted in Section 2. For example, the work
by Liu et al. (2023) focuses on trust-related concerns in LLMs, whereas Gao et
al. (2023a) investigates the issue of misinformation propagation by LLMs. This
scant attention to the identified problems suggests an imperative for
substantial engagement from the XAI community to confront these issues
adequately.

##### Open-Source Engagement

Our survey study shows that more studies are moving beyond the traditional
approach of merely describing methodologies in text. Instead, they release
them as tangible tools or open-source code, frequently hosted on platforms
such as GitHub. This evolution is a commendable step toward enhancing
transparency and reproducibility in computer science research. The trend
suggests a growing inclination among authors to release their code and
publicly publish their tools, a notable change from a few years ago. However,
we should also mention the inconsistency in the level of community engagement
with these repositories. While some repositories attract substantial interest,
fostering further development and improvement, others remain underutilised.
This disparity in engagement raises important questions about the factors
influencing community interaction with these resources.

##### Target

Predominantly, most works have directed their attention towards LLMs rather
than concentrating on more specialised or narrower subjects within AI-based
systems. This broad approach contrasts the relatively few studies that focus
specifically on Transformers or are confined to examining particular
categories of systems, such as recommendation systems. This overarching focus
on LLMs represents a positive and impactful trend within the AI community.
Given the rapid development and increasing prominence of LLM systems in
academic and practical applications, this broader focus is timely and crucial
for driving our understanding and capabilities in this domain forward. It
ensures that research keeps pace with the advancements in the field, fostering
a comprehensive and forward-looking approach essential for AI technologies’
continued growth and evolution.

##### Goal

Our analysis, as delineated in Table 1, reveals a bifurcation in the
objectives of the LLM studies under review. On the one hand, a subset of these
works is primarily dedicated to explaining and enhancing the interpretability
of these ’black box’ models. On the other hand, a larger contingent is more
task-oriented, focusing on augmenting specific tasks and models, with
interpretability emerging merely as a byproduct. This dichotomy in research
focus underscores a pivotal trend: a pressing need to shift more attention
towards demystifying the inner workings of LLMs. Rather than solely leveraging
these models to boost task performance, their inherently opaque nature should
not be overlooked. The pursuit of performance improvements must be balanced
with efforts to unravel and clarify the underlying mechanisms of LLMs. This
approach is crucial for fostering a deeper understanding of these complex
systems, ensuring their application is effective and transparent. Such a
balanced focus is essential for advancing the field technically and
maintaining ethical and accountable AI development.

##  6 Conclusion

Our SMS reveals that only a handful of works are dedicated to developing
explanation methods for LLM-based systems. This finding is particularly
salient, considering the rapidly growing prominence of LLMs in various
applications. Our study, therefore, serves a dual purpose in this context.
Firstly, it acts as a navigational beacon for the XAI community, highlighting
the fertile areas where efforts to create interpretable and transparent LLM-
based systems can effectively address the challenges the broader AI community
faces. Secondly, it is a call to action, urging researchers and practitioners
to venture into this relatively underexplored domain. The need for explanation
methods in LLM-based systems is not just a technical necessity but also a step
towards responsible AI practice. By focusing on this area, the XAI community
can contribute significantly to making AI systems more efficient, trustworthy
and accountable.

Our call for action is as follows: Firstly, researchers employing LLM models
must acknowledge and address the potential long-term challenges posed by the
opacity of these systems. The importance of explainability should be elevated
from a mere ’nice-to-have’ feature to an integral aspect of the development
process. This involves a proactive approach to incorporate explainability in
the design and implementation phases of LLM-based systems. Such a shift in
perspective is essential to ensure that these models are effective,
transparent and accountable. Secondly, we urge researchers in the XAI field to
broaden their investigative scope. The focus should not only be on devising
methodologies capable of handling the complexity of LLM-based systems but also
on enhancing the presentation layer of these explanations. Currently,
explanations provided are often too complex for non-technical stakeholders.
Therefore, developing approaches that render these explanations more
accessible and understandable to a wider audience is imperative. This dual
approach will make LLMs more understandable and user-friendly and bridge the
gap between technical efficiency and ethical responsibility in AI development.

## References

  * Zhao et al. [2023a] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  _arXiv:2303.18223_ , 2023a.
  * Amin et al. [2023] Mostafa Amin, Erik Cambria, and Björn Schuller.  Can ChatGPT’s responses boost traditional natural language processing?  _IEEE Intelligent Systems_ , 38(5):5–11, 2023.
  * Thirunavukarasu et al. [2023] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  _Nature medicine_ , pages 1–11, 2023.
  * Wu et al. [2023a] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.  Bloomberggpt: A large language model for finance.  _arXiv:2303.17564_ , 2023a.
  * Kaadoud et al. [2021] Ikram Chraibi Kaadoud, Lina Fahed, and Philippe Lenca.  Explainable ai: a narrative review at the crossroad of knowledge discovery, knowledge representation and representation learning.  In _MRC_ , volume 2995, pages 28–40. ceur-ws. org, 2021.
  * Cambria et al. [2023a] Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho.  Seven pillars for the future of artificial intelligence.  _IEEE Intelligent Systems_ , 38(6):62–69, 2023a.
  * Weidinger et al. [2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al.  Ethical and social risks of harm from language models.  _arXiv:2112.04359_ , 2021.
  * Liu [2023] Yang Liu.  The importance of human-labeled data in the era of llms.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_ , pages 7026–7032, 2023.
  * Sevastjanova and El-Assady [2022] Rita Sevastjanova and Mennatallah El-Assady.  Beware the rationalization trap! when language model explainability diverges from our mental models of language, 2022.
  * Cambria et al. [2023b] Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Navid Nobani.  A survey on xai and natural language explanations.  _Information Processing & Management_, 60(1):103111, 2023b.
  * Burkart and Huber [2021] Nadia Burkart and Marco F Huber.  A survey on the explainability of supervised machine learning.  _Journal of Artificial Intelligence Research_ , 70:245–317, 2021.
  * Mercorio et al. [2020] Fabio Mercorio, Mario Mezzanzanica, and Andrea Seveso.  exdil: A tool for classifying and explaining hospital discharge letters.  In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_ , pages 159–172. Springer, 2020.
  * Gozzi et al. [2022] Noemi Gozzi, Lorenzo Malandri, Fabio Mercorio, and Alessandra Pedrocchi.  Xai for myo-controlled prosthesis: Explaining emg data for hand gesture classification.  _Knowledge-Based Systems_ , 240:108053, 2022.
  * Alimonda et al. [2022] Nicola Alimonda, Luca Guidotto, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Giovanni Tosi.  A survey on xai for cyber physical systems in medicine.  In _2022 IEEE International Conference on Metrology for Extended Reality, Artificial Intelligence and Neural Engineering (MetroXRAINE)_ , pages 265–270. IEEE, 2022.
  * Xing et al. [2020] Frank Xing, Lorenzo Malandri, Yue Zhang, and Erik Cambria.  Financial sentiment analysis: an investigation into common mistakes and silver bullets.  In _Proceedings of the 28th international conference on computational linguistics_ , pages 978–987, 2020.
  * Castelnovo et al. [2023] Alessandro Castelnovo, Nicole Inverardi, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Andrea Seveso.  Leveraging group contrastive explanations for handling fairness.  In _World Conference on Explainable Artificial Intelligence_ , pages 332–345. Springer, 2023.
  * Yeo et al. [2023] Wei Jie Yeo, Wihan van der Heever, Rui Mao, Erik Cambria, Ranjan Satapathy, and Gianmarco Mengaldo.  A comprehensive review on financial explainable ai.  _arXiv preprint arXiv:2309.11960_ , 2023.
  * Novelli et al. [2024] Claudio Novelli, Federico Casolari, Philipp Hacker, Giorgio Spedicato, and Luciano Floridi.  Generative ai in eu law: Liability, privacy, intellectual property, and cybersecurity.  _EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity (January 14, 2024)_ , 2024.
  * Malandri et al. [2022a] Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, Navid Nobani, and Andrea Seveso.  ContrXT: Generating contrastive explanations from any text classifier.  _Inf. Fusion_ , 81:103–115, 2022a.  10.1016/j.inffus.2021.11.016.  URL https://doi.org/10.1016/j.inffus.2021.11.016.
  * Malandri et al. [2024] Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Andrea Seveso.  Model-contrastive explanations through symbolic reasoning.  _Decision Support Systems_ , 176:114040, 2024.
  * Malandri et al. [2022b] Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, Navid Nobani, and Andrea Seveso.  Contrastive explanations of text classifiers as a service.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations_ , pages 46–53, 2022b.
  * Malandri et al. [2022c] Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, Navid Nobani, Andrea Seveso, et al.  The good, the bad, and the explainer: a tool for contrastive explanations of text classifiers.  In _IJCAI_ , pages 5936–5939. AAAI Press, 2022c.
  * Shen et al. [2023] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.  ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models.  _arXiv:2308.03825_ , 2023.
  * Abd-Alrazaq et al. [2023] Alaa Abd-Alrazaq, Rawan AlSaad, Dari Alhuwail, Arfan Ahmed, Padraig Mark Healy, Syed Latifi, Sarah Aziz, Rafat Damseh, Sadam Alabed Alrazak, Javaid Sheikh, et al.  Large language models in medical education: Opportunities, challenges, and future directions.  _JMIR Medical Education_ , 9(1):e48291, 2023.
  * Yan et al. [2023] Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević.  Practical and ethical challenges of large language models in education: A systematic scoping review.  _British Journal of Educational Technology_ , 2023.
  * Salimi and Saheb [2023] Ali Salimi and Hady Saheb.  Large language models in ophthalmology scientific writing: Ethical considerations blurred lines or not at all?  _American Journal of Ophthalmology_ , 2023.
  * Rawte et al. [2023] Vipula Rawte, Amit Sheth, and Amitava Das.  A survey of hallucination in large foundation models, 2023.
  * Azaria and Mitchell [2023] Amos Azaria and Tom Mitchell.  The internal state of an llm knows when its lying.  _arXiv:2304.13734_ , 2023.
  * Barn et al. [2017] Balbir Barn, Souvik Barat, and Tony Clark.  Conducting systematic literature reviews and systematic mapping studies.  In _Innovations in Software Engineering Conference_ , pages 212–213, 2017.
  * Martínez-Gárate et al. [2023] Ángel Antonio Martínez-Gárate, José Alfonso Aguilar-Calderón, Carolina Tripp-Barba, and Aníbal Zaldívar-Colado.  Model-driven approaches for conversational agents development: A systematic mapping study.  _IEEE Access_ , 2023.
  * Oikonomidi et al. [2020] Theodora Oikonomidi, Isabelle Boutron, Olivier Pierre, Guillaume Cabanac, Philippe Ravaud, and Covid-19 Nma Consortium.  Changes in evidence for studies assessing interventions for covid-19 reported in preprints: meta-research study.  _BMC medicine_ , 18:1–10, 2020.
  * Vig [2019] Jesse Vig.  A multiscale visualization of attention in the transformer model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 37–42, 2019.
  * Swamy et al. [2021] Vinitra Swamy, Angelika Romanou, and Martin Jaggi.  Interpreting language models through knowledge graph extraction.  In _NeurIPS_ , 2021.
  * Wu et al. [2021] T Wu, M Tulio Ribeiro, J Heer, and D Weld.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _ACL-IJCNLP_ , 2021.
  * Wang et al. [2022] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  In _NeurIPS ML Safety Workshop_ , 2022.
  * Menon and Vondrick [2022] Sachit Menon and Carl Vondrick.  Visual classification via description from large language models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gao et al. [2023a] Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu.  Is chatgpt a good causal reasoner? a comprehensive evaluation.  _arXiv:2305.07375_ , 2023a.
  * Pan et al. [2023] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu.  Unifying large language models and knowledge graphs: A roadmap.  _arXiv:2306.08302_ , 2023.
  * Conmy et al. [2023] Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso.  Towards automated circuit discovery for mechanistic interpretability.  _arXiv:2304.14997_ , 2023.
  * He et al. [2022] Hangfeng He, Hongming Zhang, and Dan Roth.  Rethinking with retrieval: Faithful large language model inference.  _arXiv:2301.00303_ , 2022.
  * Yoran et al. [2023] Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant.  Answering questions by meta-reasoning over multiple chains of thought.  _arXiv:2304.13007_ , 2023.
  * Sarti et al. [2023] Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal.  Inseq: An interpretability toolkit for sequence generation models.  _arXiv:2302.13942_ , 2023.
  * Wu et al. [2023b] Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman.  Interpretability at scale: Identifying causal mechanisms in alpaca.  _arXiv:2305.08809_ , 2023b.
  * Li et al. [2023] Zongxia Li, Paiheng Xu, Fuxiao Liu, and Hyemi Song.  Towards understanding in-context learning with contrastive demonstrations and saliency maps.  _arXiv:2307.05052_ , 2023.
  * Chen et al. [2023] Zichen Chen, Ambuj K Singh, and Misha Sra.  Lmexplainer: a knowledge-enhanced explainer for language models.  _arXiv:2303.16537_ , 2023.
  * Gao et al. [2023b] Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang.  Chat-rec: Towards interactive and explainable llms-augmented recommender system.  _arXiv:2303.14524_ , 2023b.
  * Zhang et al. [2022] Hanlin Zhang, Ziyang Li, Jiani Huang, Mayur Naik, and Eric Xing.  Improved logical reasoning of language models via differentiable symbolic programming.  In _First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022_ , 2022.
  * Singh et al. [2023] Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao.  Explaining black box text modules in natural language with language models.  _arXiv:2305.09863_ , 2023.
  * Li et al. [2022] Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al.  Explanations from large language models make small reasoners better.  _arXiv:2210.06726_ , 2022.
  * Ye and Durrett [2022] Xi Ye and Greg Durrett.  The unreliability of explanations in few-shot prompting for textual reasoning.  _NeurIPS_ , 35:30378–30392, 2022.
  * Turpin et al. [2023] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  _arXiv:2305.04388_ , 2023.
  * Kang et al. [2023] Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou.  Explainable automated debugging via large language model-driven scientific debugging.  _arXiv:2304.02195_ , 2023.
  * Krishna et al. [2023] Satyapriya Krishna, Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju.  Post hoc explanations of language models can improve language models.  _arXiv:2305.11426_ , 2023.
  * Yang et al. [2023] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar.  Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19187–19197, 2023.
  * Bitton-Guetta et al. [2023] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz.  Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2616–2627, 2023.
  * Shi et al. [2023] Yucheng Shi, Hehuan Ma, Wenliang Zhong, Gengchen Mai, Xiang Li, Tianming Liu, and Junzhou Huang.  Chatgraph: Interpretable text classification by converting chatgpt knowledge to graphs.  _arXiv:2305.03513_ , 2023.
  * Lin et al. [2023] Qika Lin, Jun Liu, Rui Mao, Fangzhi Xu, and Erik Cambria.  Techs: Temporal logical graph networks for explainable extrapolation reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1281–1293, 2023.
  * Aminimehr et al. [2023] Amirhossein Aminimehr, Pouya Khani, Amirali Molaei, Amirmohammad Kazemeini, and Erik Cambria.  Tbexplain: A text-based explanation method for scene classification models with the statistical prediction correction.  _arXiv preprint arXiv:2307.10003_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Bowman [2023] Samuel R Bowman.  Eight things to know about large language models.  _arXiv:2304.00612_ , 2023.
  * Liu et al. [2023] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li.  Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  In _Socially Responsible Language Modelling Research_ , 2023.
  * Liao and Vaughan [2023] Q Vera Liao and Jennifer Wortman Vaughan.  Ai transparency in the age of llms: A human-centered research roadmap.  _arXiv:2306.01941_ , 2023.
  * Xie et al. [2023] Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang.  The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges.  _arXiv:2304.05351_ , 2023.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _NeurIPS_ , 35:2507–2521, 2022.
  * Golovneva et al. [2022] Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz.  Roscoe: A suite of metrics for scoring step-by-step reasoning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhao et al. [2023b] Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du.  Explainability for large language models: A survey.  _ACM TIST_ , 2023b.