11institutetext: Technische Universität Berlin, 10623 Berlin, Germany
22institutetext: GT-ARC gGmbH, 10587 Berlin, Germany

# LaFAM: Unsupervised Feature Attribution with Label-free Activation Maps

Aray Karjauv 1122 0000-0002-4408-7406 Sahin Albayrak 11 0000-0001-5092-4584

###### Abstract

Convolutional Neural Networks (CNNs) are known for their ability to learn
hierarchical structures, naturally developing detectors for objects, and
semantic concepts within their deeper layers. Activation maps (AMs) reveal
these saliency regions, which are crucial for many Explainable AI (XAI)
methods. However, the direct exploitation of raw AMs in CNNs for feature
attribution remains underexplored in literature. This work revises Class
Activation Map (CAM) methods by introducing the Label-free Activation Map
(LaFAM), a streamlined approach utilizing raw AMs for feature attribution
without reliance on labels. LaFAM presents an efficient alternative to
conventional CAM methods, demonstrating particular effectiveness in saliency
map generation for self-supervised learning while maintaining applicability in
supervised learning scenarios.

##  1 Introduction

Figure 1: Saliency maps comparison. The contours are overlaid to provide a
reference for the ground truth. The title on the left displays the true labels
from PASCAL VOC 2012 alongside the ImageNet labels predicted by ResNet50.
Grad-CAM utilizes the predicted labels to generate saliency maps. Noticeably,
the LaFAM saliency maps are very similar to those produced by Grad-CAM, while
RELAX produces noisy saliency maps. The first row demonstrates a
misclassification example, showcasing a situation where Grad-CAM fails to
highlight the correct region. A detailed comparison is presented in Section 4.

Self-supervised learning (SSL) has emerged as a flexible approach in computer
vision [6, 16], demonstrating the ability to learn the underlying structure
from data without manual annotations. This property enables SSL models to be
applied to a variety of downstream tasks, making them often referred to as
general purpose or foundation models [4]. However, the absence of labels
presents a challenge in evaluating whether these models have learned relevant
representations or if they captured biases and shortcuts [20]. Therefore, it
is crucial to identify and address potential shortcomings, particularly in
light of the EU AI Act [19], which requires transparency in high-risk areas
such as healthcare.

While Vision Transformers [10] show impressive results, convolutional neural
networks (CNNs) continue to be extensively used due to their well-established
effectiveness in extracting spatially coherent features and their
interpretability [26, 30, 31]. Evidence suggests that CNNs can inherently
develop detectors for semantic objects without explicit supervision [3, 15,
33], making Activation Maps (AMs) in deeper layers a key factor in the success
of current XAI methods [5, 8, 21, 23, 25, 28, 31, 34].

Surprisingly, while AMs are acknowledged for their ability to capture abstract
concepts the utilization of raw AMs has been largely overlooked in the
literature. Addressing this gap, we propose the Label-free Activation Map
(LaFAM) as an alternative to traditional occlusion and gradient-based
techniques. LaFAM does not require labels, making it ideally suited for SSL
models. It may further benefit supervised learning due to its ability to
highlight salient regions that traditional methods might ignore since they
primarily focus on generating class-specific saliency maps. For
reproducibility and further examination, both the source
code111https://github.com/karray/LaFAM and a live
demonstration222https://karay.me/examples/lafam are made available.

##  2 Related Work

Early foundational work by Zeiler and Fergus (2014) demonstrates the use of
occlusion sensitivity to show how CNN predictions decline when key image
features are masked, thereby localizing critical features for classification
[31]. Building on this concept, the Randomized Input Sampling for Explanation
(RISE) [22] method employs random masks to create occlusions, facilitating the
generation of saliency maps that identify salient regions in image
classification tasks.

Zhou et al. (2016) enhanced interpretability by introducing Class Activation
Maps (CAMs) which are a linear combination of AMs in the last convolutional
layer, multiplied by the weights of the output layer [34]. Nevertheless, this
method is limited by its architecture-specific requirements, which include a
global average pooling followed by a fully connected output layer. To mitigate
this architectural constraint, various gradient-based [25, 8, 21] and
gradient-free methods [5, 23, 28] were proposed that aim to obtain a class-
specific score to weigh the AMs.

However, these methods do not apply to SSL, as they require labels to
calculate the class scores. Given the lack of annotations in SSL, there is a
requirement for a label-free scoring function that can measure the output.
Representation learning explainability (RELAX) [29] method addresses this
issue by adapting RISE to SSL models. It utilizes cosine similarity as a
replacement for the class score function. It measures the difference between
the embedding of the original image and the embeddings of the occluded images,
which is then used to weigh the occlusion masks.

Nevertheless, it suffers from several limitations including computational
inefficiency due to multiple model inference and noisy results. Additionally,
using solid color patches for occlusion may produce out-of-distribution
samples, which can produce misleading explanations that do not reflect the
behavior of a model on typical in-distribution data.

We propose to utilize the raw AMs directly for generating saliency maps
without further manipulations, as they already encapsulate essential semantic
and spatial information [15, 33]. Given that this approach does not require
labeled data, it is particularly well-suited for SSL. Additionally, this
method may significantly benefit self-supervised models by revealing all
learned concepts, including those not explicitly labeled as shown in Figure 2.

##  3 Method

LaFAM is a post hoc analytical label and gradient-free method that generates
saliency maps by averaging the activation maps at a selected convolutional
layer. Unlike fully connected layers, convolutional layers preserve spatial
information, which is essential for constructing spatially coherent saliency
maps. LaFAM requires a single forward pass during inference to obtain the
activation of the target convolutional layer, which makes it computationally
efficient.

Let $A_{i,j}^{l,k}$ denote the activation at spatial position $(i,j)$ in the
$k$-th channel of the $l$-th convolutional layer. The average activation
$\bar{A}^{l}_{i,j}$ over channels at position $(i,j)$ for the $l$-th
convolutional layer is computed as follows:

$\bar{A}^{l}_{i,j}=\frac{1}{K}\sum_{k=1}^{K}A_{i,j}^{l,k}\text{,}$ (1)

where $K$ is the number of channels in the $l$-th convolutional layer.

The ResNet used in this work incorporates Rectified Linear Units (ReLUs),
which ensures that the AMs contain only non-negative values, representing a
positive contribution to the output. However, these values are unbounded in
the positive direction, requiring normalization. Therefore, the saliency map
$M_{\text{LaFAM}}$ is constructed by first applying min-max normalization to
the averaged map $\bar{A}^{l}$ scaling the values to the range $[0,1]$. The
resulting map is then upsampled to match the input image size:

$M_{\text{LaFAM}}=\text{Up}\left(N\left(\bar{A}^{l}\right)\right)\text{,}$ (2)

where $N$ denotes the min-max normalization function, and Up represents the
operation of upsampling the normalized map to the input image size.

##  4 Experiment

We systematically evaluate and compare LaFAM with RELAX in SSL settings. To
this end, we employ SimCLR [9] and SwAV [6] models with a ResNet50 [17]
backbone, pretrained333Pretrained models from PyTorch Lightning Bolts on
ImageNet-1k [24]. Given that the final convolutional layer produces AMs with a
7x7 spatial dimension and 2048 channels, RELAX is set to generate an equal
number of masks with the same spatial resolutions. These masks are produced
based on a Bernoulli distribution with a p-parameter of 0.5.

Table 1: Saliency maps performance comparison on ImageNet-S (higher values are better). | Supervised (ResNet50) | SSL (SimCLR) | SSL (SwAV)
---|---|---|---
Metric | Grad-CAM | LaFAM | RELAX | LaFAM | RELAX | LaFAM
Pointing-Game | 94.00 | 90.67 | 88.29 | 92.14 | 85.47 | 89.90
Sparseness | 42.74 | 34.82 | 35.26 | 49.70 | 31.49 | 39.92
Relevance Mass Accuracy | 50.28 | 45.89 | 46.19 | 53.32 | 42.96 | 50.13
Relevance Rank Accuracy | 62.22 | 59.50 | 58.13 | 61.44 | 53.64 | 64.69
Top-K Intersection | 75.07 | 69.09 | 71.21 | 76.59 | 63.83 | 71.68
AUC | 83.12 | 80.45 | 76.49 | 81.28 | 70.13 | 83.03

This evaluation is conducted using segmentation masks derived from the
validation sets of ImageNet-S [14] and the PASCAL VOC 2012 [12, 11] datasets.
To ensure a fair and consistent comparison, all methods employ nearest
neighbor interpolation for upscaling the saliency maps.

Furthermore, we isolate pertinent samples by selecting distinct categories and
discarding masks with multiple classes. For the PASCAL VOC 2012 dataset, the
image set was narrowed down to 13 labels. The refined datasets consisted of
11,294 and 753 samples from ImageNet-S and PASCAL VOC 2012, respectively.

We also compare the performance of LaFAM and Grad-CAM in supervised settings
as a baseline for reference using a pretrained ResNet50
classifier444Pretrained model from Torchvision.

Quantitative analysis is performed using the Quantus framework [18], with a
particular focus on saliency map quantification through segmentation masks.
The selected metrics are described in Appendix 0.A.1. Visual examples of the
saliency maps are presented in Figure 1, as well as in Figure 4 in the
Appendix.

Table 2: Saliency maps performance comparison on PASCAL VOC 2012 (higher values are better). | Supervised | SimCLR | SwAV
---|---|---|---
Metric | Grad-CAM | LaFAM | RELAX | LaFAM | RELAX | LaFAM
Pointing-Game | 90.83 | 91.23 | 91.63 | 94.68 | 82.73 | 93.62
Sparseness | 44.39 | 36.20 | 36.04 | 51.00 | 32.36 | 41.61
Relevance Mass Accuracy | 40.44 | 37.13 | 38.00 | 45.67 | 34.18 | 42.17
Relevance Rank Accuracy | 53.53 | 53.94 | 54.55 | 58.73 | 46.40 | 61.05
Top-K Intersection | 65.46 | 63.42 | 67.82 | 75.05 | 56.22 | 67.63
AUC | 82.87 | 84.00 | 79.88 | 85.33 | 71.24 | 87.74
Figure 2: Saliency maps comparison for scenes with two distinct objects. Left-
hand labels indicate ImageNet labels predicted by ResNet50 classifier.

Table 1 and Table 2 present the results for the ImageNet-1k and PASCAL VOC
2012 datasets, comparing LaFAM with RELAX for SimCLR and SwAV in SSL settings,
and with Grad-CAM in the supervised setting as a point of comparison.

In the SSL scenario, LaFAM outperforms RELAX across all metrics for both
datasets. Specifically, the higher Pointing-Game and Sparseness scores
indicate more precise and focused saliency maps. RELAX shows a low Sparseness
score, indicating that it tends to produce noticeably noisy saliency maps,
especially for small objects. This observation is supported by the qualitative
findings, as depicted in Figure 1 and Figure 4 in the Appendix.

In supervised settings, LaFAM is competitive with Grad-CAM across several
metrics. As expected, Grad-CAM demonstrates superior results in the Sparseness
metric. It implies that Grad-CAM generates less scattered explanations, as its
design inherently highlights discriminative regions associated with specific
class labels. However, CAM methods require labels and it can be a limiting
factor since class scores are computed with respect to the most probable
prediction. This can lead to wrong or misleading explanations when the
prediction is incorrect.

In contrast, a lower Sparseness score for LaFAM suggests that it captures a
broader range of learned features, which can be beneficial compared to CAM
methods, especially when an image contains multiple objects in a scene. It,
therefore, may not be necessarily interpreted as a shortcoming but rather as a
more universal approach as shown in Figure 2.

Figure 3: Examples of misclassifications on ImageNet-1k. The left-hand title
indicates the method used to generate the saliency maps, while the top title
indicates ImageNet-1k ground truth and labels predicted by ResNet50.

Figure 2 depicts scenes with two distinct objects, demonstrating the ability
of LaFAM to highlight multiple concepts simultaneously. This capability is
crucial not only for SSL but may also benefit supervised models by providing
insights into their behavior when processing complex scenes. Unlike
traditional CAM methods, which focus on a single object due to their class-
specific approach, LaFAM allows for a more comprehensive understanding of
learned concepts. This can be particularly beneficial when predictions are
incorrect or when assessing model robustness in real-world situations, where
scenes often contain multiple classes. By aggregating AMs without relying on
specific predictions, LaFAM can help identify and address model
misclassification.

Figure 3 shows examples of misclassifications on ImageNet-1k. It can be seen
that LaFAM accurately highlights the correct object, in contrast to Grad-CAM.
This may indicate that the model has identified the target object, but the
presence of multiple objects in the scene may lead to a higher output for an
incorrect class. Consequently, the saliency maps produced by Grad-CAM are
focused on the wrong object.

##  5 Discussion

This study evaluates methods XAI methods in the context of CNNs and compares
the performance of LaFAM and RELAX within the SSL scenario and LaFAM and Grad-
CAM under supervised conditions as a reference point.

Our findings reveal that LaFAM consistently outperforms RELAX across all
metrics for SSL models, particularly for scenes with small-sized objects.
While the performance of RELAX is suboptimal, it can be enhanced by decreasing
cell sizes and increasing the number of masks to generate more detailed
saliency maps, although this increment raises computational costs. In
contrast, LaFAM is more computationally efficient and requires no additional
hyperparameters, making it a more practical choice for saliency map generation
in SSL scenarios.

A common limitation of traditional CAM methods, including LaFAM, is low
resolution. However, it might be possible to address this limitation by
redistributing the AM values to earlier layers through Layer-wise Relevance
Propagation (LRP) [2]. This approach could potentially mitigate the issue, but
its effectiveness requires further investigation.

In conclusion, the results highlight that LaFAM emerges as a robust and
flexible method, contributing to the diversification of the XAI toolbox. Its
computational efficiency and label-free saliency map generation make it
valuable for understanding model decisions across both supervised and self-
supervised learning paradigms.

##  6 Acknowledgements

This work is based on the research conducted in the gemeinwohlorientierter KI-
Anwendungen (Go-KI) project (Offenes Innovationslabor KI zur Förderung
gemeinwohlorientierter KI-Anwendungen), funded by the German Federal Ministry
of Labour and Social Affair (BMAS) under the funding reference number
DKI.00.00032.21.

## References

  * [1] Arras, L., Osman, A., Samek, W.: CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion 81, 14–40 (2022)
  * [2] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015)
  * [3] Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: Quantifying interpretability of deep visual representations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6541–6549 (2017)
  * [4] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
  * [5] Byun, S.Y., Lee, W.: Recipro-CAM: Gradient-free reciprocal class activation map. arXiv preprint arXiv:2209.14074 (2022)
  * [6] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33, 9912–9924 (2020)
  * [7] Chalasani, P., Chen, J., Chowdhury, A.R., Wu, X., Jha, S.: Concise explanations of neural networks using adversarial training. In: International Conference on Machine Learning. pp. 1383–1391. PMLR (2020)
  * [8] Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV). pp. 839–847. IEEE (2018)
  * [9] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [11] Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2012 (VOC2012) results. 2012 http://www. pascal-network. org/challenges. In: VOC/voc2012/workshop/index. html (2012)
  * [12] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes (VOC) challenge. International journal of computer vision 88, 303–338 (2010)
  * [13] Fawcett, T.: An introduction to ROC analysis. Pattern recognition letters 27(8), 861–874 (2006)
  * [14] Gao, S., Li, Z.Y., Yang, M.H., Cheng, M.M., Han, J., Torr, P.: Large-scale unsupervised semantic segmentation. IEEE transactions on pattern analysis and machine intelligence (2022)
  * [15] Gonzalez-Garcia, A., Modolo, D., Ferrari, V.: Do semantic parts emerge in convolutional neural networks? International Journal of Computer Vision 126, 476–494 (2018)
  * [16] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33, 21271–21284 (2020)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. corr abs/1512.03385 (2015) (2015)
  * [18] Hedström, A., Weber, L., Krakowczyk, D., Bareeva, D., Motzkus, F., Samek, W., Lapuschkin, S., Höhne, M.M.M.: Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. Journal of Machine Learning Research 24(34), 1–11 (2023), http://jmlr.org/papers/v24/22-0142.html
  * [19] Madiega, T.: Artificial intelligence act. European Parliament: European Parliamentary Research Service (2021)
  * [20] Meehan, C., Bordes, F., Vincent, P., Chaudhuri, K., Guo, C.: Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=lkBygTc0SI
  * [21] Omeiza, D., Speakman, S., Cintas, C., Weldermariam, K.: Smooth Grad-CAM++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224 (2019)
  * [22] Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018)
  * [23] Ramaswamy, H.G., et al.: Ablation-CAM: Visual explanations for deep convolutional network via gradient-free localization. In: proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 983–991 (2020)
  * [24] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252 (2015)
  * [25] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [26] Smith, S.L., Brock, A., Berrada, L., De, S.: ConvNets Match Vision Transformers at Scale. arXiv preprint arXiv:2310.16764 (2023)
  * [27] Theiner, J., Müller-Budack, E., Ewerth, R.: Interpretable semantic photo geolocalization. arXiv preprint arXiv:2104.14995 (2021)
  * [28] Wang, H., Wang, Z., Du, M., Yang, F., Zhang, Z., Ding, S., Mardziel, P., Hu, X.: Score-CAM: Score-weighted visual explanations for convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 24–25 (2020)
  * [29] Wickstrøm, K.K., Trosten, D.J., Løkse, S., Boubekki, A., Mikalsen, K.ø., Kampffmeyer, M.C., Jenssen, R.: RELAX: Representation Learning Explainability. International Journal of Computer Vision 131(6), 1584–1610 (2023)
  * [30] Wightman, R., Touvron, H., Jégou, H.: Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.00476 (2021)
  * [31] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. pp. 818–833. Springer (2014)
  * [32] Zhang, J., Bargal, S.A., Lin, Z., Brandt, J., Shen, X., Sclaroff, S.: Top-down neural attention by excitation backprop. International Journal of Computer Vision 126(10), 1084–1102 (2018)
  * [33] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Object detectors emerge in deep scene CNNs. arXiv preprint arXiv:1412.6856 (2014)
  * [34] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)

##  Appendix 0.A Appendix

###  0.A.1 Metrics

A significant challenge in XAI is the absence of standardized evaluation
metrics. Quantus [18] addresses this challenge by offering a flexible toolkit
that gathers and systematizes a diverse range of evaluation metrics for
explanation methods. Within the framework, we selected the following metrics
for evaluation:

  * •

Pointing-Game [32] establishes whether the most salient attribution aligns
with the target object, reflecting the model’s precision in highlighting
relevant image areas.

  * •

Top-K Intersection [27] measures the overlap between the binarized explanation
and the ground truth mask for the top $k$ features, aiding in assessing
alignment with actual object locations.

  * •

Relevance Rank Accuracy and Relevance Mass Accuracy [1] evaluate,
respectively, the concentration of high-attribution pixels within the ground
truth mask and the extent to which positive attributions are confined to the
ground truth area.

  * •

AUC [13] assesses how well attributions correlate with the ground-truth mask
by analyzing the receiver operating characteristic curve.

  * •

Sparseness [7] measures explanatory complexity. Using the Gini Index,
Sparseness assesses whether the salient features identified are both
predictive and free of excess noise, criteria important for interpretability.

Figure 4: Additional results for PASCAL VOC 2012. The title on the left
displays the true labels from PASCAL VOC 2012, alongside ImageNet labels
predicted by ResNet50. Grad-CAM utilizes the predicted labels to generate
saliency maps.


$\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color<EMAIL_ADDRESS>and the target only has
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>their difference on values also encompasses heap locations and capabilities,
but we elide that here since this is sufficient to draw our analogy.  Thus, to
prove $\mathit{RSC}$, they rely on a cross-language relation on values, which
is scaled to trace actions, and then scaled point-wise to traces (analogously
to what we have done in Sections 3.3, 3.4 and 5.2).

Besides defining a relation on traces (which is an instance of $\sim$), they
also define a relation between source and target safety properties that
supports concurrent programs.111111They call those safety properties monitors
since they focus on safety [60] and indicate
$\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color<EMAIL_ADDRESS>with
$\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color<EMAIL_ADDRESS>and
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>with
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>Thus, they really provide an instantiation of $\tau$ that maps all safe source
traces to the related target ones. This ensures that no additional target
trace is introduced in the target property, and source safety property are
mapped to target safety ones by $\tau$.

Their compiler is then proven to generate code that respects $\tau$, so they
achieve a variation of $\mathit{RSC}^{\tau}$ Their proofs are based on
standard techniques either for secure compilation (i.e., trace-based
backtranslation [54]) and for correct compilation (i.e., forward/backward
simulation [35]).

###  0.F.2 Trace-Relating Robust Preservation of Hypersafety Properties

Patrignani and Garg [52] study the preservation of hypersafety from the
perspective of secure compilation. Again, their result can be interpreted in
our setting. They consider reactive systems, where trace alphabets are
partitioned in input actions
$\mathit{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\alpha?}}}$
and output actions
$\mathit{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\alpha!}}}$,
whose concatenation generate traces
$\mathit{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\overline{\alpha?\alpha!}}}}$.
We use the same notation as before and indicate such sequences as
$\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color<EMAIL_ADDRESS>and
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>respectively. The set of target output actions
$\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha!}}}$
includes an action
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>that has no source counterpart (i.e.,
$\nexists\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\alpha?}}}\sim\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\surd}}}}}}$),
and whose output does not depend on internal state (thus it cannot leak
secrets).121212Technically, they assume a set of
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>actions, but for this analogy a single action suffices.  By emitting
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>whenever undesired inputs are fed to a compiled program (e.g., passing a
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>when a
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>is expected), hypersafety is preserved (as
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>does not leak secrets) [52].

More formally, they assume a relation on actions $\sim$ that is total on the
source actions and injective. From there, they define
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>here corresponds to an instance of $\tau$—that maps the set of valid source
traces to the set of valid target traces (that now mention
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>as follows:

$\displaystyle{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{TPC}}}}\left(\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\right)}}}=$
$\displaystyle\
\left\\{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}\,\mathrel{}\middle|\mathrel{}\,\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}\in\bigcup_{n\in\mathbb{N}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{int_{n}}\left(\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\right)}}}\right\\}\;\text{
where}\;{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{int_{0}}\left(\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\right)}}}=\
\left\\{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}\,\mathrel{}\middle|\mathrel{}\,\exists\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{s}}}\in\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\wedge\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{s}}}\sim\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>$\displaystyle{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{int_{n+1}}\left(\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\right)}}}=$
$\displaystyle\
\left\\{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}\,\mathrel{}\middle|\mathrel{}\,(i)\
\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}\equiv\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}_{1}}}}\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha?}}}\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\surd}}}\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t_{2}}}}\wedge(ii)\
\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t_{1}\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{t}}}_{2}}}}\in{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{int_{n}}\left(\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0}{\pi_{S}}}}\right)}}}\wedge(iii)\
{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{undesired}\left(\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha?}}}\right)}}}\right\\}$

where
${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathtt{undesired}\left(\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha?}}}\right)}}}$
indicates that
$\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha?}}}$
is an undesired input (intuitively, this is an information that can be derived
from the set of source traces [52]).

Informally, given a set of source traces
$\mathsf{{\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1}\pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color<EMAIL_ADDRESS>$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>generates all target traces that are related (pointwise) to a source trace
(case $\mathtt{int_{0}}$). Then (case $\mathtt{int_{n+1}}$), it adds all
traces
($\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>with interleavings of undesired input
$\bm{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\alpha?}}}$
(point iii) followed by
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color@cmyk@fill{0}{0.77}{0.87}{0}{\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>(point i) as long as the interleavings split a trace
$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>that has already been mapped (point ii).

$\mathbf{{\color[rgb]{1,0.23,0.13}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.23,0.13}\pgfsys@color@cmyk@stroke{0}{0.77}{0.87}{0}\pgfsys@color<EMAIL_ADDRESS>is an instance of $\tau$ that maps source hypersafety to target hypersafety
(and therefore, safety to safety), thus our theory can be instantiated for the
preservation of these classes of hyperproperties as well.

## References

  * Abadi et al. [1999] M. Abadi, A. Banerjee, N. Heintze, and J. G. Riecke.  A core calculus of dependency.  _POPL_ , 1999\.
  * Abate et al. [2019] C. Abate, R. Blanco, D. Garg, C. Hriţcu, M. Patrignani, and J. Thibault.  Journey beyond full abstraction: Exploring robust property preservation for secure compilation.  _CSF_ , 2019.
  * Ahmed et al. [2018] A. Ahmed, D. Garg, C. Hriţcu, and F. Piessens.  Secure compilation (Dagstuhl Seminar 18201).  _Dagstuhl Reports_ , 8(5), 2018.
  * Anand et al. [2017] A. Anand, A. Appel, G. Morrisett, Z. Paraskevopoulou, R. Pollack, O. S. Belanger, M. Sozeau, and M. Weaver.  CertiCoq: A verified compiler for Coq.  CoqPL Workshop, 2017.
  * Backhouse and Backhouse [2004] K. Backhouse and R. Backhouse.  Safety of abstract interpretations for free, via logical relations and Galois connections.  _Science of Computer Programming_ , 51(1-2), 2004.
  * Barthe et al. [2018] G. Barthe, B. Grégoire, and V. Laporte.  Secure compilation of side-channel countermeasures: the case of cryptographic “constant-time”.  _CSF_ , 2018.
  * Beringer et al. [2014] L. Beringer, G. Stewart, R. Dockins, and A. W. Appel.  Verified compilation for shared-memory C.  _ESOP_ , 2014.
  * Besson et al. [2019] F. Besson, S. Blazy, and P. Wilke.  A verified CompCert front-end for a memory model supporting pointer arithmetic and uninitialised data.  _Journal of Automated Reasoning_ , 62(4), 2019.
  * Boldo et al. [2015] S. Boldo, J. Jourdan, X. Leroy, and G. Melquiond.  Verified compilation of floating-point computations.  _Journal of Automated Reasoning_ , 54(2), 2015.
  * Busi et al. [2019] M. Busi, P. Degano, and L. Galletta.  Translation validation for security properties.  _CoRR_ , abs/1901.05082, 2019.
  * Cao et al. [2018] Q. Cao, L. Beringer, S. Gruetter, J. Dodds, and A. W. Appel.  VST-Floyd: A separation logic tool to verify correctness of C programs.  _Journal of Automated Reasoning_ , 61(1-4), 2018.
  * Carbonneaux et al. [2014] Q. Carbonneaux, J. Hoffmann, T. Ramananandro, and Z. Shao.  End-to-end verification of stack-space bounds for C programs.  _PLDI_ , 2014.
  * Cimpanu [2019] C. Cimpanu.  Microsoft: 70 percent of all security bugs are memory safety issues.  ZDNet, 2019.
  * Clarkson and Schneider [2010] M. R. Clarkson and F. B. Schneider.  Hyperproperties.  _JCS_ , 18(6), 2010.
  * Cousot [2002] P. Cousot.  Constructive design of a hierarchy of semantics of a transition system by abstract interpretation.  _TCS_ , 277(1-2), 2002\.
  * Cousot and Cousot [1977] P. Cousot and R. Cousot.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints.  _POPL_ , 1977.
  * D’Silva et al. [2015] V. D’Silva, M. Payer, and D. X. Song.  The correctness-security gap in compiler optimization.  _S &P Workshops_, 2015.
  * Engelfriet [1985] J. Engelfriet.  Determinacy implies (observation equivalence = trace equivalence).  _TCS_ , 36, 1985.
  * Focardi and Gorrieri [1995] R. Focardi and R. Gorrieri.  A taxonomy of security properties for process algebras.  _JCS_ , 3(1), 1995.
  * Gardiner et al. [1994] P. H. Gardiner, C. E. Martin, and O. De Moor.  An algebraic construction of predicate transformers.  _Science of Computer Programming_ , 22(1-2), 1994.
  * Giacobazzi and Mastroeni [2018] R. Giacobazzi and I. Mastroeni.  Abstract non-interference: a unifying framework for weakening information-flow.  _ACM Transactions on Privacy and Security_ , 21(2), 2018.
  * Goguen and Meseguer [1982] J. A. Goguen and J. Meseguer.  Security policies and security models.  _S &P_, 1982.
  * Gu et al. [2018] R. Gu, Z. Shao, J. Kim, X. N. Wu, J. Koenig, V. Sjöberg, H. Chen, D. Costanzo, and T. Ramananandro.  Certified concurrent abstraction layers.  _PLDI_ , 2018.
  * Haller et al. [2016] I. Haller, Y. Jeon, H. Peng, M. Payer, C. Giuffrida, H. Bos, and E. van der Kouwe.  TypeSan: Practical type confusion detection.  _CCS_ , 2016\.
  * [25] Heartbleed.  The Heartbleed bug.  http://heartbleed.com/, 2014.
  * Hriţcu et al. [2019] C. Hriţcu, D. Chisnall, D. Garg, and M. Payer.  Secure compilation.  SIGPLAN PL Perspectives Blog, 2019.
  * Hur and Dreyer [2011] C. Hur and D. Dreyer.  A Kripke logical relation between ML and assembly.  _POPL_ , 2011.
  * Jeffrey and Rathke [2005] A. Jeffrey and J. Rathke.  Java Jr: Fully abstract trace semantics for a core Java language.  _ESOP_ , 2005.
  * Juglaret et al. [2016] Y. Juglaret, C. Hriţcu, A. Azevedo de Amorim, B. Eng, and B. C. Pierce.  Beyond good and evil: Formalizing the security guarantees of compartmentalizing compilation.  _CSF_ , 2016.
  * Kang et al. [2015] J. Kang, C. Hur, W. Mansky, D. Garbuzov, S. Zdancewic, and V. Vafeiadis.  A formal C memory model supporting integer-pointer casts.  _PLDI_ , 2015.
  * Kang et al. [2016] J. Kang, Y. Kim, C.-K. Hur, D. Dreyer, and V. Vafeiadis.  Lightweight verification of separate compilation.  _POPL_ , 2016.
  * Lamport and Schneider [1984] L. Lamport and F. B. Schneider.  Formal foundation for specification and verification.  In _Distributed Systems: Methods and Tools for Specification, An Advanced Course_ , 1984.
  * Lattner [2011] C. Lattner.  What every C programmer should know about undefined behavior #1/3.  LLVM Project Blog, 2011.
  * Leroy [2009a] X. Leroy.  Formal verification of a realistic compiler.  _CACM_ , 52(7), 2009a.
  * Leroy [2009b] X. Leroy.  A formally verified compiler back-end.  _JAR_ , 43(4), 2009b.
  * Leroy [2017] X. Leroy.  The formal verification of compilers (DeepSpec Summer School 2017), 2017.
  * Mastroeni and Pasqua [2018] I. Mastroeni and M. Pasqua.  Verifying bounded subset-closed hyperproperties.  _SAS_ , 2018.
  * McCarthy and Painter [1967] J. McCarthy and J. Painter.  Correctness of a compiler for arithmetic expressions.  _Mathematical Aspects Of Computer Science 1_ , 19 of Proceedings of Symposia in Applied Mathematics, 1967.
  * Melton et al. [1986] A. Melton, D. A. Schmidt, and G. E. Strecker.  Galois connections and computer science applications.  In _Proceedings of a Tutorial and Workshop on Category Theory and Computer Programming_ , 1986.
  * Milner [1982] R. Milner.  _A Calculus of Communicating Systems_.  Springer-Verlag, Berlin, Heidelberg, 1982.
  * Milner and Weyhrauch [1972] R. Milner and R. Weyhrauch.  Proving compiler correctness in a mechanized logic.  In _Proceedings of 7th Annual Machine Intelligence Workshop, volume 7 of Machine Intelligence_ , 1972.
  * Morris [1973] F. L. Morris.  Advice on structuring compilers and proving them correct.  _POPL_ , 1973\.
  * Mullen et al. [2016] E. Mullen, D. Zuniga, Z. Tatlock, and D. Grossman.  Verified peephole optimizations for CompCert.  _PLDI_ , 2016.
  * Naumann [1998] D. A. Naumann.  A categorical model for higher order imperative programming.  _Mathematical Structures in Computer Science_ , 8(4), 1998.
  * Naumann and Ngo [2019] D. A. Naumann and M. Ngo.  Whither specifications as programs.  In _International Symposium on Unifying Theories of Programming_. Springer, 2019.
  * Neis et al. [2015] G. Neis, C. Hur, J. Kaiser, C. McLaughlin, D. Dreyer, and V. Vafeiadis.  Pilsner: a compositionally verified compiler for a higher-order imperative language.  _ICFP_ , 2015.
  * New et al. [2016] M. New, W. J. Bowman, and A. Ahmed.  Fully abstract compilation via universal embedding.  In _ICFP’16_ , 2016.
  * Pasqua and Mastroeni [2017] M. Pasqua and I. Mastroeni.  On topologies for (hyper)properties.  _CEUR_ , 2017.
  * Patrignani [2015] M. Patrignani.  _The Tome of Secure Compilation: Fully Abstract Compilation to Protected Modules Architectures_.  PhD thesis, KU Leuven, Leuven, Belgium, 2015.
  * Patrignani [2020] M. Patrignani.  Why should anyone use colours? or, syntax highlighting beyond code snippets, 2020.
  * Patrignani and Clarke [2015] M. Patrignani and D. Clarke.  Fully abstract trace semantics for protected module architectures.  _Computer Languages, Systems & Structures_, 42, 2015.
  * Patrignani and Garg [2017] M. Patrignani and D. Garg.  Secure compilation and hyperproperty preservation.  _CSF_ , 2017.
  * Patrignani and Garg [2019] M. Patrignani and D. Garg.  Robustly safe compilation.  _ESOP_ , 2019\.
  * Patrignani et al. [2019] M. Patrignani, A. Ahmed, and D. Clarke.  Formal approaches to secure compilation: A survey of fully abstract compilation and related work.  _ACM Computing Surveys_ , 2019.
  * Patterson and Ahmed [2019] D. Patterson and A. Ahmed.  The next 700 compiler correctness theorems (functional pearl).  _PACMPL_ , 3(ICFP), 2019.
  * Ramananandro et al. [2015] T. Ramananandro, Z. Shao, S. Weng, J. Koenig, and Y. Fu.  A compositional semantics for verified separate compilation and linking.  _CPP_ , 2015.
  * Regehr [2010] J. Regehr.  A guide to undefined behavior in C and C++, part 3.  Embedded in Academia blog, 2010.
  * Sabelfeld and Sands [2005] A. Sabelfeld and D. Sands.  Dimensions and principles of declassification.  _CSFW_ , 2005.
  * Sabry and Wadler [1997] A. Sabry and P. Wadler.  A reflection on call-by-value.  _ACM Transactions on Programming Languages and Systems_ , 19(6), 1997.
  * Schneider [2000] F. B. Schneider.  Enforceable security policies.  _TISSEC_ , 3(1), 2000.
  * Sevcík et al. [2013] J. Sevcík, V. Vafeiadis, F. Z. Nardelli, S. Jagannathan, and P. Sewell.  CompCertTSO: A verified compiler for relaxed-memory concurrency.  _J. ACM_ , 60(3), 2013.
  * Skorstengaard et al. [2019] L. Skorstengaard, D. Devriese, and L. Birkedal.  StkTokens: Enforcing Well-bracketed Control Flow and Stack Encapsulation Using Linear Capabilities.  _Proc. ACM Program. Lang._ , 3(POPL), 2019.
  * Stewart et al. [2015] G. Stewart, L. Beringer, S. Cuellar, and A. W. Appel.  Compositional CompCert.  _POPL_ , 2015.
  * Tan et al. [2019] Y. K. Tan, M. O. Myreen, R. Kumar, A. Fox, S. Owens, and M. Norrish.  The verified CakeML compiler backend.  _Journal of Functional Programming_ , 29, 2019.
  * Wang et al. [2012] X. Wang, H. Chen, A. Cheung, Z. Jia, N. Zeldovich, and M. F. Kaashoek.  Undefined behavior: What happened to my code? _APSYS_ , 2012.
  * Wang et al. [2013] X. Wang, N. Zeldovich, M. F. Kaashoek, and A. Solar-Lezama.  Towards optimization-safe systems: Analyzing the impact of undefined behavior.  _SOSP_ , 2013.
  * Wang et al. [2019] Y. Wang, P. Wilke, and Z. Shao.  An abstract stack based approach to verified compositional compilation to machine code.  _PACMPL_ , 3(POPL), 2019.
  * Xia et al. [2020] L. Xia, Y. Zakowski, P. He, C. Hur, G. Malecha, B. C. Pierce, and S. Zdancewic.  Interaction trees: representing recursive and impure programs in Coq.  _PACMPL_ , 4(POPL), 2020.
  * Zakinthinos and Lee [1997] A. Zakinthinos and E. S. Lee.  A general theory of security properties.  _S &P_, 1997.
  * Zhao et al. [2012] J. Zhao, S. Nagarakatte, M. M. K. Martin, and S. Zdancewic.  Formalizing the LLVM intermediate representation for verified program transformations.  _POPL_ , 2012.

Open Access This chapter is licensed under the terms of the Creative
CommonsAttribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, as long as
you give appropriate credit to the original author(s) and the source, provide
a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the
chapter’s Creative Commons license, unless indicated otherwise in a credit
line to the material. If material is not included in the chapter’s Creative
Commons license and your intendeduse is not permitted by statutory regulation
or exceeds the permitted use, you will need to obtain permission directly from
the copyright holder.


# Do Large Language Models know what humans know?

Sean Trott  Cameron Jones  Tyler Chang  James Michaelov  Benjamin Bergen
Department of Cognitive Science, UC San Diego

###### Abstract

Humans can attribute mental states to others, a capacity known as Theory of
Mind. However, it is unknown to what extent this ability results from an
innate biological endowment or from experience accrued through child
development, particularly exposure to language describing others’ mental
states. We test the viability of the language exposure hypothesis by assessing
whether models exposed to large quantities of human language develop evidence
of Theory of Mind. In a pre-registered analysis, we present a linguistic
version of the False Belief Task, widely used to assess Theory of Mind, to
both human participants and a state-of-the-art Large Language Model, GPT-3.
Both are sensitive to others’ beliefs, but the language model does not perform
as well as the humans, nor does it explain the full extent of their behavior,
despite being exposed to more language than a human would in a lifetime. This
suggests that while language exposure may in part explain how humans develop
Theory of Mind, other mechanisms are also responsible.

###### keywords:

Theory of Mind, Large Language Models, language

Humans reason about the mental states of others, even when these states
diverge from their own. This capacity––sometimes called Theory of
Mind––appears critical for human social cognition (Leslie, 2001). Yet despite
consensus on the importance of Theory of Mind, there remains considerable
debate about the evolutionary (Premack Woodruff, 1978; Krupenye Call, 2019)
and developmental (Bedny ., 2009; de Villiers de Villiers, 2014) origins of
this ability. Specifically, how much of our Theory of Mind capacity results
from an innate, biologically evolved adaptation (Bedny ., 2009), and how much
is assembled from experience (Hughes ., 2005)? The answers to these questions
may also tell us what kinds of biological and artificial entities have Theory
of Mind.

A leading experience-based view proposes that Theory of Mind is built in part
from exposure to language (de Villiers de Villiers, 2014). Children develop an
understanding that others have different mental states from verbs like “know”
and “believe” (JR. Brown ., 1996), the structure of conversation (Harris,
2005), and certain syntactic structures, like sentential complements (e.g.,
“Mary thought that Fred went to the movies”) (Hale Tager-Flusberg, 2003).
However, current evidence does not address the question of to what extent
linguistic input can account for our Theory of Mind capacity. Can human-level
Theory of Mind emerge out of exposure to linguistic input alone, or does it
depend on linking that input to a distinct (possibly innate) mechanism or non-
linguistic representation? Overcoming these limitations would require a
measure of exactly how much Theory of Mind-consistent behavior can be acquired
through exposure to language alone.

Only recently has constructing such a measure become tractable, with the
advent of Large Language Models (LLMs). LLMs are computational models with
billions of parameters trained on hundreds of billions of words to predict the
next word (or a masked word). As others have noted (Bender Koller, 2020), the
training regime for these models does not include social interaction,
experience in a physical environment, or even the notion of communicative
intent. Further, their network architecture is not designed with Theory of
Mind pre-coded. And yet, LLMs display behavior consistent with the acquisition
of some linguistic structure (Manning ., 2020). LLMs have also been the
subject of recent public discussion (Johnson, 2022), including speculation
that they can acquire something akin to Theory of Mind. They thus serve as
useful baselines for what kinds of behavior can be produced merely by exposure
to distributional statistics of language in general, and for Theory of Mind in
particular.

We investigated whether GPT-3 (T. Brown ., 2020), a state-of-the-art LLM,
produced behavior consistent with Theory of Mind using the widely used False
Belief Task (Wimmer Perner, 1983). Although the False Belief Task has been
criticized on several grounds (Bloom German, 2000), it has been used
extensively to assess Theory of Mind capacity in humans and other animals
(Premack Woodruff, 1978; Krupenye Call, 2019).

We presented the same False Belief Task stimuli to humans and GPT-3. In each
passage, a character places an object in a Start location, and the object is
moved to an End location. We manipulated the Knowledge State of the main
character— whether they know the object was moved (True Belief) or not (False
Belief). We counterbalanced whether the First Mention and most Recent Mention
of a location was the Start or End location. Humans and the LLM then completed
a cue sentence indicating the character’s belief about the object location.
This Knowledge Cue was either Explicit (“Sean thinks the book is in the  ”) or
Implicit (“Sean goes to get the book from the  ”). The correct (Theory of
Mind-consistent) completion is the End location on True Belief trials, and the
Start location on False Belief trials. We measured whether participants
responded with the Start or End location, and the relative probability that
GPT-3 assigned to the Start location (the Log-Odds of Start vs. End).

We ask two key questions. First, are LLMs sensitive to false belief? If so,
then biological and artificial agents could in principle develop behavior
consistent with Theory of Mind from input-driven mechanisms alone, such as
exposure to language (de Villiers de Villiers, 2014). The second question is
whether LLMs fully explain human behavior in the False Belief Task. If so,
this would show that language exposure is not only a viable mechanism in
general but that it may in fact be sufficient to explain how humans in
particular come to display behavior indicative of Theory of Mind. If they do
not, however, we can infer that humans rely on some other crucial mechanism,
such as an innate capacity or experience with more than just language.

##  1 Method

### False Belief Passages

We constructed 12 template passages (items) that conformed to the standard
False Belief Task structure (Wimmer Perner, 1983). In each, a main character
puts an object in a Start location, and a second character moves the object to
an End location. The last sentence of each passage implies that the main
character believes the object is in some (omitted) location (e.g. “Sean thinks
the book is in the  .”). We generated 16 versions of each item (192 passages)
that varied across 4 dimensions. The primary dimension was Knowledge State:
whether the main character knows (True Belief) or does not know (False belief)
that the object has changed location; this was manipulated by the main
character being present or not when the second character moved the object. We
also manipulated whether the First Mention and most Recent Mention of a
location is the Start or End location; and Knowledge Cue: whether the main
character’s belief is stated implicitly (e.g., “goes to get the book from the
”) or explicitly (e.g., “thinks the book is in the  ”). Each location was
mentioned twice in each passage.

> Example Passage: “Sean is reading a book. When he is done, he puts the book
> in the box and picks up a sweater from the basket. Then, Anna comes into the
> room. Sean leaves to get something to eat in the kitchen. While he is away,
> Anna moves the book from the box to the basket. Sean comes back into the
> room and wants to read more of his book.”
>
> Implicit Cue: “Sean goes to get the book from the  .”
>
> Explicit Cue: “Sean thinks the book is in the  .”

### GPT-3 Predictions

We elicited from GPT-3 the log probability of each possible location (Start
vs. End) at the end of each passage version, equal to the log probabilities of
those locations in a free-response completion. Where a location comprised
multiple tokens we summed the log probabilities. We accessed GPT-3 Davinci
through the OpenAI API. Using the Log-Odds Ratio,
$\textrm{log}(\textrm{p}(\text{Start}))-\textrm{log}(\textrm{p}(\text{End})$),
higher values indicate larger relative probabilities of the Start location.

### Human Participant Responses

1161 participants from Amazon’s Mechanical Turk platform were paid $1 for
their time. Each read a single passage (except the final sentence), at their
own pace. On a new page, they were asked to complete the final sentence of the
passage by entering a single word in a free-response text input. Participants
then completed two free-response attention check questions that asked for the
true location of the object at the start and the end of the passage.

We preprocessed responses by lowercasing, and removing punctuation, stopwords,
and trailing whitespace. We excluded participants who were non-native English
speakers (13), answered $\geq 1$ attention check incorrectly (513), or
answered the sentence completion with a word that was not the start or end
location (17), retaining 613 trials. These preregistered exclusion criteria
reduced the likelihood that bots as well as participants who did not
successfully comprehend the passage for any reason were included in the human
data. All research was approved by the University of California San Diego’s
Institutional Review Board.

## Results

### Analysis of Large Language Model Behavior

In a pre-registered analysis (https://osf.io/agqwv/), nested model comparisons
determined whether GPT-3 Log-Odds changed as a function of factors such as
Knowledge State (False Belief vs. True Belief).111Code and data are available
on OSF https://osf.io/hu865/ and GitHub https://github.com/UCSD-Language-and-
Cognition-Lab/nlm-fb A linear mixed effects model predicting Log-Odds was
significantly improved by the addition of Knowledge State
[$\chi^{2}(1)=20.813,p<.001$], even accounting for other covariates (e.g.,
First Mention, Recent Mention, and Knowledge Cue). Log-Odds was lower in the
True Belief condition, reflecting the correct prediction that characters
should be more likely to look in the End Location if they are aware that the
object was moved (see Figure 1).

Critically, this main effect of Knowledge State indicates that GPT-3 is
sensitive to the manipulation of a character’s beliefs about where an object
is located. However, we note that despite this sensitivity, the model’s raw
accuracy when predicting the most probable out of the Start and End locations
was only 61%, suggesting that the model may not be attaining human-level
sensitivity to implied knowledge states.

Additionally, the model was further improved by an interaction between
Knowledge State and Knowledge Cue (Explicit vs. Implicit)
[$\chi^{2}(1)=5.74,p=.02$]; the effect of Knowledge State was attenuated in
the Implicit condition [$\beta=0.62,SE=0.26$]. Nevertheless, a main effect of
Knowledge State was still detected in each of the Explicit
[$\chi^{2}(1)=21.95,p<.001$] and Implicit [$\chi^{2}(1)=8.61,p=.003$]
conditions. Two other covariates (First Mention and Recent Mention) were not
significant predictors of Log-Odds. However, the coefficient assigned to
Knowledge Cue was significantly negative [$\beta=-2.9,SE=0.18,p<.001$],
indicating a main effect of Knowledge Cue (see Figure 1). GPT-3 was biased
towards the End location (i.e., the true location of the object) in the
Implicit condition, and towards the Start location in the Explicit condition.
GPT-3 predicts that explicit cues to belief state correlate with false
beliefs, demonstrating that this may be learnable from the statistics of
language.

Figure 1: GPT-3 Log-Odds of Start vs. End location was higher (i.e. Start was
relatively more likely) in the False Belief than True Belief condition
($\chi^{2}(1)=20.813,p<.001$). This suggests that GPT-3’s predictions are
sensitive to the character’s implied belief state: the character is unaware
that the object has moved to the End location if they did not observe it being
moved. This effect was observed both when the Knowledge Cue was Implicit
(“Sean goes to get the book from the…”) or Explicit (“Sean thinks the book is
in the…”); however, the effect was attenuated in the Implicit condition
($\chi^{2}(1)=5.74,p=.02$). Figure 2: Both human participants and GPT-3 were
more likely to say that a character believed an object was in the Start
location when the character had not observed the object being moved to the End
location (False Belief). This effect was stronger for humans than for GPT-3,
and there was a marginal effect of Knowledge State (True vs. False Belief) in
humans that could not be accounted for by GPT-3 predictions
($\chi^{2}(1)=30.4,p<0.001$).

Our second, critical question was whether Knowledge State continued to explain
human behavior even accounting for the Log-Odds obtained from GPT-3. We found
that a version of the Base model that included GPT-3 Log-Odds as a predictor
was significantly improved by adding Knowledge State as a predictor
[$\chi^{2}(1)=30.4,p<0.001$]. This final result implies that human responses
are influenced by Knowledge State in a way that is not captured by GPT-3. That
is, GPT-3 cannot fully account for human sensitivity to knowledge states. This
is highlighted by the contrast in Figure 2: while both human participants and
GPT-3 were sensitive to Knowledge State, humans displayed a much stronger
effect across conditions.

## Discussion

We asked whether exposure to linguistic input alone could account for human
sensitivity to knowledge states, a critical component of Theory of Mind.

We found that GPT-3’s predictions were sensitive to a character’s implied
knowledge states, with $61\%$ accuracy. This demonstrates that exposure to
linguistic input alone can in principle account for some sensitivity to false
belief. However, GPT-3 was markedly less sensitive than humans (who had $83\%$
accuracy—see also probabilities in Figure 2). Most critically, human behavior
was not explained fully by that of GPT-3 in a statistical model. This entails
that the capacities underlying human behavior in this Theory of Mind task
cannot be explained purely by exposure to language statistics.

Does this mean that contemporary LLMs have Theory of Mind? The evidence
presented here is mixed. State-of-the-art models display sensitivity to the
beliefs of others in a False Belief Task, but not to the degree that humans
do. This could suggest that they display a less developed form of Theory of
Mind that is qualitatively similar to that of humans. Alternatively, their
better-than-chance performance could be achieved by other means—a kind of
Clever Hans effect—that do not reflect a capacity equivalent to Theory of
Mind. Resolving this question ultimately hinges on the relationship between
the instrument used to test Theory of Mind and the construct itself. Does
having Theory of Mind merely mean displaying Theory of Mind-like behavior?

This question will become even more critical in the near future. LLMs will
continue to increase in size and will be trained on larger data sets—orders of
magnitude more words than a human is exposed to in a lifetime. This will
likely lead to improvement in Theory of Mind tasks. If machines behave
indistinguishably from humans in these tasks, the question of whether
achievement on the False Belief Task itself constitutes sufficient evidence
for Theory of Mind will have not just philosophical but also ethical
consequences. And as LLMs grow, we will also need to reckon with the question
of how big a model is too big to count as a valid demonstration of what humans
could learn from language exposure alone. A truly massive future language
model could in principle capture human behavior, but if it is trained on many
lifetimes of language, this does not entail that the human capacity similarly
derives from language exposure alone.

Where does human Theory of Mind come from? It could emerge in part from an
innate, biologically evolved capacity to reason about the mental states of
others (Bedny ., 2009). It might depend on experience, including language
input (de Villiers de Villiers, 2014). The current results help quantify the
contribution of language input. They suggest that language statistics alone
are sufficient to generate some sensitivity to false belief, but crucially,
not to fully account for human sensitivity to false belief. Human Theory of
Mind may involve linking this linguistic input to a rich array of other
embodied, social experiences.

## References

  * Bedny . (2009) bedny_2009_GrowingBlindDoesBedny, M., Pascual-Leone, A. Saxe, RR.  200907\.  Growing up Blind Does Not Change the Neural Bases of Theory of Mind Growing up blind does not change the neural bases of Theory of Mind.  Proceedings of the National Academy of Sciences1062711312–11317.  10.1073/pnas.0900010106
  * Bender Koller (2020) bender_2020_ClimbingNLUMeaningBender, EM. Koller, A.  202007\.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics ( 5185–5198).  OnlineAssociation for Computational Linguistics.  10.18653/v1/2020.acl-main.463
  * Bloom German (2000) bloom_2000_TwoReasonsAbandonBloom, P. German, TP.  200010\.  Two Reasons to Abandon the False Belief Task as a Test of Theory of Mind Two reasons to abandon the false belief task as a test of theory of mind.  Cognition771B25-B31.  10.1016/S0010-0277(00)00096-2
  * JR. Brown . (1996) brown_1996_WhyTalkMentalBrown, JR., Donelan-McCall, N. Dunn, J.  1996\.  Why Talk about Mental States? The Significance of Children’s Conversations with Friends, Siblings, and Mothers Why Talk about Mental States? The Significance of Children’s Conversations with Friends, Siblings, and Mothers.  Child Development673836–849.  10.1111/j.1467-8624.1996.tb01767.x
  * T. Brown . (2020) brown2020languageBrown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, JD., Dhariwal, P.others  2020\.  Language models are few-shot learners Language models are few-shot learners.  Advances in neural information processing systems331877–1901.
  * de Villiers de Villiers (2014) devilliers_2014_RoleLanguageTheoryde Villiers, JG. de Villiers, PA.  2014\.  The Role of Language in Theory of Mind Development The Role of Language in Theory of Mind Development.  Topics in Language Disorders344313–328.  10.1097/TLD.0000000000000037
  * Hale Tager-Flusberg (2003) hale_2003_InfluenceLanguageTheoryHale, CM. Tager-Flusberg, H.  2003\.  The Influence of Language on Theory of Mind: A Training Study The influence of language on theory of mind: A training study.  Developmental Science63346–359.  10.1111/1467-7687.00289
  * Harris (2005) harris_2005_ConversationPretenseTheoryHarris, PL.  2005\.  Conversation, Pretense, and Theory of Mind Conversation, Pretense, and Theory of Mind.  Why Language Matters for Theory of Mind Why language matters for theory of mind ( 70–83).  New York, NY, USOxford University Press.  10.1093/acprof:oso/9780195159912.003.0004
  * Hughes . (2005) hughes2005originsHughes, C., Jaffee, SR., Happé, F., Taylor, A., Caspi, A. Moffitt, TE.  2005\.  Origins of individual differences in theory of mind: From nature to nurture? Origins of individual differences in theory of mind: From nature to nurture?  Child development762356–370.
  * Johnson (2022) Johnson2022Johnson, S.  2022\.  A.I. Is Mastering Language. Should We Trust What It Says? A.I. is mastering language. Should we trust what it says?  The New York Times.  https://www.nytimes.com/2022/04/15/magazine/ai-language.html
  * Krupenye Call (2019) krupenye_2019_TheoryMindAnimalsKrupenye, C. Call, J.  2019\.  Theory of Mind in Animals: Current and Future Directions Theory of mind in animals: Current and future directions.  WIREs Cognitive Science106e1503.  10.1002/wcs.1503
  * Leslie (2001) leslie_2001_TheoryMindLeslie, AM.  200101\.  Theory of Mind Theory of Mind.  NJ. Smelser PB. Baltes (), International Encyclopedia of the Social & Behavioral Sciences International Encyclopedia of the Social & Behavioral Sciences ( 15652–15656).  OxfordPergamon.  10.1016/B0-08-043076-7/01640-5
  * Manning . (2020) manning_2020_EmergentLinguisticStructureManning, CD., Clark, K., Hewitt, J., Khandelwal, U. Levy, O.  202012\.  Emergent Linguistic Structure in Artificial Neural Networks Trained by Self-Supervision Emergent linguistic structure in artificial neural networks trained by self-supervision.  Proceedings of the National Academy of Sciences1174830046–30054.  10.1073/pnas.1907367117
  * Premack Woodruff (1978) premack_1978_DoesChimpanzeeHavePremack, D. Woodruff, G.  197812\.  Does the Chimpanzee Have a Theory of Mind? Does the chimpanzee have a theory of mind?  Behavioral and Brain Sciences14515–526.  10.1017/S0140525X00076512
  * Wimmer Perner (1983) wimmer_1983_BeliefsBeliefsRepresentationWimmer, H. Perner, J.  198301\.  Beliefs about Beliefs: Representation and Constraining Function of Wrong Beliefs in Young Children’s Understanding of Deception Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception.  Cognition131103–128.  10.1016/0010-0277(83)90004-5