# The Queen of England is not England’s Queen:
On the Lack of Factual Coherency in PLMs

Paul Youssef${\dagger}$ Jörg Schlötterer${\dagger}{\ddagger}$ Christin
Seifert${\dagger}$
${\dagger}$University of Marburg, ${\ddagger}$University of Mannheim
{paul.youssef, joerg.schloetterer<EMAIL_ADDRESS>

###### Abstract

Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their
representations and justifies their use as knowledge bases. Previous work has
focused on probing PLMs for factual knowledge by measuring how often they can
correctly predict an _object_ entity given a subject and a relation, and
improving fact retrieval by optimizing the prompts used for querying PLMs. In
this work, we consider a complementary aspect, namely the coherency of factual
knowledge in PLMs, i.e., how often can PLMs predict the _subject_ entity given
its initial prediction of the object entity. This goes beyond evaluating how
much PLMs know, and focuses on the internal state of knowledge inside them.
Our results indicate that PLMs have low coherency using manually written,
optimized and paraphrased prompts, but including an evidence paragraph leads
to substantial improvement. This shows that PLMs fail to model inverse
relations and need further enhancements to be able to handle retrieving facts
from their parameters in a coherent manner, and to be considered as knowledge
bases.

The Queen of England is not England’s Queen:
On the Lack of Factual Coherency in PLMs

Paul Youssef${\dagger}$ Jörg Schlötterer${\dagger}{\ddagger}$ Christin
Seifert${\dagger}$ ${\dagger}$University of Marburg, ${\ddagger}$University of
Mannheim {paul.youssef, joerg.schloetterer<EMAIL_ADDRESS>

##  1 Introduction

Pre-trained Language Models (PLMs) are probed for factual knowledge to
investigate their usage as knowledge bases, and gain a better understanding of
the rich representations they provide Petroni et al. (2019). Previous
extensions have focused on extracting more facts Zhong et al. (2021); Li et
al. (2022b), increasing the consistency of PLMs to paraphrased prompts Elazar
et al. (2021), identifying the parts of PLMs that are responsible for storing
knowledge Dai et al. (2022) and updating facts in them Meng et al. (2022,
2023).

Figure 1: Probing coherency in PLMs. 1) The PLM makes a prediction based on an
entity $\mathcal{S}$ and a relation. 2) The PLM makes a second prediction
based on the same relation and its first prediction $\mathcal{O}^{\prime}$. 3)
If the PLM predicts $\mathcal{S}$ in the second step it shows coherent
behavior.

More recently, Berglund et al. (2023) study the generalization abilities of
PLMs from “A is B” to “B is A”, and show that if a PLM is trained on “The
capital of Malta is Valetta” it will not be able to _correctly_ answer the
question: “Which country has Valetta as its capital?”. In this work, we
introduce an intrinsic and complementary aspect, namely the _coherency_ of
PLMs with respect to factual knowledge. Coherency is not concerned with
correctness of the PLMs’ predictions, but with the internal state of knowledge
in PLMs and its consistency. More concretely, we first ask a PLM to answer the
question: “What is the capital of Malta?”, and if it answers “Berlin”, we ask
it to answer the question: “Which country has Berlin as its capital?”, and if
it answers “Malta”, then we say that the PLM has answered coherently (even
though the answer is factually wrong). Note that in practice we use Cloze
prompts instead of questions to make the task closer to language modeling (see
Figure 1). Intuitively, if a human can tell the capital of a country given
that country’s name, then she is also able to tell the country given its
capital’s name. Given that PLMs are queried with a subject and a relation to
extract a object, we define coherency as the ability of the PLM to infer the
subject given its initial prediction for the object entity and vice versa.

Our contributions are the following: (1) we introduce coherency to investigate
the internal state of factual knowledge in PLMs; (2) we evaluate different
PLMs, showing that they have low coherency; (3) we show that optimized and
paraphrased prompts do not improve coherency, but the use of evidence
paragraphs substantially improves coherency. We make our code
available.111https://github.com/paulyoussef/coherency

##  2 Coherency

PLMs are known to capture vast amount of facts from their pre-training
corpora. This has encouraged the community to consider using them as knowledge
bases (KBs) Petroni et al. (2019); Sung et al. (2021), which can be
constructed without expensive annotations, and which can easily be queried
using natural language. However, the use of PLMs as KBs has many limitations
AlKhamissi et al. (2022). For example, PLMs are quite sensitive to their
prompts, and cannot be easily updated with new facts. Factual knowledge in
PLMs is estimated by evaluating how often PLMs can correctly predict an object
entity $\mathcal{O}$, given a subject entity $\mathcal{S}$ and a relation
$\mathcal{R}$, when provided with a prompt which contains the subject and the
relation: $t($$\mathcal{S}$,$\mathcal{R}$$)$, where $t$ is a function that
maps a subject entity and a relation to a prompt in natural language that
contains the given entity and expresses the relation in natural language,
e.g., (Malta, capital-of) $\rightarrow$ ‘‘The capital of Malta is [MASK]’’. In
this work, we focus on evaluating the coherency of PLMs with respect to the
factual knowledge stored in their parameters, i.e., how often can PLMs predict
$\mathcal{S}$ using $t($$\mathcal{O^{\prime}}$,$\mathcal{R}$$)$, given that it
predicted $\mathcal{O^{\prime}}$ using $t($$\mathcal{S}$,$\mathcal{R}$$)$. For
example, ‘‘The capital of [MASK] is Berlin’’ $\rightarrow$ Malta is coherent
with ‘‘The capital of Malta is [MASK]’’ $\rightarrow$ ‘‘Berlin’’. We do not
evaluate if the predictions are factually correct, because we are interested
in the coherency of the PLMs’ world view, regardless of its correctness .We
show and discuss correctness scores in Appendix A.

Coherency can be easily calculated for 1-1 relations, but is more challenging,
if we consider N-1 or N-M relations, since multiple entities could be correct
when trying to predict the subject entity. To address this, we exclude all
correct entities except the ground truth subject $\mathcal{S}$ in the second
inference step, following Bordes et al. (2013) and Petroni et al. (2019).
Since PLMs are known to have certain biases and are sensitive to the prompts,
we start with predicting the object given the subject in a first round. In a
second round, we start by predicting the subject given the object. The
complete algorithm for estimating coherency in PLMs for all types of relations
is shown in Algorithm 1. After estimating coherency for each relation, we
macro-average over all relations, because we are interested in the average
performance for the use case of PLMs-as-KBs, which involves storing facts from
different types of relations.

Input: PLM, dataset with $n$ relations

Output: coherency

scores_per_relation = []

// iterate over relations

for  _$i\leftarrow 1$ to $n$_ do

scores = []

// iterate over instances

for  _$j\leftarrow 1$ to $m$_ do

// round 1: start with object

$\mathcal{O}_{j}^{\prime}$ =
$PLM($$t_{i}($$\mathcal{S}_{j}$,$\mathcal{R}_{j}$$)$$)$

exclude correct answers except $\mathcal{S}_{j}$

$\mathcal{S}_{j}^{\prime}$ =
$PLM($$t_{i}($$\mathcal{O}_{j}^{\prime}$,$\mathcal{R}_{j}$$)$$)$

if  _partial_match( $\mathcal{S}_{j}^{\prime}$, $\mathcal{S}_{j}$)_ then

scores.append(1)

else

scores.append(0)

// round 2: start with subject

$\mathcal{S}_{j}^{\prime}$ =
$PLM($$t_{i}($$\mathcal{O}_{j}$,$\mathcal{R}_{j}$$)$$)$

exclude correct answers except $\mathcal{O}_{j}$

$\mathcal{O}_{j}^{\prime}$ =
$PLM($$t_{i}($$\mathcal{S}_{j}^{\prime}$,$\mathcal{R}_{j}$$)$$)$

if  _partial_match( $\mathcal{O}_{j}^{\prime}$, $\mathcal{O}_{j}$ )_ then

scores.append(1)

else

scores.append(0)

scores_per_relation.append(mean(scores))

return _mean(scores_per_relation )_

Algorithm 1 Coherency in PLMs

##  3 Experimental Setup

Here, we describe the data and PLMs, which we use, and our experiments in
detail.

###  3.1 Data

In our experiments, we use the T-REx Elsahar et al. (2018) subset of LAMA
Petroni et al. (2019), which is often used to estimate factual knowledge in
PLMs. T-REx consists of 41 relations with their corresponding templates, and
subject-object pairs, for which the relations hold in English. For each of the
relations, a manually-written template is provided, which we use to construct
the prompts. Some statistics and an example from the T-REx subset are shown in
Table 10 in Appendix D.

###  3.2 How coherent are PLMs?

In this experiment, we aim to find out how coherent are PLMs. We mostly focus
on PLMs which are trained to fill in the blanks based on context, since these
make use of a bidirectional context, and we expect them to perform better than
autoregressive PLMs on this task. More specifically, we consider BERT Devlin
et al. (2019), InformBERT Sadeq et al. (2022), T5 Raffel et al. (2020), and
T5-SSM Guu et al. (2020); Roberts et al. (2020). InformBERT adapts the masking
strategy of BERT to focus on more informative tokens. T5-SSM models are
additionally trained with Salient Span Masking objective (SSM), which masks
only named entities in the pre-training phase. More information about the
models are provided in Appendix C. If available, we consider several sizes of
the same model in order to investigate the effect of scaling PLMs on
coherency. For BERT-based models, we only consider entities that correspond to
one token, in order to adhere to the task format from pre-training. We
evaluate all models in a zero-shot setting with no finetuning, since we are
interested in the coherency of factual knowledge in PLMs after the pre-
training phase. For BERT-based models, we choose the token with the highest
probability. For T5-models, we use beam search with 10 beams. We use partial
match, which returns true if one of the two predictions is contained in the
other after converting both to lower case, when comparing the predictions
against the ground truth entities.

For completeness, we also evaluate on autoregressive PLMs. More specifically,
we consider GPT-2 Radford et al. (2019) and GPT-Neo Gao et al. (2020); Black
et al. (2021). For autoregressive PLMs, we use typed querying Kassner et al.
(2021), i.e., we extract a probability distribution over a pre-defined set of
entities from the model, and choose the most probable entity as the final
prediction. Typed querying makes it easy to extract valid answers (entities)
from the PLMs’ outputs, but also makes the task easier for PLMs since it
restricts the output space. We extend the templates from LAMA such that the
subject/object entities appear at the very end. We consider autoregressive
PLMs only in this experiment.

###  3.3 Do optimized prompts improve coherency?

Optimizing prompts leads to better fact retrieval Zhong et al. (2021). In this
experiment, we investigate whether optimized prompts lead to higher coherency
as well. We utilize Shin et al. (2020)’ optimized prompts for T-REx. These
prompts differ from one model to another, and from the models we consider,
optimized prompts are only available for BERT models.

###  3.4 Does providing an evidence paragraph increase coherency?

PLMs can fill in the blanks based on the knowledge they have stored in their
parameters (parametric knowledge), or based on information that is provided in
their inputs (contextual knowledge). The latter boils down to extracting the
right information from the input. Previous work has shown that providing
evidence paragraphs as additional inputs makes PLMs’ predictions more factual
Petroni et al. (2020). Here, we investigate how these evidence paragraphs
affect the coherency of factual knowledge in PLMs. The provided evidence
paragraphs from LAMA contain a Wikipedia paragraph that expresses the facts.
We append the evidence paragraphs to the inputs from the first experiment.

###  3.5 Is Coherency stable across paraphrased prompts?

PLMs are known to be sensitive to the provided prompts, i.e., small
insignificant changes, that preserve the meaning cause the PLMs to change
their predictions Elazar et al. (2021). As a result, retrieving facts from
PLMs is highly affected by the prompts used. In this experiment, we consider
the effect of using paraphrased prompts on coherency. Does coherency stay the
same across different prompts or is it highly variant? We evaluate whether
coherency varies with paraphrased prompts from Elazar et al. (2021)’s ParaRel
dataset. ParaRel provides paraphrases for 38 of the 41 relations in T-Rex. For
each one of the 38 relations, we randomly select a template from ParaRel, and
measure how coherency is changed over 10 runs. We consider bert-base and
t5-base for this experiment.

##  4 Results and Discussion

The results for the first three experiments are shown in Table 1. We show the
results for autoregressive PLMs separtely in Table 2, because we probe
autoregressive PLMs with typed querying. We do not evaluate if the predictions
are factually correct. For correctness scores see Table 4 in Appendix A. Since
we considered only one-token entities from T-REx for BERT models, we show a
normalized version of the results on this subset for better comparability in
Table 6, and the results with the total number of instances in Table 7 in
Appendix A.

#### PLMs show poor coherency.

We notice that all PLMs have poor coherency. Autoregressive PLMs perform even
worse than masked PLMs, even though the task is made easier for them through
typed querying (cf. Section 3.2). The poor performance of autoregressive PLMs
might be due to their unidirectional training objective, whereas masked PLMs
make use of a bidirectional context. Increasing the number of parameters in T5
models leads to consistent improvements in performance. However, this does not
generalize to the BERT models (bert-base performs better than bert-large), and
to the T5 models that are trained with SSM (t5-large-ssm performs better than
t5-3b-ssm). The SSM objective is beneficial for the large variant of T5
(t5-large-ssm improves by 6.5 percentage points over t5-large, and even
outperforms t5-3b, which has 4 times as many parameters). Contrarily, this
improvement does not generalize to the 3b variant (t5-3b outperforms its SSM
counterpart). InformBERT falls short of normal BERT, even though it was shown
to outperform BERT, when it comes to facts retrieval Sadeq et al. (2022).
Hence, better facts retrieval does not necessarily affect coherency
positively. In general, scaling and entity-centric training objectives have to
some extent a positive effect on coherency. We also notice that in most cases
models perform worse in the first round. Round 1 can be more difficult, since
it may involve predicting a specific subject based on a generic object in the
second step (e.g.,‘‘[MASK] is located in Bern’’), whereas the second round
goes into opposite and easier direction (‘‘University of Bern is located in
[MASK]’’). PLMs are known to not provide specific answers Huang et al. (2023).

We show the results per relation type in Table 5 in Appendix A. The evaluation
dataset contains 2 1-1 relations , 23 N-1 relations and 16 N-M relations with
3 of the 16 N-M relations being symmetric. Most PLMs have high coherency on
1-1 relations, but the number of instances for these relations is limited (747
at most), on N-1, N-M and symmetric relations the performance drops
significantly. This shows that N-1 and N-M relations are challenging for PLMs
not just with respect to facts retrieval Petroni et al. (2019), but also with
respect to developing a coherent knowledge state.

We also show and categorize examples from different PLMs in Table 8 in
Appendix B. In general, one can notice that incoherent predictions are due to
: 1) The answer being incorrect in the first step, making it more difficult to
predict the answer in the second step (rows 6-7); 2) The templates being not
specific enough allowing for non-factual completions (row 8); 3) missing
context to retrieve correct relation for non 1-1 relations (row 3).

#### Optimizing prompts does not help.

Optimized prompts lead to a drop in coherency in the second experiment (see
results under optimized prompts in Table 1) l. This shows that prompts that
better retrieve object entities does not help retrieve the corresponding
subject entities. Previous work showed that optimized prompts overfit the
facts distribution of objects Cao et al. (2021), which might negatively affect
their ability to retrieve the subject entities. This is also evident by the
difference in scores between the two rounds.

#### Evidence paragraphs improve coherency.

Including evidence paragraphs in the inputs substantially improves performance
(see results under evidence paragraphs in Table 1). This shows that PLMs are
better at extracting answers from their inputs than recalling them from their
parameters. In fact, adding an evidence paragraph reduces the performance gaps
among models of different sizes and pre-training objectives. This suggests
that retrieval-based approaches are indeed a promising alternative to scaling
language models Kandpal et al. (2023). Still, coherency is not high under this
setting as well. We believe this is due to the PLMs failing to extract the
correct entities or to the conflicts between contextual and parametric
knowledge in PLMs Neeman et al. (2023).

#### Coherency varies across paraphrases.

Table 3 shows the minimum, average and maximum coherency scores with
paraphrased prompts. A break-down in relations is available in Appendix A
(Fig. 2).222Note that, for this experiment, we use only 38 of the 41 relations
in T-Rex – The ones for which paraphrases exist. As with fact retrieval, the
results indicate that prompts have a significant effect on the performance.
For example, there are more than 25 percentage points difference in coherency
between the min and max scores for t5-base. Still, even when considering the
best prompts, the overall coherency score is low.

In general, our analysis shows that PLMs do not possess a coherent knowledge
state. The low coherency might be due: 1) The fact that PLMs make predictions
based on shallow surface level features Poerner et al. (2020); Li et al.
(2022a), which makes PLMs output relevant but incoherent and non-factual
predictions (for an example see row 6 in Table 8). This is inherent to all
PLMs, and requires further architectural improvements; 2) The training data
for PLMs, which might be biased towards certain entities (the more frequent
ones); 3) The uni-directional training in the case of autoregressive PLMs that
makes PLMs sensitive to the order in which the entities are observed.

Model | Round 1 | Round 2 | Avg.
---|---|---|---
bert-base-uncased | 9.74 | 11.81 | 10.78
bert-large-uncased | 9.83 | 10.29 | 10.06
InformBERT | 8.04 | 11.55 | 9.79
t5-base | 9.02 | 10.29 | 9.66
t5-large | 9.07 | 12.03 | 10.55
t5-3b | 8.62 | 23.90 | 16.26
t5-large-ssm | 9.89 | 24.23 | 17.06
t5-3b-ssm | 8.97 | 20.88 | 14.92
w/ optimized prompts |
bert-base-uncased | 1.52 | 12.80 | 7.16
bert-large-uncased | 1.87 | 7.38 | 4.62
w/ evidence paragraphs |
bert-base-uncased | 22.30 | 39.87 | 31.09
bert-large-uncased | 21.05 | 41.98 | 31.52
InformBERT | 43.07 | 46.40 | 44.74
t5-base | 41.40 | 58.31 | 49.85
t5-large | 31.46 | 55.15 | 43.31
t5-3b | 27.06 | 62.89 | 44.98
t5-large-ssm | 50.17 | 43.97 | 47.07
t5-3b-ssm | 48.52 | 41.81 | 45.17
Table 1: Coherency score per round and on average for different PLMs using manually-written, optimized prompts and evidence paragraphs. The highest performance under each category is in bold, and the best performance overall is underlined. Model | Round 1 | Round 2 | Avg.
---|---|---|---
gpt2 | 0.24 | 3.98 | 2.11
gpt-neo-1.3B | 0.44 | 12.85 | 6.65
gpt-neo-2.7B | 0.56 | 11.82 | 6.19
Table 2: Coherency score per round and on average for autoregressive PLMs using manually-written prompts. The highest performance is in bold. Autoregressive PLMs are probed using typed querying. Model | Min. | Avg. | Max. | #Instances
---|---|---|---|---
bert-base-uncased | 3.74 | 11.16 | 19.25 | 2852
t5-base | 6.51 | 16.88 | 31.69 | 27788
Table 3: Coherency scores with different paraphrases. We show the results with
the worst/average/best performing prompts per relation.

##  5 Related Work

#### Reversal curse.

Berglund et al. (2023) investigate the generalization abilities of
autoregressive PLMs from one data form, that is encountered during training (A
is B), to another (B is A), showing a generalization failure. Berglund et al.
(2023) refer to this generalization inability in autoregressive PLMs as the
_reversal curse_. Our work is close but complementary to theirs. We focus on
the coherency of the internal state of factual knowledge in autoregressive
_and_ masked PLMs, _regardless_ of how correct the PLMs’ predictions are.

#### Factual knowledge in PLMs.

PLMs contain vast amounts of linguistic Tenney et al. (2019); Jawahar et al.
(2019), commonsense Davison et al. (2019) and factual knowledge Roberts et al.
(2020) that is captured during pre-training. Many works focus on factual
knowledge in PLMs Youssef et al. (2023), since factual knowledge is said to
contribute to the rich presentations produced by PLMs, and potentially
justifies the use of PLMs as KBs Petroni et al. (2019); Ye et al. (2022). For
example, Shin et al. (2020); Zhong et al. (2021) optimize prompts to extract
more facts from PLMs, Elazar et al. (2021); Fierro and Søgaard (2022)
investigate the sensitivity of PLMs to paraphrased prompts, Malkin et al.
(2022); Wang et al. (2023) debias the outputs of PLMs for better facts
extraction, Meng et al. (2022, 2023) address editing facts in PLMs to make it
possible to correct and update facts. However, these works collectively focus
on extrinsic aspects. We focus on a more intrinsic aspect, i.e., the coherency
of factual knowledge inside PLMs. This complements aspects addressed in
previous work.

##  6 Conclusion

In this work, we focused on evaluating the coherency of factual knowledge in
PLMs. We considered the use of manually-written, optimized, and paraphrased
prompts. Our results indicate poor coherency. The inclusion of an evidence
paragraph leads to substantial improvements. This shows that PLMs can leverage
contextual knowledge better than parametric knowledge and highlights the
importance of retrieval-augmented PLMs. We believe that further improvements
are needed to improve coherency in PLMs, and to consider them as alternatives
to KBs. We believe that future work should focus on further improving PLMs on
the architectural level, the data level, and the interface between them (pre-
training objectives).

##  7 Limitations

Coherency can be easily determined using 1-1 relations. For N-1 or N-M
relations, some potential answers should be excluded. However, it is quite
difficult to exclude every possible answer for certain relations (e.g.,
everyone who is an English native speaker) from the model’s vocabulary. We
only excluded answers that are present in LAMA, following previous work Bordes
et al. (2013) and Petroni et al. (2019). This might have had a negative effect
on the results (cf. Section 4, discussion of lower scores in round 1).

## Acknowledgement

We thank Alessandro Noli for helpful discussions.

## References

  * AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022.  A review on language models as knowledge bases.  _arXiv preprint arXiv:2204.06031_.
  * Berglund et al. (2023) Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023.  The reversal curse: Llms trained on" a is b" fail to learn" b is a".  _arXiv preprint arXiv:2309.12288_.
  * Black et al. (2021) Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.  If you use this software, please cite it using these metadata.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc.
  * Cao et al. (2021) Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021.  Knowledgeable or educated guess? revisiting language models as knowledge bases.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1860–1874, Online. Association for Computational Linguistics.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018.  T-REx: A large scale alignment of natural language with knowledge base triples.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Fierro and Søgaard (2022) Constanza Fierro and Anders Søgaard. 2022.  Factual consistency of multilingual pretrained language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  ICML’20. JMLR.org.
  * Huang et al. (2023) Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2023.  Can language models be specific? how? In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 716–727, Toronto, Canada. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _International Conference on Machine Learning_ , pages 15696–15707. PMLR.
  * Kassner et al. (2021) Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021.  Multilingual LAMA: Investigating knowledge in multilingual pretrained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3250–3258, Online. Association for Computational Linguistics.
  * Li et al. (2022a) Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. 2022a.  How pre-trained language models capture factual knowledge? a causal-inspired analysis.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1720–1732, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2022b) Yiyuan Li, Tong Che, Yezhen Wang, Zhengbao Jiang, Caiming Xiong, and Snigdha Chaturvedi. 2022b.  SPE: Symmetrical prompt enhancement for fact probing.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11689–11698, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Malkin et al. (2022) Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022.  Coherence boosting: When your pretrained language model is not paying enough attention.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8214–8236, Dublin, Ireland. Association for Computational Linguistics.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in GPT.  In _Advances in Neural Information Processing Systems_.
  * Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023.  Mass-editing memory in a transformer.  In _The Eleventh International Conference on Learning Representations_.
  * Neeman et al. (2023) Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023.  DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10056–10070, Toronto, Canada. Association for Computational Linguistics.
  * Petroni et al. (2020) Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020.  How context affects language models’ factual predictions.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 803–818, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Sadeq et al. (2022) Nafis Sadeq, Canwen Xu, and Julian McAuley. 2022.  InforMask: Unsupervised informative masking for language model pretraining.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5866–5878, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Sung et al. (2021) Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021.  Can language models be biomedical knowledge bases? In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4723–4734, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019.  What do you learn from context? probing for sentence structure in contextualized word representations.  In _International Conference on Learning Representations_.
  * Wang et al. (2023) Yuhang Wang, Dongyuan Lu, Chao Kong, and Jitao Sang. 2023.  Towards alleviating the object bias in prompt tuning-based factual knowledge extraction.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4420–4432, Toronto, Canada. Association for Computational Linguistics.
  * Ye et al. (2022) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022.  ZeroGen: Efficient zero-shot learning via dataset generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11653–11669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Youssef et al. (2023) Paul Youssef, Osman Koraş, Meijie Li, Jörg Schlötterer, and Christin Seifert. 2023.  Give me the facts! a survey on factual knowledge probing in pre-trained language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15588–15605, Singapore. Association for Computational Linguistics.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021.  Factual probing is [MASK]: Learning vs. learning to recall.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5017–5033, Online. Association for Computational Linguistics.

##  Appendix A Additional Results

#### Correctness.

We investigate how correct the PLMs’ predictions are. For each instance, we
count how often the first prediction in the first round (c1), and in the
second round (c2) were correct. We only consider the first predictions in each
round, since having the incorrect answer in the first inference step of each
round makes it more difficult for the model to answer correctly in the second
inference step. We also report how often are all predictions correct (all
correct). We calculate each score for all relations, and average over all
relations. Results are shown in Table 4. We notice that for all models the c1
scores are higher than the c2 scores. We believe this is because in the first
inference step in round 1, models predict _object_ entities, whereas in the
first step of round 2 they predict _subject_ entities. Predicting subject
entities is more difficult, since their corresponding mask tokens are placed
at the beginning of the templates. This allows for valid completions that do
not contain any entities. For example, if the template is ‘‘[MASK] is the
capital of Malta’’, then ‘‘It’’ is also a valid completion with no entities.
Additionally, predicting the subject entity based on the object entity might
be ambiguous (see discussion in Section 4).

#### Coherency scores per relation type.

Coherency scores per relation type are shown in Table 5.

#### Coherency on a subset.

Table 6 shows a normalized version of the coherency scores using manually-
written prompts.

#### Coherency over relations with different paraphrases.

Figure 2 shows the average coherency scores with standard deviation over
different relations when using paraphrased prompts. Note that bert-base-
uncased has less relations than t5-base (36 vs. 38), since some relations
ended up with no instances after excluding multi-token entities. In general,
we notice high standard deviation for most relations.

#### Coherency scores with the number of instances.

Table 7 shows the coherency scores with the size of the test set in instances.

Model | c1 | c2 | All correct | #relations | #Instances
---|---|---|---|---|---
bert-base-uncased | 30.77 | 8.55 | 4.27 | 39 | 2919
bert-large-uncased | 25.96 | 8.39 | 4.22 | 39 | 2919
InformBERT | 22.33 | 5.97 | 4.34 | 39 | 2926
t5-base | 11.03 | 6.21 | 1.30 | 41 | 29672
t5-large | 14.77 | 6.26 | 1.70 | 41 | 29672
t5-3b | 20.93 | 6.10 | 2.33 | 41 | 29672
t5-large-ssm | 18.42 | 4.69 | 2.73 | 41 | 29672
t5-3b-ssm | 19.61 | 4.28 | 2.96 | 41 | 29672
Autoregressive PLMs
gpt2 | 7.70 | 0.43 | 0.04 | 41 | 29672
gpt-neo-1.3B | 17.65 | 0.93 | 0.13 | 41 | 29672
gpt-neo-2.7B | 18.50 | 1.31 | 0.22 | 41 | 29672
w/ optimized prompts |  |  |
bert-base-uncased | 25.27 | 1.49 | 0.02 | 39 | 2919
bert-large-uncased | 31.92 | 2.94 | 0.10 | 39 | 2919
w/ evidence paragraphs |  |  |
bert-base-uncased | 46.98 | 19.97 | 11.12 | 39 | 2919
bert-large-uncased | 49.66 | 20.27 | 12.98 | 39 | 2919
InformBERT | 49.42 | 45.92 | 24.95 | 39 | 2926
t5-base | 59.77 | 39.28 | 23.99 | 41 | 29672
t5-large | 59.31 | 27.57 | 15.77 | 41 | 29672
t5-3b | 57.35 | 23.17 | 11.73 | 41 | 29672
t5-large-ssm | 44.47 | 47.61 | 23.10 | 41 | 29672
t5-3b-ssm | 41.44 | 46.40 | 21.41 | 41 | 29672
Table 4: Correctness scores in the first inference step of the first round (c1), the second round (c2), and in all inference steps (all correct). Results are averaged over all relations. BERT-based models have less relations and instances, because we consider only one-token entities for these models. Relation Type | 1-1 | N-1 | N-M | symmetric | All
---|---|---|---|---|---
Model | Coherency | #Instances | Coherency | #Instances | Coherency | #Instances | Coherency | #Instances | Coherency | #Instances
bert-base-uncased | 84.11 | 232 | 5.93 | 633 | 8.10 | 2054 | 12.57 | 1927 | 10.78 | 2919
bert-large-uncased | 82.71 | 232 | 6.65 | 633 | 5.38 | 2054 | 15.10 | 1927 | 10.06 | 2919
InformBERT | 81.03 | 232 | 5.28 | 637 | 6.91 | 2057 | 18.46 | 1929 | 9.79 | 2926
t5-base | 36.84 | 747 | 8.55 | 16838 | 7.84 | 12087 | 8.61 | 2882 | 9.66 | 29672
t5-large | 48.90 | 747 | 6.90 | 16838 | 11.02 | 12087 | 14.87 | 2882 | 10.55 | 29672
t5-3b | 61.21 | 747 | 14.84 | 16838 | 12.68 | 12087 | 21.41 | 2882 | 16.26 | 29672
t5-large-ssm | 75.96 | 747 | 17.22 | 16838 | 9.46 | 12087 | 7.44 | 2882 | 17.06 | 29672
t5-3b-ssm | 76.36 | 747 | 13.94 | 16838 | 8.66 | 12087 | 13.00 | 2882 | 14.92 | 29672
Autoregressive PLMs |
gpt2 | 0.26 | 747 | 1.46 | 16838 | 3.27 | 12087 | 0.16 | 2882 | 2.11 | 29672
gpt-neo-1.3B | 3.40 | 747 | 9.71 | 16838 | 2.65 | 12087 | 0.19 | 2882 | 6.65 | 29672
gpt-neo-2.7B | 4.59 | 747 | 6.37 | 16838 | 6.14 | 12087 | 0.51 | 2882 | 6.19 | 29672
w/ optimized prompts |
bert-base-uncased | 1.46 | 232 | 7.54 | 633 | 7.35 | 2054 | 2.36 | 1927 | 7.16 | 2919
bert-large-uncased | 2.38 | 232 | 6.85 | 633 | 1.66 | 2054 | 7.23 | 1927 | 4.62 | 2919
w/ evidence paragraphs |
bert-base-uncased | 87.78 | 232 | 26.49 | 633 | 30.27 | 2054 | 22.66 | 1927 | 31.09 | 2919
bert-large-uncased | 90.30 | 232 | 28.13 | 633 | 28.65 | 2054 | 26.61 | 1927 | 31.52 | 2919
InformBERT | 84.06 | 232 | 42.05 | 637 | 43.43 | 2057 | 31.86 | 1929 | 44.74 | 2926
t5-large-ssm | 84.73 | 747 | 46.38 | 16838 | 43.35 | 12087 | 32.50 | 2882 | 47.07 | 29672
t5-3b-ssm | 86.95 | 747 | 44.80 | 16838 | 40.47 | 12087 | 27.10 | 2882 | 45.17 | 29672
t5-base | 73.86 | 747 | 49.91 | 16838 | 46.77 | 12087 | 30.37 | 2882 | 49.85 | 29672
t5-large | 66.94 | 747 | 42.65 | 16838 | 41.30 | 12087 | 26.10 | 2882 | 43.31 | 29672
t5-3b | 74.16 | 747 | 45.32 | 16838 | 40.84 | 12087 | 26.93 | 2882 | 44.98 | 29672
Table 5: Coherency scores per relation type.

(a) bert-base-uncased

(b) t5-base

Figure 2: Average coherency with standard deviation when using paraphrased prompts over different relations.  Model | Coherency | #Instances
---|---|---
bert-base-uncased | 10.78 | 2919
bert-large-uncased | 10.06 | 2919
InformBERT | 9.79 | 2919
t5-base | 10.64 | 2919
t5-large | 12.45 | 2919
t5-3b | 17.39 | 2919
t5-large-ssm | 16.76 | 2919
t5-3b-ssm | 14.57 | 2919
Autoregressive PLMs
gpt2 | 2.36 | 2919
gpt-neo-1.3B | 4.89 | 2919
gpt-neo-2.7B | 11.50 | 2919
Table 6: Coherency of different PLMs on a subset of one-token entities using BERT’s tokenizer with manually-written prompts. Model | Coherency | #Instances
---|---|---
bert-base | 10.78 | 2919
bert-large | 10.06 | 2919
InformBERT | 9.79 | 2926
t5-base | 9.66 | 29672
t5-large | 10.55 | 29672
t5-3b | 16.26 | 29672
t5-large-ssm | 17.06 | 29672
t5-3b-ssm | 14.92 | 29672
Autoregressive PLMs
gpt2 | 2.11 | 29672
gpt-neo-1.3B | 6.65 | 29672
gpt-neo-2.7B | 6.19 | 29672
w/ optimized prompts
bert-base | 7.16 | 2919
bert-large | 4.62 | 2919
w/ evidence paragraphs
bert-base-uncased | 31.09 | 2919
bert-large-uncased | 31.52 | 2919
InformBERT | 44.74 | 2926
t5-base | 49.85 | 29672
t5-large | 43.31 | 29672
t5-3b | 44.98 | 29672
t5-large-ssm | 46.78 | 29482
t5-3b-ssm | 45.17 | 29672
Table 7: Coherency for different PLMs using manually-written, optimized
prompts and evidence paragraphs. The highest performance under each category
is in bold, and the best performance overall is underlined.

##  Appendix B Examples

We show examples of several failures from different prompts and categorize
these in Table 8.

Type |  Model |  Relation |  Forward |  Backward | ID
---|---|---|---|---|---
Coherent
& Correct |  bert-base-uncased |  edmonton, alberta |  edmonton is the capital of [MASK]
$\rightarrow{}$ alberta |  [MASK] is the capital of alberta
$\rightarrow{}$ edmonton | 1
Coherent
& Incorrect |  t5-large |  Brunei, Malay |  The official language of Brunei is [MASK]
$\rightarrow{}$ Bruneian |  The official language of [MASK] is Bruneian
$\rightarrow{}$ Brunei | 2
Incoherent
& Correct (1st) |  bert-base-uncased |  lucknow, urdu |  The official language of lucknow is [MASK]
$\rightarrow{}$ urdu |  The official language of [MASK] is urdu
$\rightarrow{}$ maldives | 3
|  gpt-neo 2.7B |  Topeka, Kansas |  Topeka is the capital of [MASK]
$\rightarrow{}$ Kansas |  Kansas’s capital is [MASK]
$\rightarrow{}$ Quebec City | 4
Repetition |  informBERT |  iPhone, Apple |  iPhone is produced by [MASK]
$\rightarrow{}$ apple |  [MASK] is produced by apple
$\rightarrow{}$ apple | 5
Incoherent
& Incorrect |  bert-large-uncased |  lille, nord |  lille is the capital of [MASK]
$\rightarrow{}$ france |  [MASK] is the capital of france
$\rightarrow{}$ lyon | 6
Repetition |  t5-base |  Germany, Berlin |  The capital of Germany is [MASK]
$\rightarrow{}$ Frankfurt am Main |  The capital of [MASK] is Frankfurt am Main
$\rightarrow{}$ Frankfurt am Main | 7
Pronoun |  bert-base-uncased |  munich, germany |  munich is located in [MASK]
$\rightarrow{}$ bavaria |  [MASK] is located in bavaria
$\rightarrow{}$ it | 8
Table 8: Examples from different PLMs.

##  Appendix C Additional Details on Masked Language Models

Masked PLMs are trained to predict one or several tokens given a context. This
is considered a generalization of the conventional language modeling objective
that predicts the next token based on its left context. BERT Devlin et al.
(2019), an encoder-only model, was trained using the Maksed Language Modeling
(MLM) objective. T5, an encoder-decoder model, was also trained using a
variant of the MLM objective in addition to a mixture of supervised tasks. In
the Salient Span Masking (SSM) versions of T5, the models are additionally
trained by masking only entities to push the model to focus more on these Guu
et al. (2020); Roberts et al. (2020). Similarly, Sadeq et al. (2022) leverage
pointwise mutual information to mask salient tokens in an unsupervised manner.
Table 9 provides an overview of the architecture and the number of parameters
for each model.

Model | #Parameters | Architecture
---|---|---
bert-base | 110M | encoder-only
bert-large | 345M | encoder-only
InformBERT | 110M | encoder-only
t5-base | 220M | encoder-decoder
t5-large | 770M | encoder-decoder
t5-3B | 3B | encoder-decoder
t5-11B | 11B | encoder-decoder
gpt-2 | 117M | decoder-only
gpt-neo 1.3B | 1.3B | decoder-only
gpt-neo 2.7B | 2.7B | decoder-only
Table 9: Models with number of parameters and architectures. SSM variants of
t5 have the same number of parameters as their normal counterparts.

##  Appendix D Choice of Datasets

The LAMA probe Petroni et al. (2019) has been proposed to assess how much
factual knowledge is contained in PLMs. We believe it is suitable for the
experiments we conduct, since it consists of (subject, relation, object)
triples. This allows us to evaluate, how often PLMs can predict one entity
(either the subject or object) given the other entity and the relation.
Additionally, LAMA covers 41 relations of different types, which helps us
provide a coherency estimate based on all of these relations. See Table 10 for
an overview. We also used the ParaRel dataset Elazar et al. (2021). This
dataset has been proposed to measure the sensitivity of PLMs to paraphrased
prompts with respect to factual knowledge. Similarly, we use ParaRel to
investigate how the coherency score is affected by paraphrased prompts. All
the datasets we used are in English. Additionally, we used the prompts
obtained by Autoprompt Shin et al. (2020) to investigate the effect of having
optimized prompts on the performance. We manually create prompts for
autoregressive PLMs. These templates are included with our code.
333https://github.com/paulyoussef/coherency

#Relations | #Instances | Example
---|---|---
41 | 29672 | X was born in Y
Table 10: Statistics of LAMA and an example. Dataset | License
---|---
LAMA | CC-BY-NC 4.0
ParaRel | MIT License
Optimized prompts | Apache License 2.0
Table 11: Licenses of the datasets used in this work.

##  Appendix E Computational Resources

In all of our experiments, we use a NVIDIA A100 GPU with 80GB of memory. Our
experiments took roughly 25 GPU days.


# Random walker derivation of Archie’s law

Clinton DeW. Van Siclen<EMAIL_ADDRESS>1435 W 8750 N, Tetonia, Idaho
83452, USA

(26 March 2021)

###### Abstract

The empirical Archie’s law, relating the measured electrical resistivity of an
electrolyte-saturated rock to its connected porosity, is derived by
consideration of the fractal character of the pore space. The derivation thus
imposes some conditions on the implementation and meaning of Archie’s law.

##  I Introduction

The empirical Archie’s law (Archie, ) comes out of the laboratory, where
electrical resistivity $\rho$ and porosity $\phi$ values are obtained for
several samples of a particular rock type fully saturated by brine or some
other electrolyte. When the pairs of measured values $\phi$ and
$\rho/\rho_{e}$ ($\rho_{e}$ is the resistivity of the electrolyte) are plotted
on a log–log plot and fit to a straight line, the slope of that line is $-m$.
The utility of Archie’s law

$\frac{\rho}{\rho_{e}}\propto\phi^{-m}$ (1)

follows from the circumstance that the points fall on, or very near, that
line. In that case Archie’s law can be used in conjunction with resistivity
measurements taken in the field, to obtain an estimate of the (conductive)
porosity of the rock in question (the rock matrix itself is not conductive). A
low value may indicate the presence of non-conductive oil or gas in the pore
space.

Here it is more convenient to consider Archie’s law in the form

$\frac{\sigma}{\sigma_{e}}\propto\phi^{\mu}$ (2)

where $\sigma$ and $\sigma_{e}$ are the electrical conductivities of the
saturated rock sample and electrolyte, respectively, and $\mu$ is the
cementation exponent for that sample. (The cementation exponent $m$ is that
obtained graphically for a collection of samples of the rock type.)

This relation is derived below in a manner that makes clear Archie’s law is a
consequence of the natural fractal character of the pore space. The derivation
relies on the walker diffusion method (WDM), which is introduced in the
following section. Section III then presents the analytic derivation, and Sec.
IV gives numerical results for some model porous systems that inform on the
meaning of a $\mu$ value, and by extension an $m$ value. Final comments are
made in Sec. V.

##  II Walker Diffusion Method

The WDM was developed to calculate effective transport coefficients (e.g.,
conductivity) of composite materials and systems (PRE99, ). This method
exploits the isomorphism between the transport equations and the diffusion
equation for a collection of non-interacting walkers in the presence of a
driving force. Accordingly, the phase domains in a composite correspond to
distinct populations of walkers, where the walker density of a population is
given by the value of the transport coefficient of the corresponding phase
domain. The principle of detailed balance, which ensures that the population
densities are maintained, provides the following rule for walker diffusion
over a digitized composite: a walker at site (or pixel/voxel) $i$ attempts a
move to a randomly chosen adjacent site $j$ during the time interval
$\tau=(4d)^{-1}$, where $d$ is the Euclidean dimension of the space; this move
is successful with probability $p_{ij}=\sigma_{j}/(\sigma_{i}+\sigma_{j})$,
where $\sigma_{i}$ and $\sigma_{j}$ are the transport coefficients for the
phases comprising sites $i$ and $j$, respectively. The path of a walker thus
reflects the composition and morphology of the domains that are encountered.
Over distances greater than the correlation length $\xi$, the walker diffusion
is characterized by the diffusion coefficient $D_{w}$, which is related to the
effective transport coefficient $\sigma$ by

$\sigma=\left\langle\sigma(\mathbf{r})\right\rangle D_{w}$ (3)

where $\left\langle\sigma(\mathbf{r})\right\rangle$ is the volume average of
the constituent transport coefficients. The diffusion coefficient $D_{w}$ is
calculated from the equation

$D_{w}=\frac{\left\langle R(t)^{2}\right\rangle}{2\,d\,t}$ (4)

where the set $\left\\{R\right\\}$ of walker displacements, each occurring
over the time interval $t$, comprises a Gaussian distribution that must
necessarily be centered well beyond $\xi$. [For practical purposes, the
correlation length $\xi$ is the length scale above which the “effective”, or
macroscopic, value of a transport property is obtained. That is, all volumes
of linear dimension $\xi$ or greater are statistically identical.] Note that
$D_{w}$ is dimensionless, due to the principle of detailed balance that
governs the walker behavior. Also, it is useful to note that $D_{w}=1$ in the
case of a single-phase system, and $D_{w}<1$ otherwise.

For displacements $R<\xi$, the walker diffusion is anomalous rather than
Gaussian due to the heterogeneity of the composite at length scales less than
$\xi$ (JPA99, ). There is, however, an additional characteristic length
$\xi_{0}<\xi$ below which the composite is effectively homogeneous; this may
correspond, for example, to the average phase domain size. A walker
displacement of $\xi$ requiring a travel time $t_{\xi}=\xi^{2}/(2dD_{w})$ is
then comprised of $(\xi/\xi_{0})^{d_{w}}$ segments of length $\xi_{0}$, each
requiring a travel time of $t_{0}=\xi_{0}^{2}/(2dD_{0})$, where $D_{0}$ is the
walker diffusion coefficient calculated from displacements $R\leq\xi_{0}$.
Setting $t_{\xi}=(\xi/\xi_{0})^{d_{w}}\,t_{0}$ gives the relation

$D_{w}=D_{0}\left(\frac{\xi}{\xi_{0}}\right)^{2-d_{w}}$ (5)

between the walker diffusion coefficient $D_{w}$ (for walks of displacement
$R\geq\xi$), the walker path dimension $d_{w}>2$ (for walks of displacement
$\xi_{0}<R<\xi$), and the correlation length $\xi$.

[Note that the unit of length is the linear dimension of the pixels/voxels
used in calculations. Thus a pixel/voxel has area/volume equal to $1$, and the
values $\xi>\xi_{0}\geq 1$.]

##  III Derivation of Archie’s law

According to Eq. (3), the effective conductivity of a saturated rock sample of
linear dimension $L\geq\xi$ is

$\sigma=\sigma_{e}\,\phi\,D_{w}.$ (6)

Equation (5) shows that the value $D_{w}$ is determined by the narrow range of
length scales between $\xi_{0}$ and $\xi$ that exhibit a fractal character
(specifically, anomalous walker diffusion with $d_{w}>2$).

In the case of a statistically isotropic rock, the fractal character of the
volume $\xi^{d}$ is related to the porosity by the equation

$\phi=\frac{\left(\frac{\xi}{\xi_{0}}\right)^{D}}{\left(\frac{\xi}{\xi_{0}}\right)^{d}}=\left(\frac{\xi}{\xi_{0}}\right)^{D-d}$
(7)

where $d=3$ is the Euclidean dimension of the rock, and the right-hand side of
the equality is the volume fraction of the rock that is occupied by pore space
of (fractal) mass dimension $D<d$. Note that this expression uses $\xi_{0}$ as
the fundamental unit of length (rather than voxel size) since the pore space
is fractal only at length scales greater than $\xi_{0}$. [The rock volumes
$\left(\xi_{0}\right)^{d}$ and smaller are effectively homogeneous.] Then the
value $D$ can be obtained by the box counting method (Hansen, ), using 3D
arrays of boxes each of linear dimension $\ell$, for sizes $\xi_{0}<\ell<\xi$,
to virtually divide up the rock specimen of linear dimension $L\gg\xi$. The
number $N$ of boxes that contain a portion of the pore space is
$N\left(\ell\right)\propto\text{$\ell$}^{-D}$.

Note that, as $D$ is a function only of the gross parameters $\phi$ and
$\xi/\xi_{0}$ describing the pore space, a value $D$ is not unique to a rock
type or pore space configuration.

If $D<3$ implies some 2D character, then a smaller $D$ value must indicate a
larger pore-boundary area to volume ratio.

Archie’s law follows from combining Eq. (5), which pertains to the transport
(dynamic) properties of a multiphase material, and Eq. (7), that relates the
geometric properties of the porous rock. Both contain the ratio $\xi/\xi_{0}$.
The correlation length $\xi$ is obviously the same in both equations, as
statistically identical volumes must have identical transport properties. In
the case of Eq. (7), $\xi_{0}$ is the size of the fundamental unit of which
the fractal of linear dimension $\xi$ is made. In the case of Eq. (5),
$\xi_{0}$ is the length scale below which the multiphase material is
effectively homogeneous (meaning that $d_{w}=2$). The characteristic length
$\xi_{0}$ for porous rock satisfies both these descriptions.

Equation (7) can be written

$\frac{\xi}{\xi_{0}}=\phi^{1/\left(D-d\right)}$ (8)

giving

$\frac{\sigma}{\sigma_{e}}=\phi\,D_{w}=\phi\,D_{0}\left(\frac{\xi}{\xi_{0}}\right)^{2-d_{w}}=\kappa\,\phi^{\mu}$
(9)

with prefactor $\kappa=D_{0}\approx 1$ and cementation exponent

$\mu=1+\frac{2-d_{w}}{D-d}.$ (10)

Note that $\mu\gtrless 2$ when $d_{w}-2\gtrless d-D$ or equivalently
$D+d_{w}\gtrless d+2$.

Unfortunately the walker path dimension $d_{w}$ appears only in the context of
a transport property of the material system—it is not a geometric property of
the pore space. Indeed, there is no algebraic relation between $D$ and
$d_{w}$.

Note that the prefactor $\kappa$ is less than, but generally close to, $1$.
The maximum value $D_{0}=1$ is slightly reduced by the interactions of the
walkers with the pore-space boundary during walks of displacement
$R\leq\xi_{0}$.

Thus both the power-law form of Archie’s law, and the prefactor $\kappa\leq
1$, are obtained by this derivation.

Given that particular values of $\sigma/\sigma_{e}$ and $\phi$ generate a
particular $\mu$ value, it is remarkable that different sets of
$\sigma/\sigma_{e}$ and $\phi$ values, obtained for the same rock type, can
collectively produce a useful exponent $m$. This is due to the fact that the
values of the fractal parameters $D$ and $d_{w}$ move in opposite directions:
when $D$ is smaller/larger, reflecting a more/less complex pore space, $d_{w}$
is larger/smaller. Thus the ratio $\left(2-d_{w}\right)/\left(D-d\right)$ (and
so the exponent value $\mu$) may not change appreciably with a moderate change
in mass dimension $D$.

[To be clear, the self-similarity over length scales between $\xi_{0}$ and
$\xi$ implied by particular values $D$ and $d_{w}$ is not generally observed
in natural systems; rather the dimensions $D$ and $d_{w}$ may vary somewhat
over those length scales.]

An important result follows from the observation that the diffusion
coefficient $D_{w}$ value reflects the degree of interaction of the walkers
with the pore-space boundary. Specifically, greater interaction, due to a more
complex pore space, produces a smaller $D_{w}$ value. Note that a pore space
of volume $v$ corresponding to porosity $\phi$ can take many different
configurations, each with a different $D_{w}$ value. Then the relation
$D_{w}=\kappa\,\phi^{\mu-1}$ makes clear that for a given $\phi$ value, a
large/small $\mu$ value implies a small/large $D_{w}$ value, and so a
more/less complex pore space (that is, a larger/smaller ratio of pore–rock
interfacial area to pore volume).

##  IV Fractal character of porous rock

The 3D Sierpinski sponge is a poor representation of a porous rock, but it has
the virtues that it is recognizably fractal, and the values of the fractal
properties are known (CVS2017, ). It is a recursive, self-similar fractal of
dimension $D=\ln 26/\ln 3=2.96565$, and the walker path dimension
$d_{w}=2.0216$. Thus Archie’s law for the Sierpinski sponge has exponent
$\mu=1.6288$. Note that the $\mu$ value does not change as $\phi\rightarrow
0$. Of course, $\phi$ and the ratio $\xi/\xi_{0}$ can only take certain
values: $\phi_{i}=\left(26/27\right)^{i}$ and $\xi/\xi_{0}=3^{i}$ where
integer $i\geq 1$ is the recursion number.

Most interesting is the fact that the pore space dimension $D\approx 3$, and
the walker path dimension $d_{w}\approx 2$, even as the pore volume goes to
zero. Evidently this recursive fractal of linear dimension $\xi$ behaves much
like an effectively homogeneous system. This must be due to the character of
the pore space, which visually appears as well-connected blocks of different-
sized 3D spaces. Notably, there are no pore-space dead-ends to additionally
increase $d_{w}$ (recall that a walker will occupy every pore site for an
equal time, in accordance with the principle of detailed balance that governs
the walker behavior).

A fractal at the other extreme is the 3D percolating (conducting) cluster near
the percolation threshold at $\phi=0$ (CVS2021, ). In this case there are
conductor-site clusters of all sizes attached to the conducting “backbone”
that carries the electric current. As porosity $\phi\rightarrow 0$ (from
$\phi=1$), the mass dimension $D\rightarrow 2.52295$ (from $D=3$) and the
walker path dimension $d_{w}\rightarrow 3.84331$ (from $d_{w}=2$) in concert
with correlation length $\xi\rightarrow\infty$ and characteristic length
$\xi_{0}\rightarrow 1$. Thus $\mu\rightarrow t/\beta=4.8639$. Note that the
porosity $\phi$ is the fraction $p^{\prime}$ of sites that belong to the
percolating (conducting) cluster, and $t$ and $\beta$ are the critical
exponents in the asymptotic expressions $\sigma\sim\left(p-p_{c}\right)^{t}$
and $p^{\prime}\sim\left(p-p_{c}\right)^{\beta}$, where $p$ is the fraction of
conductor sites in the infinite system, and $p_{c}$ is the percolation
threshold.

This percolation model of a porous rock makes clear that the fractal
properties of a pore space may change dramatically as porosity
$\phi\rightarrow 0$. Obviously, Archie’s law with a particular exponent value
$m$ has validity only within a limited range of porosity values.

Interestingly, an effect of an increasing $\mu$ value as the porosity $\phi$
declines may show up in the log–log plots of $\sigma/\sigma_{e}$ vs $\phi$
which produce the best-fit line with slope $m$. The y-intercept of that line
is $\ln k$, where $k$ is the prefactor in the equation
$\sigma/\sigma_{e}=k\,\phi^{m}$. It may be noticed that many plots in the
literature show $k>1$, which seems to contradict the derivation of Eq. (9)
which finds that prefactor $\kappa\leq 1$ in the equation
$\sigma/\sigma_{e}=\kappa\,\phi^{\mu}$. But $n$ points that lie on a straight
line of slope $m$ and y-intercept $\ln k>0$ also lie on $n$ lines of slopes
$\mu_{1}<\mu_{2}<\ldots<\mu_{n}<m$ calculated for porosities
$\phi_{1}>\phi_{2}>\ldots>\phi_{n}$ respectively, all having y-intercept
$\ln\kappa\leq 0$.

Glover (Glover, ) addresses the ubiquitous use of the prefactor $k$ for a
fitting parameter.

A more realistic system, mimicking an unconsolidated sandstone, is a dense,
randomly packed assemblage of glass beads saturated with an electrolyte. A
consolidated sandstone is obtained by fusing the glass beads to a lesser or
greater degree by heating the assemblage above its softening temperature for a
lesser or greater time interval. Experiments (Wong, ) over a porosity range
$0.399>\phi>0.023$ find $\mu\approx 1.5$ for $\phi>0.2$ and $\mu\approx 2$ for
$\phi<0.2$. (In contrast, all points are fit to a straight line of slope
$m=2.3$ and prefactor $k=3.3$.) Other experiments of this sort (Johnson, ;
Guyon, ) have obtained similar results; in particular, finding that $\mu$
increases from $1.5$ as the porosity $\phi$ decreases.

The relatively small increase in the $\mu$ value suggests that the
connectivity of the pore space is little affected by the fusion of the beads
(few or no dead ends created by pore occlusion), so moderating a decrease in
the mass dimension $D$ and an increase in the path dimension $d_{w}$.

An obvious question is how predictable is the cementation exponent $\mu$.
Examination of the microstructure provides the mass dimension $D$. This is
related to walker behavior by the power law $\left\langle
S(t)\right\rangle\propto\left\langle R(t)\right\rangle^{D}$, where
$\left\langle R(t)\right\rangle$ is the average displacement for walks of time
$t$, and cluster size $\left\langle S(t)\right\rangle$ is the average number
of distinct sites visited by a walker in that time $t$. The exponent $D<3$ is
then the slope of the line fit to points $\left(\ln\left\langle
R(t)\right\rangle,\ln\left\langle S(t)\right\rangle\right)$ in the interval
$\xi_{0}<R<\xi$, visually showing that a smaller value $D$ reflects a greater
degree to which walker diffusion is inhibited and confined by the pore-space
boundary. The value of the walker path dimension $d_{w}>2$ is essentially a
measure of the extra steps (compared to walks over a homogeneous system, that
have $d_{w}=2$) a walker must take to attain a displacement $\xi_{0}<R<\xi$,
due to interactions with the pore-space boundary along the way. Thus the
factors $\left(3-D\right)$ and $\left(d_{w}-2\right)$ are similarly affected
(increased or decreased) by the complexity of the pore space. Unfortunately
they appear together as a ratio, which weakens the connection between $\mu$
value and the complexity of the pore space.

Clearly, the geometric property of “complexity”, as considered here in the
context of Archie’s law, must be better defined. The mass dimension $D$ does
not fully describe the geometry of the pore space, and the walker path
dimension $d_{w}$ cannot (yet) be obtained or inferred directly by inspection
of the rock sample. The definition should include, for example, a metric for
the prevalence of dead-ends in a pore space configuration, and the pore-
boundary area to pore volume ratio. With a more complete description of the
pore space geometry, the concept of walker diffusion in the pore space can be
used to qualitatively connect the pore space geometry to the cementation
exponent $\mu$.

##  V Concluding remarks

Archie’s law is a consequence of the fractal nature of porous rock, which is
expressed at the length scales between $\xi_{0}$ and $\xi$. Additionally, it
is a consequence of the fact that every multiphase material shows a transition
in transport properties between length scales $\xi_{0}$ and $\xi$, based on
the transition from walker diffusion coefficient $D_{0}$ to $D_{w}$.

For an example of the latter effect, consider a straight, cylindrical pore
embedded in a rock matrix. Walker displacements $R\leq\xi_{0}$ produce minimal
interactions with the pore boundary, so the walkers diffuse freely over 3D
space thereby producing the walker diffusion coefficient $D_{0}\approx 1$. For
walker displacements $R>\xi$, the collective behavior is 1D, producing the
walker diffusion coefficient $D_{w}=1/3$. The interactions with the confining
pore boundary significantly reduce the walker diffusion coefficient from
$D_{w}=1$.

Figure 1: Data obtained from walks over the linear, 3D pore space with square
cross-section. The lower and upper horizontal lines mark the values $R=W/2$
and $W$, respectively, where $W$ is the width of the pore space. Evidently
$\xi_{0}\approx W/2$ and $\xi>W$. The sequence of points (in the direction of
increasing walk time $t$) shows the transition from Gaussian diffusion
essentially unaffected by the pore boundary ($D_{0}\approx 1$), to anomalous
diffusion ($d_{w}>2$), to Gaussian diffusion significantly affected by walker
interactions with the pore boundary ($D_{w}=1/3$). Both diagonal lines have
slope $1$, indicating Gaussian diffusion.

Figure 1 shows results of WDM calculations for a straight, 3D pore with square
(100-site $\times$ 100-site) cross-section. Each apparent point $\left(\ln
t,\ln\left\langle R(t)^{2}\right\rangle\right)$ is actually the overlap of
five points, each obtained from $10^{4}$ walks of walk time $t$. The two
diagonal lines are

$\ln\left\langle R(t)^{2}\right\rangle=\ln t+\ln\left(2\,d^{\prime}\right)$
(11)

with $d^{\prime}=3$ (upper line) and $d^{\prime}=1$ (lower line). These lines
embody the diffusion equation, Eq (4). The sequence of points at short walk
times diverges from the upper line, which has y-intercept $\ln 6$, indicating
that $D_{0}\approx 1$. The points for long walk time $t$ approach the lower
line, which has y-intercept $\ln 2=\ln\left(2\,d\,D_{w}\right)$, giving
$D_{w}=1/3$.

Note that the exact values of $\xi_{0}$ and $\xi$ are difficult to discern,
but clearly they bracket the transition $D_{0}\rightarrow D_{w}$. The
transition region is characterized by the walker path dimension $d_{w}>2$, the
value of which is obtained from the slope $2/d_{w}$ of the line connecting
points $\left(\ln t_{0},\ln\xi_{0}^{2}\right)$ and $\left(\ln
t_{\xi},\ln\xi^{2}\right)$. These points lie on the upper and lower lines,
respectively.

Although this linear pore is obviously not a fractal, it presents as a fractal
system with characteristic lengths $\xi_{0}$ and $\xi$, due to the walker
interaction with the pore boundary (the nonconducting rock phase).

Walks over a complex pore system of fractal dimension $D$ would produce a
figure like Fig. 1.

###### Acknowledgements.

I thank Professor Robert “Bob” Smith (Department of Geological Sciences) for
arranging my access to the resources of the University of Idaho Library
(Moscow, Idaho).

## References

  * (1) G. E. Archie, The electrical resistivity log as an aid in determining some reservoir characteristics, Trans. AIME 146, 54–62 (1942).
  * (2) C. DeW.Van Siclen, Walker diffusion method for calculation of transport properties of composite materials, Phys. Rev. E 59 (3), 2804–7 (1999).
  * (3) C. DeW. Van Siclen, Anomalous walker diffusion through composite systems, J. Phys. A: Math. Gen. 32, 5763–71 (1999).
  * (4) J. P. Hansen and A. T. Skjeltorp, Fractal pore space and rock permeability implications, Phys. Rev. B 38 (4), 2635–8 (1988).
  * (5) C. DeW. Van Siclen, Percolation properties of the classic Sierpinski carpet and sponge, e-print arXiv:1706.03410 (2017). [Available at https://arxiv.org/abs/1706.03410]
  * (6) C. DeW. Van Siclen, Conductivity exponents at the percolation threshold, e-print arXiv:1609.01229 (2021). [Available at https://arxiv.org/abs/1609.01229]
  * (7) P. W. J. Glover, Archie’s law – a reappraisal, Solid Earth 7, 1157–69 (2016).
  * (8) P. Wong, J. Koplik, and J. P. Tomanic, Conductivity and permeability of rocks, Phys. Rev. B 30 (11), 6606–14 (1984).
  * (9) D. L. Johnson, T. J. Plona, C. Scala, F. Pasierb, and H. Kojima, Tortuosity and acoustic slow waves, Phys. Rev. Lett. 49 (25), 1840–4 (1982).
  * (10) E. Guyon, L. Oger, and T. J. Plona, Transport properties in sintered porous media composed of two particle sizes, J. Phys. D: Appl. Phys. 20, 1637–44 (1987).