# Auxiliary task demands mask the capabilities of smaller
language models

Jennifer Hu
Kempner Institute for the Study of Natural and Artificial Intelligence
Harvard University
Cambridge, MA, USA
<EMAIL_ADDRESS>
Michael C. Frank
Department of Psychology
Stanford University
Stanford, CA, USA
<EMAIL_ADDRESS>
###### Abstract

Developmental psychologists have argued about when cognitive capacities such
as language understanding or theory of mind emerge. These debates often hinge
on the concept of “task demands” – the auxiliary challenges associated with
performing a particular evaluation – that may mask the child’s underlying
ability. The same issues arise when measuring the capacities of language
models (LMs): performance on a task is a function of the model’s underlying
competence, combined with the model’s ability to interpret and perform the
task given its available resources. Here, we show that for analogical
reasoning, reflective reasoning, word prediction, and grammaticality
judgments, evaluation methods with greater task demands yield lower
performance than evaluations with reduced demands. This “demand gap” is most
pronounced for models with fewer parameters and less training data. Our
results illustrate that LM performance should not be interpreted as a direct
indication of intelligence (or lack thereof), but as a reflection of
capacities seen through the lens of researchers’ design choices.

##  1 Introduction

A shared goal in psychology and artificial intelligence is to ascribe
cognitive capacities to black-box agents. For example, we might be interested
in whether a young child has a “theory of mind”, or whether a language model
(LM) has the ability to distinguish grammatical and ungrammatical sentences.
The trouble is, although we would like to infer an underlying psychological
_construct_ , we only have access to specific observable _evaluations_ – a
child’s ability to answer a question about a character in a story, or a
model’s performance on a syntax benchmark.111Here, we use the term
“evaluation” to encompass tasks, benchmarks, measures, and experiments, for
both humans and machines. Inferences from evaluations to constructs are
ubiquitous in developmental psychology (Frank, 2023a), and cognitive
evaluations of artificial models are beginning to follow similar principles
(e.g., Momennejad et al., 2023; Ivanova, 2023).

For both humans and machines, making inferences about _failures_ is especially
tricky, because failure on a task does not always indicate the absence of the
underlying capacity. In developmental psychology, children often fail due to
_task demands_ : auxiliary challenges unrelated to the capacity of interest
(e.g., Keen, 2003; Carruthers, 2013; Turan-Küçük & Kibbe, 2024). Children of a
given age may succeed in a simple version of a task, while failing at a more
complex version that incorporates other demands. Sometimes the issue is as
basic as children not understanding the question being asked, while at other
times task demands can include issues like remembering multiple pieces of
information or inhibiting a response.

Figure 1: A: Hypothetical task demands in two evaluation settings, faced by
humans and machines. Both methods apparently measure the accuracy of target
word prediction, but the high demand setting imposes additional auxiliary
demands. B: Hypothesized pattern of results if task demands asymmetrically
affect less capable agents (e.g., younger children or smaller models). C:
Signature “demand gap” produced by hypothesized pattern in B.

Here, we argue that task demands also play an important role in determining
evaluation success or failure for LMs, especially when comparing models of
different capacities. As a conceptual illustration, Figure 1A depicts
hypothetical task demands faced by humans and machines for two different ways
of evaluating word prediction ability. The high-task-demand evaluation
presents a metalinguistic prompt, which explicitly asks the agent for a
prediction. A correct response for this stimulus requires interpreting the
question, accessing internal predictions for that target sentence, and
producing them after the prompt. In contrast, a lower-demand design for humans
might be the measurement of reaction time upon hearing the target word
(“sugar”); for LMs, a lower-demand task would be a readout of the probability
of the target word. While both of these evaluations apparently measure the
accuracy of the LM’s predictions, models might fail in the high-demand
situation due to difficulty with task demands unrelated to the accuracy of
their actual target word prediction. Indeed, recent studies have demonstrated
performance gaps between higher- and lower-demand evaluations in a variety of
linguistic domains (e.g., Hu & Levy, 2023; Hu et al., 2024; Kauf et al., 2024;
West et al., 2024; Tsvilodub et al., 2024).

Our primary question is not whether these performance gaps exist, but whether
task demands _asymmetrically_ affect less capable language models, mirroring
the findings in children. A null hypothesis might be that a low-capability
model – for example, one with fewer parameters or less training data – would
perform equally poorly on all cognitive evaluations, regardless of the method.
Inspired by the task demand debate in developmental psychology, we instead
predict that less capable agents should suffer more from task demands than
more capable agents (Figure 1B). If borne out, this prediction would produce a
signature “demand gap“, which would be larger in less capable LMs (Figure 1C).

We present experiments demonstrating this signature pattern – a larger demand
gap for less capable models – across a variety of tasks and open-source LMs.
We investigate two evaluation contrasts: production vs. forced choice, and
metalinguistic judgment vs. probability measurement. For each of these
contrasts, we measure the demand gap between the higher- and lower-demand
evaluation methods on multiple cognitive domains, collectively covering
analogical reasoning, reflective reasoning, word prediction, and
grammaticality judgments. We find evidence that demand gaps get smaller as LMs
increase in size (i.e., number of parameters) and training time. Our findings
suggest that task demands can mask the abilities of less capable models,
illustrating how inferences about LM abilities depend on our choice of
evaluation metrics.

The paper is structured as follows. In Section 2, we more fully describe the
concept of task demands and how it relates to other issues in LLM evaluation.
We describe the logic and methods behind our experiments in Section 3, and
present the results in Section 4. Finally, in Section 5 we conclude and
discuss broader implications for NLP and cognitive science.

##  2 Background and Related Work

Psychology focuses on connecting observable measurements to latent
(unobserved) psychological constructs. A _valid_ measure is one that
appropriately measures the construct (Cronbach & Meehl, 1955). Task demands
are one of many different factors that can undermine validity. In particular,
the presence of auxiliary demands in a task means that the inference from
observed performance to construct is no longer “pure.” Reducing these demands
can enable a more direct inference about the latent constructs of interest.

An evaluation with fewer task demands is not necessarily easier or more
lenient. Instead, a better gloss might be that it is a purer task, in the
sense that it better measures the targeted construct. Consider an evaluation
of arithmetic ability: the size of the numbers being manipulated and the range
of operations would likely change the overall difficulty of the evaluation. In
contrast, the use of word problems, pictorial representations, or framing in
terms of real-world objects would all change the task demands of the
evaluation, even if the difficulty of arithmetic components stayed constant.
Some of these changes might serve to make the task easier – for example, if
they engaged familiar content (Dasgupta et al., 2022) – while others might
make the task harder (at least for some models).

Our work here builds upon the field of LM evaluation – specifically, research
on links between evaluation metrics and underlying constructs, or in other
words “machine evaluation validity”. For example, Schlangen (2019) points out
that individual evaluation tasks are often only loosely related to the
underlying construct that researchers want to engineer into their system.
Other work has questioned the validity of inferences from forced choice
metrics (Tsvilodub et al., 2024; Schaeffer et al., 2023), and several studies
have noted the misalignment between production/forced choice (Lyu et al.,
2024; Röttger et al., 2024; West et al., 2024) and between
prompting/measurement of underlying probabilities (Hu & Levy, 2023; Hu et al.,
2024; Kauf et al., 2024). A related line of work asks how to design “species-
fair” comparisons between humans and models (Firestone, 2020; Lampinen, 2023).

These specific investigations of validity are part of the broader effort to
design robust LM evaluations. The results of any evaluation depend in part on
design choices, which necessarily reflect a researcher’s broader goals or
theoretical commitments. These design choices have been widely discussed
(e.g., Linzen, 2020; Bowman & Dahl, 2021; Raji et al., 2021), including
factors such as size and scope (Srivastava et al., 2023), dynamic updating
(Kiela et al., 2021), and adversarial approaches (Nie et al., 2020; Sakaguchi
et al., 2021). To our knowledge, most of this work has not specifically
analyzed the interaction between evaluation method and model capability that
is our focus (but see Schaeffer et al., 2023).

##  3 Methods

Our main research question is whether less capable language models are more
sensitive to task demands. If this is the case, then we would expect to find
the pattern in Figure 1C: the performance gap between low- and high-demand
tasks should be smaller for more capable models. To avoid conflating the
targeted cognitive construct with a specific contrast of evaluation methods,
we perform each evaluation comparison on two different domains. In the
remainder of this section, we discuss the evaluation contrasts, cognitive
constructs, and models tested in our experiments (see Table 1 for summary).

###  3.1 Evaluation contrasts

Evaluation |  Cognitive construct |  Dataset(s) |  Goal |  Example item
---|---|---|---|---
Production (high demands) vs. Forced choice (low demands) |  Analogical reasoning |  Webb et al. (2023) |  Predict next digit(s) in sequence |  [5 9 3] [8 9 2] [1 9 7] $\backslash$n [8 4 7] [1 4 3] [5 4 2] $\backslash$n [1 2 2] [5 2 7] [
Reflective reasoning |  Hagendorff et al. (2023) |  Override intuitive solution to correctly solve the problem |  A chair and a coat together cost $13. The chair costs $10 more than the coat. How much does the coat cost?
Metalinguistic judgment (high demands) vs. Probability measurement (low demands) |  Word prediction |  LAMBADA (Paperno et al., 2016) |  Predict final word of the text (underlined) |  Both its sun-speckled shade and the cool grass beneath were a welcome respite after the stifling kitchen, and I was glad to relax against the tree’s rough, brittle bark and begin my breakfast of buttery, toasted bread and fresh fruit. Even the water was tasty, it was so clean and cold. It almost made up for the lack of coffee
Grammaticality judgment |  BLiMP (Warstadt et al., 2020); Dentella et al. (2023a); Hu et al. (2024) |  Identify the grammatical sentence in a minimal pair |  (1) Rachelle had bought that chair.
(2) *Rachelle had bought that chairs.
Table 1: Overview of task contrasts and cognitive domains tested in our
experiments.

We focus on two contrasts of high- and low-demand evaluation methods: (1)
production (high) versus forced choice (low), and (2) metalinguistic judgments
(high) versus direct probability measurements (low). Each of these four
evaluation methods has previously been applied to make claims about cognitive
abilities in language models.

##### Production vs. Forced choice.

The first contrast compares _production_ of the correct answer (higher task
demands) versus _forced choice_ over a fixed set of answer options (lower task
demands). Notably, children’s ability to produce a certain linguistic form
tends to lag behind their ability to comprehend that form (e.g., Hendriks &
Koster, 2010). Forced choice paradigms have revealed knowledge in children
that would not otherwise be observed through production. For example,
6–9-month-old infants can direct their gaze to named objects, before being
able to talk themselves (Bergelson & Swingley, 2012).

It is not obvious that this asymmetry would also apply for LMs, for which the
task of generating linguistic output – which may impose auxiliary demands on
children, including planning motor actions associated with speech production –
is the primary rationale. But even without these demands, forced choice
requires less precise estimates of a probability distribution than correct
production. A correct answer might be reliably preferred over incorrect
answers in probability space, while not being generated because of auxiliary
factors such as the decoding mechanism, or simply having a low absolute
probability.222Forced choice evaluation does require researchers to design a
set of answer options, which may introduce unintended artifacts (e.g., content
or frequency effects). In line with this view, West et al. (2024) find that
generative (production) performance is better than discriminative (forced
choice) performance across a range of models and tasks. However, this finding
could be due to the complexity of the answer options in their study, which
were designed to include adversarial “hard negatives”. More broadly (and in
keeping with our general theme), researcher goals (in this case, adversarial
testing as opposed to capacity discovery) can modulate the difficulty of any
evaluation method.

##### Metalinguistic judgment vs. Probability measurement.

The second contrast compares two ways of measuring linguistic knowledge: a
_metalinguistic judgment_ versus a _direct measurement_ of how likely or well-
formed a linguistic expression is. From the developmental perspective,
children can identify well-formed sentences of their language, without
necessarily being able to articulate the underlying grammatical rules that
govern these sentences. As previously argued by Hu & Levy (2023), the
probabilities assigned to strings directly reflect a model’s linguistic
generalizations, whereas metalinguistic prompts impose additional demands such
as correctly interpreting the question (see Figure 1A).

###  3.2 Cognitive domains

####  3.2.1 Production vs. Forced choice

##### Analogical reasoning.

We used the digit matrices task introduced by Webb et al. (2023) to measure
LMs’ ability to perform analogical reasoning. Each item presents a sequence of
digits in matrix format, and the model is evaluated on its ability to predict
the continuation of the sequence. The relationships between the sequence and
continuation involve a variety of transformations and logical patterns; we
refer readers to Webb et al. (2023) for details.

We followed Webb et al.’s protocol for scoring. For the forced-choice
evaluation, we compared the log probability assigned by the model to each of
the possible answer choices (summed over tokens). The set of answer choices
for each problem was defined by Webb et al. (2023). For the production
condition, we truncated models’ responses where a closing bracket is
generated, and then evaluated the response based on whether it contained the
correct set of digits. We report results averaged over all problem types.

##### Reflective reasoning.

To evaluate LMs’ reflective reasoning abilities, we used the cognitive
reflection tests (CRTs) described by Hagendorff et al. (2023). CRTs
(Frederick, 2005) are problems that have a reflexive, intuitive solution,
which must be overridden through reflective problem-solving to arrive at the
correct answer. The CRTs used by Hagendorff et al. (2023) cover three
conditions which probe intuitive biases observed in humans.

To evaluate models in the forced-choice condition, we compared the mean log
probability assigned to the correct and intuitive answers. We average
probabilities over tokens because answer options can differ in length (e.g.,
“$10.00” vs. “$1.50”). The model succeeds on a given item if it assigns higher
probability to the correct answer than the intuitive answer, conditioned on
the problem statement. To score the freely generated responses, we manually
inspected the outputs and labeled whether they corresponded to the intuitive
answer, correct answer, or neither. Results are averaged over the three tested
conditions.

####  3.2.2 Metalinguistic judgment vs. Probability measurement

##### Word prediction.

A natural locus for a gap between direct measurement and metalinguistic
judgment is word prediction. Since the direct evaluation setting is analogous
to models’ autoregressive training objective, we can view these predictions as
a “ground-truth” measure of models’ underlying word prediction abilities,
providing a useful reference for evaluating metalinguistic abilities (Hu &
Levy, 2023). We evaluated models’ word prediction ability using the LAMBADA
dataset (Paperno et al., 2016). The items consist of short multi-sentence
passages, which are constructed such that correctly predicting the final word
of the text depends on the entire context, and not just the final
sentence.333We use the test split pre-processed by OpenAI, as exposed by the
EleutherAI/lambada_openai Huggingface dataset. We obtain the prefix and final
target word by splitting on whitespace.

To evaluate models using direct probability measurements, we computed the log
probability assigned by the model to the target final word conditioned on the
prefix (summing over sub-word tokens). To evaluate models metalinguistically,
we computed the log probability of the final word conditioned on the
metalinguistic prompt shown in 3.2.2.

. What word is most likely to come next in the following text?
Text: [prefix]
Prediction:

##### Grammaticality judgment.

A prominent question in linguistics and LM research is whether LMs capture the
distinction between grammatical and ungrammatical sentences. While a body of
work has evaluated this ability by testing whether models assign higher
probability to grammatical versus ungrammatical sentences (e.g., Warstadt et
al., 2020), there has been a growing trend to instead use metalinguistic
prompts (e.g., Dentella et al., 2023a; b; Katzir, 2023). Hu & Levy (2023) and
Hu et al. (2024) demonstrated that the probability comparison method can
reveal knowledge that is masked by metalinguistic methods. However, they did
not investigate how this gap varies with scale or training time.

We used two datasets to evaluate models’ grammatical abilities: BLiMP
(Warstadt et al., 2020),444We randomly sampled 50 items from each of the 13
categories, resulting in 650 items. and the set of minimal pairs based on
Dentella et al.’s (2023a) materials constructed by Hu et al. (2024). To
evaluate models using direct measurements, we assessed whether the model
assigned higher probability to the grammatical sentence than its ungrammatical
counterpart. This method is designed to control for factors such as lexical
items and length (see also Linzen et al., 2016; Marvin & Linzen, 2018). To
evaluate models metalinguistically, we prompted the model to compare the two
sentences in the minimal pair:

. Which of the following two sentences is more grammatically correct in
English? Respond with 1 or 2 as your answer.
Sentence 1: [sentence1]
Sentence 2: [sentence2]
Answer:

For each minimal pair, we evaluated models on two versions of 3.2.2, swapping
the order in which the sentences are presented. A model succeeds if it assigns
higher conditional probability to the token “1” or “2” (whichever one
corresponds to the index of the grammatical sentence). Accuracy is computed as
the average success across these two orders.

###  3.3 Models

Recall that our main question is whether the demand gap shrinks over the
course of model “development”, or as its general capabilities increase (Figure
1C). We tested two ways of operationalizing this concept for language models:
varying the _size_ (i.e., number of parameters) while keeping other
architectural and training details as similar as possible, and varying the
_duration of training_ for a given model instantiation.

##### Model size.

Model family | Sizes tested | Training tokens | Data cutoff
---|---|---|---
Pythia (deduped) | {1, 1.4, 2.8, 6.9, 12} B | 207 B | 2020
OLMo | {1, 7} B | {3, 2.5} T | Feb/March 2023
Gemma | {2, 7} B | {2, 6} T | unknown (before Feb 2024)
Llama-2 | {7, 13, 70} B | 2 T | Sept 2022
Mistral | 7 B | unknown | unknown (before Oct 2023)
Table 2: Models tested. For OLMo and Gemma, the first/second model size is
associated with the first/second amount of training tokens. For Gemma and
Mistral, the data cutoff date is not publicly known, but we assume it is
before publication of the associated paper.

We first use model size as a way of manipulating an LM’s general capabilities.
Models with more parameters are more expressive, suggesting that they might be
able to handle auxiliary task demands that would be difficult for less
expressive models (e.g., interpreting a metalinguistic question, or
maintaining the result of a computation across intervening content in a
prompt).

We evaluated 13 open-source autoregressive language models using the
Huggingface Transformers library. The models range in size from 1B to 70B
parameters spanning across five families: deduplicated Pythia (Biderman et
al., 2023), OLMo (Groeneveld et al., 2024), Gemma (Gemma Team et al., 2024),
Llama-2 (Touvron et al., 2023), and Mistral (Jiang et al., 2023). Table 2
summarizes the sizes we tested for each model family, as well as other
training details. We chose these models to cover a range of sizes, while also
achieving reasonable performance (ruling out smaller models such as GPT-2) and
ensuring reproducibility. Since our experiments require token-level logits, we
could not evaluate closed-API models.

This analysis is not meant to draw direct parallels between development in
children and scaling in LMs: we don’t claim, for example, that Llama-2 7B is
more like a young child and Llama-2 70B is more like an adult. Instead, we use
size as a practical way of parameterizing the “capacity” of a model, which
might affect its ability to overcome auxiliary task demands.

##### Training time.

Another way to manipulate the general capabilities of an LM is training time.
While many studies examine emergence as a function of model size (e.g., Wei et
al., 2022a), a growing line of work is concerned with “developmental
interpretability” (Hoogland et al., 2023), or how abilities and structures
emerge over the course of learning (e.g., Power et al., 2022; Simon et al.,
2023). For our experiments on training time, we evaluated OLMo 7B at 10
checkpoints over the course of training (including the final one).

Across all experiments, we computed conditional and full-sequence
probabilities using the minicons package (Misra, 2022). All models were base
models without any fine-tuning.

##### Data contamination.

Some of our tested datasets were publicly available before the data cutoff of
all our tested models (LAMBADA and BLiMP), and all of our datasets were
released before the publication of Gemma. This raises the concern of data
contamination: i.e., that these datasets were seen during training. While this
situation is not ideal, it is not a major concern because we are not
interested in performance _per se_ – rather, we are interested in the
_difference_ in performance across evaluation methods. We do not have strong
expectations that data contamination would affect one method more than
another.

##  4 Results

Figure 2: Production (high task demands) vs. forced choice (low task demands)
in two domains: analogical reasoning (top row) and reflective reasoning
(bottom row). A,C: Accuracy scores across models and evaluation methods. B,D:
Difference of log odds (forced choice $-$ production). Colored lines = best-
fit within model families. Black line = best-fit across all models. Shaded
region indicates bootstrapped 95% CI. (Log odds difference for Pythia 1B in
panel D is infinite, so it is not shown.) Figure 3: Metalinguistic judgment
(high task demands) vs. direct probability measurement (low task demands) in
two domains: word prediction (top row) and grammaticality judgments (bottom
row). A: Log probability assigned to final word in word prediction domain. B:
Difference of final-word log probability (direct $-$ metalinguistic). C:
Accuracy in gramaticality judgment domain. D: Difference of log odds (direct
$-$ metalinguistic). Colored lines = best-fit within model families. Black
line = best-fit across all models. Shaded region indicates bootstrapped 95%
CI.

###  4.1 Task demands vs. Model size

Figure 2 shows the results for our first tested contrast: production vs.
forced choice. We first briefly discuss raw task performance. For the
analogical reasoning domain, larger models generally achieve higher
performance (compared to smaller models in the same family) under both
evaluation methods (Figure 2A). Interestingly, we do not find this pattern for
the reflective reasoning domain (Figure 2C), because the larger models have a
strong preference for intuitive responses, whereas the small models slightly
prefer correct $>$ intuitive but have a much stronger preference for atypical
answers (see Hagendorff et al. 2023 for details).

Our main focus is the relationship between the demand gap (forced choice $-$
production) and number of parameters (Figure 2B,D). We measure the demand gap
as the difference in log odds, computed from the accuracy scores. In both
domains, the difference in log odds tends to _decrease_ as the number of
parameters _increases_ , demonstrating the predicted signature pattern (Figure
1C). This pattern holds for all model families except Pythia, whose log odds
differences are relatively flat across model size.

Figure 3 shows the analogous results for our second evaluation contrast:
metalinguistic judgment vs. probability measurement. For word prediction as
well as grammaticality judgment, the direct method generally yields higher
performance than the metalinguistic method, in line with Hu & Levy (2023)
(Figure 3A,C). As before, we also find that the demand gap (direct $-$
metalinguistic) decreases as model size increases for both domains. This gap
is measured as a log probability difference for the word prediction task
(Figure 3B), and log odds difference for grammaticality judgment (Figure 3D).
For the grammaticality judgment domain, we note that the log odds difference
is fairly flat for the Pythia and OLMo families. The tested Pythia and OLMo
models do not change in performance across sizes, and do not perform
appreciably above chance (50%) under the metalinguistic method. It could be
the case that due to their architectures or training, the “shrinking gap”
effect might not be seen within the range of tested sizes, but could
potentially emerge at larger sizes.

Overall, our results provide evidence for the pattern illustrated in Figure
1C: the “demand gap” between high-demand evaluation methods and low-demand
evaluation methods is smaller for more capable models, which we
operationalized here using parameter count. Next, we turn to an alternate
operationalization: training time of a single model.

###  4.2 Task demands vs. Training time

We evaluated intermediate checkpoints of OLMo 7B on analogical reasoning
(production vs. forced choice) and word prediction (metalinguistic vs.
direct).555We did not perform this analysis on the other two domains because
the fully trained OLMo 7B model achieved poor performance under the high-
demand evaluations (see Figure 2C and Figure 3C). Figure 4 compares
performance across the low- and high-demand evaluation methods across training
time. In both domains, the low-demand method yields higher performance earlier
during training. For analogical reasoning (Figure 4A), the slope of
improvement early during training is flatter for production than forced
choice. For word prediction (Figure 4B), the log probabilities achieved by the
direct method quickly plateau at a high value, whereas they increase more
consistently under the metalinguistic method.

In sum, evaluation methods with lower task demands can reveal abilities in a
model _earlier during training_ that might not otherwise be revealed by
higher-demand methods. These findings complement the results from Section 4.1,
which showed that low-demand settings can reveal abilities in _smaller_ models
that would not be revealed by higher-demand methods.

Figure 4: Performance across intermediate training checkpoints of OLMo 7B on
analogical reasoning (A) and word prediction (B).

##  5 Discussion

In studying the cognition of both humans and language models, one of the
primary challenges is linking specific evaluations to more general constructs.
Designing valid evaluations is thus critical for LM research. Here, we used a
concept from developmental psychology – auxiliary task demands – to
investigate the validity of a range of LM evaluations. We found that
evaluations with higher demands led to lower performance, and especially so
for LMs with fewer parameters or less training – a phenomenon we termed the
“demand gap.” Our findings also connect to _emergence_ (Wei et al., 2022a),
and suggest that the scale at which particular abilities are observed in LMs
is a function of both the evaluation that is being used and the auxiliary
capability of the model to handle the task demands of that evaluation. These
conclusions complement the prior findings of Schaeffer et al. (2023).

We investigated two evaluation contrasts in each of two cognitive domains,
which establishes consistency across settings but are still only samples from
the much broader literature on cognitive evaluation. More work will be needed
to investigate the impacts of other popular cognitive evaluation techniques,
such as in-context learning (Lampinen, 2023) or chain-of-thought prompting
(Wei et al., 2022b). Similarly, we made an effort to include a representative
group of openly accessible LMs (Frank, 2023b) released in multiple sizes or
with multiple training checkpoints. With the exception of the Pythia family,
all models showed qualitatively similar results, suggesting that our findings
are at least somewhat general. It is unknown, however, how our pattern might
vary across a broader scale of models; we speculate that the order of
magnitude less data used in training the Pythia models might have resulted in
the somewhat different performance we observed.

Our conception of what makes a particular task demand is informal. For
example, the decompositions given in Figure 1A rely on intuitive concepts such
as “question interpretation.” In developmental psychology, we can reason about
task demands using constructs such as memory, attention, or inhibition,
supplementing with our knowledge about how these develop in childhood. In LMs,
we judged task demands to be low when evaluations aligned with training
objectives (as in measurements of token probability) or when they constrained
the space of possibilities (as in forced choice tasks). A more granular notion
of task demand in LMs might investigate the attentional and representational
mechanisms triggered by different evaluations; we view this as a promising
avenue for future work.

Another interesting (and speculative) direction suggested by our work is using
LMs to study task demands in humans. The relative ordering of performance in
LMs under different evaluation methods could reflect more general properties
about how probabilistic predictions or representations of knowledge manifest
in different evaluation settings. These insights could help us generate
predictions about what kinds of methods might place greater demands on
children. Indeed, recent studies are consistently showing that probability
measurements reveal knowledge that can be masked by metalinguistic methods,
suggesting that direct predictions might be the most diagnostic of underlying
ability.

Finally, given the wide range of evaluation methods available to LM
researchers, and the stark differences that can emerge across methods, a
natural question is which method is the “right” one to use. When designing an
evaluation, we encourage researchers to think about the validity of using a
particular method to measure a particular construct of interest (Schlangen,
2019). As our work demonstrates, an important step is to specify the auxiliary
demands that the evaluation might impose on the model, and how these demands
interact with one’s broader evaluation goals (e.g., trying to be adversarial,
versus trying to reveal knowledge). Conversely, researchers should interpret
model performance not as a direct indication of some feature of intelligence
(or lack thereof), but as a reflection of a model’s true underlying capacities
seen through the lens of researchers’ design choices.

#### Acknowledgments

We would like to thank Noah Goodman and Michael Franke for helpful discussion.
This work has been made possible in part by a gift from the Chan Zuckerberg
Initiative Foundation to establish the Kempner Institute for the Study of
Natural and Artificial Intelligence.

##  6 Ethics Statement

Understanding language models’ limitations and abilities is becoming
increasingly important not only for academic research, but also for the
general public. There can be dangers both to overclaiming and underclaiming
the abilities of LMs (Bowman, 2022). This paper highlights task demands as a
potential threat to the validity of LM evaluation, which we hope will help
researchers make more informed conclusions about LMs’ cognitive capacities.
This work could also help non-experts better understand the broader context in
which LM evaluation results are collected and interpreted.

##  7 Reproducibility Statement

We prioritized reproducibility in our work. All of the tested models are
openly accessible through the Huggingface Transformers library. In addition,
the training data for the Pythia and OLMo model families are publicly
available. Inference was run on NVIDIA A100-40GB SXM GPUs during March 2024.

Our code and data are publicly available at https://github.com/jennhu/lm-task-
demands.

## References

  * Bergelson & Swingley (2012) Elika Bergelson and Daniel Swingley.  At 6–9 months, human infants know the meanings of many common nouns.  _Proceedings of the National Academy of Sciences_ , 109(9):3253–3258, February 2012.  doi: 10.1073/pnas.1113380109.  URL https://doi.org/10.1073/pnas.1113380109.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal.  Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, 2023.  URL https://arxiv.org/abs/2304.01373.
  * Bowman (2022) Samuel Bowman.  The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7484–7499, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.516.  URL https://aclanthology.org/2022.acl-long.516.
  * Bowman & Dahl (2021) Samuel R. Bowman and George Dahl.  What Will it Take to Fix Benchmarking in Natural Language Understanding?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4843–4855, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.385.  URL https://aclanthology.org/2021.naacl-main.385.
  * Carruthers (2013) Peter Carruthers.  Mindreading in infancy.  _Mind & Language_, 28(2):141–172, 2013.
  * Cronbach & Meehl (1955) Lee J Cronbach and Paul E Meehl.  Construct validity in psychological tests.  _Psychological Bulletin_ , 52(4):281, 1955.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill.  Language models show human-like content effects on reasoning, 2022.  URL https://arxiv.org/abs/2207.07051.
  * Dentella et al. (2023a) Vittoria Dentella, Fritz Günther, and Evelina Leivada.  Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias.  _Proceedings of the National Academy of Sciences_ , 120(51):e2309583120, December 2023a.  doi: 10.1073/pnas.2309583120.  URL https://doi.org/10.1073/pnas.2309583120.  Publisher: Proceedings of the National Academy of Sciences.
  * Dentella et al. (2023b) Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada.  Testing AI performance on less frequent aspects of language reveals insensitivity to underlying meaning, 2023b.  URL https://arxiv.org/abs/2302.12313.
  * Firestone (2020) Chaz Firestone.  Performance vs. competence in human–machine comparisons.  _Proceedings of the National Academy of Sciences_ , 117(43):26562–26571, October 2020.  URL https://doi.org/10.1073/pnas.1905334117.  Publisher: Proceedings of the National Academy of Sciences.
  * Frank (2023a) Michael C Frank.  Baby steps in evaluating the capacities of large language models.  _Nature Reviews Psychology_ , 2(8):451–452, 2023a.
  * Frank (2023b) Michael C Frank.  Openly accessible llms can help us to understand human cognition.  _Nature Human Behaviour_ , 7(11):1825–1827, 2023b.
  * Frederick (2005) Shane Frederick.  Cognitive Reflection and Decision Making.  _Journal of Economic Perspectives_ , 19(4):25–42, 2005.  doi: 10.1257/089533005775196732.  URL https://www.aeaweb.org/articles?id=10.1257/089533005775196732.
  * Gemma Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open Models Based on Gemini Research and Technology, 2024.  URL https://arxiv.org/abs/2403.08295.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  OLMo: Accelerating the Science of Language Models, 2024.  URL https://arxiv.org/abs/2402.00838.
  * Hagendorff et al. (2023) Thilo Hagendorff, Sarah Fabi, and Michal Kosinski.  Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT.  _Nature Computational Science_ , 3(10):833–838, October 2023.  ISSN 2662-8457.  doi: 10.1038/s43588-023-00527-x.  URL https://doi.org/10.1038/s43588-023-00527-x.
  * Hendriks & Koster (2010) Petra Hendriks and Charlotte Koster.  Production/comprehension asymmetries in language acquisition.  _Asymmetries in Language Acquisition_ , 120(8):1887–1897, August 2010.  ISSN 0024-3841.  doi: 10.1016/j.lingua.2010.02.002.  URL https://www.sciencedirect.com/science/article/pii/S0024384110000215.
  * Hoogland et al. (2023) Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, and Stan van Wingerden.  Towards Developmental Interpretability, 2023.  URL https://www.lesswrong.com/posts/TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability.
  * Hu & Levy (2023) Jennifer Hu and Roger Levy.  Prompting is not a substitute for probability measurements in large language models.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 5040–5060, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.306.  URL https://aclanthology.org/2023.emnlp-main.306.
  * Hu et al. (2024) Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy.  Language models align with human judgments on key grammatical constructions, 2024.  URL https://arxiv.org/abs/2402.01676.
  * Ivanova (2023) Anna A. Ivanova.  Running cognitive evaluations on large language models: The do’s and the don’ts, 2023.  URL https://arxiv.org/abs/2312.01276.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7B, 2023.  URL https://arxiv.org/abs/2310.06825.
  * Katzir (2023) Roni Katzir.  Why large language models are poor theories of human linguistic cognition: A reply to Piantadosi (2023).  2023\.  URL https://ling.auf.net/lingbuzz/007190.
  * Kauf et al. (2024) Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A. Ivanova.  Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models, 2024.  URL https://arxiv.org/abs/2403.14859.
  * Keen (2003) Rachel Keen.  Representation of objects and events: Why do infants look so smart and toddlers look so dumb?  _Current Directions in Psychological Science_ , 12(3):79–83, 2003.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams.  Dynabench: Rethinking Benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 4110–4124, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.324.  URL https://aclanthology.org/2021.naacl-main.324.
  * Lampinen (2023) Andrew Kyle Lampinen.  Can language models handle recursively nested grammatical structures? A case study on comparing models and humans, 2023.  URL https://arxiv.org/abs/2210.15303.
  * Linzen (2020) Tal Linzen.  How Can We Accelerate Progress Towards Human-like Linguistic Generalization?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 5210–5217, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.465.  URL https://aclanthology.org/2020.acl-main.465.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535, 2016.  doi: 10.1162/tacl_a_00115.  URL https://aclanthology.org/Q16-1037.  Place: Cambridge, MA Publisher: MIT Press.
  * Lyu et al. (2024) Chenyang Lyu, Minghao Wu, and Alham Fikri Aji.  Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models, 2024.  URL https://arxiv.org/abs/2402.13887.
  * Marvin & Linzen (2018) Rebecca Marvin and Tal Linzen.  Targeted Syntactic Evaluation of Language Models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 1192–1202, Brussels, Belgium, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1151.  URL https://aclanthology.org/D18-1151.
  * Misra (2022) Kanishka Misra.  minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models, 2022.  URL https://arxiv.org/abs/2203.13112.
  * Momennejad et al. (2023) Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson.  Evaluating Cognitive Maps and Planning in Large Language Models with CogEval.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=VtkGvGcGe3.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial NLI: A New Benchmark for Natural Language Understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4885–4901, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.441.  URL https://aclanthology.org/2020.acl-main.441.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In Katrin Erk and Noah A. Smith (eds.), _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1144.  URL https://aclanthology.org/P16-1144.
  * Power et al. (2022) Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra.  Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, 2022.  URL https://arxiv.org/abs/2201.02177.
  * Raji et al. (2021) Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada.  AI and the Everything in the Whole Wide World Benchmark.  In J. Vanschoren and S. Yeung (eds.), _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran, 2021.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf.
  * Röttger et al. (2024) Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy.  Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models, 2024.  URL https://arxiv.org/abs/2402.16786.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  WinoGrande: An Adversarial Winograd Schema Challenge at Scale.  _Commun. ACM_ , 64(9):99–106, August 2021.  ISSN 0001-0782.  doi: 10.1145/3474381.  URL https://doi.org/10.1145/3474381.  Place: New York, NY, USA Publisher: Association for Computing Machinery.
  * Schaeffer et al. (2023) Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.  Are Emergent Abilities of Large Language Models a Mirage?  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=ITw9edRDlD.
  * Schlangen (2019) David Schlangen.  Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research, 2019.  URL https://arxiv.org/abs/1908.10747.
  * Simon et al. (2023) James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht.  On the Stepwise Nature of Self-Supervised Learning.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.  URL https://proceedings.mlr.press/v202/simon23a/simon23a.pdf.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=uyTL5Bvosj.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.  URL https://arxiv.org/abs/2307.09288.
  * Tsvilodub et al. (2024) Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke.  Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods, 2024.  URL https://arxiv.org/abs/2403.00998.
  * Turan-Küçük & Kibbe (2024) Esra Nur Turan-Küçük and Melissa M Kibbe.  Three-year-olds’ ability to plan for mutually exclusive future possibilities is limited primarily by their representations of possible plans, not possible events.  _Cognition_ , 244:105712, 2024.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman.  BLiMP: The Benchmark of Linguistic Minimal Pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8, 2020.  URL https://doi.org/10.1162/tacl_a_00321.  Publisher: MIT Press.
  * Webb et al. (2023) Taylor Webb, Keith J. Holyoak, and Hongjing Lu.  Emergent analogical reasoning in large language models.  _Nature Human Behaviour_ , 7(9):1526–1541, September 2023.  ISSN 2397-3374.  doi: 10.1038/s41562-023-01659-w.  URL https://doi.org/10.1038/s41562-023-01659-w.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent Abilities of Large Language Models.  _Transactions on Machine Learning Research_ , 2022a.  URL https://openreview.net/forum?id=yzkSU5zdwD.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain of Thought Prompting Elicits Reasoning in Large Language Models.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022b.  URL https://openreview.net/forum?id=_VjQlMeSB_J.
  * West et al. (2024) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi.  The Generative AI Paradox: “What It Can Create, It May Not Understand”.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=CF8H8MS5P8.


# Diplomarbeit

Karim Adiprasito  Karim ADIPRASITO111The final preparation of this paper was
supported by the Deutsche Forschungsgemeinschaft within the research training
group ‘Methods for Discrete Structures’ (GRK1408).
Institute of Mathematics, Discrete Geometry Group, Free University Berlin,
Arnimallee 2, 14195 Berlin
e-mail<EMAIL_ADDRESS>
# Characterization of polytopes via tilings with similar pieces

Karim Adiprasito  Karim ADIPRASITO111The final preparation of this paper was
supported by the Deutsche Forschungsgemeinschaft within the research training
group ‘Methods for Discrete Structures’ (GRK1408).
Institute of Mathematics, Discrete Geometry Group, Free University Berlin,
Arnimallee 2, 14195 Berlin
e-mail<EMAIL_ADDRESS>
###### Abstract

Generalizing results by Valette, Zamfirescu and Laczkovich, we will prove that
a convex body $K$ is a polytope if there are sufficiently many tilings which
contain a tile similar to $K$. Furthermore, we give an example that this can
not be improved.

Consider a convex body (a compact convex set with nonempty interior) $K$ in
$\mathbb{R}^{n}$, which is tiled into a finite number of convex bodies. A
tiling $T$, as a formal object, will here be the set consisting of all its
tiles, and all tiles will be assumed to be convex bodies. Also, this
investigation will only consider proper tilings, that is, tilings which are
not trivial (meaning they consist of more than just one tile).
We consider the following problem: Let there be tiles similar to $K$. Does it
follow that $K$ is a polytope?
For dimension 2, M. Laczkovich ([1]) could show that if one tile is similar to
$K$, and if the tiling is proper, $K$ is in fact a polytope. He generalized a
remark by G. Valette and T. Zamfirescu in [2].
Now the question arises: is this extendable to dimension 3? Already the
original paper by Laczkovich contained a remark that the immediate
generalization must be wrong and left the problem open for higher dimensions.
Indeed, a circular cone can easily be tiled in such a way that one tile is
similar to the cone by just cutting near the apex. T. Zamfirescu asked whether
this was optimal, i.e.: Consider a convex body $K$ in 3-space which is tiled
in such a way such that 2 tiles are similar to $K$. Then is $K$ a polytope?
This will turn out to be true, and is a very special case of the general
theorem. In dimensions higher than 3, a condition only on the number of
similar tiles will never be sufficient (see the example at the end of this
paper). The additional condition will encode some information on how the tiles
are located relative to the convex body itself. In consistency with our
observations, the condition will degenerate in dimensions 2 and 3. We thus are
able to answer Zamfirescu’s question as well as solve the problem.
Let us state the theorem:

######  Theorem 1.

Let $K$ be a $n$-dimensional convex body, and let $T_{i},\
\\{1,2,3,...,n-1\\}$ be $n-1$ proper tilings of $K$, each of which contains a
tile $L_{i}$ similar to $K$.
If the convex hull of the fixed points $x_{L_{i}}$ forms a nondegenerate
$n-2$-dimensional simplex, then $K$ is a polytope.

Here, the point $x_{L}$ is the fixed point of the similarity $f_{L}$ from $K$
to $L$. If there is more than one similarity from $K$ to $L$, choose one.
The simplex referred to in theorem 1 will be called the tip simplex of $K$
with respect to the tilings involved.
Often, one is in the situation to have a single tiling, and several tiles are
similar to $K$. This is a special case of above theorem, which is a bit harder
to prove, since then one has to deal with interdependencies between the tiles
similar to $K$ when deforming tilings.

######  Corollary 2.

Let $K$ be a $n$-dimensional convex body, which is properly tiled into a
finite number of convex bodies, $n-1$ of which are similar to $K$. Denote
these particular tiles by $L_{i}$.
If the convex hull of the fixed points $x_{L_{i}}$ forms a nondegenerate
$(n-2)$-dimensional simplex, then $K$ is a polytope.

######  Corollary 3.

Let $K$ be a convex body in $\mathbb{R}^{n},\ n\leq 3$, which is properly
tiled into a finite number of convex bodies, $n-1$ of which are similar to
$K$. Then $K$ is a polygon/ a polyhedron.

Even more, in dimension 3, symmetry also plays a role in this calculation:

######  Corollary 4.

Let $K$ be a convex body in $\mathbb{R}^{3}$, which is properly tiled into a
finite number of convex bodies, $1$ of which is similar to $K$. If there are
two similarities $f_{L},f_{L}^{\prime}$ from $K$ to the tile $L$ with
different fixpoints respectively, then $K$ is a polytope.

###### Proof.

We can regard this situation in the following way: $K$ has two tilings $T$ and
$T^{\prime}$ which are the same, but in $T$ $f_{L}$ is the similarity from $K$
to $L$, and in $T^{\prime}$ $f_{L}^{\prime}$ is considered to be the
similarity from $K$ to $L$. Theorem 1 then concludes the proof. ∎

Denote by $\textsc{bd}(M),\ \textsc{int}(M),\ \mathscr{E}(M),\
\textsc{conv}(M)$ the the boundary/ the interior/ set of extremal points/ the
convex hull of a set $M$. Also, $B_{\varepsilon}(x)$ denotes the set of all
points in $\mathbb{R}^{n}$ with distance less than $\varepsilon$ to $x$.
Consider the general situation of a convex body $K$ tiled into convex bodies
$K\supset P_{j}\in T,\ j\in[s]:=\\{1,2,3,...s\\}$. For every 2 tiles $P_{i},\
P_{j}$ there is a hyperplane $H_{ij}$ seperating the two. For a fixed
$i\in[s]$, let $F_{ij}$ be the halfspace with boundary $H_{ij}$ containing
$P_{i}$. Then, $P_{i}=K\cap\bigcap_{i\neq j}F_{ij}$, and every extremal point
$x\in\textsc{int}(K)$ of $P_{i}$ is a vertex of $\bigcap_{i\neq j}F_{ij}$. In
particular, $\mathscr{E}(T):=\cup{\mathscr{E}(P_{j})}$ has finitely many
elements in common with $\textsc{int}(K)$, i.e.
$\mathscr{E}(T)\cap\textsc{int}(K)$ is finite. Suppose $L=P_{1}$ is similar to
$K$, then the similarity transformation $f_{L}$ from $K$ to $L$ maps
$\mathscr{E}(K)$ to $\mathscr{E}(L)$ bijectively. If the fixed point $x_{L}$
of $f_{L}$ lies in $\textsc{int}(K)$, then for some iteration of $f_{L}$,
$f_{L}^{n}(K)$ lies in the interior of $K$. Since
$\mathscr{E}(f_{L}^{n}(K))=\mathscr{E}(f_{L}^{n}(K))\cap\textsc{int}(K)$ is
finite, so is $\mathscr{E}(K)$, and $K$ is a polytope. Therefore, we can
assume that $x_{L}\in\textsc{bd}(K)$.

## Adapting a tiling

In this section, we focus on adapting tilings until they have features that
come handy in the later proof. First, we want $L$ to be homothetic to $K$.
Then we want to move the fixed point to an arbitrary point of the tip simplex.

Before we begin, let us state our methods to deform a tiling:
Let $K$ be a convex body, and let $T$ be a tiling of $K$, containing a tile
$L$ similar to $K$. Let $f_{L}$ be a similarity mapping from $K$ to $L$.
Define $f_{L}(T)$ to be the function $f_{L}$ applied to all the tiles of $T$,
and thus a tiling of $L$. The idea is that we can refine the tiling $T$ to
$T^{\prime}$ by defining $T^{\prime}:=f_{L}(T)\cup T\setminus\\{L\\}$. In this
situation, we will also write $T^{\prime}=f_{L}(T)+T$. The similarity mappings
of newly created tiles will be defined by the composition of the similarities
involved. We call this procedure iterating a tiling.
In a related situation, suppose we are given two tilings $T_{1}$ and $T_{2}$,
we could form the tiling $T_{1}*T_{2}:=\\{P\cap Q|\ P\in T_{1},\ Q\in
T_{2}\\}$. If $T_{1}$ tiles a set $K_{1}$, and $T_{2}$ tiles $K_{2}$, then
$T_{1}*T_{2}$ is a tiling of the intersection of $K_{1}$ and $K_{2}$. In
particular, if $K_{1}=K_{2}$, then $T_{1}*T_{2}$ is a (possibly refined)
tiling of $K_{1}$. Let us apply the above methods to concrete tilings to show
that simplifications are indeed possible.

######  Lemma 5.

Let $K$ be a convex body as above, $T$ a tiling which contains a tile $L$
similar to $K$. Then there is a tiling $T^{\prime}$ of $K$ which contains a
homothetic copy of $K$ whose fixed point coincides with $x_{L}$.

###### Proof.

Denote by $f_{L}$ the similarity transformation from $K$ to $L$. We may assume
$x_{L}=0$. Let $O_{n}$ denote the set of orthogonal transformations of
$R^{n}$. Let $\lambda$ denote the similarity ratio of $f_{L}$, and define
$M_{L}:=\lambda^{-1}f_{L}$. Then $M_{L}\in O_{n}$. Since $O_{n}$ forms a
compact subset of the space of n-dimensional matrices, it follows that the
powers of $M_{L}$ can get as close to the identity matrix as we please.
Write $B_{\varepsilon}:=B_{\varepsilon}(0)$ for the open ball of radius
$\varepsilon$ around $0$ and $TC(K):=TC_{0}(K)=TC_{x_{L}}(K)$ for the solid
tangent cone of $K$ at 0. Obviously, the solid tangent cones of later
iterations of $f_{L}(K)$ are included in earlier:

$TC(K)\supset TC(L)\supset TC(f^{2}_{L}(K))\supset TC(f^{3}_{L}(K))...$

But since we can get $f^{i}_{L}$ as close to a homothety as we want, we can
get $B_{\varepsilon}\cap TC(f^{i}_{L}(K))$ as close to $B_{\varepsilon}\cap
TC(K)$ (with respect to, for example, the Pompeiu-Hausdorff metric and a fixed
$\varepsilon>0$) as we want by choosing $i\in\mathbb{N}$. Thus, the solid
tangent cones of $K$ and $f^{i}_{L}(K),\ i\in\mathbb{N}$ at $x_{L}$ coincide.
In particular, because all tiles are compact, convex and have nonempty
interior, their tangent cones are never degenerate, and it follows that $L$ is
the only tile of $T$ which contains $0=x_{L}$. Then there is an
$\varepsilon>0$ such that

$K\supset B_{\varepsilon}\cap K=B_{\varepsilon}\cap L=B_{\varepsilon}\cap
TC(K).$

In particular, $x_{L}$ is not an accumulation point of extremal points of $K$.
Choose $R$ such that $B_{R}\supset K$, set $t:=R/\varepsilon$ and $d$ such
that $\lambda^{d}<\varepsilon$ and $t\lambda^{d}<1$. There is a tiling $T$ of
$K$ such that $f_{L}^{d}(K)$ is a tile of $K$. $tM_{L}^{-d}(T)$ gives a tiling
of $tM_{L}^{-d}(K)$, and by the choice of $t$, $tM_{L}^{-d}(K)\supset K$.
Thus, $tM_{L}^{-d}(T)*\\{K\\}$ is a tiling of $K$ which contains
$L^{\prime}:=tM_{L}^{-d}(f_{L}^{d}(K))=t\lambda^{d}K\subsetneq K$, which is a
homothetic copy of $K$. ∎

Before we go on, let us consider the above constructed tiling. Recall the
notion of hyperplanes $H_{1j}$ seperating the tiles $P_{1}=L^{\prime}$ and
$P_{j}$ of $K$, then $L^{\prime}$ is the convex hull of

$\\{x_{L}\\}\cup(H_{1j}\cap TC(K))=\\{x_{L}\\}\cup(H_{1j}\cap K).$

Thus, we have the following Lemma:

######  Lemma 6.

$K$ can be written as

$\textsc{conv}(\\{x_{L}\\}\bigcup_{i}B_{i}),$

where the $B_{i}\subset\textsc{bd}(K)$ are finitely many convex compact
$n-1$-dimensional sets with boundary (which can be chosen to be disjoint from
$\\{x_{L}\\}$). We will call these sets a base of $K$.

Now that the tiles are directed nicely, we want to turn to adjusting their
position.

######  Lemma 7.

Let $K$ be a convex body as above, $T_{i}$ tilings fulfilling the conditions
of theorem 1, $S$ the tip simplex, $x_{0}$ a point in the tip simplex. Then
there is a tiling $T^{\prime}$ with a tile $L$ similar to $K$ such that
$x_{L}=x_{0}$.

###### Proof.

If the tip simplex is just a point, the Lemma is trivial. Suppose therefore
that there are at least two tilings with $T_{1}$ and $T_{2}$ with tiles
$L_{1}$ respectively $L_{2}$ which are similar to $K$ and have distinct fixed
points, and suppose the corresponding similarities are chosen to be
homotheties. Then the tiling $T_{1}+f_{L_{1}}(T_{2})$ will contain a
homothetic copy of $K$: $L_{3}=f_{L_{1}}(L_{2})$. $L_{3}$ is new to us,
because the corresponding fixed point will not coincide with $x_{L_{1}}$ or
$x_{L_{2}}$, but will lie in their convex hull. By iterating this procedure,
we see that there is a dense subset $M$ of
$\textsc{conv}(\\{x_{L_{1}},x_{L_{2}}\\})$ we can make the fixed points of
similarities lie in.
Next, suppose $x_{0}\in\textsc{conv}(\\{x_{L_{1}},x_{L_{2}}\\})\setminus M$.
Find a tiling $T$ with a similar copy $L$ of $K$ such that
$x_{L}\in\textsc{conv}(\\{x_{L_{1}},x_{0}\\})$. (Again, we assume the
similarities to be homotheties.) Define $H(x)$ to be the homothety which fixes
$x_{L_{1}}$ and maps $f_{L}(x_{0})$ to $x_{0}$, and let $T^{\prime}:=H(T)$,
which covers $K$. $T^{\prime}*\\{K\\}$ is a tiling of $K$. If $L$ is chosen
small enough, $H(L)$ is a proper subset of $K$ and thus a tile of
$T^{\prime}*\\{K\\}$ with fixed point $x_{0}$. With higher dimensional tip
simplices, this construction works just in the same way. ∎

## Conclusion of proof

We will prove theorem 1 by induction. In dimension 1, it is trivial, in
dimension 2, it was proven by Laczkovich.

Let us assume theorem 1 is proven in dimension $n-1$. Let $K$ be a convex body
fulfilling the conditions of the theorem 1 in Dimension $n$, let $x$ be some
point in the relative interior of the tip simplex $S$, and let $H$ be some
$n-1$-dimensional hyperplane of $\mathbb{R}^{n}$ containing $x$. $H\cap K$ is
a $n-1$-dimensional convex set. It could even be of smaller dimensions, so let
us just assume $H\cap K$ is not a point. Let $T^{\prime}_{i}$ be proper
tilings of $K$ whose similarities are homotheties and whose fixed points are
also the extremal points of the $n-3$-dimensional simplex $S\cap H$.
$H\cap K$ inherits a tiling structure from $K$ by means of intersection:
$T^{\prime}_{i}$ induces the tiling $\\{H\\}*T^{\prime}_{i}$ on $H\cap K$.
This tiling is proper if $H$ does not coincide with any of the hyperplanes
separating an $L_{i}$ from an adjacent tile, which is true for all but
finitely many choices of $H$. Note that because $L^{\prime}_{i}$ is a
homothetic copy of $K$ whose fixed point lies in $H$, $H\cap L^{\prime}_{i}$
will be an element of $T^{\prime}_{i}$ which is a homothetic copy of $H\cap
K$. The fixed point of $H\cap L^{\prime}_{i}$ in $H\cap K$ coincides with the
fixed point of $L^{\prime}_{i}$ in $K$ and is therefore an extremal point of
$S\cap H$. Since $x$ is a relative interior point of $S$, $S\cap H$ is a
nondegenerate $n-3$-dimensional simplex spanned by the fixed points of $H\cap
L^{\prime}_{i}$, which in turn are elements of the proper tilings
$\\{H\\}*T^{\prime}_{i}$ of $H\cap K$.
Using the induction hypothesis, we see that $H\cap K$ must be a polytope.

######  Proposition 8.

Let $K$ be a convex body in $\mathbb{R}^{n}$, which is tiled as in the
description of theorem 1. Let $S$ be the tip simplex, $x$ a point in the
relative interior of this simplex, and let $H$ be some hyperplane of
$\mathbb{R}^{n}$ containing $x$. Then $H\cap K$ is a polytope in all but
possibly a finite number of cases.

###### Proof of theorem 1.

Assume theorem 1 is proven in dimension $n-1$.
As already stated, $K$ is the convex hull of the union of a finite number of
$n-1$ dimensional compact convex sets and a point $x$ in the relative interior
of the tip simplex. Since we supposed $K$ is not a polytope, one of these sets
has infinitely many extremal points which are also extremal points of $K$.
Call this set $B$, and recall that $B\subset\textsc{bd}(K)$. Pick another
point $y$ in the relative interior of the tip simplex of $K$. Choose a
$n-1$-dimensional affine subspace $H$ parallel to $B$ and containing $y$. If
$H\cap\textsc{conv}(\\{x\\}\cup B)=\emptyset$, interchange the roles of $y$
and $x$. If $H$ contains $x$, tilt $H$ just a little such that it intersects
$\textsc{conv}(\\{x\\}\cup B)$ but neither intersects $\\{x\\}$ nor the base
$B$. Also, assume $H$ is not one of the finite hyperplanes excluded in
proposition 8.
The intersection of $H$ and $K$ is, as we know from proposition 8, a polytope.
But $H\cap B$ will have infinitely many extremal points, as it is a (possibly
dilated, if we tilted $H$) homothetic copy of $B$. Thus, $K$ and $B$ can only
share a finite number of extremal points, in contradiction with the
assumption. ∎

### Sharpness of results

We could now ask if the above results are optimal, and indeed, they are. First
note that as soon as we have constructed (in any dimension) a tiling $T$ of a
convex body $K$ which contains (at least) 2 similar copies $L,L^{\prime}$ of
$K$, then we could create a tiling with more similar copies by regarding the
tiling $f_{L}(T)+T$. Note however that we can never make a degenerate tip
simplex nondegenerate using this method. The dimension of the tip simplex is
thus the real condition to make $K$ a polytope, a property not visible in
dimensions 2 or 3.
Using induction on dimensions, we can construct a convex body which even
allows us to see that the condition on the tip simplex is optimal, in
particular, $n-3$-dimensional tip simplices are not enough to conclude that
$K$ is a polytope. Figuratively speaking, we take a circular cone, which shows
theorem 1 to be sharp in dimension 3, and take it as a base for a
4-dimensional cone, which in turn forms a base of a five-dimensional cone etc.
This example is just an extension of the example known for dimension 3.
To make a concrete example with $n-3$ dimensional tip simplex in dimension
$n>2$, use the following construction: Regard the convex body $K$ which is the
set of all points

$\sqrt{x_{1}^{2}+x_{2}^{2}}+\sum_{i\in\\{3,4,5,...,n\\}}x_{i}\leq 1;\
x_{i}\geq 0\ \forall i\in\\{3,4,5,...,n\\}$

where the $x_{i}$ are coordinates with respect to some base
$\\{e_{1},e_{2},e_{3},...,e_{n}\\}$ of $\mathbb{R}^{n}$. We will show that the
tip simplex $S$ is
$\textsc{conv}(\bigcup_{i\in\\{3,4,5,...,n\\}}\\{e_{i}\\})$.
We regard the homotheties

$f_{i}(x)=\frac{1}{2}(x-e_{i})+e_{i}$

for $i\in\\{3,4,5,...,n\\}$. Their fixed points span the said tip simplex, and
the interior of the images of $K$ does not intersect. Thus, it remains to show
that the remaining tile is convex. But this is simple, since it can be written
as intersection of the convex sets $K$ and $X_{i},\ i\in\\{3,4,5,...,n\\}$,

$X_{i}:=\\{x\in\mathbb{R}^{n}|x_{i}\leq\frac{1}{2}\\}.$

## References

  * [1] M. Laczkovich; Decomposition of convex figures into similar pieces, Discrete and Comp. Geometry Vol. 13, 143-148, 1995
  * [2] G. Valette, T. Zamfirescu; Les partages d’un polygone convexe en 4 polygones semblambes au premier, J. Combin. Theory Ser. B, Vol 16, 1-16, 1974