# CoLLEGe: Concept Embedding Generation
for Large Language Models

Ryan Teehan, Brenden M. Lake, Mengye Ren

New York University
<EMAIL_ADDRESS>
###### Abstract

Current language models are unable to quickly learn new concepts on the fly,
often requiring a more involved finetuning process to learn robustly.
Prompting in-context is not robust to context distractions, and often fails to
confer much information about the new concepts. Classic methods for few-shot
word learning in NLP, relying on global word vectors, are less applicable to
large language models. In this paper, we introduce a novel approach named
CoLLEGe (Concept Learning with Language Embedding Generation) to modernize
few-shot concept learning. CoLLEGe is a meta-learning framework capable of
generating flexible embeddings for new concepts using a small number of
example sentences or definitions. Our primary meta-learning objective is
simply to facilitate a language model to make next word predictions in
forthcoming sentences, making it compatible with language model pretraining.
We design a series of tasks to test new concept learning in challenging real-
world scenarios, including new word acquisition, definition inference, and
verbal reasoning, and demonstrate that our method succeeds in each setting
_without task-specific training_.

##  1 Introduction

Imagine a student first attending a philosophy lecture on epistemology,
wherein their professor discusses and critiques the positions of idealists,
pragmatists, and foundationalists, among others. Some concepts and terms, such
as idealism or pragmatism, may be familiar from past experience but in this
new and unfamiliar context they seem to have taken on new meaning. Other
concepts may be entirely new, including the concept of “epistemology” itself.
During the lecture, the examples the professor provides for each concept, as
well as the sentences they use when discussing them, allow the student to form
an initial sense of their meaning and usage. With time, additional examples,
and using the concepts directly in writing, the student’s knowledge solidifies
and what was once unfamiliar is now intuitive.

Figure 1: Our model generates an embedding for an unseen token given two
example sentences. The ground truth word is _pendant_ , and the model is able
to generate an accurate definition using the embedding produced by CoLLEGe.

Building intuitions about the meaning of unfamiliar concepts in this way, with
only a few examples of their usage, is common in real-world human learning,
but remains difficult for language models, particularly when we want to
consolidate this knowledge into discrete tokens. Providing a few in-context
examples of how to use these new tokens can be a stopgap, but is often less
powerful, leads to increased context length, and the additional examples can
at times serve as distractors that confuse the language model, to say nothing
about how unnatural it is (imagine if, each time the professor wanted the
student to answer a question about epistemological idealism, they began by
repeating a few sentences containing “idealism”). Instead, with a few examples
showing how a concept is used, language models should know general semantic
knowledge about this new concept, similar to the knowledge encoded in their
pretrained representations. We frame this as a few-shot learning problem,
where, given a few example sentences, the goal is generate an embedding for a
new concept token with expressive and task-general semantic information.

Prior work on few-shot word learning in natural language focused on leveraging
the seminal works on global word vector representations (Mikolov et al., 2013;
Pennington et al., 2014; Khodak et al., 2018; Lampinen & McClelland, 2017).
However, these methods are less well suited to augmenting the knowledge of
contemporary large language models. First, the embeddings generated from
methods based on global word vector representations may be difficult to adapt
to the representation space of contemporary language models. Additionally,
learned contextual representations from pretrained language models provide a
more powerful and semantically rich source for few-shot concept learning,
allowing for complex usage of new concepts using embeddings generated from
only a few examples.

Furthermore, prior evaluation methods for new concept learning were limited to
noisy proxy measures, such as the correlation between embedding cosine
similarity and human similarity judgements (Lazaridou et al., 2017), or the
cosine similarity between the few-shot concept embeddings and “ground truth”
Word2Vec embeddings (e.g. the definitional nonce in Herbelot & Baroni (2017)).
Current language models are able to use language in highly sophisticated and
complex ways; true evaluation of few-shot concept learning in this setting
should assess whether new concepts can be used in similarly complex and
sophisticated ways. What older evaluations measure (e.g. correlation with
human similarities or with ground-truth embeddings) tell us little about how
well language models can use learned concepts. How well can they define a
concept given only a few examples? Can they correctly answer fill-in-the-blank
questions for difficult words? We can, and often do, ask humans the same
questions to determine how well they have internalized a new concept. Our
evaluations of language models should follow suit.

In this paper, we present CoLLEGe, a novel and conceptually simple framework
to enable large language models to quickly learn new concept tokens. To
evaluate our method, we develop a suite of tasks to directly evaluate how well
these concepts are learned, including GRE-style fill-in-the-blank verbal
reasoning, definition inference and generation, and Internet slang usage. Our
method leverages the vast amount of pre-training data and learning can be
seamlessly embedded in the model pre-training process. We discover that
training techniques such as an example buffer, negative example sampling, and
knowledge distillation contributed significantly to the model’s concept
learning performance. Moreover, thanks to our general pre-training procedure,
our model is able to transfer to these concept learning tasks in a zero-shot
manner with no additional task-specific training needed, while maintaining the
LLM’s original performance on regular data.

In summary, our contributions are:

  * •

A simple add-on learnable module for few-shot, LLM concept learning;

  * •

An approach to training our algorithm which combines an example buffer,
negative sampling, and knowledge distillation. We show that each of these
components plays an important role in learning;

  * •

Challenging datasets to measure the effectiveness of few-shot concept learning
methods for LLMs. They test both general and complex concept knowledge,
naturalistic acquisition of new concepts, and relational abstraction;

  * •

Experiments showing that, by training an embedding generation modules in a
task-general manner, we can generate embeddings that, without additional
training, allow a pretrained LLM to: a) generate plausible definitions for new
concepts, b) correctly solve fill-in-the-blank tasks for difficult words, and
c) correctly identify the meaning of new slang terms.

Figure 2: Our proposed CoLLEGe framework for concept embedding generation.
Support sequences are embedded by a pretrained masked language model (MLM)
(e.g. RoBERTa) with an additional Transformer encoder to produce pooled
sequence embeddings for each support sequence. These are aggregated and
projected into the input and output embedding space for the pretrained LLM
(e.g. LLaMA). According to UrbanDictionary, _beige flag_ , an Internet slang
appeared in mid-2023, means “a benign but annoying trait or
habit.”222https://www.urbandictionary.com/define.php?term=Beige%20flag

##  2 Related Work

#### Few-Shot Word Learning:

A classic and related task in NLP is rare word learning (Luong et al., 2015).
Lazaridou et al. (2017) create the synthetic “chimera” concepts and provide
early evidence that summation over (global) word vectors in the context
surrounding a new or rare word can produce a useful embedding. Khodak et al.
(2018) build on this, presenting a method which includes a learned linear
transformation to account for shared features across global word vectors.
Lampinen & McClelland (2017) present an even simpler method, involving
freezing the majority of the weights of the network and using gradient descent
to tune only the weights related to the new word(s). For a more complex
approach, Herbelot & Baroni (2017) modify the Word2Vec algorithm for more
effective few-shot learning. More modern approaches include HiCE (Hu et al.,
2019), which uses Transformer layers to induce a new word embedding from
Word2Vec embeddings, and Mem2Vec (Sun et al., 2018), which uses a long-term
memory system. Similarly, Weston et al. (2015) model new words by using
contextual examples from memory. They store a bag-of-words for the left and
right context surrounding the new word, and simulate new word learning with a
fixed percent of words encountered during training. By contrast, we use a
frozen MLM to learn a representation for the new concept token. Other
approaches incorporate morphological information (Luong et al., 2013; Schick &
Schütze, 2019). While these methods are useful, particlarly for learning
global word vector representations, they are less useful for augmenting the
embeddings of pretrained LLMs. In part, this is because global word vectors do
not map easily to the pretrained LLM embedding space. Additionally, global
word vector representations are often less informative than the pretrained
representations from BERT-style models.

#### Meta-Learning:

Matching Networks (Vinyals et al., 2016) and Prototypical Networks (Snell et
al., 2017) both approach few-shot learning as a meta-learning problem. Online
prototypical networks (Ren et al., 2020) build on the latter for novel concept
learning in an online and continual fashion. Our approach is also related to
fast-weight networks (Schmidhuber, 1992), since we use the support sequences
to generate a fast embedding weight for the new concept.

For language, meta-learning is often used for knowledge augmentation (Hu et
al., 2023), task adaptation (Chen et al., 2022; Zhong et al., 2021; Bansal et
al., 2020), domain adaptation (Qian & Yu, 2019; Li et al., 2020; Geng et al.,
2019), rare word recognition (Lux & Vu, 2021) (in ASR), and word sense
disambiguation Holla et al. (2020), among other applications (Lee et al.,
2022). Some recent methods frame meta-learning as a sequence modeling problem,
drawing inspiration from in-context learning (Chen et al., 2022; Fifty et al.,
2023). Finally, Lake & Baroni (2023) recently developed a method for learning
compositional concepts. In our work, context sentences are encoded to generate
a new embedding, conceptually similar to a prototype for the new concept, to
optimize a general language modeling objective, rather than a collection of
task objectives.

#### Compression:

A number of methods exist to compress sentences into new embeddings or tokens.
Prior work in NLP developed methods for generating task-general embeddings
from natural language sentences (Conneau et al., 2017; Kiros et al., 2015;
Wang et al., 2020). ReadOnce (Lin et al., 2021) is a more recent method for
generating compressed document representations which can be used across a
variety of downstream tasks. Similarly, recent methods compress prompts
(Chevalier et al., 2023; Ge et al., 2024; Mu et al., 2024) or documents Xu et
al. (2024) into summary vectors either as a form of memory or a method for re-
using prompts (e.g. when specifying instructions). RMT (Bulatov et al., 2022)
learns memory tokens during pretraining in order to extend the effective
context window. Nugget (Qin & Van Durme, 2023) dynamically chooses which
tokens are aggregated into the encoded representation. Rather than compressing
the entire meaning of each context sentence, our method extracts and
aggregates information relevant to the new concept.

##  3 CoLLEGe: Concept Learning with Language Embedding
Generation

In this section, we describe our proposed approach for enabling LLMs to
quickly learn new concept tokens. Given a new word or concept and a set of
example sentences containing that word or concept, we want to produce an
embedding that captures its semantically meaningful features.

Framing this as a few-shot learning problem, we use a set of $K$ support
sequences $\\{\bm{s}_{1},...,\bm{s}_{K}\\}$, containing a new token, `<nonce>`
to produce a useful embedding for this new token. The new embedding can then
be used to augment the knowledge of a frozen autoregressive language model.
During training, we encourage the LM to use the new embedding to correctly
generate a query sequence $\bm{q}$.

#### Concept Embedding Generation:

To do this, the new token is replaced with a `<mask>` token in each support
sequence, and each is embedded with a frozen masked language model (MLM) used
for feature extraction. The contextual embeddings for each sequence are then
passed through an additional Transformer self-attention layer to process the
contextual embeddings for each sequence to obtain $\\{\bm{h}_{i,t}\\}$. These
are then aggregated using mean pooling, producing $k$ sequence embeddings
$\\{\bm{e}_{1},...,\bm{e}_{k}\\}$:

$\displaystyle\bm{e}_{i}=\frac{1}{n_{i}}\sum_{t=1}^{n_{i}}\bm{h}_{i,t},$ (1)

where $n_{i}$ is the length of each sequence. The sequence embeddings are
aggregated once more using mean pooling, producing a single output embedding
$\bm{e}_{\text{new}}$:

$\displaystyle\bm{e}_{\text{new}}=\frac{1}{K}\sum_{i=1}^{K}\bm{e}_{i}.$ (2)

Mean pooling can also facilitate incremental consolidation of new concepts
without having to store past examples. To integrate the embedding with a
frozen autoregressive LM, we apply two distinct linear layers to produce an
input and output embedding for the new token $\bm{e}_{\text{in}}$ and
$\bm{e}_{\text{out}}$:

$\displaystyle[\bm{e}_{\text{in}},\bm{e}_{\text{out}}]=\mathrm{Linear}(\bm{e}_{\text{new}}).$
(3)

The autoregressive LM’s input and output token embedding matrices are then
expanded with these generated embeddings, and used to model the query
sequence.

$\displaystyle W_{\text{emb\\_in, new}}$
$\displaystyle=[W_{\text{emb\\_in}},\bm{e}_{\text{in}}],$ (4) $\displaystyle
W_{\text{emb\\_out, new}}$
$\displaystyle=[W_{\text{emb\\_out}},\bm{e}_{\text{out}}].$ (5)

We visualize this process for a simple language modeling example in Figure 2.

#### Sampling Few-Shot Learning Episodes:

One novel aspect of our framework is that, unlike many meta-learning
approaches, our training procedure follows the same style as pretraining by
directly leveraging text data from the pretraining datasets. We hypothesize
that a good way to rapidly learn new concept is to actually “use” the concept
in another sentence—we let an LLM consume the newly learned embedding to
generate another sentence. Moreover, the autoregressive cross entropy loss is
the same as the pretraining objective, so, in theory, our meta-learning
procedure can be perfectly blended into the pretraining stage.

To efficiently sample support and query sequences, we save sentences
containing a new token in an example buffer to serve as _support_ sequences,
and when we encounter the same token being used again in the training corpus,
we will use the sequence as a _query_ sequence. The query sequence can then be
saved in the example buffer and used as a support sequence again. We find that
reusing query sequences as support sequences is helpful for training and
allows the model to make use of examples it has already learning when learning
new examples. Often, query sequences are longer, comprising a few sentences or
a paragraph of text. The sentence which contains the new token is extracted
from each and used as a support sequence for a different query sequence
concerning the same concept. Not every such sequence ends up in the final set,
however, since we filter and rank the examples, see Section 4 for details.

#### Negative Examples:

Initial experiments training on only positive examples, i.e. examples
containing the new token, tended to yield generated embeddings with abnormally
high norm compared to other LLM input or output token embeddings. One
hypothesis was that, since every example the model has to learn contains a new
token, it converges to an abnormally high norm embedding, to ensure that the
new token appears in the query sequence. During normal pretraining, few tokens
are shared across all sequences, and language models learn both when to
generate and when not to generate each token. To likewise teach our model when
not to generate a new token, we sample a sequence without a new token, which
we call a negative example, and take the sum of the cross entropy loss on the
positive example, $L_{\text{ce}}^{+}$, and the cross entropy loss on the
negative example, $L_{\text{ce}}^{-}$.

#### Knowledge Distillation:

Since the pretrained LLM has already seen all the words before, we know the
“true” language model embeddings and logits for the remainder of the sequence.
Ideally, we want the generated embeddings from our model to match the ground
truth embeddings and logits as faithfully as possible, to better approximate
the underlying language model distribution. To do this, we retain the original
sequence, before masking a word with a new token, and compute the output
embeddings and logits for the rest of the sequence with a non-augmented
version of our pretrained autoregressive model. We then compute the cosine
distance between those output embeddings and the output embeddings from
CoLLEGe, $L_{\text{cos}}$, as well as the MSE between the CoLLEGe LLM logits
and the “true” LLM logits using the original embeddings, $L_{\text{mse}}$.
Using a more standard objective with a distillation temperature (Hinton et
al., 2015) was less effective during training.

In order to compute these two distillation loss terms, we define the positive
example token sequence as $t_{1,+},....,t_{n,+}$ and original example token
sequence as $t_{1,\text{orig}},....,t_{l,\text{orig}}$, and additionally
construct a deterministic mapping $\sigma:\mathbb{N}\to\mathbb{N}$ that maps a
token at index $i$ in the positive example to its corresponding token at index
$k$ in the original sequence. The tokens following the new token in the
positive sequence are guaranteed to have a match in the original example
sequence, by definition, but the index may be different (if, for example, the
word replaced with `<nonce>` is subtokenized in the original example
sequence). We compute our distillation loss terms using:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{cos}}}$
$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}=\frac{1}{n-|I_{\text{new}}|}\sum_{k\not\in
I_{\text{new}}}1-\cos(\bm{e}_{t_{k,+}},\bm{e}_{t_{\sigma(k),\text{orig}}})},$
(6)
$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{mse}}}$
$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}=\frac{1}{n-|I_{\text{new}}|}\sum_{k\not\in
I_{\text{new}}}(\bm{\ell}_{t_{k,+}}-\bm{\ell}_{t_{\sigma(k),\text{orig}}})^{2}},$
(7)

where $I_{\text{new}}$ is the set of new token indices in the positive example
sequence, $E_{i,\cdot}$ denotes the output embedding at token position $i$ for
the positive or negative example sequence, and $\ell_{i,\cdot}$ denotes the
logit vector at token position $i$ for the positive or negative example
sequence.

#### Final Loss:

Our final training loss is simply a sum of these individual loss terms. We
explored using a linear combination of the loss terms to weight each
differently, but found it to be at best no more effective than an unweighted
sum and at times worse. Our final loss, $L_{\text{total}}$, is:

$\displaystyle L_{\text{total}}$
$\displaystyle=\underbrace{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{ce}}^{+}}+{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{ce}}^{-}}}_{\text{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}Cross
Entropy
Losses}}}+\underbrace{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{cos}}}+{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}L_{\text{mse}}}}_{\text{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}Distillation
Losses}}}.$ (8)

##  4 Datasets for Training CoLLEGe

In contrast to many other meta-learning methods, which use a specific set of
tasks during training, we adopt a training approach that mirrors general-
purpose pretraining. In a sense, we treat each query sequence, and each new
token in turn, as its own “task” to solve. Pretrained language model
representations are highly adaptable, and can be successfully applied to a
variety of tasks with simple prompting strategies. By adopting a task-general
training method, we train a module that can produce similarly adaptable
embeddings on the fly.

Because CoLLEGe is designed to learn a single new token per sequence, and the
LLM is frozen, training is highly sensitive to data quality, both for the
support and query sequences. Additionally, three forms of mismatch between
support and query sequences are important to guard against: language,
contextual meaning, and knowledge mismatch. The first case is mostly self-
explanatory, non-English support sequences for an English query sequence cause
difficulties in training. Contextual meaning mismatch was particularly
important to avoid when training with the Pile (Gao et al., 2020), whose
examples are drawn from a variety of sources. Creating support and query
sequences from WikiText, as in HiCE, often implicitly controls for contextual
meaning (all examples are from one sources (Wikipedia), and support and query
sequences for a word are often unintentionally drawn from the same article and
thus share contextual meaning). Likewise, knowledge mismatch is more prominent
when training with the Pile, since it contains more diverse sources. If one,
or many, support sequences are more confusing than the query sequence, this
can destabilize training.

Source | Num. Examples
---|---
Pile-CC | 79,606
Books3 | 51,850
BooksCorpus2 | 3,436
Table 1: The top Pile subsets represented in our dataset.

Using the Squeakily333https://github.com/CarperAI/squeakily library, we
filtered for English text at threshold 0.90 using the FastText (Joulin et al.,
2016) model for language identification, applied perplexity filtering at
threshold 1000 (filtering examples above 1000 perplexity) using KenLM,
following de la Rosa et al. (2022). We also filtered examples with too much
character repetition as well as examples with words flagged as obscene.
Afterwards, we cleaned examples by normalizing white space and punctuation.
Each query sequence is constructed from 4 sentence, non-overlapping, chunks of
the text examples from the Pile samples.

To build a set of support sequences for each query sequence, we first split
all examples into individual sentences, and matched each query sequence with
sentences that use the same new word. We removed sentences that appear in the
query sequence, examples with a large number of newline characters (these
often were article titles, tables of contents, or lists), and examples with
fewer than 15 words. In earlier experiments, we found it helpful to embed each
query sequence and candidate set of support sequences with DistilRoBERTa (Sanh
et al., 2019), extract the contextual embedding of the new word from both the
query sequence and each candidate, and score candidates by the cosine
similarity of the embeddings, in order to reduce contextual meaning mismatch.
Later analysis revealed that the majority of these low-quality or mixed-
meaning examples came from FreeLaw, USPTO, PubMed, or NIH ExPorter, which
often have specialized meaning and formatting. Excluding those from the
samples made the filtering process much more straightforward. Table 1
summarizes the top subsets from the Pile represented in our dataset.

| Without Definition | With Definition
---|---|---
Method | 1-Shot | 2-Shot | 3-Shot | 4-Shot | Def + 1-Shot | Def + 2-Shot | Def + 3-Shot | Def + 4-Shot
TT-1 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 10.9 $\pm$ 3.0 | 8.6 $\pm$ 3.1 | 8.1 $\pm$ 3.9 | 8.8 $\pm$ 4.3
TT-2 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 6.8 $\pm$ 0.0 | 12.3 $\pm$ 3.5 | 9.3 $\pm$ 2.5 | 8.1 $\pm$ 5.3 | 7.6 $\pm$ 2.5
HiCE | 11.5 $\pm$ 5.0 | 12.5 $\pm$ 2.7 | 13.1 $\pm$ 3.6 | 16.1 $\pm$ 4.2 | 19.3 $\pm$ 3.5 | 15.9 $\pm$ 8.2 | 11.4 $\pm$ 2.3 | 10.6 $\pm$ 1.3
Additive | 13.6 $\pm$ 2.3 | 9.1 $\pm$ 3.9 | 13.6 $\pm$ 3.9 | 11.4 $\pm$ 0.0 | 12.9 $\pm$ 1.3 | 10.6 $\pm$ 6.6 | 12.9 $\pm$ 3.5 | 7.6 $\pm$ 1.3
Prompting | 13.9 $\pm$ 4.3 | 17.7 $\pm$ 3.0 | 19.8 $\pm$ 4.2 | 21.8 $\pm$ 3.0 | 21.6 $\pm$ 5.7 | 20.5 $\pm$ 6.8 | 19.3 $\pm$ 4.8 | 24.0 $\pm$ 4.7
CoLLEGe w/o KD / Neg. Ex. | 32.2 $\pm$ 5.2 | 37.7 $\pm$ 4.9 | 38.6 $\pm$ 2.9 | 42.2 $\pm$ 3.1 | 40.0 $\pm$ 6.6 |  49.3 $\pm$ 5.8 |  49.6 $\pm$ 3.5 | 48.0 $\pm$ 5.0
CoLLEGe w/o KD | 25.9 $\pm$ 4.7 | 33.6 $\pm$ 5.0 | 35.0 $\pm$ 4.7 | 35.7 $\pm$ 3.2 | 40.5 $\pm$ 1.9 | 40.2 $\pm$ 3.8 | 43.2 $\pm$ 3.5 | 42.3 $\pm$ 3.2
CoLLEGe w/o Neg. Ex. | 28.9 $\pm$ 6.0 | 31.6 $\pm$ 5.1 | 34.1 $\pm$ 4.8 | 33.6 $\pm$ 4.3 | 37.5 $\pm$ 6.2 | 38.4 $\pm$ 4.8 | 42.5 $\pm$ 4.6 | 44.1 $\pm$ 4.6
CoLLEGe |  35.0 $\pm$ 5.6 |  40.5 $\pm$ 4.3 |  42.7 $\pm$ 3.9 |  44.5 $\pm$ 3.9 |  42.5 $\pm$ 3.9 | 46.8 $\pm$ 3.7 | 45.2 $\pm$ 3.2 |  49.3 $\pm$ 1.5

Table 2: Accuracy in percentage on the GRE Verbal Reasoning task, averaged
over 10 trials. In each trial, a new set of $K$ example sentences are sampled
and used for the $K$-shot task. For Token Tuning, we use LR = 1e-3, as that
gave the best performance.

##  5 Experiments

In this section, we show experimental results on four different evaluation
tasks that we designed: GRE verbal reasoning, definition generation, and slang
identification. Note that CoLLEGe is a task-general concept embedding
generation network, and all of the evaluations are performed _zero-shot_ ,
_without_ further training or fine-tuning, just like pretrained LLMs. In the
following, we first discuss implementation and training details, then describe
the baseline methods. Afterwards, we present the core results in Subsections
5.1-LABEL:sec:relation.

#### Implementation Details:

As our pretrained MLM model for the Sequence Encoder, we use RoBERTa-Large
(Liu et al., 2019), and apply a trainable Transformer Encoder layer to encode
the RoBERTa sequence embeddings. These embeddings are aggregated using mean-
pooling to produce a single embedding per sequence, which is further mean-
pooled into our Concept Embedding. We use a pretrained LLaMA-2 7B model
(Touvron et al., 2023) as the pretrained autoregressive language model in all
our experiments.

During training, we provide 1 support sequence for each query sequence and
generalize to $K>1$ during testing. We train our model for 28000 steps at
batch size 32, with a learning rate of 1e-3, a linear learning rate schedule
with warmup, and using the AdamW optimizer (Loshchilov & Hutter, 2019) with
weight decay = 0.1 and default beta values. We experimented with different
beta values during training, but found they had little effect. During training
we clip gradients to a norm of 1.0. We save checkpoints based on both LM loss
on the test set as well as cross entropy loss on new tokens in the test set.
The final model checkpoint is selected based on its GRE score.

Using the default initialization for our Encoder produces input and output
embeddings that are significantly larger in norm than those of the pretrained
LLaMA model. During training with the default initialization, a lot of
training time is spent reducing the norm. To address this inefficiency, we
apply a Layer Normalization (Ba et al., 2016) layer before the input and
output linear layers, and initialize those layers so that the expected norm is
as close to the average input or output token embedding norm as possible.

#### Baselines:

In order to evaluate the effectivness of our method, evaluate against
baselines from prior work on new concept learnign as well as prompting and
gradient descent tuning. More details on implementation for the baselines can
be found in Appendix C.

  * •

Token Tuning (TT) (Lampinen & McClelland, 2017) finetunes only the new token
embedding(s) using gradient descent. The support sequences are treated as the
training batch for each step. TT-$N$ denotes $N$ gradient descent steps.
Unlike Lampinen & McClelland (2017), $N$ is kept small here since we found
that a large $N$ results in degraded performance. A similar approach has been
proposed in Textual Inversion (Gal et al., 2023) for image few-shot learning
and generation.

  * •

HiCE (Hu et al., 2019) consists of a Transformer sequence encoder as well as a
Transformer layer to aggregate the sequence embeddings. It is trained to
output an embedding with minimal cosine distance to the true Word2Vec
embedding.

  * •

Additive (Lazaridou et al., 2017) is a simple baseline that consists of
summing the Word2Vec embeddings for all tokens in the context that are not the
new token.

  * •

Prompting uses randomly initialized new token embeddings and includes the
support sequences in the prompt. It is a strong baseline with direct context
access. Since prompting allows the LM to reason over the tokens in context, it
can be combined with our embedding generation approach (see Appendix D for
additional experiments). However, prompting original sentences can also make
the context window too long and distracting.

###  5.1 GRE Verbal Reasoning

The GRE verbal reasoning task is a challenging type of question appearing in
Graduate Record Examinations that not only tests the understanding of rare
vocabulary but also their logical placement in a sentence. We increase the
difficulty here by making each multiple choice answer an unknown vocabulary
word with a few example sentences as hints. We test whether the CoLLEGe
generated concept embeddings can directly support downstream verbal reasoning.

#### Dataset:

Using actual GRE practice questions from a Kaplan GRE prep book (Kaplan,
2019), we design a task where a language model has to either select the top or
top-2 choices for a sequence with blanks. Examples for each of these questions
are provided in Table 7 in Appendix A. Questions were hand-annotated from the
Kaplan book and details about the cleaning process can be found in Appendix B.
We produce a high-quality selection of 43 GRE Verbal Reasoning problems with a
single blank (i.e. not multi-part). On an version of this task without new
tokens, a pretrained LLaMa-2 7B scores 75%, which serves as an upper bound on
our potential performance.

To evaluate, we create a version of the sequence for each possible choice, and
calculate the log probability of each such sequence. The highest log
probability sequence is selected as the chosen answer. The final scores
reflect the average accuracy over 10 trials of sampling different example
sentences from GPT-4.

#### Results:

Results are reported in Table 2, with additional results for “Prompting+…”
reported in Table 8 in Appendix D.1. Token tuning does not seem to help much,
and even sometimes hurts performance. Prompting by including the definition of
each term alongside the example sentences improves performance for the
baselines the most, but CoLLEGe significantly outperforms by between 4% and
19%. Model performance also increases with more examples, showing effective
utilization of multiple example sentences. By contrast, more examples can
sometimes harm Prompting, and Prompting+CoLLEGe, performance, likely due to
the additional in-context examples distracting the LLaMA model. We note that,
although our model outperforms the baselines with and without support
sentences provided in-context, it performs better without them. We hypothesize
that not including instruction-tuning data during the pretraining process for
the new embedding reduces performance when prompted.

Example Sentence |  CoLLEGe Definition |  True Definition |  Word/
Phrase
---|---|---|---
The eerie creak of the attic door, coupled with the flickering candlelight, was enough to give anyone the `<nonce>`. |  a feeling of unease, usually in the stomach, caused by anxiety or fear. |  feelings of uneasiness |  willies
Intrigued by holistic therapies, she found herself lying on a soft mat as the therapist applied `<nonce>` to various points on her body to alleviate her chronic migraines. |  a substance that is used to heal or soothe a part of the body. |  treatment of symptoms by applying pressure with the fingers to specific pressure points on the body |  acupressure
Nestled in the far corner of the bustling newsroom, the diligent `<nonce>` worked tirelessly, transcribing reporter’s notes into clean, easy-to-read articles. |  a person who writes or edits for a newspaper, magazine, or other publication. |  someone employed to make written copies of documents and manuscripts |  copyist
The delicate `<nonce>` sprouted from the forest floor, adding a touch of alien beauty to the woodland scene. |  a plant that resembles a mushroom. |  a fungus composed of several apothecia that look like elongated rabbit ears; |  Wynnea americana
Table 3: Definitions generated with CoLLEGe, using the prompt “The word
|<nonce>| is defined as”. Each definition is generated using the single
example sentence shown. None of the example sentences are provided in-context
to the model.

###  5.2 Definition Generation

To probe how well our model understands a new word, we prompt the LLM to
generate a definition for the word given a few example sentences, as shown in
Figure 1.

#### Dataset:

To construct the dataset for evaluation, we selected 954 words from WordNet
(Miller, 1994). We then prompt GPT-4 (OpenAI, 2023) to generate an example
sentence for the word using the prompt: Give me a unique, descriptive sentence
using the word “[WORD]” without defining it or making it obvious what the word
means. Without the latter half of the prompt, many generated examples
rephrased the definition. Examples are generated at temperature = 0.8. Since
both our model and the baselines continue to generate text, we select the
first sentence generated as the definition for scoring.

#### Results:

Generation examples are reported in Table 3. The generated embeddings often
capture high- and low-level semantic details of the example sentences.
Sometimes this is fairly precise, for example the generated definition for
willies is exactly correct and similarly with copyist. CoLLEGe is also able to
identify Wynnea americana as a mushroom. Even when the definition is not quite
right, it may capture general features of the concept correctly, and may
reflect the limited information contained in the example sentence. The
definition for acupressure, for example, is not exactly correct but a very
good inference based on the example provided. Additional generated
definitions, including those generated using more than one example sentence,
are shown in Table 11 in Appendix E.1. We also show side-by-side comparisons
with the baselines (without prompting) in Table 12 in Appendix E.2. Some
failure cases are described in Appendix E.3.

In order to evaluate the quality of generated definitions, we compare a
definition generated from our model to one generated from a baseline model as
well as to a ground truth definition. For comparison between models, we
simulate a head-to-head competition and compute the ELO score (Elo, 1978) of
each model. Specifically, for each example in the task dataset, we choose a
$k$-shot setting and sample a baseline at random. We then compare the
definition generated by CoLLEGe with the one generated by the baseline. Based
on the result—win, lose, or tie—we update the ELO score, starting with an
initial score of 1000 for all models. To choose a winner in each “round”, we
use the Outlines package444https://github.com/outlines-dev/outlines, we ask
GPT-3.5 to select which definition is best for the word in question or if they
are tied. The order of the choices (both generated definitions and “tie”) are
randomized. We compute ELO separately for Table 9. We also compare the
generated definitions with ground truth definitions for each word. Using the
ground truth definition as the reference and the generated definition as the
candidate, we compute the BERTScore F1 and report average values for our model
as well as each baseline. In both quantitative evaluations, CoLLEGe
outperforms the baselines. Qualitatively, only Prompting produces generated
definitions that are somewhat competitive. Definitions generated by the other
baselines are often incoherent, generating repetitive text or unrelated words
and characters.

Model | BERTScore F1 | ELO |  |
---|---|---|---|---
TT-1 |  0.752 $\pm$ 0.081 | 980.78 |  $\pm$ | 18.48
TT-2 |  0.752 $\pm$ 0.081 | 978.49 |  $\pm$ | 18.42
HiCE |  0.767 $\pm$ 0.022 | 975.64 |  $\pm$ | 7.86
Additive |  0.801 $\pm$ 0.023 | 967.22 |  $\pm$ | 8.50
Prompting |  0.825 $\pm$ 0.028 | 1032.28 |  $\pm$ | 28.27
CoLLEGe |  0.848 $\pm$ 0.023 | 1065.57 |  $\pm$ | 24.01
Table 4: Results for the definition generation task. We compare the model generated definitions with a reference definition generated by GPT-4 using BERTScore. Additionally, we simulate random challenges between CoLLEGe and each baseline and compute and ELO rating. For Token Tuning, we report results for LR = 3e-4, as that yielded the best results. Model | 1-Shot | 2-Shot | 3-Shot | 4-Shot
---|---|---|---|---
TT-1 |  32.2 $\pm$ 1.4 |  32.1 $\pm$ 2.6 |  32.6 $\pm$ 1.3 |  34.5 $\pm$ 0.4
TT-2 |  32.3 $\pm$ 1.6 |  32.8 $\pm$ 3.9 |  33.1 $\pm$ 0.2 |  32.5 $\pm$ 3.0
Additive |  27.0 $\pm$ 1.0 |  28.3 $\pm$ 1.9 |  28.0 $\pm$ 0.7 |  29.0 $\pm$ 1.0
HiCE |  34.0 $\pm$ 1.1 |  32.7 $\pm$ 1.9 |  31.3 $\pm$ 2.3 |  32.8 $\pm$ 1.0
Prompting |  41.0 $\pm$ 1.0 |  47.0 $\pm$ 2.1 |  51.7 $\pm$ 1.8 |  53.8 $\pm$ 1.4
CoLLEGe |  49.3 $\pm$ 1.0 |  53.2 $\pm$ 1.6 |  54.8 $\pm$ 2.6 |  60.0 $\pm$ 0.7
Table 5: Accuracy in percentage on the Twitter Slang task.

Slang Term |  Definition |  Example Tweet
---|---|---
rizz |  The ability to confidently approach people and talk to them, in a more romantic or flirty way. |  Imagine having so little rizz that even the AI girlfriend rejects you. Just complete negative game.
hits different |  When something is significantly better than usual or is way better under certain circumstances. |  getting called pretty in person just hits different. people be making my day.
gorpcore |  A fashion style that is similar to that of
hiking/wilderness/utility tech wear. |  An anecdote from my coverage of Gorpcore as a trend: This vintage seller put 4 Gore-Tex hats up for sale on his website at $135….

Table 6: Examples from the Twitter Slang task, showing the slang term, its
definition, and an example tweet.

###  5.3 Twitter Slang

To emulate new word learning in a more natural setting, we construct a task
based on identifying the correct definition for a slang term, using Tweets as
example sentences.

#### Dataset:

We hand-curate a set of 80 recent slang terms as well as their definitions.
Alongside each term is a list of up to 8 high-quality example Tweets which use
the term in an informative way. For this hand-curated set, example tweets are
predominantly from 2022 and 2023. Definitions are taken from UrbanDictionary,
Dictionary.com’s Pop Culture and Slang sections, the recent American Dialect
Society meeting, Bark, and Wiktionary. To supplement these, we then sample 120
additional slang terms from UrbanDictionary and the Online Slang Dictionary.
We select example tweets from the Twitter archive, using the downloading and
processing pipeline from Hu et al. (2022). Some examples from the hand-crafted
set are shown in Table 6. More information about the filtering process for
this dataset can be found in Appendix B and links to the websites and curation
details can be found Appendix F.

To evaluate the different models on this task, we select the true slang term,
its example tweets, and its true definition. We then select 3 different
incorrect slang terms and their examples. We score the log probability of the
true definition conditioned on each set of examples. The highest probability
is selected as the “choice”. If it corresponds to the correct combination
definition for the slang term, that is counted as a correct choice, otherwise
not. We score the model based on its accuracy across the whole set of slang
terms.

#### Results:

Results are presented in Table 5. Without providing example tweets in-context,
CoLLEGe outperforms each baseline. In fact, CoLLEGe is able to outperform
prompting directly, showing that in novel contexts (such as Twitter slang),
in-context examples may be more confusing than a concise embedding
representation. Notably, when including example tweets in-context, the
baselines hurt performance compared to a simple prompting baseline. Only using
the CoLLEGe generated embeddings in this setting improve performance over
prompting with randomly initialized new token embeddings. We also compare
CoLLEGe to our baselines in the prompting setting in Table 10.

##  6 Conclusion and Discussion

In this paper we present CoLLEGe, a few-shot learning framework for new
concept acquisition and knowledge augmentation for pretrained LLMs. We model
our meta-learning approach on the original pretraining task by sampling few-
shot learning episodes directly from language model pretraining datasets (e.g.
the Pile) and use next-word prediction, the pretraining objective, as our
primary meta-learning objective.

We find that training CoLLEGe with a single support sequence for each query
sequence generalizes well to multiple support sequences at test time, and that
the generated embeddings contain rich and task-general semantic information.
To thoroughly evaluate the quality of these embeddings, we propose three
challenging tasks including verbal reasoning, definition generation, and slang
identification. We find that the generated embeddings transfer directly to
these tasks zero-shot, without additional finetuning or training, and are
particularly useful for more complex reasoning tasks.

While CoLLEGe achieves the best performance in all of the benchmarks, we
summarize a few limitations in our current framework. First, the generated
embeddings sometimes miss precise details in the examples, instead encoding
higher level semantic information. This can be seen in some incorrect
generated definitions, where general features of the unknown concept are
correctly inferred even though specific details are missed. Second, we find
the averaging mechanism cannot fully achieve parity with pretrained
embeddings, even with more support sequences. For example, our best score on
the few-shot GRE Verbal Reasoning task is still lower than the 75% accuracy
LLaMA achieves on the task without new tokens. Additionally, Token Tuning may
be more useful when we initialize the new token embedding with the embedding
generated by CoLLEGe, but we do not investigate that in this paper.

Our work points to a number of future research directions. In the short term,
work needs to be done to investigate different data mixes for training CoLLEGe
and understand the effect of data sources on the generated embeddings (e.g.
inclusion/exclusion of instruction-tuning data, non-English text, code data,
etc.). More broadly, CoLLEGe only scratches the surface of exciting research
in concept learning. This research is a first step in an exciting direction
for future research: online continual concept acquisition performed jointly
with pretraining—incrementally identifying and compressing new concepts from
an online stream of sequential experience. Furthermore, this approach can be
expanded towards grouping of composite novel concepts for more flexible and
hierarchical organization of knowledge.

## Acknowledgment

We would like to thank the Microsoft Accelerating Foundation Models Research
program for providing cloud compute credits for running some parts of our LLM
experiments. The compute was also supported by the NYU High Performance
Computing resources, services, and staff expertise.

## References

  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bansal et al. (2020) Bansal, T., Jha, R., and McCallum, A.  Learning to few-shot learn across diverse natural language classification tasks.  In Scott, D., Bel, N., and Zong, C. (eds.), _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 5108–5123, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.448.  URL https://aclanthology.org/2020.coling-main.448.
  * Bulatov et al. (2022) Bulatov, A., Kuratov, Y., and Burtsev, M.  Recurrent memory transformer.  _Advances in Neural Information Processing Systems_ , 35:11079–11091, 2022.
  * Chen et al. (2022) Chen, Y., Zhong, R., Zha, S., Karypis, G., and He, H.  Meta-learning via language model in-context tuning.  In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 719–730, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.53.  URL https://aclanthology.org/2022.acl-long.53.
  * Chevalier et al. (2023) Chevalier, A., Wettig, A., Ajith, A., and Chen, D.  Adapting language models to compress contexts.  In Bouamor, H., Pino, J., and Bali, K. (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 3829–3846, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.232.  URL https://aclanthology.org/2023.emnlp-main.232.
  * Conneau et al. (2017) Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A.  Supervised learning of universal sentence representations from natural language inference data.  In Palmer, M., Hwa, R., and Riedel, S. (eds.), _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 670–680, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1070.  URL https://aclanthology.org/D17-1070.
  * de la Rosa et al. (2022) de la Rosa, J., uardo González Ponferrada, E., Villegas, P., de Prado Salas, P. G., Romero, M., and Grandury, M.  Bertin: Efficient pre-training of a spanish language model using perplexity sampling.  _Proces. del Leng. Natural_ , 68:13–23, 2022.  URL https://api.semanticscholar.org/CorpusID:250526558.
  * Elo (1978) Elo, A. E.  _The Rating of Chessplayers, Past and Present_.  Arco Pub., New York, 1978.  ISBN 0668047216 9780668047210.  URL http://www.amazon.com/Rating-Chess-Players-Past-Present/dp/0668047216.
  * Fifty et al. (2023) Fifty, C., Duan, D., Junkins, R. G., Amid, E., Leskovec, J., Ré, C., and Thrun, S.  Context-aware meta-learning.  _arXiv preprint arXiv:2310.10971_ , 2023.
  * Gal et al. (2023) Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=NAQvF08TcyG.
  * Gao et al. (2020) Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Ge et al. (2024) Ge, T., Jing, H., Wang, L., Wang, X., Chen, S.-Q., and Wei, F.  In-context autoencoder for context compression in a large language model.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=uREj4ZuGJE.
  * Geng et al. (2019) Geng, R., Li, B., Li, Y., Zhu, X., Jian, P., and Sun, J.  Induction networks for few-shot text classification.  In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3904–3913, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1403.  URL https://aclanthology.org/D19-1403.
  * Herbelot & Baroni (2017) Herbelot, A. and Baroni, M.  High-risk learning: acquiring new word vectors from tiny data.  In Palmer, M., Hwa, R., and Riedel, S. (eds.), _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 304–309, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1030.  URL https://aclanthology.org/D17-1030.
  * Hinton et al. (2015) Hinton, G., Vinyals, O., and Dean, J.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Holla et al. (2020) Holla, N., Mishra, P., Yannakoudakis, H., and Shutova, E.  Learning to learn to disambiguate: Meta-learning for few-shot word sense disambiguation.  In Cohn, T., He, Y., and Liu, Y. (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 4517–4533, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.405.  URL https://aclanthology.org/2020.findings-emnlp.405.
  * Hu et al. (2022) Hu, H., Sener, O., Sha, F., and Koltun, V.  Drinking from a firehose: Continual learning with web-scale natural language.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5684–5696, 2022.
  * Hu et al. (2023) Hu, N. Z., Mitchell, E., Manning, C. D., and Finn, C.  Meta-learning online adaptation of language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.  URL https://openreview.net/forum?id=jPrl18r4RA.
  * Hu et al. (2019) Hu, Z., Chen, T., Chang, K.-W., and Sun, Y.  Few-shot representation learning for out-of-vocabulary words.  In Korhonen, A., Traum, D., and Màrquez, L. (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4102–4112, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1402.  URL https://aclanthology.org/P19-1402.
  * Joulin et al. (2016) Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T.  Fasttext.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ , 2016.
  * Kaplan (2019) Kaplan.  _GRE Prep 2019_.  Kaplan Publishing, New York, 2019.
  * Khodak et al. (2018) Khodak, M., Saunshi, N., Liang, Y., Ma, T., Stewart, B., and Arora, S.  A la carte embedding: Cheap but effective induction of semantic feature vectors.  In Gurevych, I. and Miyao, Y. (eds.), _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 12–22, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1002.  URL https://aclanthology.org/P18-1002.
  * Kiros et al. (2015) Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S.  Skip-thought vectors.  _Advances in neural information processing systems_ , 28, 2015.
  * Lake & Baroni (2023) Lake, B. and Baroni, M.  Human-like systematic generalization through a meta-learning neural network.  _Nature_ , 623:115–121, 2023.
  * Lampinen & McClelland (2017) Lampinen, A. K. and McClelland, J. L.  One-shot and few-shot learning of word embeddings.  _arXiv preprint arXiv:1710.10280_ , 2017.
  * Lazaridou et al. (2017) Lazaridou, A., Marelli, M., and Baroni, M.  Multimodal word meaning induction from minimal exposure to natural text.  _Cognitive science_ , 41 Suppl 4:677–705, 2017.  URL https://api.semanticscholar.org/CorpusID:205032138.
  * Lee et al. (2022) Lee, H.-y., Li, S.-W., and Vu, N. T.  Meta learning for natural language processing: A survey.  _arXiv preprint arXiv:2205.01500_ , 2022.
  * Li et al. (2020) Li, R., Wang, X., and Yu, H.  Metamt, a meta learning method leveraging multiple domain data for low resource machine translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 8245–8252, 2020.
  * Lin et al. (2021) Lin, S.-T., Sabharwal, A., and Khot, T.  ReadOnce transformers: Reusable representations of text for transformers.  In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 7129–7141, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.554.  URL https://aclanthology.org/2021.acl-long.554.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Loshchilov & Hutter (2019) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Luong et al. (2013) Luong, T., Socher, R., and Manning, C.  Better word representations with recursive neural networks for morphology.  In Hockenmaier, J. and Riedel, S. (eds.), _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_ , pp. 104–113, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.  URL https://aclanthology.org/W13-3512.
  * Luong et al. (2015) Luong, T., Sutskever, I., Le, Q., Vinyals, O., and Zaremba, W.  Addressing the rare word problem in neural machine translation.  In Zong, C. and Strube, M. (eds.), _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 11–19, Beijing, China, July 2015. Association for Computational Linguistics.  doi: 10.3115/v1/P15-1002.  URL https://aclanthology.org/P15-1002.
  * Lux & Vu (2021) Lux, F. and Vu, N. T.  Meta-learning for improving rare word recognition in end-to-end asr.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 5974–5978. IEEE, 2021.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G. S., and Dean, J.  Efficient estimation of word representations in vector space.  In _International Conference on Learning Representations_ , 2013.  URL https://api.semanticscholar.org/CorpusID:5959482.
  * Miller (1994) Miller, G. A.  WordNet: A lexical database for English.  In _Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994_ , 1994.  URL https://aclanthology.org/H94-1111.
  * Mu et al. (2024) Mu, J., Li, X., and Goodman, N.  Learning to compress prompts with gist tokens.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  2023\.  URL https://api.semanticscholar.org/CorpusID:257532815.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C.  GloVe: Global vectors for word representation.  In Moschitti, A., Pang, B., and Daelemans, W. (eds.), _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi: 10.3115/v1/D14-1162.  URL https://aclanthology.org/D14-1162.
  * Qian & Yu (2019) Qian, K. and Yu, Z.  Domain adaptive dialog generation via meta learning.  In Korhonen, A., Traum, D., and Màrquez, L. (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 2639–2649, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1253.  URL https://aclanthology.org/P19-1253.
  * Qin & Van Durme (2023) Qin, G. and Van Durme, B.  Nugget: Neural agglomerative embeddings of text.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 28337–28350. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/qin23a.html.
  * Ren et al. (2020) Ren, M., Iuzzolino, M. L., Mozer, M. C., and Zemel, R. S.  Wandering within a world: Online contextualized few-shot learning.  _ArXiv_ , abs/2007.04546, 2020.  URL https://api.semanticscholar.org/CorpusID:220424770.
  * Sanh et al. (2019) Sanh, V., Debut, L., Chaumond, J., and Wolf, T.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108, 2019.
  * Schick & Schütze (2019) Schick, T. and Schütze, H.  Learning semantic representations for novel words: Leveraging both form and context.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):6965–6973, Jul. 2019.  doi: 10.1609/aaai.v33i01.33016965.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4675.
  * Schmidhuber (1992) Schmidhuber, J.  Learning to control fast-weight memories: An alternative to dynamic recurrent networks.  _Neural Computation_ , 4:131–139, 1992.  URL https://api.semanticscholar.org/CorpusID:16683347.
  * Snell et al. (2017) Snell, J., Swersky, K., and Zemel, R.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Sun et al. (2018) Sun, J., Wang, S., and Zong, C.  Memory, show the way: Memory based few shot word representation learning.  In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 1435–1444, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1173.  URL https://aclanthology.org/D18-1173.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vinyals et al. (2016) Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Wang et al. (2020) Wang, S., Fang, Y., Sun, S., Gan, Z., Cheng, Y., Liu, J., and Jiang, J.  Cross-thought for sentence encoder pre-training.  In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 412–421, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.30.  URL https://aclanthology.org/2020.emnlp-main.30.
  * Weston et al. (2015) Weston, J., Chopra, S., and Bordes, A.  Memory networks.  In Bengio, Y. and LeCun, Y. (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1410.3916.
  * Xu et al. (2024) Xu, F., Shi, W., and Choi, E.  RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=mlJLVigNHp.
  * Zhong et al. (2021) Zhong, R., Lee, K., Zhang, Z., and Klein, D.  Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections.  In _Conference on Empirical Methods in Natural Language Processing_ , 2021.  URL https://api.semanticscholar.org/CorpusID:237304362.

##  Appendix A GRE Verbal Reasoning Examples

The GRE task consists of two different types of fill-in-the-blank questions.
The first type asks you to select the best possible choice to complete the
provided sentence, so it serves as a test of the top-1 prediction. The second
type asks for which two words best complete the sentence. Often these words
are similar, but distinct. It tests how the top-2 predictions of the LLM.

Question |  Answer Choices |  Correct
Answer(s) |  Evaluation Type
---|---|---|---
Mary’s former classmates were taken aback by her [BLANK] behavior at the reunion for, during her school years, she was frequently reprimanded for creating disturbances with her exuberant outbursts and playful antics. |  a) gregarious b) discourteous c) obsequious d) reticent e) scurrilous |  d) reticent |  Choose the word for each blank that best fits the meaning of the sentence as a whole.
The firefighter, desperate to save the children on the second floor of the fiery house, rushed into their bedroom; his colleagues, more wary of the [BLANK] structure, remained outside. |  a) stalwart b) precarious c) stout d) irrefragable e) tottering f) fecund |  b) precarious e) tottering |  Select the two answer choices that, when inserted into the sentence, fit the meaning of the sentence as a whole and yield complete sentences that are similar in meaning.

Table 7: Examples of both types of questions for the GRE task dataset

##  Appendix B Data Processing

We provide further details on cleaning and processing for some of our task
datasets.

#### GRE:

The cleaning process for the GRE dataset involved normalizing the different
formats for blanks (i.e. empty spaces, underlines, “(a)”, etc.), removing
artifacts from the conversion to text from PDF, and associating each question
with its answer from the answer key.

#### Twitter Slang:

We filter examples from this archive for flagged obscene words using
Squeakily, but it is important to note that online slang is often obscene.
This is especially true for the sources used to define the slang terms
(UrbanDictionary in particular).

##  Appendix C Baseline Implementations

#### Hice:

To train HiCE, we follow the method outlined in the paper, and use the
WikiText-103 dataset to train the model with a morphology network. We use
hyperparameters from the authors’ implementation.

#### Word2Vec Projection:

For the Additive baseline as well as HiCE, the baseline outputs a Word2Vec
embedding. To make this compatable with LLaMa, we train a linear layer on
shared tokens between LLaMa and Word2Vec to map between the embedding spaces.

#### Token Tuning:

For Token Tuning, I treat the example sentences as a ”batch” and perform N=1,2
steps of gradient descent on only the input and output embeddings for the new
token.

##  Appendix D Results for Prompting+

Since our Prompting baseline provides all the support sentences in the context
window, allowing the LM to attend to the new token embeddings directly, we
also show using CoLLEGe-generated embeddings improves performance over random
initialization and our other baselines (denoted “Prompting+…”).

###  D.1 GRE Results for Prompting+

We show results for the GRE task when prompting with examples in-context,
using embeddings for the new token generated by the model or baseline
following the “+”.

| Without Definition | With Definition
---|---|---
Method | 1-Shot | 2-Shot | 3-Shot | 4-Shot | Def + 1-Shot | Def + 2-Shot | Def + 3-Shot | Def + 4-Shot
Prompting | 13.9 $\pm$ 4.3 | 17.7 $\pm$ 3.0 | 19.8 $\pm$ 4.2 | 21.8 $\pm$ 3.0 | 21.6 $\pm$ 5.7 | 20.5 $\pm$ 6.8 | 19.3 $\pm$ 4.8 | 24.0 $\pm$ 4.7
\+ CoLLEGe w/o KD / Neg. Ex. | 25.7 $\pm$ 3.9 | 31.1 $\pm$ 3.9 | 32.7 $\pm$ 5.7 | 32.1 $\pm$ 6.1 | 35.4 $\pm$ 4.3 | 31.8 $\pm$ 3.8 | 33.4 $\pm$ 4.8 |  33.4 $\pm$ 5.8
\+ CoLLEGe w/o KD | 26.1 $\pm$ 4.5 | 29.6 $\pm$ 3.4 | 31.8 $\pm$ 4.7 | 29.1 $\pm$ 4.0 | 33.9 $\pm$ 4.7 | 28.9 $\pm$ 4.1 |  33.9 $\pm$ 4.7 | 35.7 $\pm$ 2.3
\+ CoLLEGe w/o Neg. Ex. |  35.0 $\pm$ 3.7 |  33.2 $\pm$ 2.3 | 31.4 $\pm$ 3.6 | 25.9 $\pm$ 10.3 | 31.1 $\pm$ 7.1 | 31.1 $\pm$ 4.6 | 28.6 $\pm$ 3.7 | 28.4 $\pm$ 4.9
\+ CoLLEGe | 34.3 $\pm$ 4.3 | 33.0 $\pm$ 5.6 |  34.8 $\pm$ 3.4 |  33.6 $\pm$ 3.6 |  37.5 $\pm$ 5.6 |  34.1 $\pm$ 3.8 | 30.5 $\pm$ 4.6 |  33.4 $\pm$ 4.6

Table 8: Accuracy in percentage on the GRE Verbal Reasoning task when
prompting with examples in-context, using new token embeddings generated by
each model or baseline. Results for prompting with randomly initialized
embeddings are reproduced here for clarity.

All CoLLEGe models outperform the Prompting baseline, where new token
embeddings are randomly initialized. Each “Prompting+CoLLEGe” model performs
worse than the unprompted version, which we hypothesize is due to a lack of
instruction tuning data in the training dataset.

###  D.2 Definition Generation Results for Prompting+

We present results for our definition generation task with examples presented
in-context, using our baselines or CoLLEGe to generate the new token
embedding. ELO scores are calculated separately from those in Table 4 by
sampling a random baseline challenger to the “Prompting+CoLLEGe” model.

Model | BERTScore F1 | ELO |  |
---|---|---|---|---
Prompting |  0.825 $\pm$ 0.028 | 1002.09 |  $\pm$ | 22.87
\+ TT-1 |  0.762 $\pm$ 0.035 | 959.88 |  $\pm$ | 17.22
\+ TT-2 |  0.763 $\pm$ 0.045 | 962.10 |  $\pm$ | 12.65
\+ HiCE |  0.740 $\pm$ 0.098 | 949.32 |  $\pm$ | 10.77
\+ Additive |  0.735 $\pm$ 0.098 | 950.61 |  $\pm$ | 3.48
\+ CoLLEGe |  0.858 $\pm$ 0.029 | 1176.00 |  $\pm$ | 12.69
Table 9: Results for the definition generation task, when prompting with
examples in-context. We compare the model generated definitions with a
reference definition generated by GPT-4 using BERTScore and simulate random
challenges between CoLLEGe and each baseline and compute and ELO rating. Token
Tuning results are for LR = 3e-4, as in the main paper.

Generated definitions improve with in-context examples, and we note that our
model far outperforms the baselines. The only competitive baseline is
prompting with randomly initialized embeddings.

###  D.3 Twitter Results for Prompting+

When examples Tweets are provided in-context, our CoLLEGe model’s accuracy on
the slang identification task increases. For other baselines, aside from
prompting with randomly initialized embeddings, however, performance either
degrades or remains about the same. With the Word2Vec-based baselines, this
may be due to the difficulty of mapping between Word2Vec embeddings and the
LLaMA input and output embedding space.

Model | 1-Shot | 2-Shot | 3-Shot | 4-Shot
---|---|---|---|---
Prompting |  41.0 $\pm$ 1.0 |  47.0 $\pm$ 2.1 |  51.7 $\pm$ 1.8 |  53.8 $\pm$ 1.4
\+ TT-1 |  30.5 $\pm$ 3.5 |  28.2 $\pm$ 2.9 |  26.0 $\pm$ 1.4 |  24.1 $\pm$ 1.2
\+ TT-2 |  29.3 $\pm$ 3.0 |  28.3 $\pm$ 2.7 |  25.8 $\pm$ 3.1 |  24.0 $\pm$ 1.6
\+ Additive |  30.7 $\pm$ 3.8 |  25.2 $\pm$ 0.4 |  24.5 $\pm$ 1.2 |  24.1 $\pm$ 1.7
\+ HiCE |  25.0 $\pm$ 4.9 |  26.0 $\pm$ 1.1 |  27.8 $\pm$ 3.0 |  25.5 $\pm$ 0.8
\+ CoLLEGe |  56.5 $\pm$ 2.0 |  60.5 $\pm$ 2.1 |  67.4 $\pm$ 1.0 |  69.8 $\pm$ 0.8
Table 10: Accuracy in percentage on the Twitter Slang task, where example
Tweets are provided in-context.

##  Appendix E Generated Definitions

###  E.1 Additional CoLLEGe Definitions

We show additional definitions generated from CoLLEGe, including definitions
generated with more than one example. CoLLEGe is able to generate plausible
definitions that capture important features of the new word. In some cases,
the CoLLEGe definition is incorrect, but plausible based on the examples
provided. For example, the generated definition for opera company is
incorrect, but reasonable in light of the example sentences.

Example Sentence |  CoLLEGe Definition |  True Definition |  Word/
Phrase
---|---|---|---
During the complex abdominal surgery, the surgeon carefully moved the `<nonce>` aside to gain better access to the patient’s damaged organs. |  a surgical procedure in which a portion of the intestine is brought through an opening in the abdominal wall. |  a fold of peritoneum supporting the viscera |  omentum
The yellow blooms of the `<nonce>` added a vibrant contrast to the green canvas of the wetlands. |  ”a plant of the genus Ficus, having a milky sap and large, often edible, fruit.” |  aromatic evergreen or deciduous dioecious shrubs or trees of eastern Asia and North America |  Lindera
The prestigious `<nonce>`, clad in elaborate costumes, filled the auditorium
with their mesmerizing harmonies and dramatic performances.
After months of rigorous rehearsals, the `<nonce>` finally brought their magnum opus to life, filling the ornate theater with powerful harmonies that resonated with every member of the riveted audience. |  a person who is skilled in the art of dancing. |  a company that produces operas |  opera
company
Despite countless imitations flooding the market, only her grandmother’s
secret recipe for apple pie was the `<nonce>`.
After tasting many alternatives, he finally found the `<nonce>` of artisanal
cheeses in a quaint little shop in Paris.
Despite all the replica paintings she had seen, it was breathtaking to stand
before the `<nonce>` in the museum.
|  the most beautiful or perfect specimen of its kind. |  informal usage attributing authenticity |  real McCoy
Table 11: Additional definitions generated with CoLLEGe, using the prompt “The
word |<nonce>| is defined as”. Each definition is generated using the
examples. None of the example sentences are provided in-context to the model.

###  E.2 Qualitative Comparison of Generated Definitions

We present a side-by-side comparison of definitions generated by CoLLEGe as
well as the baselines in Table 12. Definitions generated by the baselines
(with the exception of Prompting) are essentially unusable.

Word/
Phrase |  True
Definition |  CoLLEGe |  HiCE |  Additive |  TT-1 |  TT-2
---|---|---|---|---|---|---
horsecar |  an early form of streetcar that was drawn by horses |  a “motorized vehicle with a cabin and a platform for passengers to stand on.” |  a word” “the a word a” is a |  that the place where you are the place. |  follows. |  follows.
popishly |  like the Pope; in a popish manner |  in a manner that is intended to attract attention or admiration. |  a ” a “a” a a word a word a a a |  that the same. |  ’the opposite of the word ” ” in the dictionary |  the opposite of the word,
Table 12: Side-by-side comparison between CoLLEGe-generated definitions and
definitions generated by the baselines without prompting.

###  E.3 Failures of Generated Definitions

#### Word2Vec Baselines:

Both Word2Vec baselines tended to produce embeddings that were unusable for
generating definitions.

#### CoLLEGe Failures:

When analyzing generated definitions, it is clear that there are some
“default” definitions the model will generate when the embedding for the new
token is not informative enough.

Some of these common “default” definitions, listed in order of frequency, are:

  * •

a person who is not a member of a particular group or class

  * •

a place of refuge or shelter

  * •

a person who is a source of annoyance or irritation

  * •

a noun

##  Appendix F Slang Sources

To build the dataset of slang terms and definitions, we used
UrbanDictionary555https://www.urbandictionary.com/ Dictionary.com’s Pop
Culture666https://www.dictionary.com/e/pop-culture/ and
Slang777https://www.dictionary.com/e/slang/ sections, the recent American
Dialect Society meeting888https://americandialect.org/nominations-for-words-
of-the-year-2023/, Bark 999bark.us,
Wiktionary101010https://www.wiktionary.org/, and the Online Slang
Dictionary111111http://onlineslangdictionary.com/

.