# Implementing Continuation based language in GCC

Shinji KONO, Kento YOGI
<EMAIL_ADDRESS>
Information Engineering, University of the Ryukyus
Nishihara-cyo 1, Okinawa, 903-01, Japan

(2008 Apr 13)

We have implemented C like Continuation based programming language.
Continuation based C, CbC was implemented using micro-C on various
architecture, and we have tried several CbC programming experiments. Here we
report new implementation of CbC compiler based on GCC 4.2.3. Since it
contains full C capability, we can use CbC and C in a mixture.

##  1  A Practical Continuation based Language

If CPS theory is successful, it should also be working well in practical area.
Our idea is simple. How about a programming language which has continuation
passing style only? How about it runs as fast as current GNU C compiler?

Instead of creating complete new programming language, we designed a lower
language of C, so called Continuation based C, here after CbC. Using CPS
transformation like method, we can compile C into CbC, that is, we have some
kind of backward compatibility.

We have implemented CbC using micro-C on various architecture, and we have
tried several CbC programming experiments. Here we report new partial
implementation of CbC compiler[5] based on GCC 4.2.3[1]. Since it contains
full C capability, we can use CbC and C in a mixture, so when call the mixture
C with C, here after CwC.

First we show CbC language overview.

##  2  Continuation based C

CbC’s basic programming unit is a code segment. It is not a subroutine, but it
looks like a function, because it has input and output. We can use C struct as
input and output interfaces.

       struct interface1  { int i; };
       struct interface2  { int o; };

       __code f(struct interface1 a) {
           struct interface2 b; b.o=a.i;
           goto g(b);
       }

In this example, a code segment `f` has `input a` and sends `output b` to a
code segment `g`. There is no return from code segment `b`, `b` should call
another continuation using `goto`. Any control structure in C is allowed in
CwC language, but in case of CbC, we restrict ourselves to use `if` statement
only, because it is sufficient to implement C to CbC translation. In this
case, code segment has one input interface and several output interfaces
(fig.1).

Figure 1: code

`__code` and parameterized global goto statement is an extension of
Continuation based C. Unlike `C--` [4]’s parameterized goto, we cannot goto
into normal C function.

###  2.1  Intermix with C

In CwC, we can go to a code segment from a C function and we can call C
functions in a code segment. So we don’t have to shift completely from C to
CbC. The later one is straight forward, but the former one needs further
extensions.

       void *env;
       __code (*exit)(int);

       __code h(char *s) {
            printf(s);
            goto (*exit)(0),env;
       }

       int main() {
            env  = __environment;
            exit = __return;
            goto h("hello World\n");
       }

In this hello world example, the environment of `main()` and its continuation
is kept in global variables. The environment and the continuation can be get
using `__environment`, and `__return`. Arbitrary mixture of code segments and
functions are allowed (in CwC). The continuation of `goto` statement never
returns to original function, but it goes to caller of original function. In
this case, it returns result 0 to the operating system.

##  3  What’s good?

CbC is a kind of high level assembler language. It can do several original C
language cannot do. For examples,

        Thread Scheduler
        Context Switch
        Synchronization Primitives
        I/O wait semantics

are impossible to write in C. Usually it requires some help of assembler
language such as `__asm` statement extension which is of course not portable.

###  3.1  Scheduler example

We can easily write these things in CbC, because CbC has no hidden information
behind the stack frame of C. A thread simply go to the scheduler,

        goto scheduler(self, task_list);

and the scheduler simply pass the control to the next thread in the task
queue.

        code scheduler(Thread self,TaskPtr list)
        {
            TaskPtr t = list;
            TaskPtr e;
            list = list->next;
            goto list->thread->next(list->thread,list);
        }

Of course it is a simulator, but it is an implementation also. If we have a
CPU resource API, we can write real multi CPU scheduler in CbC.

This is impossible in C, because we cannot access the hidden stack which is
necessary to switch in the scheduler. In CbC, everything is visible, so we can
switch threads very easily.

This means we can use CbC as an executable specification language of OS API.

###  3.2  Self Verification

Since we can write a scheduler in CbC, we can also enumerate all possible
interleaving of a concurrent program. We have implement a model checker in
CwC. CbC can be a self verifiable language[7].

SPIN[3] is a very reliable model checker, but it have to use special
specification language PROMELA. We cannot directly use PROMELA as an
implementation language, and it is slightly difficult to study its concurrent
execution semantics including communication ports.

There are another kind of model checker for real programming language, such as
Java PathFinder[2]. Java PathFinder use Java Virtual Machine (JVM) for state
space enumeration which is very expensive some time.

In CbC, state enumerator itself is written in CbC, and its concurrency
semantics is written in CbC itself. Besides it is very close to the
implementation. Actually we can use CbC as an implementation language. Since
enumerator is written in the application itself, we can perform abstraction or
approximation in the application specific way, which is a little difficult in
Java PathFinder. It is possible to handle JVM API for the purpose, although.

We can use CPS transformed CbC source code for verification, but we don’t have
to transform all of the source code, because CwC supports all C constructs.
(But not in C++… Theoretically it is possible with using cfront converter, it
should be difficult).

###  3.3  As a target language

Now we have GCC implementation of CbC, it runs very fast. Many popular
languages are implemented on top of C. Some of them uses very large switch
statement for the byte code interpreter. We don’t have to use these hacks,
when we use CbC as an implementation language.

CbC is naturally similar to the state charts. It means it is very close to UML
diagrams. Although CbC does not have Object Oriented feature such as message
passing nor inheritance, which is not crucial in UML.

##  4  Transformation (C2CbC)

Conversion from C to CbC is straight forward, but it generates a lot of code
segments. Since CbC does not have heap management itself, the stack area have
to be allocated explicitly.

We find GCC can perform better optimization in translated code segment. We
will discuss it later.

We have an easy implementation of C to CbC compilation, but it is not a
practical level, but we need good converter for backward compatibility.

We can also consider possible conversion from C++ to CbC. In this case, all
hidden operation in C++ should become explicit, for examples, object
allocations and deallocations in the stack, handling of auto pointer and so
on.

##  5  GNU CC implementation

So how to implement CwC in GCC. The idea itself is simple[6], forcing C tail
call elimination for all code segment.

But before GCC version 4.x, tail call elimination (here after TCE) is not so
cleanly implemented , it is very difficult to implement it. But in GCC 4.x,
basically TCE can be applied for all possible functions.

`__code` is implemented as a new type keyword in GCC. You may think `__code`
is an attribute of a function, which means that the function can call in tail
call elimination only.

Because of this implementation, we can actually call code segment as a normal
function call.

###  5.1  How to force tail call elimination

There many enable conditions for tail call elimination, for example, there
should be no statement after tail call, return value type have to be the same,
arguments size should be compatible, and so on. We find almost half of lines
in `calls.c` spends to check TCE possibilities.

Our conclusion is this. It is not practical to make sure to pass all the TCE
tests, instead, we write TCE only version of `expand_call()` separately in 783
lines.

        4463   18527  145469 calls.c
            expand_call() for function
         783    2935   23651 cbc-goto.h
            expand_cbc_goto() for code segment

All code segment has the same virtual argument size and void return type, that
is argument register or argument value in the memory is shared among all code
segments. This leads a problem.

###  5.2  Parallel Assignment

Consider the next code,

        __code carg4(struct arg args0,struct arg args1,
             int i, int j,int k,int l)
        {
            goto carg5(args1,args0,j,k,l,i);
        }

In this case, simple sequential assignments does not work. It override `args1`
or `args0`. In normal function case, GCC simply give up TCE, and pushes all
arguments in new register or stack area. We are not allowed that. That is we
have to implement parallel assignment in the code segment goto.

This is done by simple copy overlapped arguments in a stack. We hope to
eliminate unnecessary copy during GCC optimization.

###  5.3  Not yet done

Currently we have not yet implemented goto with environment and `__return`,
`__environment`.

In some GCC 4.x supported architecture, TCE itself is not supported in special
case. Our method does not work for the architecture.

Since we made modifications on GCC compiler itself, our method is GCC version
sensitive. We have to do necessary modifications for coming new version of
GCC.

##  6  Result

Here is our bench mark program.

        f0(int i) {
            int k,j;
            k = 3+i;
            j = g0(i+3);
            return k+4+j;
        }

        g0(int i) {
            return h0(i+4)+i;
        }

        h0(int i) {
            return i+4;
        }

It is written in C, we perform CPS transformation in several steps by hands.
There are several optimization is possible.

        /* straight conversion case (1) */

        typedef char *stack;

        struct cont_interface {
          // General Return Continuation
            __code (*ret)();
        };

        __code f(int i,stack sp) {
            int k,j;
            k = 3+i;
            goto f_g0(i,k,sp);
        }

        struct f_g0_interface {
           // Specialized Return Continuation
            __code (*ret)();
            int i_,k_,j_;
        };

        __code f_g1(int j,stack sp);

        __code f_g0(int i,int k,stack sp) { // Caller
            struct f_g0_interface *c =
                (struct f_g0_interface *)(
             sp -= sizeof(struct f_g0_interface));

            c->ret = f_g1;
            c->k_ = k;
            c->i_ = i;

            goto g(i+3,sp);
        }

        __code f_g1(int j,stack sp) {  // Continuation
            struct f_g0_interface *c =
              (struct f_g0_interface *)sp;
            int k = c->k_;
            sp+=sizeof(struct f_g0_interface);
            c = (struct f_g0_interface *)sp;
            goto (c->ret)(k+4+j,sp);
        }

        __code g_h1(int j,stack sp);

        __code g(int i,stack sp) { // Caller
            struct f_g0_interface *c =
                (struct f_g0_interface *)(
               sp -= sizeof(struct f_g0_interface));

            c->ret = g_h1;
            c->i_ = i;

            goto h(i+3,sp);
        }

        __code g_h1(int j,stack sp) {
            // Continuation
            struct f_g0_interface *c =
              (struct f_g0_interface *)sp;
            int i = c->i_;
            sp+=sizeof(struct f_g0_interface);
            c = (struct f_g0_interface *)sp;
            goto (c->ret)(j+i,sp);
        }

        __code h(int i,stack sp) {
            struct f_g0_interface *c =
              (struct f_g0_interface *)sp;
            goto (c->ret)(i+4,sp);
        }

        struct main_continuation {
            // General Return Continuation
            __code (*ret)();
            __code (*main_ret)();
            void *env;
        };

        __code main_return(int i,stack sp) {
            if (loop-->0)
                goto f(233,sp);
            printf("#0103:%d\n",i);
            goto (( (struct main_continuation *)sp)->main_ret)(0),
                   ((struct main_continuation *)sp)->env;
        }

This is awfully long, but it is straight forward. Several forward prototyping
is necessary, and we find strict prototyping is painful in CbC, because we
have to use many code segments to perform simple thing. CbC is not a language
for human, but for automatic generation, verification or IDE directed
programming.

We can shorten the result in this way.

        /* little optimized case (3) */

        __code f2_1(int i,char *sp) {
            int k,j;
            k = 3+i;
            goto g2_1(k,i+3,sp);
        }

        __code g2_1(int k,int i,char *sp) {
            goto h2_11(k,i+4,sp);
        }

        __code f2_0_1(int k,int j,char *sp);
        __code h2_1_1(int i,int k,int j,char *sp) {
            goto f2_0_1(k,i+j,sp);
        }

        __code h2_11(int i,int k,char *sp) {
            goto h2_1_1(i,k,i+4,sp);
        }

        __code f2_0_1(int k,int j,char *sp) {
            goto (( (struct cont_interface *)
               sp)->ret)(k+4+j,sp);
        }

        __code main_return2_1(int i,stack sp) {
            if (loop-->0)
                goto f2_1(233,sp);
            printf("#0165:%d\n",i);
            goto (( (struct main_continuation *)sp)->main_ret)(0),
                   ((struct main_continuation *)sp)->env;
        }

In this example, CPS transformed source is faster than original function call
form. There are not so much area for the optimization in function call form,
because function call API have to be strict. CbC does not need standard call
API other than interface which is simply a struct and there are no need for
register save. (This bench mark is designed to require the register save).

Here is the result in IA32 architecture (Table.1). Micro-C is our previous
implementation in tiny C. `conv1 1` is function call. `conv1 2`, `conv1 3` is
optimized CPS transformed source.

| ./conv1 1 | ./conv1 2 | ./conv1 3
---|---|---|---
Micro-C | 8.97 | 2.19 | 2.73
GCC | 4.87 | 3.08 | 3.65
GCC (+omit) | 4.20 | 2.25 | 2.76
GCC (+fast) | 3.44 | 1.76 | 2.34
Table 1: Micro-C, GCC bench mark (in sec)

There are two optimization flag for GCC. `-fomit-frame-pointer` eliminates
frame pointer (%ebp). The frame pointer itself is useful in code segment, but
it generates unnecessary push and pop or `leave` instruction. Using `fastcall`
option, GCC ignore the standard call convention such as all argument have be
on stack in IA32. In Micro-C implementation, these optimization is naturally
implemented in code segment, so it is faster than GCC without these options.

But with these options, GCC is faster than Micro-C. Of course, in more complex
source, GCC’s complex optimization should work well.

##  7  Conclusion

We have designed and implemented Continuation based language for practical
use. We have partial implementation of CwC using GCC 4.2.3. Using suitable
optimized options CPS transformed source sometimes runs faster than original
function call version.

This gcc implementation should be portable on all architectures supporting
tail call elimination, but we have tested only on i386 now.

## References

  * [1] Free Software Foundation, Inc.  GCC, the GNU Compiler Collection, March 2008.
  * [2] K. Havelund and T. Pressburger.  Model checking java programs using java pathfinder, 1998.
  * [3] Gerard J. Holzmann.  The model checker SPIN.  Software Engineering, Vol. 23, No. 5, pp. 279–295, 1997.
  * [4] Norman Ramsey and Simon Peyton Jones.  A single intermediate language that supports multiple implementations of exceptions.  In ACM SIGPLAN 2000 Conference on Programming Language Design and Implementation, June 2000.
  * [5] Shinji KONO.  CbC, March 2008.
  * [6] 河野 真治.  継続を基本とした言語CbCのgcc上の実装.  日本ソフトウェア科学会第19回大会論文集, Sep 2002.
  * [7] 河野 真治 .  検証を自身で表現できるハードウェア、ソフトウェア記述言語 Continuation based C と¿、そのCell への応用.  電子通信学会VLD研究会, March 2008.


# Curating Grounded Synthetic Data with Global Perspectives for Equitable AI

Elin Törnquist
Emergent Methods, CO
<EMAIL_ADDRESS>
Robert Alexander Caulk
Emergent Methods, CO
<EMAIL_ADDRESS>
###### Abstract

The development of robust AI models relies heavily on the quality and variety
of training data available. In fields where data scarcity is prevalent,
synthetic data generation offers a vital solution. In this paper, we introduce
a novel approach to creating synthetic datasets, grounded in real-world
diversity and enriched through strategic diversification. We synthesize data
using a comprehensive collection of news articles spanning 12 languages and
originating from 125 countries, to ensure a breadth of linguistic and cultural
representations. Through enforced topic diversification, translation, and
summarization, the resulting dataset accurately mirrors real-world
complexities and addresses the issue of underrepresentation in traditional
datasets. This methodology, applied initially to Named Entity Recognition
(NER), serves as a model for numerous AI disciplines where data
diversification is critical for generalizability. Preliminary results
demonstrate substantial improvements in performance on traditional NER
benchmarks, by up to 7.3%, highlighting the effectiveness of our synthetic
data in mimicking the rich, varied nuances of global data sources. This paper
outlines the strategies employed for synthesizing diverse datasets and
provides such a curated dataset for NER.

##  1 Introduction

In the rapidly evolving field of artificial intelligence, the adequacy of
training data significantly influences the accuracy and fairness of AI models
(Mehrabi et al., 2021). Traditionally, datasets have skewed towards
predominately available contexts and languages, leading to models that perform
inconsistently across diverse global scenarios Osoba and Welser IV (2017).
Recognizing the limitations posed by non-representative datasets, there is an
urgent need for a methodological shift towards the creation of more inclusive
and diverse datasets. Synthetic data generation emerges as a powerful strategy
to address this need, by artificially constructing data that reflects a wider
array of linguistic and cultural backgrounds. By grounding synthetic datasets
in real-world diversity and enriching them through strategic diversification,
we can significantly enhance the performance and generalizability of AI models
across different languages and cultures.

News texts offer an exceptional foundation for grounding synthetic data due to
their inherent diversity and comprehensive coverage of a multitude of topics
and cultures. As dynamic reflections of global events, news articles
encapsulate a wide array of subjects, from politics and economy to science and
culture, each embedded with nuanced linguistic and contextual specifics
characteristic of different geographical regions. This variety makes news
texts the ideal base for synthesizing datasets that aim to mirror real-world
complexities and societal variations. The recent maturation of Large Language
Models (LLMs), such as OpenAI’s GPT models (Brown et al., 2020; Achiam et al.,
2023), Meta’s Llama models (Touvron et al., 2023a, b), and Anthropic’s Claude
models, makes it possible to generate high-quality synthetic text data (Ziems
et al., 2024) using news articles to incorporate rich, multifaceted inputs
that enhance the representativeness and authenticity of the resulting dataset.

Named Entity Recognition (NER), a fundamental task in natural language
processing (NLP) that involves identifying and classifying key elements in
text into predefined categories, is a prime example of a field where dataset
diversity is a key necessity for off-the-shelf performance. NER systems are
integral for a variety of applications, including information retrieval,
recommendation systems, and automated content analysis, making their
reliability critically important. Traditional NER models, such as spaCy
(Honnibal et al., 2020) and Flair (Akbik et al., 2019), are efficient but
limited to a predefined set of entity types. LLMs make it possible to identify
any entity type using simple natural language instructions. However, this
comes at a significant increase in computational cost. The release of GLiNER
(Zaratiana et al., 2023) bridges the gap between computationally expensive yet
entity type permissive LLMs and traditional, restrictive NER frameworks. As
opposed to other LLM-based NER models, such as InstructUIE (Wang et al.,
2023), GoLLIE (Sainz et al., 2023), and UniversalNER (Zhou et al., 2023), that
use autoregressive language models (LMs), GLiNER uses computationally
efficient Bidirectional Language Models (BiLM), such as BERT (Devlin et al.,
2018) and deBERTa He et al. (2021). This allows GLiNER to predict multiple
entity types in parallel, with low computational cost, and permits
bidirectional context processing, resulting in richer representations.

In this paper, we use GLiNER to showcase performance enhancements resulting
from our proposed methodology for grounding and diversifying synthetic data.
Our approach leverages a voluminous collection of news articles across 125
countries (Figure 1A) and in 12 languages, ensuring a rich tapestry of global
perspectives and narratives. This diverse base is further augmented through
processes like topic diversification, translation, and summarization, thereby
creating a synthetic dataset that not only supports robust model training but
also significantly reduces the bias inherent in more homogenous datasets. We
present a detailed description of the methodology and show that the resulting
dataset improves performance on common NER benchmark datasets by up to 7.3%.
Further, we provide the open-sourced dataset, comprising 5,049 samples of
summarized news articles covering 73 unique topics and 54 unique entity types,
as well as fine-tuned models available under Apache 2.0 license. Initially
applied to the domain of NER, our methodology demonstrates a scalable strategy
for enhancing data representativeness in numerous AI disciplines.

Figure 1: Country distribution. Distribution of country of origin for the
articles in the full dataset (A) and in each of the splits (B-D). A and B
share the color bar in the top row; C and D share the color bar in the bottom
row. Detailed counts are available in Supplementary Tables 1-3

##  2 Related work

Veselovsky et al. (2023) investigated three strategies for improving
faithfulness in synthetically generated data: grounding, filtering, and
taxonomy-based generation. Applied to the task of identifying sarcasm, they
concluded that grounding the prompt with a real example text resulted in the
highest improvement. Josifoski et al. (2023) created synthetic data for closed
information extraction (cIE) using graph triplets and showed that cIE models
trained on this curated and grounded dataset outperformed models trained on
manually annotated data. Eldan and Li (2023) enforced diversity in LLM-
generated text by incorporating a grounding vocabulary and showed how training
small language models (SMLs) on the resulting dataset improved creativity in
the SLMs’ output.

##  3 Method

This section details the steps taken to diversify the news data underpinning
the grounded synthetic data generation, the data generation itself, as well as
the selection of data for the NER dataset. The full procedure is shown in
Figure 2.

Figure 2: Dataset engineering. Procedure used for enforcing diversity and
generating synthetic data for AskNews-NER-v0.

###  3.1 Synthetic data diversification

We parsed close to 1 million open-web, non-paywalled, news articles per day
from news outlets across 212 countries. Every five minutes, approximately
3,500 articles are passed through the following pipeline:

  1. 1.

Keep all articles corresponding to the following 12 languages: English,
Spanish, Portuguese, German, Russian, French, Arabic, Italian, Ukrainian,
Norwegian, Swedish, and Danish (this list is determined by the LLM used in
Step 5 for translation and summarization).

  2. 2.

Identify the distribution of country of origin for the news outlets in the
selected articles (e.g., Figure 1A)

  3. 3.

Sort the articles by Majestic Million page rank, which preferences outlets
with more backlinks.

  4. 4.

Sample a subset of articles, enforcing the country of origin distribution from
Step 2.

  5. 5.

Translate, summarize, and classify each article using WizardLM 13B V1.2 (Xu et
al., 2023).

  6. 6.

Embed the titles and summaries using the thenlper/gte-base embedding model (Li
et al., 2023).

  7. 7.

Store the embeddings in a vector database (we are using Qdrant (Qdrant,
2024)).

The vector database is accessible via an API endpoint as detailed here:
AskNews API documentation.

###  3.2 The AskNews-NER-v0 dataset

#### Topic diversity

Topic diversification was enforced by grouping the articles based on semantic
similarity through clustering of their embeddings, then sampling from the
clusters while enforcing country distributions. The topic diversification
pipeline follows:

  1. 1.

Select articles indexed between Tuesday, February 20, 2024 6:10 PM and Sunday,
March 31, 2024 2:10 PM UTC. 600k articles.

  2. 2.

Sub-sample 29 evenly spaced 4 hour buckets of embedded articles

  3. 3.

Cluster each bucket of embeddings after L2-normalizarion using scikit-learn’s
(Pedregosa et al., 2011) HDBSCAN algorithm using the Euclidean distance
metric, a minimum cluster size of 30, and a minimum sample size of 20 (595
total clusters were identified across all buckets, each cluster corresponding
to semantically similar articles).

  4. 4.

Filter embeddings for cluster probabilities > 98% (the resulting clusters
contained between 30 and 112 articles, totalling 22,636 articles).

  5. 5.

Identify the distribution of countries. 22,636 articles.

  6. 6.

Enforce country distribution by ensuring to sample low-representation
countries and not over-sampling any high-representation country (Figure 3).
5,049 articles.

Figure 3: Country representation. Country representation in the AskNews-NER-v0
dataset after enforcing country distribution, as per step 6 of the topic
diversification pipeline.

The procedure outlined above yielded 5,049 articles covering 73 unique topics.
The top ten most frequent topics were (in descending order) Politics, Sports,
News, Finance, Crime, Entertainment, Business, Technology, Weather, and
Science (see Supplementary Figure 2 for details). For an easier overview, the
73 topics were grouped using GPT4 (Figure 4).

Figure 4: Topics distribution. Topics summarized by GPT4. The full set of
topics are available in Supplementary Figure 2.

#### Entity labeling

We used meta-llama/Meta-Llama-3-70B-Instruct (AI@Meta, 2024) for labeling
entities in the news articles. The instructions provided to the LLM are shown
in Figure 5. We restricted the requested entity types to 54 choices as shown
in Figure 6.

Figure 5: Entity extraction prompt. We gave the following instructions to
Llama-3-70B-Instruct regarding identification and labeling of entities. We
also provided two example input texts and expected outputs, as shown in
Supplementary Figure 1.

#### Prompt

A chat between a User and an artificial intelligence Assistant that is an
expert at identifying and extracting named entities from text. The Assistant’s
task is to analyze a given text and extract all entities and identify their
entity types.
Instructions for the Assistant:
1\. **read the text** The Assistant reads the text provided by the User.
2\. **identify entities** As the Assistant reads the text, the Assistant
identifies entities in the text in the order that they exist in the text.
3\. **determine entity types** For each Found Entity, the Assistant determines
an entity type that best describes the Found Entity from the following list of
entity types: {entity_types}
4\. **verbatim extraction** The Assistant extracts the Found Entity exactly as
found in the text. The Assistant does not invent entities that are not present
in the text.
The output should be a JSON dictionary according to the following schema:
{{’properties’: {{’entities’: {{’items’: {{’items’: {{’type’: ’string’}},
’maxItems’: 2, ’minItems’: 2, ’type’: ’array’}}, ’title’: ’Entities’, ’type’:
’array’}}}}, ’required’: [’entities’], ’title’: ’ENTITY_MODEL’, ’type’:
’object’}}
Example: Text Input: "In 2021, Apple released the iPhone 13 at 75% battery
power." Expected Output: {{ "entities" : [ [ "2021" , "Date" ] , [ "Apple" ,
"Organization" ] , [ "iPhone 13" , "Product" ] , [ "75%" , "Percentage" ] ] }}
Given the following text, please extract all entities and their entity types:
<doc> {input} </doc>

Figure 6: Requested entity types. We specified 54 entity types that we
requested Llama-3-70B-Instruct to identify and label in the news articles. The
order was shuffled for each article to reduce the risk of influencing the
labeling.

#### Entity types

Person, Politician, Actor, Athlete, Artist, Writer, Character, Musician,
Scientist, Director, Title, Organization, Political Party, Election, Facility,
Location, Country, City, Nationality, Language, Year, Date, Time, Event,
Award, Song, Movie, Book, Media, Band, Album, TV Show, Game, Software, Brand,
Money, Price, Law, Quantity, Number, Percentage, Sports, Politics, Weapon,
Product, Food, Material, Transportation, Vehicle, Religion, Technology, Space,
Medicine, Science.

Despite the instructions to limit the entity types, the labeled articles
sported 182 unique entity types (Supplementary Figure 3). Entity types that
were not requested were subsequently removed from the AskNews-NER-v0 dataset,
resulting in the distribution seen in Figure 7.

Figure 7: Entity type frequency for requested labels.

#### Dataset splits

The dataset was split temporally such that data from the four latest time
windows (between Thursday, March 28, 2024 2:10 PM and Wednesday, March 27,
2024 8:10 AM) was selected as test data, amounting to 730 samples. The
remaining data was stratified based on time window and cluster. 730 samples,
to be used as the validation set, were selected such that each time window and
cluster were sampled at least once. The remaining data, amounting to 3,589
samples, was assigned as training data. Country distributions for the splits
are shown in Figure 1B-D. Detailed counts are available in Supplementary
Tables 1-3.

###  3.3 GLiNER fine-tuning

#### Training hyperparameters

We fine-tuned GLiNER-Small-v2.1 (50M parameters), GLiNER-Medium-v2.1 (90M
parameters), and GLiNER-Large-v2.1 (0.3B parameters) on the AskNews-NER-v0
dataset. All three models were trained on the NuNER dataset (Bogdanov et al.,
2024) and are available under the Apache 2.0 license.

Training was done using the AdamW optimizer. The base learning rate for both
pretrained layers and non-pretrained layers was set to 1e-6 and adjusted using
ReduceLROnPlateau with factor 0.5 and patience 5. The models were trained for
25 epochs, with a batch size of 5, for 1,000 steps. Evaluation using the
validation data was done with a batch size of 12 and a span probability
threshold of 0.5 every epoch, using the top 30 most common entity types
present in the news dataset (see Supplementary Table W for details). Both
shuffling and random drop of entity types were enabled for regularization
purposes (Sainz et al., 2023). The best model was chosen based on the highest
micro-F1 score on the evaluation data.

#### Compute

Training was done using a NVIDIA RTX A4500 with 20GB RAM and took $\sim$1h
15min, $\sim$1h 48min, and $\sim$4h 3min for the small, medium, and large
GLiNER models, respectively.

###  3.4 Evaluation

#### Datasets

We evaluated our fine-tuned models in a zero-shot context on common NER
benchmarks, as were presented in (Zaratiana et al., 2023). These benchmark
datasets include the Out-of-Domain (OOD) NER Benchmark (Table 2), which
contains seven NER datasets from the CrossNER (Liu et al., 2020) and MIT (Liu
et al., 2013a, b) benchmark collections, as well as 18 NER datasets (Table 1)
that are commonly used to train NER models.

#### Models

We compared the zero-shot performance on the aforementioned datasets between
the base GLiNER models, i.e., Small-v1, Medium-v1, Large-v1 (trained on the
Pile-NER dataset (Zhou et al., 2023) and available under CC BY NC 4.0
license), and Small-v2.1, Medium-v2.1, and Large-v2.1 (trained on the NuNER
dataset (Bogdanov et al., 2024) and available under Apache 2.0 license).

#### Metric

Evaluation was done using micro-F1 score, computed based on the exact match
between predicted and ground-truth entity types.

#### Compute

Evaluation was performed using a NVIDIA GeForce RTX 3090 with 32GB of RAM and
took $\sim$6min, $\sim$8min, and $\sim$15min for the small, medium, and large
GLiNER models, respectively.

##  4 Results

The quality of our AskNews-NER-v0 dataset was assessed through micro-F1 scores
on 18 common NER benchmark datasets after fine-tuning of the commercially
permissible GLiNER models, i.e., GLiNER-Small-v2.1, GLiNER-Medium-v2.1, and
GLiNER-Large-v2.1.

Tables 1 and 2 show that our fine-tuned models improve the performance of the
base models by 4.6%, on average, and outperforms other LLM-based models. The
fine-tuned small model showed increased scores of 2.5 points (5.5%) on the 18
NER datasets and 0.9 points (1.7%) on the OOD benchmarks. The scores for the
fine-tuned medium model also showed improvements with an additional 3.3 points
(7.2%) on the 18 NER datasets and 2.1 points (3.6%) on the OOD benchmarks.
Similarly, the fine-tuning of the large model improved the score by and 3.3
points (7.3%) on the 18 NER datasets and 1 point (1.7%) on the OOD benchmarks.

Note that the performance of the GLiNER models detailed in Zaratiana et al.
(2023) does not align perfectly with the scores presented here in Tables 1 and
2 as the GLiNER models were refactored before publication to Huggingface (see
discussion in GitHub Issue here:  Reproduction questions #54). As our models
were fine-tuned on the Huggingface versions, we evaluate those models rather
than present the previous scores. A comparison between the scores published in
Zaratiana et al. (2023) and in this paper can be found in Supplementary Tables
6 and 7.

Table 1: Zero-shot performance on 18 NER datasets. Performance comparison
between NuNERZero-span, GLiNER base models ("v2.1"), and fine-tuned models
("News") on common NER benchmarks as well as on the test split of the AskNews-
NER-v0 dataset. Bold indicates the highest score per dataset. Green and red
values indicate increase or decrease in score for the fine-tuned model
compared to the base model. Comparison also to GLiNER-v1 models are found in
Supplementary Table 4. Commit hashes for all models are found in Supplementary
Table 5.

Dataset | NuNerZero-span | S-v2.1 | S-News | M-v2.1 | M-News | L-v2.1 | L-News
---|---|---|---|---|---|---|---
ACE 2005 | 23.6 | 32.8 | 23.2 -9.6 | 31.3 | 23.3 -8.0 | 33.3 | 28.9 -4.4
AnatEM | 29.2 | 35.1 | 36.5 +1.4 | 29.2 |  36.8 +7.6 | 24.7 | 34.8 +10.1
Broad Tweet Corpus | 60.2 | 63.0 |  69.2 +6.2 | 63.9 | 67.7 +3.8 | 59.9 |  69.2 +9.3
CoNLL 2003 | 63.6 | 59.8 | 60.9 +1.1 | 61.5 | 63.0 +1.5 | 59.5 | 56.4 -3.1
FabNER | 24.0 | 17.9 | 18.9 +1.0 | 17.0 | 20.6 +3.6 | 19.2 | 22.5 +3.3
FindVehicle | 43.7 | 38.0 | 43.1 +5.1 | 36.0 | 38.7 +2.7 | 51.9 |  52.9 +1.0
GENIA_NER | 55.0 | 47.7 | 47.2 -0.5 | 55.4 | 54.0 -1.4 | 55.3 | 55.0 -0.3
HarveyNER | 24.9 | 18.3 | 23.1 +4.8 | 23.2 |  26.7 +3.5 | 18.7 | 15.8 -2.9
MultiNERD | 63.9 | 57.3 |  68.3 +11.0 | 55.3 | 67.1 +11.8 | 48.7 | 64.0 +15.3
Ontonotes | 31.6 | 28.1 |  35.1 +7.0 | 25.7 | 33.2 +7.5 | 19.5 | 32.0 +12.5
PolyglotNER | 42.8 | 40.4 | 43.9 +3.5 | 42.1 |  45.3 +3.2 | 39.6 | 42.5 +2.9
TweetNER7 | 40.1 | 36.5 | 35.6 -0.9 | 38.2 | 38.6 +0.4 | 36.1 | 35.5 -0.6
WikiANN en | 58.1 | 55.3 | 53.6 -1.7 | 55.5 | 53.0 -2.5 | 55.5 | 52.5 -3.0
WikiNeural | 72.3 | 64.7 | 76.0 +11.3 | 67.5 |  77.9 +10.4 | 62.9 | 70.3 +7.4
bc2gm | 52.7 | 50.4 | 49.4 -1.0 | 54.0 |  54.6 +0.6 | 45.2 | 46.9 +1.7
bc4chemd | 50.8 | 49.0 | 49.4 +0.4 | 47.3 | 51.1 +3.8 | 53.1 |  55.1 +2.0
bc5cdr | 69.7 | 66.1 | 70.9 +4.8 | 67.0 |  72.5 +5.5 | 68.4 | 71.2 +2.8
ncbi | 61.4 | 56.6 | 57.9 +1.3 | 60.3 | 64.5 +4.2 | 59.7 |  65.5 +5.8
Average | 48.2 | 45.4 | 47.9 +2.5 | 46.1 |  49.4 +3.3 | 45.1 | 48.4 +3.3
AskNews-NER-v0 | 61.1 | 54.4 | 71.8 +17.4 | 53.3 | 71.9 +18.6 | 48.6 |  72.9 +24.3

Table 2: Zero-Shot Scores on Out-of-Domain NER Benchmarks. Performance for
LLM-based models and comparison between base models ("v2.1") and fine-tuned
models ("News"). Bold indicates the highest score per dataset. Green and red
values indicate increase or decrease in score for the fine-tuned model
compared to the base model. Results for ChatGPT and UniNER are from (Zhou et
al., 2023); InstructUIE is from (Wang et al., 2023); GoLLIE is from (Sainz et
al., 2023). Huggingface  commit hashes for the NuNER model and GLiNER models
are found in Supplementary Table 5.

Model | AI | Literature | Music | Politics | Science | Movie | Restaurant | Average
---|---|---|---|---|---|---|---|---
ChatGPT | 52.4 | 39.8 | 66.6 | 68.5 | 67.0 | 5.3 | 32.8 | 47.5
InstructUIE | 49.0 | 47.2 | 53.2 | 48.1 | 49.2 | 63.0 | 21.0 | 47.2
UniNER-7B | 53.6 | 59.3 | 67.0 | 60.9 | 61.1 | 42.4 | 31.7 | 53.7
UniNER-13B | 54.2 | 60.9 | 64.5 | 61.4 | 63.5 | 48.7 | 36.2 | 55.6
GoLLIE | 59.1 | 62.7 | 67.8 | 57.2 | 55.5 | 63.0 | 43.4 | 58.0
NuNerZero-span | 60.1 | 64.9 | 69.9 | 72.7 | 66.4 | 55.0 | 41.1 | 61.5
S-v1 | 50.7 | 60.0 | 60.9 | 61.5 | 55.6 | 46.9 | 33.3 | 52.7
S-v2.1 | 52.6 | 64.9 | 66.7 | 64.3 | 64.3 | 47.2 | 20.8 | 54.4
S-News | 57.0 +4.4 | 63.5 -1.4 | 64.7 -2.0 | 65.6 +1.3 | 63.9 -0.4 | 46.9 -0.3 | 25.7 +4.9 | 55.3 +0.9
M-v1 | 51.8 | 59.7 | 69.4 | 68.6 | 58.1 | 42.9 | 37.3 | 55.4
M-v2.1 | 52.0 | 62.6 | 68.9 | 65.7 | 65.2 | 46.5 | 30.9 | 56.0
M-News | 57.1 +5.1 | 64.3 +1.7 |  70.3 +1.4 | 67.1 +1.4 |  67.9 +2.7 | 45.2 -1.3 | 35.1 +4.2 | 58.1 +2.1
L-v1 | 57.2 | 64.4 | 69.6 | 72.6 | 62.6 | 57.2 | 42.9 | 60.9
L-v2.1 | 53.4 | 57.9 | 67.1 | 64.9 | 62.0 | 51.3 | 46.0 | 57.5
L-News | 57.0 +3.6 | 60.4 +2.5 | 65.2 -1.9 | 62.9 -2.0 | 65.9 +3.9 | 56.6 +5.3 | 41.8 -4.2 | 58.5 +1.0

##  5 Discussion

In this paper, we presented a methodology for grounding synthetic data in
real-world diversity by using news texts originating from global sources that
cover numerous topics. Using this methodology, we curate a diverse dataset for
Named Entity Recognition (NER) and demonstrated that training NER models on
this grounded, synthetic dataset significantly enhances model performance by
enabling the models to recognize and generalize across a broader spectrum of
linguistic and cultural nuances.

The diversity of data sources is pivotal in developing robust AI systems. By
integrating news texts from a wide geographical and linguistic spread, we
captured diverse contexts that are often underrepresented. This approach
addresses critical gaps in common dataset that tend to be biased towards more
readily available data sources. As a result, models trained on diversified and
grounded data are better equipped to handle real-world applications across
different languages and regions, reducing the cultural and linguistic biases
prevalent in many current AI systems. Our methodology incorporated topic
diversification, translation, and summarization to enrich the synthetic
dataset. These strategies collectively improve the quality of the generated
dataset and can be implemented to create data for real-world tasks beyond NER,
such as text classification and sentiment analysis. In fact, while the dataset
described in this paper is curated for diversity, the methodology can easily
be extended to domain-specific text corpora as the framework used incorporates
a vector database containing both sparse and dense text embeddings, hence
allowing for key-word as well as natural language queries to obtain articles
within a specific domain.

The grounding of synthetic datasets in diversified real-world data introduces
significant advancements with both substantial benefits and noteworthy
challenges. On the positive side, these datasets enhance the inclusivity and
fairness of AI systems. By reflecting a broader array of cultural and
linguistic backgrounds, they help create models that perform equitably across
diverse populations, reducing biases that traditionally affect AI
implementations. This contributes to more reliable and universally applicable
AI solutions, fostering greater trust and broader adoption across various
regions and communities. However, the potential for perpetuating existing
biases in source materials does exist, particularly if certain regions or
groups are underrepresented in global news coverage. Additionally, the
complexities of translating and summarizing content from various languages
might introduce inaccuracies. Yet, these challenges highlight the importance
of refining data synthesis techniques and establishing rigorous oversight
processes. By continually improving these methodologies, the field can
leverage the full potential of diversified data to develop AI systems that are
not only more effective but also truly equitable and beneficial for a global
audience.

###  5.1 Limitations

#### Language limitations

Our methodology is currently limited to 12 languages due to the choice of LLM
for translation and summarization.It is possible to extend the method to cover
more languages by, e.g., employing multiple LLMs, each with different language
coverage, and adaptively choosing which one to use based on the language of
the news article. Furthermore, the accuracy of translations and the risk of
losing nuances in summarization are concerns that could affect the fidelity of
the synthetic data.

#### Model bias

Although the goal of the dataset is to reduce bias, and improve diversity, it
is still biased to western languages and countries. This limitation originates
from the abilities of WizardLM 13B v1.2 for the translation and summary
generations. Further, any bias originating in WizardLM 13B v1.2 training data
will also be present in this dataset, since WizardLM 13B v1.2 was used to
summarize the open-web articles. Further, any biases present in
Llama-3-70B-Instruct will be present in the present dataset since
Llama-3-70B-Instruct was used to extract entities from the summaries

#### Model availability

All models used in the creation of the dataset, namely WizardLM 13B V1.2 and
Llama-3-70B-Instruct are open-sourced and available both via API endpoints and
for running locally. Similarly, the GLiNER models used for evaluating our
methodology are open-sourced and available on Huggingface. Whilst the GLiNER
models are compact and can be run on most hardware, the Llama models require
more compute or paying for API calls.

##  6 Conclusion

Our research presents a compelling blueprint for generation of diversified
synthetic data aimed at training AI models. By grounding synthetic data in
real-world diversity, we enhance the generalizability and fairness of AI
systems. This study not only advances our understanding of effective dataset
synthesis but also underscores the critical need for diversity in training AI
to tackle real-world challenges across various domains.

#### Data, model, and code availability

The AskNews-NER-v0 dataset and fine-tuned NER models are available on
Huggingface at AskNews-NER-v0 and GLiNER-Small-News, GLiNER-Medium-News, and
GLiNER-Large-News, respectively, under Apache 2.0 license. The code used for
the analysis in this paper is available on GitHub at *not yet available*.

## References

  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM computing surveys (CSUR)_ , 54(6):1–35, 2021.
  * Osoba and Welser IV [2017] Osonde Osoba and William Welser IV.  An intelligence in our image.  _Santa Mônica: RAND corporation_ , 2017.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Ziems et al. [2024] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang.  Can large language models transform computational social science?  _Computational Linguistics_ , 50(1):237–291, 2024.
  * Honnibal et al. [2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd.  spacy: Industrial-strength natural language processing in python.  2020\.  doi: 10.5281/zenodo.1212303.
  * Akbik et al. [2019] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf.  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In _NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 54–59, 2019.
  * Zaratiana et al. [2023] Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois.  Gliner: Generalist model for named entity recognition using bidirectional transformer.  _arXiv preprint arXiv:2311.08526_ , 2023.
  * Wang et al. [2023] Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al.  Instructuie: multi-task instruction tuning for unified information extraction.  _arXiv preprint arXiv:2304.08085_ , 2023.
  * Sainz et al. [2023] Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre.  Gollie: Annotation guidelines improve zero-shot information-extraction.  _arXiv preprint arXiv:2310.03668_ , 2023.
  * Zhou et al. [2023] Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon.  Universalner: Targeted distillation from large language models for open named entity recognition.  _arXiv preprint arXiv:2308.03279_ , 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * He et al. [2021] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _arXiv preprint arXiv:2111.09543_ , 2021.
  * Veselovsky et al. [2023] Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil Arora, Martin Josifoski, Ashton Anderson, and Robert West.  Generating faithful synthetic data with large language models: A case study in computational social science.  _arXiv preprint arXiv:2305.15041_ , 2023.
  * Josifoski et al. [2023] Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West.  Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction.  _arXiv preprint arXiv:2303.04132_ , 2023.
  * Eldan and Li [2023] Ronen Eldan and Yuanzhi Li.  Tinystories: How small can language models be and still speak coherent english?  _arXiv preprint arXiv:2305.07759_ , 2023.
  * Xu et al. [2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ , 2023.
  * Li et al. [2023] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.  Towards general text embeddings with multi-stage contrastive learning.  _arXiv preprint arXiv:2308.03281_ , 2023.
  * Qdrant [2024] Qdrant.  Qdrant: Vector search engine for the next generation of ai applications.  https://github.com/qdrant/qdrant, 2024.  Accessed: 2024-05-20.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011.
  * AI@Meta [2024] AI@Meta.  Llama 3 model card.  2024\.  URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  * Bogdanov et al. [2024] Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne Bernard.  Nuner: Entity recognition encoder pre-training via llm-annotated data, 2024.
  * Liu et al. [2020] Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung.  Crossner: Evaluating cross-domain named entity recognition.  2020\.
  * Liu et al. [2013a] Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass.  Asgard: A portable architecture for multilingual dialogue systems.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 8386–8390. IEEE, 2013a.
  * Liu et al. [2013b] Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass.  Query understanding enhanced by hierarchical parsing structures.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 72–77. IEEE, 2013b.

##  Appendix A Appendix

###  A.1 Entity labeling

We used meta-llama/Meta-Llama-3-70B-Instruct [AI@Meta, 2024] for labeling
entities in the news articles. The instructions provided to the LLM are shown
in Figure 5. We restricted the requested entity types to 54 choices as shown
in Figure 6. In addition to the prompt instructions, we provided the following
examples (Supplementary Figure 1) to guide the LLM.

Supplementary Figure 1: Entity extraction prompt examples. We provided the
following two example input texts and expected outputs, in addition to the
prompt in Figure 5.

#### Prompt examples

Q: Given the following text, please extract all named entities: <doc> French
Prime Minister Gabriel Attal, 61, announced at the ongoing United Nations (UN)
climate summit that France will be investing £500 million into the French
renewable energy sector, specifically, a Parisian hydroelectric dam. The news
comes after the conservative French government’s recent election victory and
new laws being added to the French constitution. </doc>
A: { "entities" : [ [ "French" , "Nationality" ] , [ "Prime Minister" ,
"Title" ] , [ "French Prime Minister" , "Title" ] , [ "French Prime Minister
Gabriel Attal" , "Politician" ] , [ "Gabriel Attal" , "Person" ] , [ "61" ,
"Number" ] , [ "United Nations" , "Organization" ] , [ "UN" , "Organization" ]
, [ "United Nations (UN) climate summit" , "Event" ] , [ "climate summit" ,
"Event" ] , [ "France" , "Country" ] , [ "£500 million" , "Money" ] , [
"French" , "Nationality" ] , [ "Parisian" , "Nationality" ] , [ "hydroelectric
dam" , "Facility" ] , [ "French" , "Nationality" ] , [ "French government" ,
"Organization" ] , [ "election victory" , "Event" ] , [ "French" ,
"Nationality" ] , [ "French constitution" , "Law" ] , [ "constitution" , "Law"
] ] }
Q: Given the following text, please extract all named entities: <doc> Daniel
is 5-foot-2 and 181 pounds, while his brother, John, is 6-foot-1 and 200
pounds. They both play for the New York Giants in the NFL. Both scored 3
touchdowns, 10% of the match’s total, in the last game against the Dallas
Cowboys, ending with a score of 21-17 after 45 minutes. Their average scores
for the season are 12.5 points per game, after 4,000+ games. The first 5 games
had -56 odds for the home team. </doc>
A: { "entities" : [ [ "Daniel" , "Athlete" ] , [ "5-foot-2" , "Quantity" ] , [
"181 pounds" , "Quantity" ] , [ "John" , "Athlete" ] , [ "6-foot-1" ,
"Quantity" ] , [ "200 pounds" , "Quantity" ] , [ "New York" , "City" ] , [
"New York Giants" , "Organization" ] , [ "New York Giants" , "Sports" ] , [
"NFL" , "Organization" ] , [ "NFL" , "Sports" ] , [ "3 touchdowns" ,
"Quantity" ] , [ "10%" , "Percentage" ] , [ "Dallas" , "City" ] , [ "Dallas
Cowboys" , "Organization" ] , [ "Dallas Cowboys" , "Sports" ] , [ "21-17" ,
"Quantity" ] , [ "45 minutes" , "Time" ] , [ "12.5 points" , "Quantity" ] , [
"4,000+" , "Quantity" ] , [ "5" , "Quantity" ] , [ "-56" , "Number" ] ] }

###  A.2 Dataset information

Following topic diversification, as detailed in Section 3.2, the AskNews-
NER-v0 dataset comprised 5,049 articles covering 73 unique topics
(Supplementary Figure 2) and originating from 125 different countries. The
distribution of countries in the full dataset, as well as in the train, test,
and validation splits, are shown in Supplementary Tables 1-3. meta-llama/Meta-
Llama-3-70B-Instruct [AI@Meta, 2024] was used for labeling entities in these
news articles, resulting in the entity types shown in Supplementary Figure 3.

Supplementary Figure 2: Topics distribution. Percentage coverage of the 73 unique topics assigned to the collection of articles used for the AskNews-NER-v0 dataset. For legibility, the topics have been split up into three separate charts where A shows topics with $\geq$1% coverage and "other" corresponding to the remaining topics, B shows topics with $<$1% but $\geq$0.1% coverage and "other" corresponding to the remaining topics, and C shows topics with $<$0.1% coverage. Supplementary Table 1: Country distribution. Article frequency for the top 50 countries represented in the AskNews-NER-v0 dataset. Country | Full dataset | Train split | Validation split | Test split
---|---|---|---|---
United States | 129 | 86 | 22 | 21
Canada | 129 | 100 | 20 | 9
United Arab Emirates | 129 | 92 | 12 | 25
Australia | 129 | 102 | 14 | 13
Austria | 129 | 91 | 17 | 21
Switzerland | 129 | 87 | 21 | 21
Lebanon | 129 | 91 | 17 | 21
Colombia | 129 | 91 | 16 | 22
Türkiye | 129 | 92 | 13 | 24
Malaysia | 129 | 87 | 18 | 24
Brazil | 129 | 89 | 26 | 14
Portugal | 129 | 103 | 17 | 9
Nigeria | 129 | 101 | 21 | 7
Venezuela | 129 | 87 | 24 | 18
Ukraine | 129 | 91 | 13 | 25
Spain | 129 | 93 | 22 | 14
India | 129 | 79 | 24 | 26
Egypt | 129 | 97 | 13 | 19
France | 129 | 90 | 24 | 15
Russia | 129 | 83 | 24 | 22
Germany | 129 | 93 | 15 | 21
Argentina | 129 | 87 | 24 | 18
Mexico | 129 | 88 | 23 | 18
United Kingdom | 129 | 93 | 25 | 11
Italy | 129 | 99 | 16 | 14
Saudi Arabia | 118 | 91 | 13 | 14
Chile | 103 | 77 | 14 | 12
Norway | 100 | 74 | 11 | 15
Peru | 86 | 66 | 10 | 10
Dominican Republic | 83 | 63 | 8 | 12
Pakistan | 83 | 49 | 13 | 21
Ireland | 83 | 59 | 15 | 9
Sweden | 77 | 62 | 9 | 6
Tuvalu | 72 | 43 | 16 | 13
Jordan | 63 | 52 | 5 | 6
Philippines | 62 | 27 | 7 | 28
Kazakhstan | 47 | 30 | 4 | 13
Denmark | 45 | 30 | 7 | 8
Ecuador | 41 | 27 | 6 | 8
South Africa | 38 | 30 | 4 | 4
China | 36 | 26 | 7 | 3
El Salvador | 29 | 19 | 5 | 5
Belarus | 28 | 23 | 2 | 3
Singapore | 26 | 14 | 5 | 7
Morocco | 26 | 23 | 2 | 1
Costa Rica | 25 | 19 | 4 | 2
Honduras | 24 | 18 | 1 | 5
Paraguay | 23 | 20 | 1 | 2
Netherlands | 23 | 14 | 6 | 3
Belgium | 23 | 20 | 3 | 0
Supplementary Table 2: Country distribution. Article frequency for the mid 51-100 countries represented in the AskNews-NER-v0 dataset. Continued from Supplementary Table 1. Country | Full dataset | Train split | Validation split | Test split
---|---|---|---|---
Israel | 19 | 10 | 7 | 2
Guatemala | 18 | 12 | 1 | 5
Iraq | 18 | 14 | 1 | 3
Uruguay | 18 | 16 | 1 | 1
Qatar | 17 | 12 | 5 | 0
Panama | 16 | 12 | 3 | 1
Japan | 16 | 9 | 2 | 5
Bolivia | 15 | 10 | 2 | 3
Kuwait | 14 | 8 | 4 | 2
Senegal | 14 | 11 | 3 | 0
New Zealand | 14 | 12 | 2 | 0
Algeria | 13 | 9 | 1 | 3
Iceland | 13 | 8 | 4 | 1
Iran | 11 | 6 | 2 | 3
Kenya | 11 | 7 | 1 | 3
Ghana | 11 | 7 | 1 | 3
Sierra Leone | 10 | 8 | 2 | 0
Luxembourg | 10 | 8 | 0 | 2
Cuba | 10 | 5 | 2 | 3
South Korea | 10 | 8 | 1 | 1
Macau | 9 | 7 | 1 | 1
Puerto Rico | 9 | 6 | 2 | 1
Nicaragua | 9 | 6 | 1 | 2
Bangladesh | 9 | 6 | 3 | 0
Yemen | 8 | 6 | 2 | 0
Palestine | 8 | 5 | 2 | 1
Azerbaijan | 8 | 6 | 1 | 1
Syria | 8 | 6 | 0 | 2
Sri Lanka | 7 | 4 | 3 | 0
Hong Kong | 7 | 6 | 0 | 1
Latvia | 7 | 2 | 2 | 3
Micronesia, Fed. Sts. | 7 | 4 | 1 | 2
Croatia | 6 | 6 | 0 | 0
EU | 5 | 4 | 0 | 1
Oman | 5 | 4 | 1 | 0
Estonia | 5 | 3 | 1 | 1
Thailand | 4 | 2 | 0 | 2
Indonesia | 4 | 3 | 0 | 1
Armenia | 3 | 1 | 2 | 0
Kyrgyz Republic | 3 | 3 | 0 | 0
Libya | 3 | 3 | 0 | 0
Tunisia | 3 | 3 | 0 | 0
Malta | 3 | 3 | 0 | 0
Taiwan | 3 | 3 | 0 | 0
Tokelau | 2 | 2 | 0 | 0
Finland | 2 | 2 | 0 | 0
Poland | 2 | 2 | 0 | 0
Anguilla | 2 | 2 | 0 | 0
Afghanistan | 2 | 1 | 1 | 0
Uzbekistan | 2 | 1 | 0 | 1
Supplementary Table 3: Country distribution. Article frequency for the bottom 101-125 countries represented in the AskNews-NER-v0 dataset. Continued from Supplementary Tables 1 and 2. Country | Full dataset | Train split | Validation split | Test split
---|---|---|---|---
Bahrain | 2 | 1 | 0 | 1
Angola | 2 | 0 | 1 | 1
Romania | 1 | 0 | 1 | 0
Mauritania | 1 | 1 | 0 | 0
Georgia | 1 | 1 | 0 | 0
Benin | 1 | 1 | 0 | 0
Hungary | 1 | 1 | 0 | 0
British Indian Ocean Territory | 1 | 1 | 0 | 0
St. Lucia | 1 | 1 | 0 | 0
Fiji | 1 | 1 | 0 | 0
Sudan | 1 | 1 | 0 | 0
Mozambique | 1 | 1 | 0 | 0
Greece | 1 | 1 | 0 | 0
Uganda | 1 | 1 | 0 | 0
Nepal | 1 | 1 | 0 | 0
Sao Tome and Principe | 1 | 0 | 1 | 0
Laos | 1 | 1 | 0 | 0
Reunion | 1 | 1 | 0 | 0
Sint Maarten | 1 | 1 | 0 | 0
Burundi | 1 | 0 | 0 | 1
Barbados | 1 | 0 | 0 | 1
Lithuania | 1 | 0 | 0 | 1
Bulgaria | 1 | 0 | 0 | 1
San Marino | 1 | 0 | 0 | 1
Zambia | 1 | 1 | 0 | 0
Total | 5049 | 3589 | 730 | 730
Supplementary Figure 3: Entity type frequency for the raw labels. Labeling of
news articles using Llama3 resulted in 182 unique entity types, despite
requesting 54 as per Figure 6. Note that one of the entities types in the
final 25% of the raw labels is an empty string.

###  A.3 Evaluation

The fine-tuned NER models were evaluated in a zero-shot context on common NER
benchmarks. Supplementary Table 4 shows the scores for GLiNER v1 models,
GLiNER v2.1 models, and the fine-tuned models on 18 NER benchmark datasets.
Performance improvements for the fine-tuned models compared to the base models
(GLiNER v2.1) are found in Table 1.

Supplementary Table 4: Zero-shot performance on 18 NER datasets. Comparison
between base models (S - GLiNER-Small, M - GLiNER-Medium, L - GLiNER-Large)
and fine-tuned models ("News"). Performance on the test split of the anNER-v0
dataset is shown at the bottom. Bold indicates the highest score per dataset.
Commit hashes for all models are found in Supplementary Table 5.

Dataset | S-v1 | S-v2.1 | S-News | M-v1 | M-v2.1 | M-News | L-v1 | L-v2.1 | L-News
---|---|---|---|---|---|---|---|---|---
ACE 2005 | 23.8 | 32.8 | 23.2 | 23.6 | 31.3 | 23.3 | 27.1 | 33.3 | 28.9
AnatEM | 49.0 | 35.1 | 36.5 | 33.2 | 29.2 | 36.8 | 33.7 | 24.7 | 34.8
Broad Tweet Corpus | 60.8 | 63.0 | 69.2 | 63.1 | 63.9 | 67.7 | 64.1 | 59.9 | 69.2
CoNLL 2003 | 61.4 | 59.8 | 60.9 | 62.8 | 61.5 | 63.0 | 60.7 | 59.5 | 56.4
FabNER | 21.8 | 17.9 | 18.9 | 24.1 | 17.0 | 20.6 | 25.4 | 19.2 | 22.5
FindVehicle | 32.3 | 38.0 | 43.1 | 30.3 | 36.0 | 38.7 | 45.1 | 51.9 | 52.9
GENIA_NER | 51.2 | 47.7 | 47.2 | 57.3 | 55.4 | 54.0 | 58.4 | 55.3 | 55.0
HarveyNER | 22.8 | 18.3 | 23.1 | 19.4 | 23.2 | 26.7 | 16.9 | 18.7 | 15.8
MultiNERD | 58.2 | 57.3 | 68.3 | 54.9 | 55.3 | 67.1 | 54.8 | 48.7 | 64.0
Ontonotes | 27.6 | 28.1 | 35.1 | 24.5 | 25.7 | 33.2 | 27.0 | 19.5 | 32.0
PolyglotNER | 40.6 | 40.4 | 43.9 | 42.0 | 42.1 | 45.3 | 40.7 | 39.6 | 42.5
TweetNER7 | 39.3 | 36.5 | 35.6 | 39.9 | 38.2 | 38.6 | 40.6 | 36.1 | 35.5
WikiANN en | 56.4 | 55.3 | 53.6 | 58.4 | 55.5 | 53.0 | 57.4 | 55.5 | 52.5
WikiNeural | 68.7 | 64.7 | 76.0 | 71.0 | 67.5 | 77.9 | 67.7 | 62.9 | 70.3
bc2gm | 51.5 | 50.4 | 49.4 | 52.9 | 54.0 | 54.6 | 52.0 | 45.2 | 46.9
bc4chemd | 44.2 | 49.0 | 49.4 | 46.4 | 47.3 | 51.1 | 48.4 | 53.1 | 55.1
bc5cdr | 66.8 | 66.1 | 70.9 | 66.5 | 67.0 | 72.5 | 67.8 | 68.4 | 71.2
ncbi | 55.0 | 56.6 | 57.9 | 58.6 | 60.3 | 64.5 | 63.8 | 59.7 | 65.5
Average | 46.2 | 45.4 | 47.9 | 46.0 | 46.1 | 49.4 | 47.3 | 45.1 | 48.4
AskNews-NER-v0 | 55.4 | 54.4 | 71.8 | 57.0 | 53.3 | 71.9 | 57.0 | 48.6 | 72.9

###  A.4 Model versions

For reproducibility purposes, we provide the Huggingface commit hashes for the
models used in this paper (Supplementary Table 5). The GLiNER models were
refactored from the versions published in Zaratiana et al. [2023] before being
published to Huggingface. As we have evaluated our fine-tuned models compared
to the versions available on Huggingface, we also compare benchmark scores of
the GLiNER models available on Huggingface to those reported in in
Supplementary Tables 7 and 6.

Supplementary Table 5: Model versions. Huggingface  commit hashes for the models used for fine-tuning and evaluation in this paper. Model | Commit hash
---|---
Small-v1 | 0f0f4e7d3f10e48844110162d6b5c6072ddd5a4e
Small-v2.1 | 4e091416cf7c3481db542c2a3d26156916f3a47f
Medium-v1 | b63645e3596b70a7d66e0597b3e739ad17bffe9d
Medium-v2.1 | a3f776a34b65ed1218fb4765b89d564260f0dc01
Large-v1 | 1f55b526b24c7576857d4eb2b047cc77b0143594
Large-v2.1 | abd49a1f1ebc12af1be84d06f6848221cf96dcad
NuNERZero_span | 6b5d591b348c8775ce065737e267b4aa6a907538
Small-AskNews-NER-v0 | dba3032b4f4426c827fed54fc9993aaece5dbf2a
Medium-AskNews-NER-v0 | 44c9b148230c65df75661c38c4fa7c1825892133
Large-AskNews-NER-v0 | 8f4f822f21b393b6eb577f4efd8558894b4e2b62
Supplementary Table 6: Comparison between models currently available on
Huggingface  and scores reported in Table 1 in Zaratiana et al. [2023]. Zero-
shot performance on Out-of-Domain NER benchmark datasets for the different
model sizes (S- small, M - medium, L - large). Bold indicates the highest
score per dataset. Green and red values indicate increase or decrease in score
for  models compared to published scores. Commit hashes for the  models are
found in Supplementary Table 5.

Model | AI | Literature | Music | Politics | Science | Movie | Restaurant | Average
---|---|---|---|---|---|---|---|---
S-v1 | 50.7 | 60.0 | 60.9 | 61.5 | 55.6 | 46.9 | 33.3 | 52.7
S-v1  | 49.8 -0.9 | 62.8 +2.8 | 66.2 +5.3 | 64.0 +2.5 | 57.0 +1.4 | 43.1 -3.8 | 28.7 -4.6 | 53.1 +0.4
M-v1 | 51.8 | 59.7 | 69.4 | 68.6 | 58.1 | 42.9 | 37.3 | 55.4
M-v1  | 51.9 +0.1 | 59.5 -0.2 | 66.5 -2.9 | 67.3 -1.3 | 55.9 -2.2 | 46.8 +3.9 | 34.4 -2.9 | 54.6 -0.8
L-v1 | 57.2 | 64.4 | 69.6 | 72.6 | 62.6 | 57.2 | 42.9 | 60.9
L-v1  |  57.2 +0.0 | 63.7 -0.7 |  69.6 +0.0 | 72.5 -0.1 |  65.3 +2.7 | 48.6 -8.6 | 37.6 -5.3 | 59.2 -1.7

Supplementary Table 7: Comparison between large model currently available on Huggingface  and scores reported in Table 2 in Zaratiana et al. [2023]. Zero-shot performance on 20 NER benchmark datasets for Large models. Bold indicates the highest score per dataset. Green and red values indicate increase or decrease in score for  models compared to published scores. The commit hash for the  model is found in Supplementary Table 5. Dataset | Large-v1 | Large-v1
---|---|---
ACE 2005 | 27.3 | 27.1 -0.2
AnatEM | 33.7 |  33.7 +0.0
Broad Tweet Corpus | 61.2 |  64.1 +2.9
CoNLL 2003 | 64.6 | 60.7 -3.9
FabNER | 23.6 |  25.4 +1.8
FindVehicle | 41.9 |  45.1 +3.2
GENIA_NER | 55.5 |  58.4 +2.9
HarveyNER | 22.7 | 16.9 -5.8
MultiNERD | 59.7 | 54.8 -4.9
Ontonotes | 32.2 | 27.0 -5.2
PolyglotNER | 42.9 | 40.7 -2.2
TweetNER7 | 41.4 | 40.6 -0.8
WikiANN en | 58.9 | 57.4 -1.5
WikiNeural | 71.8 | 67.7 -4.1
bc2gm | 47.9 |  52.0 +4.1
bc4chemd | 43.1 |  48.4 +5.3
bc5cdr | 66.4 |  67.8 +1.4
ncbi | 61.9 |  63.8 +1.9
Average | 47.6 | 47.3 -0.3