# OWLOOP: A Modular API to Describe
OWL Axioms in OOP Objects Hierarchies⋆

Luca Buoncompagni*  |, Syed Yusha Kareem*  |, and Fulvio Mastrogiovanni*  |,
Department of Informatics, Bioengineering, Robotics and Systems Engineering,
University of Genoa, Via Opera Pia 13, 16145, Genoa, Italy.  Teseo s.r.l.,
Piazza Montano 2a, 16151, Genoa, Italy.

###### Abstract

OWLOOP is an Application Programming Interface (API) for using the Ontology
Web Language (OWL) by the means of Object-Oriented Programming (OOP). It is
common to design software architectures using the OOP paradigm for increasing
their modularity. If the components of an architecture also exploit OWL
ontologies for knowledge representation and reasoning, they would require to
be interfaced with OWL axioms. Since OWL does not adhere to the OOP paradigm,
such an interface often leads to boilerplate code affecting modularity, and
OWLOOP is designed to address this issue as well as the associated
computational aspects. We present an extension of the OWL-API to provide a
general-purpose interface between OWL axioms subject to reasoning and modular
OOP objects hierarchies.
Keywords
Object-Oriented Programming (OOP) , Ontology Web Language (OWL) , Application
Programming Interface (API) , Software Architecture.
⋆ This manuscript has been submitted to the SoftwareX Elsevier journal on the
12th of January 2021, revised on the 18th of November 2021, accepted on the
14th of December 2021, and published on the 30th of December 2021. This
document contains the published version of the manuscript. Please, cite this
work as:

Luca Buoncompagni, Syed Yusha Kareem, and Fulvio Mastrogiovanni, “OWLOOP: A
Modular API to Describe OWL Axioms in OOP Objects Hierarchies”, SoftwareX,
January 2022, 100952, Vol. 17, Elsevier,
https://doi.org/10.1016/j.softx.2021.100952.

## Code Metadata

C1 |  Current code version |  v2.1
---|---|---
C2 |  Permanent link to code/repository used for this code version |  https://github.com/TheEngineRoom-UniGe/OWLOOP
C4 |  Legal Code License |  GNU General Public License v3.0
C5 |  Code versioning system used |  git
C6 |  Software code languages, tools, and services used |  code language: Java
C7 |  Compilation requirements, operating environments & dependencies |  Java v1.8.0, Gradle v5.2.1, Junit v4.12, aMOR v2.2, OWL-API v5.0.5, openllet v2.5.1
C8 |  Link to developer documentation/manual |  github.com/TheEngineRoom-UniGe/OWLOOP/wiki
C9 |  Support email for questions |<EMAIL_ADDRESS><EMAIL_ADDRESS>
Table 1: Code metadata

##  1 Motivation and Significance

The Web Ontology Language (OWL) is a semantic language standardised by the
World Wide Web Consortium (W3C) [1]. It is used to formalise implicit and
explicit knowledge as logic-based _axioms_ among symbols to represent _things_
, groups of things, and relations between things in a domain of interest.
Ontologies provide the ability to reason on a semantic knowledge
representation, which is particularly important for applications involving
autonomous systems and human-machine interaction.

From a software architecture perspective, an ontology is meant to be used in
synergy with other software components [2], which are usually implemented with
the Object-Oriented Programming (OOP) paradigm. However, issues occur since
OWL axioms are not structured in an ontology following the OOP formalism.
Although there are similarities between the knowledge in an ontology and OOP-
based data structures (_e_._g_.,  _classes_ , _objects_ , or _properties_)
there are also non-trivial differences, an exhaustive list of which can be
found in [3]. This is one of the reasons why software developers are typically
reluctant to use ontologies in their architectures [4].

As surveyed in [4], for accessing and integrating ontologies in a software
architecture, the active and passive _OWL to OOP mapping_ can be used. The
_active_ mapping transforms an ontology from its syntactic form, _e_._g_.,
based on the eXtensible Markup Language (XML), to code statements in a target
programming language. With an active mapping, there is the possibility of
reasoning over the executable ontology at runtime [5, 6, 7], but this process
is limited to the amount of fitness between the OWL language and its OOP
counterpart. Instead, with the _passive_ OWL to OOP mapping [4], an ontology
is integrated with the OOP language through an external inference engine,
_i_._e_., a reasoner like Hermit [8] or Pellet [9], among others. The reasoner
exploits the _factory pattern_ to create immutable OOP objects containing
snapshots of the knowledge in the ontology, which is an OWL-based data
structure loaded in memory.

The passive mapping involves an _API-based_ strategy, while the active mapping
concerns an _ontology-oriented programming_ strategy [4]. On the one hand, the
passive mapping focuses on performance but it might lead to complex and
voluminous source codes. Furthermore, with a passive mapping, OWL axioms which
are related to each other in the ontology are always represented as
independent OOP objects. Therefore, passive mapping allows to use OWL axioms
through OOP objects but it does not exploit the benefits of the OOP paradigm
as active mapping does. On the other hand, we are not aware of an active
mapping that supports all the reasoning mechanisms implemented by the OWL
reasoners used with passive mapping.

B

A

$n$

OOP Composition:

A _instanciates_ $n$ B.

OOP Inheritance:

A _extends_ B.

B

A

OOP Realization:

A _implements_ B.

A

«interface»

B

Figure 1: The UML notations used in this paper.

To develop a software architecture exploiting the OWL formalism, we identify
two aspects concerning the design and implementation of ($i$) the OWL-based
knowledge representation from data and prior semantics, and ($ii$) the
interfaces among the components of the software architecture and OWL-based
knowledge. The former aspect can be tackled with different ontology design
patterns111A collection of state of the art ontology design patterns is
available at
http://ontologydesignpatterns.org/wiki/Ontology_Design_Patterns_._org_(ODP).,
which can be implemented using available tools, _e_._g_., OWL-API [10], Jena-
API [11] or the Protégé editor [12]. These tools exploit string-based
identifiers of OWL symbols, and they allow to define the procedures that
retrieve specific knowledge through queries. Although these tools support well
the ontology design process, they might not be suitable for the latter aspect
since it has different requirements, which include modularity, maintainability
and synchronisation of the software architecture.

Both aspects might be tackled by developers with different expertise,
_i_._e_., on OWL formalism and software architecture solutions, respectively,
and they require a constant joint effort to develop an effective system. Thus,
we aim for an API which can be used to develop a modular abstraction layer
interfacing ontologies and software components. In our approach, on the one
hand, experts on OWL ontologies can design semantic data representation and
store it in a file as prior knowledge; also, they can develop OWLOOP-based
interfaces to manipulate and query the ontology at run time. On the other
hand, the ontology and OWLOOP-based interfaces can be used by the developers
focusing on architectural aspects through the OOP paradigm.

This paper presents the OWLOOP API, which implements a passive OWL to OOP
mapping for integrating ontologies and software architectures. We aim to
entirely support OWL reasoners and exploit the benefits of the OOP paradigm by
avoiding the factory design pattern. The OWLOOP API interfaces OWL axioms with
_descriptors_ , which are OOP classes representing a fragment of the ontology,
but they are neither immutable nor independent. In other words, descriptors
can be designed within an OOP-based hierarchy of classes. OWLOOP supports the
implementation of general-purpose and modular OWL to OOP mapping, which can
improve flexibility, reusability and maintainability of software architectures
exploiting ontologies.

owl:THING

DOOR

LOCATION

ROOM

CORRIDOR

ROBOT

Room1

Room2

Corridor1

Robot1

xsd:int

24

hasTemperature

isIn

isLinkedTo

Figure 2: A simple ontology used as an example throughout the paper.

##  2 Software Description

###  2.1 OWL Formalism Overview

Figure 2 depicts an ontology used as an illustrative example throughout the
paper. An OWL ontology is made by _axioms_ , _i_._e_., symbolic statements
describing some _entities_ (listed in Table 2) with a specific logic-based
semantic expression. Axioms are organised in three structures, _i_._e_., the
Terminological and the Rule graphs (TBox and RBox), and the Assertion set
(ABox).

The TBox describes concepts (OWLClasses or _classes_ in short) as a hierarchy
of logic implications (_e_._g_., a ROOM is a _sub-class_ of LOCATION) with the
THING class as default root, _i_._e_., the _super-class_ of all possible
classes, and its counterpart NOTHING. Instead, the ABox collects instances
(OWLNamedIndividual or _individuals_) of concepts, _e_._g_., Room1 is an
_instance_ of the ROOM class, which could be related among each other through
roles (or _properties_), _e_._g_., Room1 isLinkedTo Corridor1. The RBox
represents properties and sub-properties similar to the TBox, and it includes
specific definitions, such as transitive, symmetric, _etc._ A property in the
Rbox can describe its _inverse_ property, and its _range_ and _domain_
classes, _e_._g_., hasDoor relates instances of LOCATION and DOOR classes,
respectively. OWL standard distinguishes _object_ properties from _data_
properties; OWLObjectProperty and OWLDataProperty, respectively. As introduced
above, the former relates two individuals, while the latter relates an
individual domain with a _literal_ (OWLLiteral) range, _e_._g_., a number. A
literal is considered as an instance of _concrete_ classes (OWLDataTypes),
_e_._g_., the set of positive numbers.

OWL defines the logic _disjunction_ and _conjunction_ among classes, as well
as _cardinality restrictions_ to represent classes through properties,
_e_._g_., a CORRIDOR is a LOCATION that has (_i_._e_., hasDoor) _at least_ 2
Doors. In addition, each pair of entities within the same box can be set to be
_equivalent_ or different (_i_._e_.,  _disjoint_), _e_._g_., Robot1 is a
different individual from Room1. The axioms in the three boxes of the ontology
can be processed by OWL-based reasoners, which check the consistency of an
ontology and _infer_ new implicit axioms under the _open word_ assumption. In
addition, it can solve _queries_ that are given as incomplete axioms,
_e_._g_., find all instances of the ROBOT class. To better appreciate the
reasoning ability, more details of the OWL formalisms are available in [13,
14].

To deal with such a variety of axioms semantics, the standard implementation
of OWL associates each axiom $\mathcal{A}_{j}$ to a certain $k$-th
_expression_ $\mathcal{E}_{k}$, _e_._g_., Sub, Super, Instance, Equivalent,
Disjoint, _etc._ In turn, the expression defines the types of entities that
need to be specified in the axiom
${\mathcal{A}_{j}=\mathcal{E}_{k}(e_{1},e_{2},\ldots)}$, _e_._g_., the sub-
class expression requires that $e_{1}$ and $e_{2}$ are both classes, while the
instance expression requires an individual and a class. To represent axioms
suitably for OOP mapping, OWLOOP provides an $\mathcal{E}_{k}$ counterpart for
each OWL expression, _e_._g_., an OWL ObjectPropertyAssertion is an OWLOOP
ObjectLink, while an OWL ClassAssertion is an OWLOOP Type. This paper presents
OWLOOP expressions as shown in Table 3 and introduced above, _e_._g_., some
OWL axioms in our illustrative ontology are

Super $\displaystyle\big{(}\text{{{ROOM}}},\;\text{{{LOCATION}}}\big{)},$ (1)
Instance $\displaystyle\big{(}\text{{{Room1}}},\;\text{{{ROOM}}}\big{)},$
ObjectLink
$\displaystyle\big{(}\text{{{isLinkedTo}}},\;\text{{{Room1}}},\;\text{{{Corridor1}}}\big{)},$
DataLink
$\displaystyle\big{(}\text{{{hasTemperature}}},\;\text{{{Room1}}},\;24\big{)},$
EquivalentRestriction $\displaystyle\big{(}\text{{{CORRIDOR}}},\;\text{{{min
2}}},\;\text{{{hasDoor}}},\;\text{{{DOOR}}}\big{)}.$

The OWLOOP API enables interaction with OWL entities in an ontology by using
_descriptors_ , _i_._e_., Java classes encapsulating reusable code to exploit
OWL axioms, _e_._g_., string-based identifiers of OWL symbols and procedures
based on queries. The next Section presents the structure of OWL axioms in a
descriptor, while Section 2.3 focuses on the descriptor functionalities to
assert and retrieve knowledge subject to reasoning.

###  2.2 Software Architecture

The OWL to OOP mapping implemented by OWLOOP stores the axioms involving an
$\mathcal{E}_{k}$ expression into a data structure ${D_{k}=\langle
x,Y_{k}\rangle}$. In it, $x$ is an OWL entity called _ground_ , and
${Y_{k}=k{:}[\\{y_{1}\\},\\{y_{2}\\},\ldots]}$ is an _entity set_ such that
the OWL axiom ${\mathcal{E}_{k}(x,y_{i})}$ is derived for each $y_{i}\in
Y_{k}$. For example, we encode the OWL axioms in (1) as

$\displaystyle D_{1}$
$\displaystyle=\big{\langle}\text{{{ROOM}}},\;\;\text{{{Super}}}{:}\big{[}\\{\text{{{LOCATION}}}\\},\;\;\\{\text{{{THING}}}\\}\big{]}\big{\rangle},$
(2) $\displaystyle D_{2}$
$\displaystyle=\big{\langle}\text{{{ROOM}}},\;\;\text{{{Instance}}}{:}\big{[}\\{\text{{{Room1}}}\\},\;\\{\text{{{Room2}}}\\}\big{]}\big{\rangle},$
$\displaystyle D_{3}$
$\displaystyle=\big{\langle}\text{{{Room1}}},\;\;\text{{{ObjectLink}}}{:}\big{[}\\{\text{{{isLinkedTo}}},\;\text{{{Corridor1}}}\\}\big{]}\big{\rangle},$
$\displaystyle D_{4}$
$\displaystyle=\big{\langle}\text{{{Room1}}},\;\;\text{{{DataLink}}}{:}\big{[}\\{\text{{{hasTemperature}}},\;\text{{{24}}}\\}\big{]}\big{\rangle},$
$\displaystyle D_{5}$
$\displaystyle=\big{\langle}\text{{{CORRIDOR}}},\;\text{{{EquivalentRestriction}}}{:}\big{[}\\{\text{{{min}}}\,2,\;\text{{{hasDoor}}},\;\text{{{DOOR}}}\\}\big{]}\big{\rangle}.$

Each $i$-th element of the entity set $Y_{k}$ is an OWLOOP entity, which can
either be an OWL entity (as in $D_{1}$ and $D_{2}$) or a structure of OWL
entities, _i_._e_., as specified in Table 2, an OWLOOPObject (in $D_{3}$), an
OWLOOPData (in $D_{4}$) or an OWLOOPRestriction (in $D_{5}$).

OWLEntity
---
OWLClass,
OWLDataType,
OWLNamedIndividual,
OWLLiteral,
OWLObjectProperty,
OWLDataProperty.

OWLOOPEntity _(extends_ OWLEntity _)_
---
OWLOOPObject: |  $\big{\langle}\texttt{OWLObjectProperty},\texttt{OWLNamedIndividual}\big{\rangle}$,
OWLOOPData: |  $\big{\langle}\texttt{OWLDataProperty},\texttt{OWLLiteral}\big{\rangle}$,
OWLOOP Restriction: |  $\big{\langle}\texttt{OWLClass}\big{\rangle}$, or
$\big{\langle}\mathcal{C},\texttt{OWLClass}\big{\rangle}$, or
$\big{\langle}\mathcal{C},\texttt{OWLObjectProperty},\texttt{OWLClass}\big{\rangle}$,
or
$\big{\langle}\mathcal{C},\texttt{OWLDataProperty},\texttt{OWLDataType}\big{\rangle}$.

Table 2: On the left–hand side, the possible types of entities in an OWL ontology. OWLOOP entities can either be OWL entities or structures of OWL entities, which are shown on the right. The cardinality $\mathcal{C}$ spans in an OWL axiom among $\\{\text{{{some}}},\;\text{{{only}}},\;\text{{{min}}}\,n,\;\text{{{max}}}\,n,\;\text{{{exact}}}\,n\\}$ where ${n\in\mathbb{N}^{+}}$.  Ground ($x$) | Expression ($\mathcal{E}_{k}$) | Entity set element ($y_{i}$)
---|---|---
| OWLClass
---
(Class
Descriptor)
| Equivalent
---
Disjoint
Super
Sub
Instance
EquivalentRestriction
| OWLClass
---
OWLClass
OWLClass
OWLClass
OWLNamedIndividual
OWLOOPRestriction
| OWLNamedIndividual
---
(Individual
Descriptor)
| Type
---
Equivalent
Disjoint
ObjectLink
DataLink
| OWLClass
---
OWLNamedIndividual
OWLNamedIndividual
OWLOOPObject
OWLOOPData
| OWLObjectProprty
---
(ObjectProperty
Descriptor)
| Equivalent
---
Disjoint
Sub
Super
Inverse
Domain
Range
| OWLObjectProperty
---
OWLObjectProperty
OWLObjectProperty
OWLObjectProperty
OWLObjectProperty
OWLOOPRestriction
OWLOOPRestriction
| OWLDataProperty
---
(DataProperty
Descriptor)
| Equivalent
---
Disjoint
Sub
Super
Domain
Range
| OWLDataProperty
---
OWLDataProperty
OWLDataProperty
OWLDataProperty
OWLOOPRestriction
OWLOOPRestriction
Table 3: The expressions that OWLOOP can currently map into axioms (in the
second column), organised by the four types of descriptors, _i_._e_., class,
individual, object and data property descriptors. The type of each descriptor
is given by the type of its ground entity (in the first column), and it can
describe the shown expressions, each with a specific entity set (third
column).

OWLOOP represents the data structure $D$ in the OOP interface named
Descriptor, which is implemented as shown in Figure 3 through the Unified
Modelling Language (UML), whose notation is summarised in Figure 1. From (2)
it is possible to deduce that descriptors with the same type of ground might
be merged to encode different axioms, _e_._g_., $D_{1}$ and $D_{2}$ can be
represented in a _compound_ descriptor ${D_{c}=\langle x,Y_{1},Y_{2}\rangle}$,
where $Y_{1}$ and $Y_{2}$ concern the Super and Instance expressions. Hence,
OWLOOP provides four different descriptors based on their ground, _i_._e_.,
ClassDescriptor, IndividualDescriptor, ObjectPropertyDescriptor, and
DataPropertyDescriptor.

In order to address an application, developers should design suitable compound
descriptors with sets of axioms based on Table 3. Four steps should be
followed to design a compound descriptor based on the functionalities that an
OOP class can inherit from OWLOOP interfaces, _e_._g_., as shown in Listing 4:
($i$) assign a ground entity by inheriting from one of the interfaces provided
by the ${}^{\star}\\!$.descriptorGround package; ($ii$) inherit from
extensions of the Descriptor interfaces (available in the package
${}^{\star}\\!$.descriptorExpression) the representation of some expressions
$\mathcal{E}_{k}$; ($iii$) consistently for each $\mathcal{E}_{k}$ expression,
instantiate an empty entity set based on the
${}^{\star}\\!$.descriptorEntitySet package; ($iv$) for each
$\mathcal{E}_{k}$, specify a descriptor to _build_ (addressed in the next
Section).

1

$x$

$1$

$Y$

$0..n$

$y$

$1$

$e$

$1$

$\mathcal{E}$

«interface»

Expression

«interface»

Descriptor

«interface»

ObjectProperty
Descriptor

«interface»

DataProperty
Descriptor

«interface»

Individual
Descriptor

«interface»

Class
Descriptor

«interface»

Ground

«interface»

Entity
Set

Class
Ground

Individual
Ground

ObjectProperty
Ground

DataProperty
Ground

$0..n$

$e$

«interface»

OWLOOP
Entity

«interface»

OWL

Entity

Java Packages (⋆it.emarolab.owloop)

${}^{\star}\\!$.descriptor.construction.descriptorExpression

${}^{\star}\\!$.descriptor.construction.descriptorGround

${}^{\star}\\!$.descriptor.construction.descriptorEntitySet

${}^{\star}\\!$.core

org.semanticweb.owl.model

${}^{\star}\\!$.descriptor.utility.classDescriptor

Figure 3: The general-purpose definition of OWLOOP Descriptor. Colours
identify OOP packages, which are also consistently used in Figures 4–6.

Figure 4 shows the implementation of the FullClassDescriptor, which is a
compound descriptor involving all the class expressions shown in the first row
of Table 3. Therefore it requires as EntitySet four sets of Classes, a set of
Individuals and a set of Restrictions. Remarkably, any compound descriptors
with a ClassGround that can be implemented with OWLOOP concerns a subset of
expressions involved in the FullClassDescriptor. Figures 5 and 6 respectively
show the implementation of the FullIndividualDescriptor, and the
FullObjectPropertyDescriptor, from which it is possible to derive the
implementation of the FullDataPropertyDescriptor. Indeed, since our mapping
exploits the same structure $D$ for all OWL axioms, OWLOOP always relies on
the same pattern to address different OWL expressions. Thus, OWLOOP is
modular, and this also facilitates the implementation of the OWL axioms not
considered in the current version, as discussed in Section 5.

$0..n$

$y$

$0..n$

$y$

$0..n$

$y$

$1$

$x$

$4$

$1$

$1$

«interface»

Class
Descriptor

Class
Ground

OWLOOP
Restriction

Classes

Restrictions

Individuals

«interface»

Sub

«interface»

Super

«interface»

Instance

«interface»

Equivalent
Restriction

«interface»

Equivalent

«interface»

Disjoint

FullClass
Descriptor

«interface»

Entity
Set

«interface»

OWL
Class

«interface»

OWLNamed
Individual

Figure 4: The implementation of the FullClassDescriptor.

###  2.3 Software Functionalities

A descriptor derived from Table 3 can be instantiated given ($i$) a ground,
_i_._e_., the identifier to an OWL entity (_e_._g_., “Corridor1”), and ($ii$)
the reference to an ontology as required by aMOR222OWLOOP is interfaced with
the ontology through the aMOR library, which wraps the OWL-API to provide
helper classes. We discuss the portability of aMOR in Section 4., _i_._e_., an
OntologyReference. For instance, Listing 1 shows at Line 1 a function to
create the ontology with an associated reasoner, which should be manually
invoked. As an example, Line 12 constructs a LinkIndividualDescr, which is a
compound descriptor grounded on Corridor1 and it is concerned with the
expressions $\mathcal{E}_{k}$ inherited from the descriptors addressing
ObjectLink and DataLink.

More generally, an OOP interface D that extends Descriptor for an expression
$\mathcal{E}_{k}$ inherits the OOP _methods_ addressed in the following
paragraphs. These functionalities are based on the _internal state_ of a
descriptor (_i_._e_., the ground $x$ and the entity set $Y_{k}$) that can be
synchronised with the ontology. A compound descriptor is an OOP object cd
realising some Descriptor interfaces, whose functionalities can be accessed
with cd.$\mathcal{E}_{k}$. For instance, the interfaces implemented by the
object cd returned at Line 12 are accessible with cd.ObjectLink or
cd.DataLink333In this paper, we slightly simplify the syntax. See the
${}^{\star}\\!$.articleExamples package for the runnable version of the
illustrated examples..

D.getGround(): This method returns the ground entity $x$. For a compound
descriptor, cd.$\mathcal{E}_{k}$.getGround() returns the same entity for all
the implemented expressions $\mathcal{E}_{k}$.

D.getEntities(): It returns the entity set $Y_{k}$ associated to the
$\mathcal{E}_{k}$ expression. To enable the definition of compound
descriptors, each descriptor D defines this method with a different name,
_e_._g_., cd.getObjects() for the ObjectLink expression, and
cd.getEquivalentIndividuals() for Equivalent associated to an individual
ground. These methods are used to access entities $y_{i}$, as well as
manipulate them, _e_._g_., through cd.removeObject().

⬇

1private OntologyReference createEmptyOntology() {

2 OntologyReference.activateAMORlogging(false); // Disable logs.

3 return OntologyReference.newOWLReferencesCreatedWithPellet(

4 "robotAtHomeOnto", // Ontology refeference name.

5 "src/test/resources/robotAtHomeOntology.owl", // File path.

6 "http://www.semanticweb.org/emaroLab/robotAtHomeOntology", // IRI.

7 true // Synchronize the Pellet reasoner manually.

8 );

9}

10private LinkIndividualDesc newCorridorDescriptorr() {

11 OWLReferences ontoRef = createEmptyOntology();

12 LinkIndividualDesc cd = LinkIndividualDesc("Corridor1", ontoRef);

13 return cd; // Compound descriptor concerning
$\mathcal{E}{:}\,\\{\text{{{ObjectLink, DataLink}}}\\}$.

14}

Listing 1:  Example showing the instantiation of an OWLOOP descriptor.

$1$

$x$

$2$

$1$

$1$

$1$

«interface»

Individual
Descriptor

Individual
Ground

FullIndividual
Descriptor

«interface»

Object
Link

«interface»

Data
Link

«interface»

Equivalent

«interface»

Disjoint

«interface»

Type

$0..n$

$y$

$0..n$

$y$

$0..n$

$y$

$0..n$

$y$

OWLOOP
Object

OWLOOP
Data

Individual

Classes

Object
Links

Data
Links

«interface»

Entity
Set

«interface»

OWLNamed
Individual

«interface»

OWL
Class

Figure 5: The implementation of the FullIndividualDescriptor.

D.query(): This method returns the knowledge related to the $k$-th expression
that involves the ground entity in the ontology. It returns a set structured
as $Y_{k}$ but it does not affect the internal state of the descriptor. It is
mainly used by the Descriptor interface itself.

D.readAxioms(): It relies on D.getEntities() and D.query() to compare the
internal state of the descriptor with the state of the ontology. It changes
the entity set of the descriptor to be equal to the ontology. It returns a
list of _intents_ containing all the performed changes, which can be used to
recover from possible inconsistencies. A compound descriptor provides the
method cd.readAxioms(), which invokes cd.$\mathcal{E}_{k}$.readAxioms() for
each $k$-th expression concerned by cd.

D.writeAxioms(): It is similar to readAxioms() but it changes the ontology to
become equal to the entity set.

$1$

$x$

$5$

$2$

$0..n$

$y$

$0..n$

$y$

«interface»

ObjectProperty
Descriptor

ObjectProperty
Ground

OWLOOP
Restriction

«interface»

Equivalent

«interface»

Sub

«interface»

Super

«interface»

OWLObject
Property

«interface»

Inverse

«interface»

Domain

«interface»

Range

«interface»

Disjoint

FullObject
PropertyDescriptor

«interface»

Entity
Set

Object
Properties

Restrictions

Figure 6: The implementation of the FullObjectPropertyDescriptor.

D.build(): It allows to use a descriptor in an OOP manner since it returns a
new compound descriptor nd grounded in each entity $y_{i}$. While designing
compound descriptors, developers should specify the type of nd with a ground
consistent with the elements in the entity set of D. Then, the entity set of
the new descriptors is populated through the nd.readAxioms() method. Similarly
to D.getEntities(), each descriptor D defines building methods with different
names to enable the definition of compound descriptors. Remarkably, if
entities $y_{i}$ are not OWL entities, the build() method would have multiple
definitions, _e_._g_., for an ObjectLink it is possible to return descriptors
nd grounded on an object property or on an individual.

There are also other useful functionalities implemented by aMOR, _e_._g_., to
get OWL entities or save the ontology. Moreover, the function
ontoRef.synchroniseReasoner() performs OWL reasoning, which involves all the
OWL expressions. Remarkably, OWL querying is a time-consuming task if
reasoning is required. Hence, the reasoning process should only be used when
required, _i_._e_., to affect the results of query() and, consequently, the
outcomes of readAxioms() and WriteAxioms(). In addition, to be compatible with
other software using ontologies, aMOR provides full access to factory-based
OWL to OOP mapping implemented with OWL-API.

##  3 Illustrative Examples

###  3.1 Asserting and Inferring OWL Axioms

Listing 2 shows how to use compound descriptors to manipulate an ontology by
adding and removing some of the axioms shown in Figure 2. At Line 5 the axiom
$\text{{{ObjectLink}}}(\text{{{isLinkedTo}}},\text{{{Corridor1}}},\text{{{Room1}}})$
is added to the relative entity set, and the same operation is performed at
Line 6 with a different type of input. At Line 7, the ontology changes such
that Corridor1 is linked to Room1 and Room2.

Lines 11–12 define axioms concerning disjoint classes, and at Line 13, the
ontology is changed to contain them, _i_._e_.,
$\text{{{Disjoint}}}(\text{{{ROBOT}}},\text{{{LOCATION}}})$ and
$\text{{{Disjoint}}}(\text{{{ROBOT}}},\text{{{DOOR}}})$. Lines 17–18 add
axioms to specify the domain and range of the hasDoor property, _i_._e_.,
$\text{{{Domain}}}(\text{{{hasDoor}}},\text{{{LOCATION}}})$ and
$\text{{{Range}}}(\text{{{hasDoor}}},\text{{{DOOR}}})$, which are applied to
the ontology at Line 19.

Line 21 updates the reasoner to infer new knowledge in the ontology, which is
then synchronised with the internal states of the descriptors at Lines 22–24.
By reading axioms, we can populate the entity set with implicit knowledge
required for implementing procedures that are based on queries. Finally, Line
26 shows a way to remove an element from the internal state of a descriptor,
and Line 27 applies those changes to the ontology.

⬇

1OWLReferences ontoRef = this.createEmptyOntology();

2// Assert OWL Individual Expression Axioms in the ontology.

3ObjectLinkIndividualDesc corrdor1 = //
$\mathcal{E}{:}\,\\{\text{{{ObjectLink, DataLink}}}\\}$.

4 new ObjectLinkIndividualDesc("Corridor1", ontoRef);

5corrdor1.addObject("isLinkedTo", "Room1"); // Add to ObjectLink EtitySet.

6corrdor1.addObject(new OWLOOPObject("isLinkedTo", "Room2"));

7corrdor1.writeAxioms(); // Synchronise changes to the ontology.

8// Add OWL Class Expression Axioms to the ontology.

9DisjointClassDesc robotClass = //
$\mathcal{E}{:}\,\\{\text{{{Disjoint}}}\\}$.

10 new RestrictionClassDesc("ROBOT", ontoRef);

11robotClass.addDisjointClass("LOCATION"); // Add to Disjoint EntitySet.

12robotClass.addDisjointClass(ontoRef.getOWLClass("DOOR"));

13corrdor1.writeAxioms(); // Synchronise changes to the ontology.

14// Add OWL ObjectProperty Expression Axioms to the ontology.

15DomainRangeObjectPropertyDesc hasDoor = //
$\mathcal{E}{:}\,\\{\text{{{Domain, Range}}}\\}$.

16 new DomainRangeObjectPropertyDesc("hasDoor", ontoRef);

17hasDoor.addDomainClassRestriction("LOCATION"); // Add to Domain EntitySet.

18hasDoor.addRangeClassRestriction(new OWLOOPRestriction("DOOR"));

19corrdor1.writeAxioms(); // Synchronise changes to the ontology.

20// Synchronise changes to the entity sets based on the ontology.

21ontoRef.synchroniseReasoner(); // Invoke OWL reasoning.

22corrdor1.readAxioms();

23hasDoor.readAxioms();

24robotClass.readAxioms();

25// Remove an axiom from the internal state of a descriptor.

26corridor1.removeObject("isLinkedTo"));

27corridor1.writeAxioms(); // Synchronise changes to the ontology.

Listing 2:  Example showing how to manipulate axioms through descriptors.

⬇

14// Load the ontology shown in Figure 2 similarly to Line 1.

15OWLReferences ontoRef = this.loadOntology();

16// Retrieve the robot location (robotLoc) by assuming that it is unique.

17LinkIndividualDesc robot1 = new LinkIndividualDesc("Robot1",ontoRef);

18robot1.addObject("isIn", true); // Set to read only the isIn property once.

19robot1.readAxioms();

20OWLNamedIndividual robotLoc = robot1.getIndividualFromObjectProp("isIn");

21// Ground an individual descriptor on robotLoc concerning
$\mathcal{E}{:}\,\\{\text{{{Type}}}\\}$.

22TypeIndividualDesc locIndiv = new TypeIndividualDesc(robotLoc, ontoRef);

23locIndiv.readAxioms(); // read the classes of robotLoc.

24// Build descriptors grounded on the classes of robotLoc.

25Set<SubclassDesc> locClasses = locIndiv.buildTypes(); //
$\mathcal{E}{:}\,\\{\text{{{Sub, Super}}}\\}$

26for(SubclassDesc locClass : locClasses)

27 // A class of robotLoc that only subsumes owl:NOTHING is a leaf.

28 if(locClass.getdsubClasses().size() == 1)

29 // Print, _e_._g_., "Robot1 is in Corridor1, which is a CORRIDOR".

30 System.out.println(robot1.getGround() + " is in " + robotLoc

31 + ", which is a " + locClass.getGround());

Listing 3:  Example showing the usage of the descriptor build method.

###  3.2 Accessing OWL Axioms with the OOP Paradigm

The example in Listing 3 has the objective of finding the type of room where
the robot is located, _e_._g_., CORRIDOR. The example shows the retrieval of
the individual related to the isIn property, _e_._g_., Corridor1, that we
store in a variable named robotLoc. Then, it shows how to find the OWL classes
that classify robotLoc such that they are a leaf in the TBox. Within Lines
17–20 the robot location is retrieved by assuming in the ontology that only an
isIn axiom involving Robot1 exists. Lines 22–23 initialise a descriptor
grounded on robotLoc that maps OWL classes concerning the Type expression.
Line 25 builds the Type of robotLoc, _i_._e_., it computes a set of
descriptors that are grounded in a class representing the robot location
(_e_._g_., CORRIDOR, LOCATION and THING), and involving Sub expressions. For
each class of robotLoc, Line 28 checks whether it has only one sub-class,
_i_._e_., NOTHING, which implies that it is a leaf class in the ontology.

The latter example shows that OWLOOP allows to get OWL classes and sub-classes
in an OOP fashion, and a similar approach can also be used for the other
expressions. In particular, at Lines 25–30, we iterate for each individual to
retrieve their class and related sub-classes without using the OWL factory but
only relying on the OOP-based internal states of the descriptors. This is
possible thanks to the build() method since it can be used to relate entity
sets of different descriptors and, as an example, Lines 20–23 show a possible
implementation of it. However, to retrieve locIndiv from robot1 directly
through building, the descriptor LinkIndividualDescr should have been designed
to build new individual descriptors concerning the Type expression, while it
builds descriptors concerning ObjectLink and DataLink expressions, _i_._e_.,
other LinkIndividualDescr.

A simplistic approach could be to always build _full_ descriptors for
accessing all possible expressions from each build() method, but this might
impact computation complexity since OWLOOP would synchronise axioms that might
be unnecessary.

###  3.3 Compound Descriptor Implementation

In order to highlight the modularity of OWLOOP, Listing 4 shows the
implementation of a ClassDescriptor that inherits the functionalities listed
in Section 2.3 for two types of expressions, _i_._e_., Instance and Sub.
Indeed, for all the Descriptors based on Table 3, the same pattern can be used
to implement OOP classes encoding OWL knowledge, and that are related among
each other based on the ontology, _e_._g_., it is possible to _get_ or _set_
the individuals or the sub-classes of the ground class in an OOP-like manner.
In particular, the get or set entities are not encoded in immutable classes as
in the case of the factory design pattern but in OWLOOP descriptors, which can
be related to other descriptors through the building methods.

In addition, a developer can exploit OOP-based _polymorphism_ to implement a
hierarchy of OWLOOP descriptors, which encode string-based identifiers of OWL
symbols and procedures to retrieve implicit knowledge based on OWL reasoning
in a modular way. Hence, experts on OWL formalism can build an abstract layer
concerned with OWL-related aspects by the means of descriptors, which can be
provided to experts in software architecture who aim to use an ontology within
an OOP-like paradigm.

For example, Listing 3 could be implemented as a method of the descriptor in
Listing 4. In this way, experts in software architectures may rely on an OOP-
like method to retrieve the robot position and, at the same time, OWL-
formalism experts may exploit a modular interface to the ontology for
representing knowledge about the robot position.

⬇

27public class CDescr extends ClassGround

28 implements ClassExpression.Sub<CDescr>,

29 ClassExpression.Instance<IDescr> {

30

31 private Classes subclasses = new Classes();

32 private Individuals individuals = new Individuals();

33

34 public CDescr(OWLClass instance, OWLReferences onto) {

35 super(instance, onto); // A constructor based on ClassGround.

36 }

37 @Override // Override from ClassGround.

38 public List<MappingIntent> readAxioms() { // Write in the ontology.

39 List<MappingIntent> r = Sub.super.readAxioms();

40 r.addAll(Instance.super.readAxioms());

41 return r;

42 }

43 @Override // Override from ClassGround.

44 public List<MappingIntent> writeAxioms() { // Read from the ontology.

45 List<MappingIntent> r = Sub.super.writeAxioms();

46 r.addAll(Instance.super.writeAxioms());

47 return r;

48 }

49

50 @Override // Override from the Sub expression.

51 public Classes getsubclasses() {

52 return subclasses; // A factory-based getter of sub-classes names.

53 }

54 @Override // Override from Sub, and used in the Sub.build() method.

55 public CDescr getsubclassDescriptor(OWLClass cls, OWLReferences onto){

56 // Implements an OOP-like getter of sub-classe descriptors.

57 return new CDescr(cls, onto);

58 }

59

60 @Override // Override from the Instance expression.

61 public Individuals getIndividuals() {

62 return individuals; // A factory-based getter of instance names.

63 }

64 @Override // Override from Instance, and used in Instance.build().

65 public IDescr getIndividualDescriptor(OWLNamedIndividual instance,

66 OWLReferences onto) {

67 // Implements an OOP-like getter of instance descriptors.

68 return new IDescr(instance, onto);

69 }

70}

Listing 4:  The implementation of CDescr: a ClassDescriptor that covers Sub
and Instance expressions. CDescr is designed to build other instances of
CDescr for each OWL class related to the Sub expression, and a IDescr for each
OWL individuals related to the Instance expression.

##  4 Impact

OWLOOP enables the implementation of modular OOP hierarchies of descriptors
that represent specific knowledge in the ontology. Descriptors can encode both
string-based identifiers of OWL symbols, _i_._e_., immutable OOP objects, and
procedures to exploit implicit knowledge based on polymorphism and OWL
reasoning. Since descriptors can build other descriptors such that they are
related to each other based on the ontology, it is possible to exploit OWL-
based knowledge through OOP objects that are mutable. Moreover, the process of
adapting a system when its ontology changes is simplified with OWLOOP because
all OWL-related aspects are encapsulated in the descriptors rather than being
defined by software components. Hence, OWLOOP allows to decouple the OWL-based
representation of knowledge and its usage within a software system.

OWLOOP is based on aMOR, which is a library providing utilities to use OWL-
API. Since aMOR does not hide the access to OWL-API, developers can keep using
any procedure based on OWL-API. However, other OWL-based APIs (_e_._g_., Jena)
enable working with different ontology formats, which are currently not
compatible with OWLOOP. Nevertheless, there is no reason for which our OWL to
OOP mapping cannot be implemented with other APIs that encompass the OWL
formalism. In addition, other available tools can be used to design an
ontology that can be later interfaced with the components of software
architectures using OWLOOP descriptors. Indeed, within the OWL domain, there
are no particular features that OWLOOP tackles but other tools do not.
Nonetheless, in the domain of software design, development and maintenance,
OWLOOP has features that are not directly provided by other OWL-related APIs,
especially when reasoning is concerned. In particular, OWLOOP descriptors can
inherit and extend procedures to evaluate axioms in an ontology by hard-coding
only a few string-based identifiers of OWL symbols. Also, through the building
methods, descriptors can access a piece of knowledge without explicitly using
any farther axioms.

Ontologies have been used in a large range of applications, _e_._g_.,
concerning semantic web [15], data access [16], information retrieval [17],
medical knowledge representation [18], internet of things [19], robotics [20],
and ambient assisted living [21]. Usually, OWL ontologies are deployed in
software architectures by assuming that they only contain knowledge that does
not change over time. Other systems exploit ontology structures where
individuals are dynamic, while classes and properties are static. However,
some applications also require changing OWL classes over time, _e_._g_., to
learn scene classification from images [22]. In that work, which has been
deployed in a software architecture to address human supervision [23], and to
classify scenes in a long-term interaction [24], we observed that OWLOOP
allows to develop a more modular architecture than OWL-API since it can
decouple the procedures related to the knowledge representation and its
deployment. For complex systems involving a robot, OWLOOP has been
integrated444A scenes learner implemented with OWLOOP and injected in the
aRMOR service can be found at https://github.com/TheEngineRoom-
UniGe/ARMOR_OWLOOP. in the aRMOR service [25], which allows to use ontologies
in the Robot Operating System (ROS). Furthermore, by investigating the problem
of human activity recognition [26], we observed that OWLOOP is efficient for
system prototyping and evaluation of activity models designed by domain
experts.

##  5 Conclusions and Future Work

OWLOOP API performs a passive OWL to OOP mapping that allows to use knowledge
inferred from OWL reasoners within the OOP paradigm. We presented OWLOOP
descriptors, which are modular Java classes interfacing a hierarchy of OOP
objects with the knowledge structured in memory as an ontology. Descriptors
enable designing general-purpose interfaces between ontologies and components
of software architectures such that drawbacks in the computation are avoided.
In addition, they encapsulate boilerplate code to simplify the development and
maintenance of an architecture. OWLOOP is compatible with other tools to
design ontologies, and it is suitable for integrating dynamic ontologies
within complex software architectures.

The current version of OWLOOP concerns only the expressions shown in Table 3,
which do not encompass the whole OWL axioms. For instance, it cannot represent
classes through structured disjunctions and conjunctions of class restrictions
as defined by OWL. Instead, it considers all the restrictions to be in
conjunction without a specific order. This is not a limitation for all the
applications where the definition of OWL classes do not change dynamically,
because the ontology can be created with other tools and be interfaced with
OWLOOP afterwords. Although there are no reasons for which our OWL to OOP
mapping cannot consider all OWL axioms, we present this preliminary
implementation to propose our approach. Nevertheless, in the future, we want
to modularly extend OWLOOP to support all OWL axioms by designing new types of
OWLOOP entities and related descriptors.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## References

  * [1] D. L. McGuinness, F. Van Harmelen, OWL Web Ontology Language Overview, W3C Recommendation 10 (2004).
  * [2] H. Wache, T. Voegele, U. Visser, H. Stuckenschmidt, G. Schuster, H. Neumann, S. Hübner, Ontology-based integration of information – A survey of existing approaches, in: Proceedings of Ontologies and Information Sharing at the International Joint Conferences on Artificial Intelligence (IJCAI), Seattle, Washington, USA, 2001, pp. 108–117.
  * [3] H. Knublauch, D. Oberle, P. Tetlow, E. Wallace, A Semantic Web Primer for Object-Oriented Software Developers, W3C Working Group Note, W3C (2006).
  * [4] S. S. Baset, K. Stoffel, Object-oriented Software Modeling with Ontologies Around - A Survey of Existing Approaches, in: Proceedings of the 30th International Conference on Software Engineering and Knowledge Engineering (SEKE), San Francisco, California, USA, 2018, pp. 29–45.
  * [5] G. Stevenson, S. Dobson, Sapphire: Generating java runtime artefacts from owl ontologies, in: Proceedings of the 23rd International Conference on Advanced Information Systems Engineering (CAiSE), Springer, London, UK, 2011, pp. 425–436.
  * [6] S. Baset, K. Stoffel, Ontojit: Parsing native OWL DL into executable ontologies in an object oriented paradigm, in: Proceedings of the 13th Internaltional Workshop on OWL: Experiences and Directions (OWLED) and 5th International Workshop on Reasoner Evaluation (ORE), Springer, Bologna, Italy, 2016, pp. 1–14.
  * [7] J.-B. Lamy, Owlready: Ontology-oriented programming in Python with automatic classification and high level constructs for biomedical ontologies, Artificial Intelligence in Medicine 80 (2017) 11–28.
  * [8] B. Glimm, I. Horrocks, B. Motik, G. Stoilos, Z. Wang, Hermit: an OWL 2 reasoner, Journal of Automated Reasoning 53 (3) (2014) 245–269.
  * [9] E. Sirin, B. Parsia, B. C. Grau, A. Kalyanpur, Y. Katz, Pellet: A practical OWL-DL reasoner, Web Semantics: science, services and agents on the World Wide Web 5 (2) (2007) 51–53.
  * [10] M. Horridge, S. Bechhofer, The OWL API: A Java API for OWL ontologies, Semantic Web 2 (1) (2011) 11–21.
  * [11] J. J. Carroll, I. Dickinson, C. Dollin, D. Reynolds, A. Seaborne, K. Wilkinson, Jena: Implementing the semantic web recommendations, in: Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters, Association for Computing Machinery, New York, NY, USA, 2004, p. 74–83.  doi:10.1145/1013367.1013381.
  * [12] M. A. Musen, The protégé project: a look back and a look forward, AI matters 1 (4) (2015) 4–12.
  * [13] B. Motik, P. F. Patel-Schneider, B. Parsia, C. Bock, A. Fokoue, P. Haase, R. Hoekstra, I. Horrocks, A. Ruttenberg, U. Sattler, et al., OWL 2 Web Ontology Language Structural Specification and Functional-Style Syntax, Tech. rep., W3C recommendation (2009).
  * [14] B. Motik, P. F. Patel–Schneider, B. C. Grau, OWL 2 Web Ontology Language Direct Semantics (Second Edition), Tech. rep., W3C recommendation (2012).
  * [15] T. Berners-Lee, J. Hendler, O. Lassila, A new form of web content that is meaningful to computers will unleash a revolution of new possibilities, Scientific American (2002).
  * [16] G. Xiao, D. Calvanese, R. Kontchakov, D. Lembo, A. Poggi, R. Rosati, M. Zakharyaschev, Ontology-based data access: A survey, in: Proceedings of the 27th International Joint Conferences on Artificial Intelligence (IJCAI), Stockholm, Sweden, 2018, pp. 5511–5519.
  * [17] M. Fernández, I. Cantador, V. López, D. Vallet, P. Castells, E. Motta, Semantically enhanced information retrieval: An ontology-based approach, Web semantics: Science, Services and Agents on the World Wide Web 9 (4) (2011) 434–452.
  * [18] K. W. Fung, O. Bodenreider, Knowledge representation and ontologies, in: Clinical Research Informatics, Springer, 2019, pp. 313–339.
  * [19] J. Gubbi, R. Buyya, S. Marusic, M. Palaniswami, Internet of Things (IoT): A vision, architectural elements, and future directions, Future Generation Computer Systems 29 (7) (2013) 1645–1660.
  * [20] J. I. Olszewska, M. Barreto, J. Bermejo-Alonso, J. Carbonera, A. Chibani, S. Fiorini, P. Goncalves, M. Habib, A. Khamis, A. Olivares, et al., Ontology for autonomous robotics, in: proceedings of 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Lisbon, Portugal, 2017, pp. 189–194.
  * [21] J. Rafferty, C. D. Nugent, J. Liu, L. Chen, From activity recognition to intention recognition for assisted living within smart homes, IEEE Transactions on Human-Machine Systems 47 (3) (2017) 368–379.
  * [22] L. Buoncompagni, F. Mastrogiovanni, Teaching a robot how to spatially arrange objects: Representation and recognition issues, in: Proceedings of the 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), New Delhi, India, 2019, pp. 1–8.
  * [23] L. Buoncompagni, F. Mastrogiovanni, Dialogue-based supervision and explanation of robot spatial beliefs: a software architecture perspective, in: Proceedings of the 27th IEEE International Symposium on Robot and Human Interactive Communication (RO–MAN), Nanjing, China, 2018, pp. 977–984.
  * [24] L. Buoncompagni, F. Mastrogiovanni, A framework inspired by cognitive memory to learn planning domains from demonstrations, in: Proceedings of the 6th Italian Workshop on Artificial Intelligence and Robotics. A workshop of the XVIII International Conference of the Italian Association for Artificial Intelligence, Vol. 2594, CEUR-WS, Rende, Italy, 2019, pp. 13–18.
  * [25] L. Buoncompagni, A. Capitanelli, F. Mastrogiovanni, A ROS multi–ontology references service: OWL reasoners and application prototyping issues, in: Proceedings of the 5th Italian Workshop on Artificial Intelligence and Robotics. A workshop of the XVII International Conference of the Italian Association for Artificial Intelligence, Vol. 2352, CEUR-WS, Trento, Italy, 2018, pp. 36–41.
  * [26] S. Y. Kareem, L. Buoncompagni, F. Mastrogiovanni, Arianna+: Scalable human activity recognition by reasoning with a network of ontologies, in: proceedings of the 17th International Conference of the Italian Association for Artificial Intelligence, Springer International Publishing, Trento, Italy, 2018, pp. 83–95.


# PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise
Training

Dawei Zhu †‡ Nan Yang ‡ Liang Wang ‡ Yifan Song † Wenhao Wu †
Furu Wei ‡ Sujian Li † 🖂
† School of Computer Science, Peking University
‡ Microsoft Corporation
https://github.com/dwzhu-pku/PoSE Work done during internship at MSRA.

###### Abstract

In this paper, we introduce Positional Skip-wisE (PoSE) training for efficient
adaptation of large language models (LLMs) to extremely long context windows.
PoSE decouples train length from target context window size by simulating long
inputs using a fixed context window with manipulated position indices during
training. Concretely, we select several short chunks from a long input
sequence, and introduce distinct skipping bias terms to modify the position
indices of each chunk. These bias terms, along with the length of each chunk,
are altered for each training example, allowing the model to adapt to all
positions within the target context window without training on full length
inputs. Experiments show that, compared with fine-tuning on the full length,
PoSE greatly reduces memory and time overhead with minimal impact on
performance. Leveraging this advantage, we have successfully extended the
LLaMA model to 128k tokens. Furthermore, we empirically confirm that PoSE is
compatible with all RoPE-based LLMs and various position interpolation
strategies. Notably, by decoupling fine-tuning length from target context
window, PoSE can theoretically extend the context window infinitely,
constrained only by memory usage for inference. With ongoing advancements for
efficient inference, we believe PoSE holds great promise for scaling the
context window even further.

##  1 Introduction

Large Language Models (LLMs) have revolutionized language modeling and
demonstrated impressive abilities to perform various tasks [2]. However, even
with their remarkable capacity, these LLMs remain restricted by pre-defined
context window sizes, suffering from notable performance decline when input
tokens exceeds these limits. Nevertheless, numerous application scenarios
demand extremely long input sequences, including long document summarization
[13], in-context learning with numerous examples [18], and long document
retrieval [38], etc. This naturally poses a significant challenge of context
window extension: Extending the context window of a pre-trained LLM to
accommodate longer sequences.

Naively extending pre-trained models to longer context windows by fine-tuning
on longer inputs has received limited success due to the large disruption
introduced by the new position indices [4]. In response to this challenge,
various Position Interpolation (PI) strategies have been proposed [4, 22],
demonstrating improved context window extension performance when fine-tuned on
sequences of the target length. Nevertheless, even with the use of position
interpolation, fine-tuning LLMs on sequences of the target length, referred to
as Full-length fine-tuning in this paper, remains both memory and time-
intensive. Moreover, it becomes impractical when the target context window
size is extremely large, due to the computational complexity that increases
quadratically with input length.

In this paper, we introduce Positional Skip-wisE (PoSE) fine-tuning to
decouple the fine-tuning length from the target context window length,
unleashing the possibility of efficiently extending context window to an
extreme size. The key idea of PoSE is to simulate long inputs by manipulating
position indices within a fixed context window. As depicted in Figure 1, we
partition the original context window into several chunks, and adjust the
position indices of each chunk by adding a distinct skipping bias term. These
bias terms, as well as the length of each chunk, are altered for each training
example, so that the model can adapt to all positions (including both absolute
and relative) within the target context window through fine-tuning. Meanwhile,
by maintaining continuous position indices within each chunk, PoSE bears a
close resemblance to pre-training. As a result, the model’s pre-trained
capacity for language modeling and comprehension is retained to the greatest
degree.

The advantages of our PoSE are threefold: 1) Memory and Time Efficiency: By
only requiring the original context size for fine-tuning, PoSE circumvents the
quadratic increase in computational complexity with respect to target length
during the fine-tuning stage, thereby significantly reducing memory and time
overhead. 2) Potential for Extremely-Long Context: We manage to extend the
context window of LLaMA [31] by up to 64 times (2k$\xrightarrow{}$128k,
k=1,024) while preserving decent ability of language modeling and
understanding. 3) Compatible with all RoPE-based LLMs and PI strategies: The
effectiveness of PoSE has been empirically validated across several RoPE-based
LLMs, including LLaMA, LLaMA2 [32], and GPT-J [33]. Additionally, PoSE has
been demonstrated to be compatible with a variety of position interpolation
methods, such as linear [4], Neural Tangent Kernel (NTK) [21], and YaRN [22]
interpolation.

Notably, by decoupling the fine-tuning length and the target context window
length, PoSE can theoretically extend context window to an infinite length.
The only constraint is the memory usage during the inference phase. Hopefully,
with the continuous advancements in efficient inference techniques, including
Flash Attention [9, 8], xFormers [17], vLLM [16], etc, we believe PoSE can
promisingly push the context window size to a even larger scale.

Figure 1: Position indices of Full-length fine-tuning v.s. PoSE fine-tuning
for extending the context window size from 2,048 to 4,096. At each iteration,
the former directly takes 4,096 tokens for fine-tuning, while PoSE manipulates
the position indices of 2,048 tokens to simulate longer inputs. For example,
we partition the original context window of 2,048 tokens into two chunks, and
adjust the position indices of the second chunk by adding a distinct skipping
bias term. These bias terms, as well as the length of each chunk, are altered
for each training example, so that the model can adapt to all relative
positions of the target context window through fine-tuning.

##  2 Related Work

###  2.1 Length Extrapolation

Length extrapolation aims to ensure that the model continues to perform well,
even when the number of input tokens during inference exceeds the size of the
context window on which the model is trained [23]. To this end, a series of
positional embedding schemes have been proposed, including ALibi [23], xPos
[30], NoPos [12], etc.

Similar to our work, Ruoss et al. [28] also attempted to simulate longer
sequences during training time to mitigate out-of-distribution lengths. They
proposed randomized positional encoding (RandPos), which randomly selected an
ordered subset of position indices from longer sequences. Our proposed method,
PoSE, diverges from their approach in several key aspects: First, RandPos is a
positional embedding scheme designed for pre-training encoder-only models from
scratch to enhance length generalization ability. In contrast, PoSE is a fine-
tuning method that aims to efficiently extend the context window of pre-
trained LLMs, the majority of which follow a decoder-only architecture.
Second, in RandPos, the position indices between adjacent tokens are not
continuous. However, in PoSE, the position indices within each chunk are
intentionally made continuous to closely resemble the pre-training phase,
therefore reducing the risk of disrupting the abilities learned during the
pre-training stage.

###  2.2 Context Window Extension

Differing from length extrapolation, which primarily involves training a model
from scratch to support lengths exceeding those it was initially trained for,
context window extension focuses on extending the context window of a pre-
trained LLM. Directly fine-tune an existing pre-trained Transformer with a
longer context window has been shown to progress slowly [4]. To expedite and
stabilize training, Chen et al. [4] first down-scaled position indices to
match original context size through linear position interpolation, followed by
full-length fine-tuning. Subsequently, a range of positional interpolation
strategies have been introduced, including NTK [21] and YaRN [22]. While also
leveraging position interpolation, our method managed to simulate long inputs
by manipulating position indices, requiring only the original context size for
fine-tuning.

###  2.3 Memory Transformers

An alternative strategy for managing exceedingly lengthy input sequences
involves the implementation of memory mechanisms [7, 3, 35, 34, 20].
Generally, there are two predominant techniques for exploiting memory:
recurrence-based and retrieval-based methodologies. With respect to
recurrence-based approaches, the Transformer-XL [7] processes inputs within
fixed-length windows, allowing each token to attend not only to tokens within
the current window but also those within the preceding window. The RMT [3]
incorporates dedicated memory tokens at the beginning of the input sequence
for the purpose of encoding and transmitting memory. However, a significant
limitation of this approach is its reliance on a compressed memory format,
which may result in the loss of information, and its restricted capacity for
random access.

In the retrieval-based paradigm, Wu et al. [35] encode previous sequences as
(key, value) pairs. These pairs are subsequently retrieved through kNN lookup.
LongMem [34] further advances this approach by incorporating a frozen Language
Model as a memory encoder. This is supplemented by an adaptive residual side-
network, which operates as both a memory retriever and reader. The major
constraint of this approach is the lack of interaction between distinct memory
chunks.

More recently, Mohtashami and Jaggi [20] introduced landmark attention, which
facilitates random access to any chunk of the input by introducing landmark
tokens. In contrast, our method achieves full access to the entire input
without requiring any modifications to the attention mechanism.

##  3 Methodology

###  3.1 Preliminaries

#### Rotary Position Embedding (RoPE).

The use of RoPE [29] has become pervasive in contemporary LLMs, including
LLaMA [31], GPT-J [33], etc. It encodes position information of tokens with a
rotation matrix that naturally incorporates explicit relative position
dependency. To elucidate, given a hidden vector
${\boldsymbol{h}}=[h_{0},h_{1},...,h_{d-1}]$, where $d$ is the hidden
dimension, and a position index $m$, RoPE operates as follows:

$f({\boldsymbol{h}},m)=\begin{pmatrix}h_{0}\\\ h_{1}\\\ h_{2}\\\ h_{3}\\\
\vdots\\\ h_{d-2}\\\
h_{d-1}\end{pmatrix}\otimes\begin{pmatrix}\cos{m\theta_{0}}\\\
\cos{m\theta_{0}}\\\ \cos{m\theta_{1}}\\\ \cos{m\theta_{1}}\\\ \vdots\\\
\cos{m\theta_{d/2-1}}\\\
\cos{m\theta_{d/2-1}}\end{pmatrix}+\begin{pmatrix}-h_{1}\\\ h_{0}\\\ -h_{3}\\\
h_{2}\\\ \vdots\\\ -h_{d-1}\\\
h_{d-2}\end{pmatrix}\otimes\begin{pmatrix}\sin{m\theta_{0}}\\\
\sin{m\theta_{0}}\\\ \sin{m\theta_{1}}\\\ \sin{m\theta_{1}}\\\ \vdots\\\
\sin{m\theta_{d/2-1}}\\\ \sin{m\theta_{d/2-1}}\end{pmatrix}$ (1)

where $\theta_{j}=10000^{-2j/d},j\in\\{0,1,...,d/2-1\\}$. Unlike previous
absolute position encodings that are directly applied to the input vector
${\boldsymbol{x}}$, RoPE is employed on the query and key vectors at each
layer. Considering a query vector ${\boldsymbol{q}}$ at position $m$ and a key
vector ${\boldsymbol{k}}$ at position $n$, the attention score
$a({\boldsymbol{q}},{\boldsymbol{k}})$ is defined as follows:

$\displaystyle a({\boldsymbol{q}},{\boldsymbol{k}})$
$\displaystyle=<f({\boldsymbol{q}},m),f({\boldsymbol{k}},n)>$
$\displaystyle=\sum_{j=0}^{d/2-1}[(q_{2j}k_{2j}+q_{2j+1}k_{2j+1})\cos{(m-n)\theta_{j}}+(q_{2j}k_{2j+1}-q_{2j+1}k_{2j})\sin{(m-n)\theta_{j}}]$
$\displaystyle:=g({\boldsymbol{q}},{\boldsymbol{k}},\boldsymbol{\theta},m-n)$
(2)

Hence, RoPE encodes position information in a relative manner, as the
attention score depends on the relative distances between positions rather
than their absolute position values.

#### Problem Formulation.

Given a Large Language Model pre-trained with a context window size of
$L_{c}$, our objective is to extend this context size to a target length
$L_{t}$, so that the model maintains good performance when processing input
sequences containing a maximum of $L_{t}$ tokens.

#### Position Interpolation (PI).

In contrast to directly extending the position indices to $L_{t}-1$ when
dealing with an input text ${\boldsymbol{x}}=\\{x_{0},x_{1},...,x_{L_{t}}\\}$,
position interpolation down-scales the position indices to align with the
original context window size $L_{c}$. This approach effectively mitigates the
risk of encountering extreme values and has been empirically demonstrated to
enhance stability during fine-tuning. Various interpolation strategies have
been proposed, with $\alpha=L_{t}/L_{c}$ denoting the scaling factor:

  * •

Linear Interpolation. As described by Chen et al. [4] and kaiokendev [14],
linear interpolation involves a proportional down-scaling of the position
index $m$ to $m/\alpha$. Consequently, the attention score between a query
${\boldsymbol{q}}$ at position $m$ and a key ${\boldsymbol{k}}$ at position
$n$ becomes
$g({\boldsymbol{q}},{\boldsymbol{k}},\boldsymbol{\theta},(m-n)/\alpha)$, as
defined in Equation 2. Theoretical analysis has substantiated that the
interpolated attention score exhibits significantly greater stability compared
to the extrapolated counterpart.

  * •

Neural Tangent Kernel (NTK) Interpolation. In contrast to linear
interpolation, NTK Interpolation alters the base of RoPE, effectively
modifying the rotational "speed" of each dimension of RoPE [21]. Specifically,
the original $\theta_{j}=10000^{-2j/d},j\in\\{0,1,...,d/2-1\\}$ in RoPE is
transformed into $\theta^{\prime}_{j}=(10000\lambda)^{-2j/d}$, where
$\lambda=\alpha^{d/d-2}$. It is noteworthy that the value of $\lambda$ is
chosen to ensure that $m\theta^{\prime}_{d/2-1}=(m/\alpha)\theta_{d/2-1}$.

  * •

YaRN Interpolation. Different from Linear and NTK interpolation that treat
each dimension of RoPE equally, YaRN [22] employs a ramp function to combine
Linear and NTK interpolation at varying proportions across different
dimensions. Simultaneously, it introduces a temperature factor to mitigate
distribution shift of attention matrix caused by long inputs.

###  3.2 Proposed Approach: Positional Skip-wise Training (PoSE)

Although position interpolation effectively addresses out-of-distribution
position indices, extending to an extreme length by fine-tuning on context
window of this size remains impractical, owing to the quadratic growth in
computational complexity of attention as sequence length increases. Instead,
we explore to train within the original context window $L_{c}$ and achieve
context window extension via manipulating position indices to simulate longer
inputs.

There are two designing desiderata for this endeavor: First, to avoid out-of-
distribution positions during inference, the relative distance of manipulated
position indices should comprehensively cover the range of
$\\{0,1,\ldots,L_{t}-1\\}$. Second, fine-tuning with the manipulated position
indices should not harm the original abilities of LLMs, so the structure of
manipulated position indices should closely adhere to the original structure
to the greatest extent possible.

Initially, we divide the original context window $L_{c}$ into $N$ chunks
$c_{0},c_{1},\ldots,c_{N-1}$, each with lengths $l_{0},l_{1},\ldots,l_{N-1}$,
where $\sum_{i=0}^{N-1}l_{i}=L_{c}$. We introduce the starting index $st_{i}$
for each chunk $c_{i}$, which facilitates the formulation of its position
indices as follows:

$\text{Pos}(c_{i})=\\{st_{i},st_{i}+1,\ldots,st_{i}+l_{i}-1\\},\quad
st_{i}=\sum_{j=0}^{i-1}l_{j}$ (3)

Subsequently, we employ the discrete uniform distribution $\mathcal{U}(S)$ to
sample a skipping bias term
$u_{i}\sim\mathcal{U}(\\{u_{i-1},\ldots,L_{t}-\sum_{j=i}^{N}l_{j}\\})$ for
each chunk $c_{i}$. This bias term is applied to the corresponding chunk to
transform the original position indices into:

$\text{PoSE{}}(c_{i})=\\{u_{i}+st_{i},u_{i}+st_{i}+1,\ldots,u_{i}+st_{i}+l_{i}-1\\}$
(4)

Note that the constraint of $u_{i}>u_{i-1}$ is applied to prevent position
index overlaps between chunks.

Intuitively, the introduction of skipping bias terms exposes model to a more
diverse range of relative positions. It is essential to highlight that, at
each iteration, we dynamically adjust both the length and skipping bias term
for each chunk to achieve comprehensive coverage of the target context window.
Moreover, the continuity of position indices within each chunk closely
resembles the structure employed during pre-training. Consequently, fine-
tuning the model on these new position indices for language modeling does not
compromise its original capabilities.

Concerning the text contained within each chunk, a similar procedure is
followed to select continuous spans of tokens from the input text
${\boldsymbol{x}}=\\{x_{0},x_{1},...\\}$. To elaborate, we begin by sampling a
bias term
$v_{i}\sim\mathcal{U}(\\{v_{i-1},\ldots,L_{t}-\sum_{j=i}^{N}l_{j}\\})$
followed by assigning the content of chunk $c_{i}$ as below:

$c_{i}={\boldsymbol{x}}[v_{i}+st_{i}:v_{i}+st_{i}+l_{i}]$ (5)

It is noteworthy that we have conducted experiments under various settings,
including scenarios where $v_{i}=0$, which results in genuinely continuous
content for the chunks, or $v_{i}=u_{i}$, aligning the manipulated position
indices with actual positions in the original text. However, we observe that
these variations have relatively little impact on the outcomes of fine-tuning.

After position indices for each chunk is settled, we perform position
interpolation for stabilized fine-tuning of language modeling. For simplicity,
we set the initial bias terms $u_{0}$ and $v_{0}$ to 0. Number of chunks
explored in this paper is $N=2$.

##  4 Experiments

In this section, we conduct experiments to verify the effectiveness of PoSE
for context window extension. Our method demonstrates impressive results on
context lengths of both 16k and 32k for language modeling as well as passkey
retrieval. Other advantages of PoSE are elaborated upon in Section 5.

###  4.1 Setups

#### Training Procedure.

For each setting in the main experiments, we train LLaMA-7B with the next
token prediction objective. This training process comprises 1,000 steps,
employing a global batch size of 64 on 8 V100 GPUs using Deepspeed ZeRO stage
3 [27]. The fine-tuning dataset is sourced from The Pile [11], with a minimum
length requirement of 2,048 tokens. Our default choice for interpolation
strategies is linear interpolation. During the inference phase, we employ
Flash Attention V2 [8] to enable the model to process extremely long
documents, accommodating up to 128k tokens (k=1,024) on a single A100 GPU.

#### Evaluation Tasks and Datasets.

We examine the ability of long text modeling on two tasks: language modeling
and passkey retrieval. The language modeling task is a fundamental task that
reflects the overall capability of a model in handling long text. Passkey
retrieval, on the other hand, can effectively measure the maximum distance
that a token can attend to during the inference stage. For language modeling,
we use two datasets, GovReport [13] and Proof-pile [37]. For passkey
retrieval, we follow Mohtashami and Jaggi [20] to construct synthetic prompts
for evaluation.

#### Baseline Methods.

We compare our PoSE training method against following baselines:

  * •

Full-length fine-tuning takes input tokens of target length for fine-tuning.
For this method, computation complexity scales quadratically with target
context window size.

  * •

RandPos [28] is initially designed to train an encoder-only model from scratch
for length extrapolation. However, since it shares similar idea of simulating
longer sequences via changing position indices, we include it for a
comprehensive comparison. Given the original context length $L_{c}$ and the
target context length $L_{t}$, it uniquely samples $L_{c}$ positions from the
set $\\{0,...,L_{t}-1\\}$, arranges them in ascending order, and employs them
as new position indices for training.

###  4.2 Language Modeling

Table 1: Perplexity of models trained with different methods on GovReport and Proof-pile. Our PoSE, with a fixed training window size of 2k, effectively extended to a target context size of 16k / 32k for inference while receiving only minimal performance degradation compared to Full-length. Method | Context size | GovReport | Proof-pile
---|---|---|---
Train / Target | 2k | 4k | 8k | 16k | 32k | 2k | 4k | 8k | 16k | 32k
None | \- / - | 4.74 | $>10^{3}$ | $>10^{3}$ | $>10^{3}$ | $>10^{3}$ | 2.83 | $>10^{3}$ | $>10^{3}$ | $>10^{3}$ | $>10^{3}$
Full-length | 16k / 16k | 4.87 | 4.70 | 4.61 | 4.59 | - | 2.93 | 2.71 | 2.58 | 2.53 | -
RandPos | 2k / 16k | 11.63 | 11.17 | 11.54 | 15.16 | - | 7.26 | 6.83 | 6.76 | 7.73 | -
2k / 32k | 93.43 | 95.85 | 91.79 | 93.22 | 97.57 | 60.74 | 63.54 | 60.56 | 63.15 | 66.47
PoSE (Ours) | 2k / 16k | 4.84 | 4.68 | 4.60 | 4.79 | - | 2.95 | 2.74 | 2.61 | 2.65 | -
2k / 32k | 4.91 | 4.76 | 4.68 | 4.64 | 4.74 | 3.01 | 2.78 | 2.66 | 2.60 | 2.62

First, we investigate the impacts of different fine-tuning methods on long
sequence language modeling using the GovReport and Proof-pile datasets.
GovReport is a summarization dataset comprising 19,402 reports published by
the Congress and the U.S. Government, with an average document length of 7,866
tokens. We randomly select 50 reports containing more than 32,768 tokens for
evaluation. Similarly, Proof-pile is a mathematical dataset of 13 gigabytes
containing numerous lengthy mathematical documents. In line with the approach
taken for GovReport, we choose 50 samples from Proof-pile that contain more
than 32,768 tokens for evaluation.

Table 1 presents the results of scaling to 16k and 32k using Full-length
training, RandPos, and PoSE. For each scaled model, as well as the non-fine-
tuned LLaMA model (None), we report perplexity scores at various evaluation
context window sizes, ranging from 2k to 32k, employing the sliding window
approach proposed by Press et al. [23]. For evaluation efficiency, we set the
stride of the sliding window to 1,024.

First, we observe an overall decreasing trend of perplexity for both models
scaled to 16k and 32k via PoSE as evaluation context window size increases,
proving their abilities to leverage longer context. Second, with significantly
shorter context length during fine-tuning, our PoSE achieves comparable
results with Full-length, consolidating its effectiveness. Third, our method
achieves much stronger results than RandPos. We suppose it is because our
manipulated position indices closely resembles that of pre-training, hereby
preserving the pre-trained language modeling ability to the greatest extent.

We also notice that all the scaling methods suffers certain performance
degradation as the supported context length increases. We perceive this as a
trade-off between the quantity of tokens the model can process and the level
of granularity in the attention the model can pay to each individual token.

###  4.3 Passkey Retrieval for Effective Context Window

(a)

(b)

Figure 2: (a) Prompt template used for passkey retrieval; (b) retrieval
accuracy for the non-fine-tuned LLaMA model (None), and the PoSE-extended
counterparts for 16k and 32k context window size. Both PoSE-extended models
maintain a high retrieval accuracy ($\geq 90\%$) within their respective
target context window.

To effectively measure the maximum distance that a token can attend to during
the inference stage, we adopt the passkey retrieval test proposed by
Mohtashami and Jaggi [20]. In this test, models are tasked with recovering a
random passkey hidden within a lengthy document. Prompt template used for this
task is presented in Figure 2(a).

Specifically, we compare the non-fine-tuned LLaMA model (denoted as None) with
the PoSE-extended version for 16k and 32k context window size. For each model,
we vary the prompt length from 2k to 32k. In each case, we conduct the passkey
retrieval test for 50 times, with a random passkey of 5 digits generated and
placed at a random position inside the prompt for each trial. Figure 2(b)
illustrates the results. For the none-fine-tuned LLaMA model (None), the
retrieval accuracy rapidly drops to 0 when the prompt length exceeds 2k. In
contrast, both PoSE-extended models managed to maintain a high retrieval
accuracy ($\geq 90\%$) within their respective target context window. This
indicates that models trained via PoSE genuinely possess the capability to
attend to all tokens within the extended context windows.

##  5 Analysis

In this section, we analyze the advantages of PoSE, including 1) memory and
time efficiency; 2) potential for extremely-long context; 3) compatibility
with all RoPE-based LLMs and diverse interpolation strategies. In Section 5.4,
We also verify that model performance within the original context window only
receives minimal degradation.

###  5.1 Memory and Time Efficiency

We study the memory and time efficiency of PoSE compared with Full-length
fine-tuning. For each method, we scale LLaMA-7B to 16k through 1,000 training
steps with a global batch size of 64 on 8 V100 GPUs. Especially, for Full-
length fine-tuning, we adjust training batch size to 2 and gradient
accumulation steps to 4 on each device to achieve an equivalent configuration.
Experiment results are demonstrated in Figure 3. Figure 3(a) illustrates time
consumption for 1,000 steps of full training versus PoSE. While full training
requires 56.37 hours to complete, our PoSE method only takes 6.82 hours,
effectively achieving an eight-fold reduction in total training time. Figure
3(b) compares model perplexity of the two training methods at different steps.
Notably, both models achieve relatively low perplexity levels within the
initial 100 training steps. Moreover, at each step, our proposed PoSE, while
keeping a context size of 2k tokens during training, exhibits very close
language modeling ability to Full-length fine-tuning, which requires an
extended training context of 16k. Consequently, we can confidently assert that
our proposed approach is both memory and time-efficient.

Figure 3: (a) Time consumption of Full-length fine-tuning v.s. PoSE for
finishing 1,000 steps; (b) perplexity at different training steps of both
models. We show that PoSE attains a comparable level of PPL performance to
Full-length fine-tuning at each step, while concurrently achieving an eight-
fold reduction in total training time.

###  5.2 Compatibility with RoPE-Based LLMs and Diverse Interpolation
Strategies

We also delve into the effectiveness of PoSE when applied to different RoPE-
based LLMs, as well as various interpolation strategies. Specifically, we
employ PoSE on three distinct models: LLaMA-7B, LLaMA2-7B, and GPT-J-6B, all
of which encompasses RoPE in their architectures. For each model, we examine
the integration with Linear, NTK, and YaRN interpolation, as well as the non-
fine-tuned original version for comparative purposes. The same GovReport
dataset as described in Section 4.2 is utilized. The test set is truncated to
the first 1k 16k tokens for plotting the perplexity curve, as depicted in
Figure 4. First, it is evident that PoSE is effective across all three models
and three interpolation strategies, as evidenced by the low perplexities
achieved by all nine combinations in comparison to the non-fine-tuned original
model. Second, we observe that NTK and YaRN interpolation generally yields
superior results compared to Linear interpolation. However, it is noteworthy
that NTK exhibits a significant increase in perplexity after a certain turning
point, which occurs prior to reaching the target context length. This behavior
is consistent with previous findings, indicating that for a given scaling
factor $\alpha$, NTK is unable to genuinely expand the context window by
$\alpha$ times [21, 25, 22].

Figure 4: Perplexity of LLaMA-7B, LLaMA2-7B, GPT-J-6B extended to 16k via PoSE
with Linear / NTK / YaRN interpolation, along with the non-fine-tuned Original
model. The consistently low perplexity observed across all nine combinations
serves as an indication of the effectiveness of our method across RoPE-based
LLMs and diverse interpolation strategies.

###  5.3 Potential for Extremely-Long Sequences

Because PoSE only takes a fixed context window at training stage to extend to
target context window size, in theory we can extend LLMs to support infinite
input lengths via PoSE. In this section, we extend context window size to 96k
and 128k to explore PoSE’s potential for context window extension. Given the
need to assess documents of extreme length, we have opted to employ two book
datasets, namely Books3 [24] and Gutenberg (PG-19) [26]. Both of these
datasets consist of extensive collections of literary works, rendering them
well-suited subjects for the investigation of long-range modeling. For our
evaluation, we randomly selected 20 books from each dataset, each containing
more than 128k tokens.

Fine-tuning LLaMA models using PoSE, we experimented with Linear / NTK / YaRN
interpolation for context extension of 96k and 128k, respectively. For each
fine-tuned model, we adhere to the sliding window strategy adopted in Section
4.2 for perplexity calculation, with the only modification being an increase
in the sliding window steps to 16k to enhance evaluation efficiency. The
outcomes of these experiments are detailed in Table 2. Notably, we observe
that, when coupled with linear interpolation, PoSE successfully extends the
model’s context window to 96k, and when paired with YaRN, it can further
extends the context window to 128k.

Table 2: Perplexity of models extended to extreme context size via PoSE on PG-19 and Books3. We show that our training method can effectively extend context window size to 128k when combined with YaRN interpolation. Model | Gutenberg (PG-19) | Books3
---|---|---
32k | 64k | 96k | 128k | 32k | 64k | 96k | 128k
PoSE-Linear-96k | 10.18 | 11.11 | 13.57 | - | 9.98 | 10.90 | 13.42 | -
PoSE-NTK-96k | 7.98 | 20.39 | 38.73 | - | 8.29 | 20.82 | 40.39 | -
PoSE-YaRN-96k | 8.31 | 8.65 | 9.36 | - | 8.90 | 9.40 | 10.38 | -
PoSE-Linear-128k | 16.90 | 22.47 | 26.77 | 31.18 | 26.20 | 43.62 | 57.08 | 70.87
PoSE-NTK-128k | 8.04 | 14.84 | 29.48 | 34.80 | 8.34 | 16.04 | 31.42 | 37.00
PoSE-YaRN-128k | 9.32 | 10.36 | 10.77 | 11.33 | 10.56 | 12.30 | 13.07 | 13.81

###  5.4 Standard Benchmarks on Original Context Window Size

Table 3: Performance of PoSE-extended LLaMA model on standard benchmarks in comparison with Full-length fine-tuning and the original LLaMA. We show that PoSE-extended models exhibit only marginal performance degradation compared with Full-length fine-tuning and the original version. Model | Zero-Shot | Few-Shot
---|---|---
BoolQ | PIQA | WinoGrande | TruthfulQA | ARC-C | HellaSwag
LLaMA | 75.11 | 78.67 | 69.85 | 34.08 | 51.19 | 77.75
Full-Linear-16k | 70.95 | 77.64 | 69.06 | 31.89 | 48.55 | 74.19
Full-NTK-16k | 75.80 | 78.08 | 68.98 | 33.83 | 48.81 | 76.57
Full-YaRN-16k | 73.88 | 77.64 | 68.15 | 34.12 | 50.60 | 77.18
PoSE-Linear-16k | 74.50 | 78.13 | 68.59 | 32.05 | 48.29 | 75.56
PoSE-Linear-128k | 67.71 | 76.22 | 67.56 | 36.16 | 39.93 | 66.04
PoSE-NTK-16k | 74.28 | 78.24 | 68.90 | 33.89 | 49.83 | 76.82
PoSE-NTK-128k | 75.35 | 78.18 | 68.98 | 32.71 | 49.66 | 76.19
PoSE-YaRN-16k | 74.28 | 78.02 | 69.06 | 34.00 | 49.23 | 77.04
PoSE-YaRN-128k | 73.61 | 77.80 | 70.01 | 34.47 | 48.46 | 75.54

In this section, we examine the capabilities of the model within the original
context window size using standard benchmarks. We combine the Hugging Face
Open LLM Leaderboard [10] with a subset of LLaMA benchmarks to assess zero-
shot and few-shot performance. For zero-shot evaluation, we employ BoolQ [5],
PIQA [1], WinoGrande [15], and TruthfulQA [19]. For few-shot evaluation, we
utilize 25-shot ARC-Challenge [6] and 10-shot HellaSwag [36]. Our evaluation
metrics are benchmark-specific: for BoolQ, PIQA, and WinoGrande, we report
accuracy; for TruthfulQA, we report mc2; and for ARC-C and HellaSwag, we
report normalized accuracy.

Table 3 summarizes the results. It can be discerned that, PoSE-extended models
exhibit only marginal performance degradation compared with Full-length fine-
tuning and the original version, with the only exception of the 128k model
employing linear interpolation. For both NTK and YaRN, the extension from a 2k
context window to a 16k and even a 128k context window resulted in minimal
reductions in model performance, indicating their ability to effectively
preserve the model’s capabilities while extending the window size.

##  6 Conclusion

Context window extension aims at extending the context window of a pre-trained
LLM to accommodate longer sequences. In this paper, we introduce Positional
Skip-wisE (PoSE) training to efficiently extend the context window of Large
Language Models. PoSE simulates long inputs by manipulating position indices,
thereby requiring only the original context window for fine-tuning,
successfully decoupling train length and test length. Experiments have shown
that, compared with fine-tuning on the full length, PoSE greatly reduces
memory and time overhead. Taking advantage of this, we have managed to extend
LLaMA model to 128k with very limited training budget, observing only minimal
performance degradation on standard benchmarks. We have also empirically
verified that PoSE is compatible with all RoPE-based LLMs and position
interpolation strategies.

## References

  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_ , 2020\.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bulatov et al. [2022] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev.  Recurrent memory transformer.  _Advances in Neural Information Processing Systems_ , 35:11079–11091, 2022.
  * Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.  Extending context window of large language models via positional interpolation.  _ArXiv_ , abs/2306.15595, 2023.  URL https://api.semanticscholar.org/CorpusID:259262376.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL_ , 2019.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, 2019.
  * Dao [2023] Tri Dao.  FlashAttention-2: Faster attention with better parallelism and work partitioning.  2023\.
  * Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Face [2023] Hugging Face.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Haviv et al. [2022] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy.  Transformer language models without positional encodings still learn positional information.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1382–1390, Abu Dhabi, United Arab Emirates, December 2022\. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.99.  URL https://aclanthology.org/2022.findings-emnlp.99.
  * Huang et al. [2021] Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang.  Efficient attentions for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1419–1436, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.112.  URL https://aclanthology.org/2021.naacl-main.112.
  * kaiokendev [2023] kaiokendev.  Things i’m learning while training superhot.  https://kaiokendev.github.io/til#extending-context-to-8k, 2023.
  * Keisuke et al. [2019] Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin.  Winogrande: An adversarial winograd schema challenge at scale.  2019\.
  * Kwon* et al. [2023] Woosuk Kwon*, Zhuohan Li*, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang, , and Ion Stoica.  vllm: Easy, fast, and cheap llm serving with pagedattention.  https://vllm.ai/, 2023.
  * Lefaudeux et al. [2022] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza.  xformers: A modular and hackable transformer modelling library.  https://github.com/facebookresearch/xformers, 2022.
  * Li et al. [2023] Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong.  In-context learning with many demonstration examples.  _arXiv preprint arXiv:2302.04931_ , 2023.
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, 2022\.
  * Mohtashami and Jaggi [2023] Amirkeivan Mohtashami and Martin Jaggi.  Landmark attention: Random-access infinite context length for transformers, 2023.
  * Peng and Quesnelle [2023] Bowen Peng and Jeffrey Quesnelle.  Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.  https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have, 2023\.
  * Peng et al. [2023] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models, 2023\.
  * Press et al. [2021] Ofir Press, Noah A Smith, and Mike Lewis.  Train short, test long: Attention with linear biases enables input length extrapolation.  _arXiv preprint arXiv:2108.12409_ , 2021.
  * Presser [2020] Shawn Presser.  https://twitter.com/theshawwn/status/1320282149329784833, 2020.
  * Quesnelle [2023] Jeffrey Quesnelle.  Dynamically scaled rope further increases performance of long context llama with zero fine-tuning.  https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/, 2023\.
  * Rae et al. [2019] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  _arXiv preprint_ , 2019.  URL https://arxiv.org/abs/1911.05507.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE, 2020.
  * Ruoss et al. [2023] Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness.  Randomized positional encodings boost length generalization of transformers.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1889–1903, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.161.  URL https://aclanthology.org/2023.acl-short.161.
  * Su et al. [2021] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  _arXiv preprint arXiv:2104.09864_ , 2021.
  * Sun et al. [2023] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei.  A length-extrapolatable transformer.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14590–14604, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.816.  URL https://aclanthology.org/2023.acl-long.816.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang and Komatsuzaki [2021] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * Wang et al. [2023] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei.  Augmenting language models with long-term memory.  _arXiv preprint arXiv:2306.07174_ , 2023.
  * Wu et al. [2022] Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing transformers.  _arXiv preprint arXiv:2203.08913_ , 2022.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, 2019.
  * Zhangir et al. [2022] Azerbayev Zhangir, Ayers Edward, and Bartosz Piotrowski.  Proof-pile.  https://github.com/zhangir-azerbayev/proof-pile, 2022.
  * Zhou et al. [2022] Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Guodong Long, Can Xu, and Daxin Jiang.  Fine-grained distillation for long document retrieval.  _arXiv preprint arXiv:2212.10423_ , 2022.