Ştefan Minică logo_ttlM. Antonia Huertas, João Marcos, María Manzano, Sophie
Pinchinat,
François Schwarzentruber54th International Conference on Tools for Teaching
Logic11147TTL2015

#  RAESON: A Tool for Reasoning Tasks Driven by Interactive Visualization of
Logical Structure

Ştefan Minică<EMAIL_ADDRESS>
###### Abstract.

The paper presents a software tool for analysis and interactive engagement in
various logical reasoning tasks. A first feature of the program consists in
providing an interface for working with logic-specific repositories of formal
knowledge. A second feature provides the means to intuitively visualize and
interactively generate the underlying logical structure that propels customary
logical reasoning tasks. Starting from this we argue that both aspects have
didactic potential and can be integrated in teaching activities to provide an
engaging learning experience.

###### Key words and phrases:

Logic Teaching Software, Reasoning Tools, Interactive Visualization

###### 1991 Mathematics Subject Classification:

I.2 ARTIFICIAL INTELLIGENCE, I.2.4 Knowledge Representation Formalisms and
Methods: Modal Logic, Predicate Logic, K.3 COMPUTERS AND EDUCATION, K.3.1
Computer Uses in Education: Collaborative learning, Computer-assisted
instruction, Computer-managed instruction

##  1\. Introduction

The tradition of using diagrammatic representations of formal structure in
logical reasoning stretches back to times when sand or papyrus were the media
for visualizations. The tradition of using interactive software tools in
logical reasoning is a recent refinement of an ancient interest. This
stretches back to the age of compact disk distributed software alongside paper
printed books, with [1] as a paradigmatic example, and plenty others, many of
which have been presented at this venue [3]. Using didactic software online
for logic education and e-learning is an equally respectable tradition going
back to the age when java applets roamed on the web, with [2] as a
representative example, and plenty others, many of which have been also
presented at this venue [4]. Latest trends in this evolution are dissemination
of logic courseware freely available online, with [5] as a pertinent example,
alongside didactic software tools built on a technology stack up to date with
current web standards, with [15] as a relevant illustration. This is the
general trend for many other fields across the curriculum [13]. When compared
to what exists in other fields, many reasoning and modeling techniques
emerging in current branches of logic remain in many ways still
underrepresented, e.g. modal logics, interrogative logics, epistemic logics,
etc. However, consult [16, 17, 18, 19, 20, 21] as a short list of notable
exceptions containing robust software tools for reasoning and modeling
techniques emerging in nonclassical logics that have been recently integrated
into this tradition, frequently accompanied by web workers or didactically
tailored online demos. Inside this general landscape, the paper will revisit
classical challenges for teaching logic and reasoning skills and implement
possibly new software tools for interactive reasoning tasks driven by
automated visualization of logical structure emerging in logics that are often
underrepresented in or overlooked by existing approaches.

Overview The paper is structured as follows: in the first section we introduce
the topic and describe the main architecture of the program and some design
options. Section 2 describes the use of the repositories of formal knowledge
underlying the program. Section 3 describes the reasoning tasks and
interactive visualization tools provided by the software. Sections 4 and 5
explore teaching use case scenarios and provide evidence for didactic
relevance. Section 6 draws conclusions and hints towards possible directions
for further work.

The Overall Architecture The rest of this section contains information about
implementation and development aspects. Readers only interested in using the
program will probably want to skip to Section 2 where the program’s features
are described.

The Backend Repositories of Formal Knowledge The program consists of two
layers: remote repositories of formal knowledge are stored and managed on the
server side and the client side consists of the user interface to interact
with and visualize the repositories’ formal content. The backend application
layer is a REST-like API built using NodeJS and Express to provide the usual
CRUD operations on repository items. The storage layer consists of several
document structured databases built using MongoDB and Mongoose to define
repository items [10]. The databases contain items from the standard domains
in a logic curriculum: propositional logic, modal logic, predicate logic as
well as quantified modal logic. Database entries are JSON documents with each
item containing a basic ASCII representation of its content, as received from
an input device [11], together with a set of additional representation formats
(MathJax-LaTeX, D3-SVG, Unicode, etc.) used in visualizations, interactive
reasoning tasks and natural language information. The left and middle panels
in Figure 1 illustrate intuitively the list representation and the item
features for the formulae library database. Section 2 contains further
information about how this data is managed via CRUD operations on the REST-
like API. In total, the databases contain over five hundred items from four
logic domains. Moreover, besides the predefined, ready to be used items, and
perhaps more importantly, the backend application layer also serves as a tool
for students or instructors to add items of their own interest to the database
and use the reasoning and visualization tools in an approach to learning or
teaching that fits their needs. The right panel in Figure 1 shows the formula
editor GUI, details on how the GUI maps to CRUD actions are in Section 2. The
server contains mainly databases of formulae libraries and models corpora in
the four logic domains mentioned and a repository of interactive reasoning
tasks (dialogic/semantic games, etc.) using both formulae and models.

The Frontend User Interface The client side layer of the program uses HTML5
and CSS3 for structure respectively presentation and JavaScript for handling
application logic as well as for implementing logic specific functionality.
The framework used to build the frontend application layer is Backbone
enhanced with Marionette modules, together with their standard dependencies
JQuery and Underscore [8]. For the graphical user interface we use Bootstrap
and for rendering formulae in a browser MathJax. For displaying logical
structure we use SVG and for interacting with reasoning tasks visualizations
we use D3 [9], particularly tree and force graph layouts. Figure 1 presents
the main user interface components and Section 2 describes standard use
scenarios for the GUI. Figure 2 illustrates some visualizations of logical
structure and Section 3 describes how these can be interactively generated,
transformed and explored. The logic specific application component consists of
several modules containing functionality handling formal aspects concerning
the syntax and semantics of the logics and dealing with their associated
reasoning tasks. These include a term unification package and an expression
parser [12], a module for translating syntax into D3 tree layout, a tableau
based theorem prover [6] with a corresponding module to translate the analytic
tableau proof trace into a D3 force graph layout with a hierarchical tree
structure, and a (counter)model generator [7] with a corresponding module for
translating the (closed) tableau branches into (counter)example relational
structures and further into a D3 force graph layout.

Further information about implementation details and the technology stack used
are included in the electronic version of this paper and in the documentation
available on [22]. The program was developed and tested using primarily
Chromium in an Ubuntu desktop environment and will run optimally in WebKit-
based browsers and using a desktop format factor. A live demo of the program
is openly available online at [22].

##  2\. Managing Repositories of Formal Knowledge

A first feature of the program consists in providing an interface for
accessing and working with the following logic-specific repositories of formal
knowledge:

  1. (1)

Formulae Libraries provide an interface for accessing databases containing
syntactic items from various logical languages. Currently, these include:
Propositional Logic, Modal Logic, Predicate Logic, Quantified Modal Logic.

  2. (2)

Models Corpora provide an interface for accessing databases containing
semantic items associated with various logics. These are, depending on domain:
truth assignments, relational structures, first-order models, QML models.

  3. (3)

Games Repositories provide an interface for accessing databases containing
interactive reasoning tasks. These are not entries for separate entities but
merely documents associating, formulae and/or models with logic modules.

To facilitate access to repository items and reasoning tasks, the main
interface layout has three component regions containing distinct information
panels.

The first panel, in the left in Figure 1, is a collection view, its role is to
visualize the repository content in an items list format. These can be either
a formulae library or a corpus of models from those enumerated above. The
second panel, shown in the middle of Figure 1, is an item view, its role is to
display item features. These features are specific to each repository type and
logic domain and link to various reasoning tasks, additional natural language
information or further symbolic representations for a particular repository
item selected from the items list panel. The third panel, in the right of
Figure 1, is also an item view. It will display, depending on the chosen
feature, the specific visualization content and/or an interface for
interaction. This can be, fore example, a SVG canvas, a editor form, a table,
etc. The specific details of each interactive reasoning task are described in
Section 3. The subviews are interlinked and in a usual workflow will be
navigated from left to right by first choosing an item of interest from a
collection then a specific feature and finally working with its interactive
visualization content in the working details panel. The main navigation menu
also provides links to instances of the three layout regions. More detailed
descriptions of possible workflows applied to concrete examples are presented
in Section 4.

Working with the Formula and Model Editors  The first level of feedback guided
interaction with logical structure takes place starting from syntactic aspects
and is mediated by the formula and model editors. The right side panel in
Figure 1 illustrates the editors’ user interface. This includes examples of
well formed entities for each logic domain and detailed information about the
formation rules. Feedback is provided as the user types and parser errors are
displayed whenever an incorrect string is entered. A modular parsing
expression grammar is used for input validation. This extends naturally from
logical fragments to more expressive logics, from object language to the
specification metalanguage, and from formulae to model syntaxes. Section 7
contains an illustration of the the grammar modules. A concrete editor use
case example is given below in Section 4. Further details are included in the
electronic version of this paper and in the program documentation available on
[22].

Figure 1. Graphical user interface components: library view, formula features
and formula editor

Working with Repository Item Features A second level of feedback guided
interaction with logical structure is mediated by reasoning tasks
visualizations, accessed through specific item features. The main feature
categories are:

  1. (1)

Reasoning Tasks is the main category, it contains various entries such as:
visualizing syntactic construction trees, building semantic tableaus,
generating countermodels and models, etc. Each task is described in detail in
Section 3.

  2. (2)

Item Information contains various categories of information about the chosen
items in natural language, such as name, description, sources, domain, etc.

  3. (3)

Symbolic Representation Formats used for reference, linking, translation, etc.

Clicking on a formula feature will display its corresponding working details
in the right hand side panel of the main layout presented in Figure 1. While
the first two panels are used for organization and navigation of formal
repositories, the working details panel is used to host and manage most of the
interactive aspects by coupling interface actions with a feedback loop
allowing the user to generate and/or explore the formal structure underlying
and propelling various logical reasoning tasks.

##  3\. Interactively Visualizing Reasoning Tasks

The main interactive feature of the program is visualization of logical
structure. This takes place for several kinds of analysis and reasoning tasks
specific to each logic domain. In this section we describe how to use the main
reasoning tasks.

Figure 2. Examples of interactively visualizing logical structure in reasoning
tasks: syntactic construction tree (left), semantic tableau proof trace
(middle), generated counterexamples/models (right)

Syntax Construction Trees A first analysis task is the interactive generation
of syntactic construction trees. This uses the syntactic formation rules of
each logic domain to generate a tree layout of the formula and all its
subformulae, all the way down to its atomic components. The left side of
Figure 2 illustrates a syntactic tree layout generated in this fashion for a
first order logic formula. The interface distinguishes between nodes
containing fully extended components and the nodes containing further
analyzable components. The fully extended tree branches and the nodes
containing atomic syntactic elements are marked with a blue color while the
not yet fully analyzed nodes are colored in red, the user can further extend
these nodes by clicking on their content. In the illustration the second
branch is not yet fully decomposed, this is marked in red and can be
interactively extended further by clicking on the terminal subformula.
Concrete teaching context are included in Sections 4, 5.

Semantic Tableaus A second analysis task is the interactive generation of
semantic tableau proofs. This uses the semantic decomposition rules specific
to each logic domain to generate a force tree layout by analyzing the head
formula in a branch and so on in all resulting tableau branches, all the way
down to exhausted branches. The middle region of Figure 2 illustrates a
semantic tableau generated in this way for an example formula from the modal
logic domain. The interface distinguishes between nodes representing fully
expanded branches and the nodes standing for branches that can be further
expanded i.e. contain formulae that can be further analyzed. The layout also
distinguishes between closed, i.e. containing contradictory formulae, and open
branches, i.e. fully expanded but not containing a formula and its negation.
Nodes are labeled by the head position formula in the branch, this is the
formula logically analyzed and decomposed in that expansion step. The user can
interactively display the full content of a tableau branch by hovering over
any node of the tableau. Branches which are not fully expanded yet can be
extended by clicking on a terminal node. This triggers the process of logical
decomposition on the head formula in the branch and adds the resulting
branch(es) to the tableau. Clicking on an expanded node reverts the tableau
expansion step and removes the outgoing branch(es). For modal logics in
addition to decomposition rules for formulas in the logical object language
the tableau construction process also uses a specification metalanguage and
corresponding decomposition rules for hybrid logic with nominals. These
decomposition rules are based on applying metanominals to possible worlds as
Skolem terms constructed from formulae containing modal operators and using
relational properties of accessible successors during tableau expansion. The
theoretical background behind this technique has been described in full detail
in [6], and the logical grammar used for parsing hybrid logic formulae in the
specification metalanguage is also included for reference in Section 7.
Concrete teaching context are included in Sections 4, 5.

Countermodel Generator A third reasoning task consists in generating
countermodels for a given formula. This starts from the proof trace generated
by a semantic tableau and from each open branch in the terminal nodes of the
tree layout extracts a corresponding model. Hybrid logic formulae with
nominals are used for expressing relational components of a model inside the
specification metalanguage. These are then translated in a force graph layout
representing a relational structure which is displayed for visualization and
can be further interactively inspected. The right hand side of Figure 2
presents two graphs generated in this fashion. The interface distinguishes
counterexamples falsifying the formula, which are colored in red, from models
satisfying the formula, which are colored in green. The satisfying models are
obtained in a similar way starting from a complementary formula. Each node of
the graph is labeled with a relevant extensional component, for modal logic
this is the world’s propositional valuation. The user ca obtain further
information about the graph by hovering over the nodes. This will reveal the
metanominal corresponding to each possible world. This nominal is a Skolem
term generated during tableau construction and contains further data about the
stage in the proof trace it was introduced and the formula and other metaterms
it was generated from. Concrete teaching contexts are included in Sections 4,
5.

Several additional reasoning tasks are described and/or exemplified in
Sections 4 and 5, in the extended version of this paper and the documentation
available on [22]. These reasoning tasks are either specific to a particular
logic domain, such as Truth Tables for Propositional Logic, or only relevant
for corpora of models, such as Model Inspection etc.

##  4\. The Didactic Touch

In this section we will present a series of concrete use case scenarios: in
order to illustrate the way in which the program can be useful for didactic
purposes.

Use Case Scenario 1: Exploring Formal Knowledge Repositories This use case
will step through a typical repository navigation. Step (1): From the Main
Navigation Menu select Formulae Libraries $>$ Modal Logic, alternatively, you
can click on the Modal link from the Go to a Formula Library green div in the
Items List panel in the left of the main layout. The left side panel will
display the content of the Modal Logic Library. Step (2): Search in the list
the formula named Distribution Axiom (K), alternatively, you can hit ctrl+f
and type the formula’s ascii content (#(p>q)>(#p>#q)) or its name, as a
shortcut to finding the formula faster (using native browser search). When you
click on the formula, the middle panel will display the formula’s features.
Step (3): Inspect the list of available features for the formula listed in the
middle panel and find either the Name or the Domain item. When you click on
the chosen feature the right hand side panel will display additional
information about the item of interest. In this case this will be additional
information in natural language about the formula. Item features related with
reasoning tasks are explored in the next use case.

Use Case Scenario 2: Interactively Visualizing Reasoning Tasks This use case
will present step by step a possible scenario of exploring reasoning tasks.
Repeat steps (1) to (3) from the previous use case, but this time use
(#(p|q)>(#p|#q)) to find formula named Distribution Axiom (F), note the (F) at
the end which distinguishes it from the previous example. Step (4): Click on
the Syntactic Tree feature and explore the construction tree by clicking on
the nodes of the tree. Note that the syntactic construction of the formula is
the same with the one displayed in the previous example (both formulae
generate the same tree structure). Step (5): Click on the Semantic Tableau
feature and fully expand all the tableau branches by clicking on the nodes of
the tree layout. By hovering over the nodes in the tableau you can see the
entire structure of the branch containing all the extended formulae up to that
point. Continue until all the formulae are logically decomposed and every
branch of the tableau is exhausted. Note that the semantic tableau constructed
in this case is different from the one for the previous formula. Find the
green colored open branch that makes the difference and inspect its content,
try to imagine a counterexample starting from it. Step (6): Click on the Model
Generator feature and scroll the panel of models generated in the Working
Details panel until you find one that is colored in red. The nodes in the
graph represent possible worlds and are labeled by their propositional
valuation. Hovering over possible worlds in the models will reveal their
construction nominal, compare it with the open branch found in Step (5).

Use Case Scenario 3: Update Formal Knowledge Repositories In the previous use case scenario you have found a counterexample to a formula labeled as an axiom, now you are going to fix this. If needed, go through steps (1-6) in the previous use case to obtain the open tableau and the countermodel illustrated in Figure 2 (middle, respectively, right). Step (7): Click on the Copy to Sandbox Library feature. The Sandbox Library repository will be displayed in the Items List panel. Find the formula and click the Edit Formula Information feature. The rightmost panel will display a form with editable entries. Step (8): Change formula’s name from Distribution Axiom (F) to Distribution Formula (fails for disjunction). Optionally, add a description and sources, then click the Submit Changes button. Step (9): Find the modified formula in the Sandbox Library, click the Edit Ascii Content feature. The Formula Editor will open with the formula’s ASCII content available for editing. Step (10): Change all disjunctions in the formula to conjunctions, as indicated in the available instructions panels this is done by replacing all occurrences of | with &. Feedback will be provided as you type, when the new formula is parsed correctly, click on the Submit Formula button. Eventually, repeat Step (8): renaming the formula Distribution over Conjunction and adding the description “This fails for disjunction”.

Use Case Scenario 4: Semantic Tableaus Expansion Heuristics Step (1): Use the
semantic tableau feature to find a counterexample model falsifying the modal
formula
$(\Box{}(\lnot{}p\lor{}\Box{}q)\leftrightarrow{}(\Box{}\lnot{}p\lor{}\Box{}q))$,
Theorem (S5) 3 in the Modal Library, using no more than seven expansion steps.
Step (2): Can you do this with less than seven clicks? If yes, how many? If
no, what syntactic property of the formula can be used to prove this for the
general case.

Several other use cases are possible including features such as: interactive
visualization and inspection of models, linking formulae with structures
through model checking tasks, playing dialogic or semantic games, etc. More
use cases are described and explored in the electronic version of this paper
and in the program documentation available on [22].

##  5\. Classroom Settings Using the Tool

In this section we describe several possible scenarios of using the program in
a classroom setting and the didactic experience that has been accumulated. The
program can be used both during live classroom activities and also remotely by
students. The live teaching classroom settings can integrate the tool
alongside traditional teaching methods also during theory exposition but
mostly for solving exercises. The following list describes typical exercise
templates (concrete examples can be randomly selected from repositories):

Exercise Model One:

Given: A list of several formulae in symbolic notation (MathJax-LaTeX) and a
matching list of formulae represented as syntax construction trees (D3-SVG).
Required: Establish the correct correspondence between the items in the two
given lists.

Exercise Model Two:

Given: A list of structures in extensional notation (MathJax-LaTeX) and a
matching list of models represented as labeled graphs (D3-SVG force layout).
Required: Establish the correct correspondence between the items in the two
given lists.

Exercise Model Three:

Given: Syntax construction trees of several formulae with matching structure
graphs (pointed in the case of modal logics, i.e. with designated actual
worlds). Required: Decide for each pair if the (pointed) model satisfies or
not the given formula.

Exercise Model Four:

Given: A list of formulae in symbolic notation (MathJax) and a matching list
of (pointed) models in extensional representation (not as D3-SVG graphs).
Required: For each formula in the list select the (points)/structures that
satisfy/falsify it.

Exercise Model Five:

Given: A list of (pointed) models represented in graphical format (D3).
Required: For each structure in the list upload into the formulae library M
formulae that are true/false in the model (at the actual world) and having
(modal) depth at least N.

Exercise Model Six:

Given: A list of (pointed) models represented in extensional format. Required:
For each structure in the list upload into the formulae library N formulae
that are false/true in the model (at the actual world) and having (modal)
depth at least M.

Exercise Model Seven:

Given: A list of formulae represented as syntactic trees (D3-SVG). Required:
For each formula either upload into the corresponding model corpus a
counterexample structure with at least N possible worlds(/objects) or claim
validity otherwise.

Exercise Model Eight:

Given: A list of formulae represented symbolically (MathJax-LaTeX). Required:
For each formula either find and write down in extensional form a
counterexample structure with at least M possible worlds(/objects) or claim
validity otherwise.

The previous exercise models can be used in classroom activities, progress
evaluations and final exams. To asses the didactic relevance of the program a
nonequivalent groups design can be used to measure the average grades in
student groups that used the program or not. Another measure can be the
response time for accurate answers. Early adopters have (only recently)
started to use the program in teaching activities111 Dr. Adrian Luduşan is
using R  E A SON in teaching activities for logic courses at “Babeş-Bolyai”
University and relevant data can be obtained by logging user interaction based
on the previous list of exercise templates. As the design of the exercises
suggests, the main testing parameter is the usefulness of interactively
visualizing logical structure in reasoning tasks. Preliminary results [14]
have been used so far only for internal feedback loop. These show that the
test group significantly outperforms the control group for exercise model one,
which is as expected. For exercise model three the grades average is also
better in the test group, which is also not surprising. The really interesting
result is that the test group outperforms the control group also for exercise
model four (no visualizations). Moreover, when comparing averages for exercise
model three with those for exercise models four, inside the same group, the
former significantly outperform the later. This is even more relevant as this
outcome holds not only for the test group, but also for the control group. All
these suggest that interactive visualizations of logical structure in
reasoning tasks can be didactically relevant. At the moment, not enough data
is available for the remaining exercise models in order to warrant conclusion
validity as a rigorous statistical experiment (more results will become
available online as more data is collected).

A final didactic experience that deserves to be mentioned is that the tool
facilitates types of exercise that might be impractical (if not impossible) to
stage using traditional teaching media (like pen and paper or even chalk and a
wide blackboard). Consider an example:

Example 1:

Given: Rule of consensus, exception 1 from the propositional logic formulae
library. Required: Find a counterexample structure using the semantic tableau
method.

##  6\. Conclusions

The paper presented a software tool for analysis and interactive engagement in
various logical reasoning tasks. The main feature of the program is to provide
the means to intuitively visualize and interactively generate the logical
structure that propels customary logical reasoning tasks. A second feature
consists in providing an interface for working with logic-specific
repositories of formal knowledge. Starting from this we showed how both
aspects have didactic potential and can be integrated in teaching activities
to provide an engaging learning experience. Overall, this creates an ecosystem
of interactive software tools modeling reasoning tasks driven by automated
visualization of logical structure emerging in logics that are often
underrepresented in or overlooked by existing approaches. We also illustrated
how this approach can be useful for didactic purposes. Besides presenting a
possibly new logical reasoning software suite, the paper also revisited
classical challenges for teaching logic and reasoning skills using a
completely open source software stack for development, testing, deployment and
hosting. The resulting program is openly available online [22], it is
implemented completely platform independent and can be run in any modern
browser.

Topics of further work include adding relational theories to the modal
metatheory and customizing the modal tableau prover. Extending the modeled
reasoning tasks to currently uncovered logic domains. Adding additional layers
of interactivity to the interface and a customizable strategy for tableau
expansion. Adding dynamic logic elements, etc.

##  7\. Logical Syntax Grammars

Listing 1: Atomic Grammar Components (common for both Formulae and Model
syntaxes)

⬇

1propsym "proposition" = chars:([p-s0-9_\\-]+) {return chars.join("")}

2nominal "nominal" = chars:([i-k0-9_\\-]+) {return chars.join("")}

3metanominal "metanominal" = chars:([lt-w0-9_\\-]+) {return chars.join("")}

4predicate "predicate" = chars:([P-SA-EMN0-9_\\-]+) {return chars.join("")}

5function "function" = chars:([f-h0-9_\\-]+) {return chars.join("")}

6variable "variable" = chars:([X-Zx-z0-9_\\-]+) {return chars.join("")}

7constant "constant" = chars:([a-emn0-9_\\-]+) {return chars.join("")}

Listing 2: Formula Syntax: Complex Grammar Components

⬇

1start = metaformula / formula

2formula "formula" = atomic / unary / binary / quanty / metaterm2form

3metaformula "metaformula" = formula "@" metaterm

4quanty "quantified" = "(" quantifier variable formula ")"

5quantifier "quantifier" = "$" / "!"

6binary "binary" = "(" formula "&" formula ")" / "(" formula "|" formula ")"

7 / "(" formula ">" formula ")" / "(" formula "^" formula ")"

8unary "unary" = "~" formula / "#" formula / "*" formula

9atomic "atomic" = nominal / propsym / equality / predicate "(" term ")"

10 / predicate "(" term "," term ")"

11equality "equality" = "(" term "=" term ")"

12term "term" = constant / variable / function "(" term ")"

13 / function "(" term "," term ")"

14metaterm "metaterm" = metanominal / metafunction "(" metaterm "," formula
")"

15metaterm2form "metaterm2form" = "{" metaterm // reductionaxiom = "<" atomic

16metafunction "metafunction" = "F*" / "F~#"

Listing 3: Model Syntax: Complex Grammar Components

⬇

1start = model

2model "model" = left:world ";" right:model / world

3world "world" = metanominal ":" extensionnotationslist

4extensionnotationslist "extensionnotationslist"

5 = left:extensionnotation right:extensionnotationslist / extensionnotation /
""

6extensionnotation "extensionnotation" = valsymb valuation / relsymb relations

7 / domainsymb domain / monadicsymb monadicext / diadicsymb diadicext

8extsymb "extsymb" = valsymb / domainsymb / monadicsymb / diadicsymb

9valsymb "valsymb" = "VL"

10valuation "valuation" = left:literal "," right:valuation / literal / ""

11relsymb "relsymb" = "RL"

12relations "relations" = left:metanominal "," right:relations / metanominal /
""

13literal "literal" = atomic / "~" atomic:atomic {return "~" + atomic;}

14atomic "atomic" = nominal / propsym / equality / predicate "(" groundterm
")"

15 / predicate "(" groundterm "," groundterm ")"

16domainsymb "domainsymb" = "DO"

17domain "domain" = left:groundterm "," right:domain / groundterm / ""

18monadicsymb "monadicsymb" = "MP"

19monadicext "monadicext" = predicate "{" objectslist "}"

20monadicextslist "monadicextslist" = left:monadicext ","
right:monadicextslist / monadicext / ""

21objectslist "objectslist" = left: groundterm "," right: objectslist /
groundterm / ""

22diadicsymb "diadicsymb" = "DP"

23diadicext "diadicext" = predicate "{" pairslist "}"

24diadicextslist "diadicextslist" = left:diadicext "," right:diadicextslist /
diadicext / ""

25pairslist "pairslist" = left: tuple2 "," right: pairslist / tuple2 / ""

26tuple2 "tuple2" = "(" groundterm "," groundterm ")"

27groundterm "groundterm" = constant / function "(" groundterm ")"

28 / function "(" groundterm "," groundterm ")"

## References

  * [1] Barwise, Jon and Etchemendy, John and Allwein, Gerard and Barker-Plummer, Dave and Liu, Albert: Language, Proof and Logic, CSLI Publications, Stanford, USA, 2000.
  * [2] Sieg, Wilfried: The AProS Project: Strategic Thinking and Computational Logic, Logic Journal of IGPL, vol. 5, no. 4, pages: 359-368, 2007.
  * [3] van Ditmarsch, Hans and Manzano, María: Proceedings of SICTTL’06, Tools for Teaching Logic, Salamanca, Spain, Logic Journal of IGPL, vol. 5, no. 4, 2007.
  * [4] Blackburn, Patrick and van Ditmarsch, Hans and Soler-Toscano, Fernando and Manzano, María: Proceedings of TICTTL’11, Tools for Teaching Logic, Springer-Verlag, 2011
  * [5] van Benthem, Johan and van Ditmarsch, Hans and van Eijck, Jan and Jaspars, Jan: Logic in Action, Open Logic Courseware Project, 2009-2014.
  * [6] Minică, Ştefan and Khodadadi, Mohammad and Schmidt, Renate A. and Tishkovsky, Dmitry: Synthesizing and Implementing Tableau Calculi for Interrogative Epistemic Logics, in Fontaine, P., Schmidt, R.A., Schulz, S. (eds.), PAAR, The Third Workshop on Practical Aspects of Automated Reasoning, pp. 109-123, Manchester, United Kingdom, 2012.
  * [7] Minică, Ştefan: Dynamic-Epistemic Logic of Questions in Inquiry, Ph.D. Thesis, Institute of Logic, Language and Computation, University of Amsterdam, The Netherlands, 2011.
  * [8] JQuery, available online from: http://jquery.com/, Underscore, available online from: http://underscorejs.org/, Backbone, available online from: http://backbonejs.org/, Marionette, available online from: http://marionettejs.com/.
  * [9] MathJax, from: http://www.mathjax.org/, Bootstrap, at: http://getbootstrap.com/, Scalable Vector Graphics, available online from: http://www.w3.org/Graphics/SVG/, Data Driven Documents, available online from: http://d3js.org/
  * [10] NodeJS, available online from: http://nodejs.org/, ExpressJS, available online from: http://expressjs.com/, MongoDB, available online from: http://www.mongodb.org/, BENM, available online from: https://github.com/jkat98/benm.
  * [11] JSON: JavaScript Object Notation: http://www.json.org/, ASCII: American Standard Code for Information Interchange: http://www.ascii-code.com/, CRUD: persistent storage functions: http://en.wikipedia.org/wiki/Create,_read,_update_and_delete.
  * [12] PEG.js, parser generator for JavaScript, available online from: http://pegjs.org/, Unify.js, package available online from NPM: https://www.npmjs.com/package/unify
  * [13] Khan Academy, available online from: https://www.khanacademy.org/a/0z1s
  * [14] https://www.academia.edu/12056013/Class_experience_using_RAESON
  * [15] Modal Logic Playground, available online at: http://rkirsling.github.io/modallogic/
  * [16] MetTeL2, Metric Tesselation Logic, a generic tableau prover, demo available online from: http://www.mettel-prover.org./demo.php
  * [17] DEMO, a demo of epistemic modelling, code and examples available online from: http://homepages.cwi.nl/~jve/demo/
  * [18] LoTREC, logical tableaux research engine companion, demo available online from: http://www.irit.fr/ACTIVITES/LILaC/Lotrec/webstart
  * [19] TWB, the tableau WorkBench, a generic tableau framework for building tableau/sequent-based theorem provers for non-classical propositional logics, demo available online from: http://twb.rsise.anu.edu.au/demolist
  * [20] InToHyLo: Inference Tools for Hybrid Logics, source code and demos available online from: http://www.glyc.dc.uba.ar/intohylo/index.php
  * [21] MleanTAP: A Modal Theorem Prover, at: http://www.leancop.de/mleantap/
  * [22] R  E A SON, a software tool for interactively visualizing logical structure in reasoning tasks, available online from: http://raeson.mybluemix.net


# Artificial neural networks on graded vector spaces

T. Shaska  Department of Mathematics and Statistics,
College of Liberal Arts and Sciences
Oakland University, Rochester, MI, 48326<EMAIL_ADDRESS>
###### Abstract.

We develop new artificial neural network models for graded vector spaces,
which are suitable when different features in the data have different
significance (weights). This is the first time that such models are designed
mathematically and they are expected to perform better than neural networks
over usual vector spaces, which are the special case when the gradings are all
1s.

###### Key words and phrases:

artificial neural networks, equivariant networks, weighted varieties, weighted
heights

###### 2020 Mathematics Subject Classification:

xx, yy

###### Contents

  1. 1 Introduction
  2. 2 Mathematical foundations of artificial neural networks
    1. 2.1 Artificial Neural Networks
    2. 2.2 Symmetries
    3. 2.3 Groups acting on sets
    4. 2.4 Invariant and equivariant maps
    5. 2.5 Quotient spaces
    6. 2.6 Group representations
    7. 2.7 Quotient representation:
    8. 2.8 Tensor products
    9. 2.9 Topological groups
    10. 2.10 Clebsch-Gordan decomposition
  3. 3 Equivariant Neural Networks
    1. 3.1 Equivariant neural networks
    2. 3.2 Convolution Neural Networks: translation equivariance
    3. 3.3 Integral transforms
    4. 3.4 Translation equivariant bias summation
    5. 3.5 Translation equivariant local nonlinearities
    6. 3.6 Translation equivariant local pooling operations
    7. 3.7 Affine group equivariance and steerable Euclidean CNNs
  4. 4 Graded vector spaces
    1. 4.1 Integer gradation
    2. 4.2 General gradation
    3. 4.3 Graded linear maps
    4. 4.4 Operations over graded vector spaces
    5. 4.5 Inner graded vector spaces
  5. 5 Artificial neural networks over graded vector spaces
    1. 5.1 Artificial neural networks on weighted projective spaces

##  1\. Introduction

Artificial neural networks are widely used in artificial intelligence for a
variety of problems, including problems that rise from pure mathematics. A
neural network model is a function $f:k^{n}\to k^{m}$, for some feld $k$ and
in the majority of cases $k=\mathbb{R}$. Many different architectures and
models are used for such networks. The coordinates of $\mathbf{v}\in k^{n}$
are called _input features_ and the coordinates of the vector
$\mathbf{u}=f(\mathbf{v})$ the _output features_.

There are many scenerios when the input features are characterized by
different values from some set, say $I$. For example, if the entries of the
data are document and each one has a different significance and could be
associated with different values. Consider for example if
$\mathbf{v}=[x_{0},\ldots,x_{n}]$ we can assign to any $x_{i}$ some value
$\mathbf{wt}(x_{i})\in I$. Such values are called _weights_ . A vector space
in which coordinates of each vector are assigned some other value are known in
mathematics as graded vector spaces (cf. Section 4). In this paper we
investigate whether one can design neural networks over such graded vector
spaces. One can think of many scenarios where neural networks defined over
graded vector spaces can make a lot os sense from the applications point of
view.

Our motivation came from studying the weighted projective space
$\mathbb{WP}_{(2,4,6,10),\mathbb{Q}}$ which is the moduli space of genus two
curves; see [sh-2024], in which case the weights are positive integers. The
space of homogenous polynomials graded by their degree is a classical example
of such graded vector spaces, when again the grading is done over the set of
positive integers.

If one intends to carry the theory of neural networks to such graded vector
spaces there are some mathematical obstacles that need to be cleared. Are
there linear maps between such spaces? How will the activation functions look
like? Will such graded or weighted neural networks have any advantages over
the classical neural networks?

This paper is organized as follows. In Section 2 we give the mathematical
background of artificial neural networks. We briefly define group action on
sets, invariant and equivariant maps, quotient spaces, group and quotient
representations, tensor products, topological groups, and state the Clebsch-
Gordan decomposition. While some of these definitions are basic knowledge for
mathematicians, they become necessary since this paper is intended to a larger
audience of the AI community. Section 2 is a prelude to defining equivariant
neural networks what we intend to develop for the analog of such networks over
graded vector spaces.

In Section 3 we give the basic definitions of equivariant neural networks. We
define convolutional neural networks or translation equivariant networks,
integral transforms, square integrable functions, regular translation
intertwiners, and describe some of the properties of the translation
equivariant local pooling operations. Part of Section 3 are also affine group
equivariance and steerable Euclidean convolutional neural networks. For more
details on such new and exciting topics the reader can check the wonderful
book [weiler-book].

In many ways we want to reproduce the results of Section 3 for neural networks
over graded vector spaces, but the upshot is to go even further and define
such neural networks that are equivariant under coordinate changes (i.e. work
for weighted projective spaces) and to do this over any field $k$ so we can
study not just applications from everyday life, but to use such neural
networks to study arithmetic applications (i.e. when $k$ is a number field),
cryptography and cybersecurity (when $k$ is a finite field), etc. Of course,
this is probably unrealistic currently since such methods are not fully
understood even on classical neural networks.

In Section 4 we go over the mathematical foundations of graded vector spaces.
We define gradations, graded linear maps, operations on graded vector spaces,
inner graded vector spaces, and discuss how to define a norm on such spaces.
Defining a norm is very important since it will be based on this norm that one
would define a cost function for the neural network. An adjusted homogenous
norm seems as the best option to capture the significance of the weights,
similar to the discussion on linear bundles and weighted heights in [SS]. This
is open to further investigation.

In Section 5 we define graded neural networks, graded activation functions. In
general a graded neural network is defined, as expected, as a neural network
which handles data where every input feature has a certain weight. It seems as
under mild conditions, we can replicate all the machinery of the artificial
neural networks to work for such artificial graded neural networks. It is
worth pointing out that when the weights are all ones the graded neural
network is just the usual neural network. It is interesting both
mathematically and from the application point of view to understand the
performance of such neural networks and whether they perform better for
certain applications.

From the mathematical point of view many questions arise, but the main one is
the understanding of the geometry of weighted projective spaces. In view of
[SS, vojta, sh-2024] understanding the geometry of such spaces possibly could
shed light to many intriguing arithmetic questions on weighted projective
varieties.

##  2\. Mathematical foundations of artificial neural networks

In this section we establish the notation and give basic definitions of
equivariant neural networks. We assume the reader has basic knowledge on the
subject on the level of [weiler-book], [roman] Throughout this paper $k$
denotes a field, $\mathbb{A}^{n}(k):=k^{n}$ the affine space, and
$\mathbb{P}^{n}(k)$ the projective space over $k$.

###  2.1. Artificial Neural Networks

Let the input vector be $\mathbf{x}=(x_{0},\ldots,x_{m})$ and the output say
some $\mathbf{y}=(y_{0},\ldots,y_{n})$. We denote by
$\operatorname{\mathcal{X}}$ the space of in-features and
$\operatorname{\mathcal{Y}}$ the space of out-features. A neuron is a function
$f:k^{n}\to k$ such that

$f(\mathbf{x})=\sum_{i-0}^{n}w_{i}x_{i}+b,$

where $b\in k$ is a constant called bias. We can generalize neurons to tuples
of neurons via

$\begin{split}L:\;k^{n}&\to k^{n}\\\
\mathbf{x}&\to\left(f_{0}(\mathbf{x}),\ldots,f_{n}(\mathbf{x})\right)\end{split}$

Then $L$ is a function given by

$L(\mathbf{x})=W\cdot\mathbf{x}+\mathbf{b},$

where $W$ is an $n\times n$ matrix (of weights) with integer entries and
$\mathbf{b}\in k^{n}$. A non-linear function $g:k^{n}\to k^{n}$ is called an
activation function.

A network layer is a function

$\begin{split}k^{n}&\to k^{n}\\\ \mathbf{x}&\to
g\left(W\cdot\mathbf{x}+\mathbf{b}\right)\end{split}$

for some $g$ some activation function. A neural network is the composition of
many layers. The $i$-th layer

$\begin{split}\cdots\longrightarrow k^{n}&\stackrel{{\scriptstyle
L_{i}}}{{\longrightarrow}}k^{n}\longrightarrow\cdots\\\
\mathbf{x}&\longrightarrow
L_{i}(\mathbf{x})=g_{i}\left(W_{i}\mathbf{x}+\mathbf{b}_{i}\right),\end{split}$

where $g_{i}$, $W_{i}$, and $\mathbf{b}_{i}$ are the activation, matrix, and
bias of the corresponding to this layer.

After $m$ layers the output (predicted values) will be denoted by
$\hat{\mathbf{y}}=[\hat{y}_{1},\ldots,\hat{y_{n}}]^{t}$, where

$\hat{\mathbf{y}}=L_{m}\left(L_{m-1}\left(\ldots(L_{1}(\mathbf{x}))\ldots\right)\right),$

while the true values by $\mathbf{y}=[y_{1},\ldots,y_{n}]^{t}$. The
composition of all layers is called the model function, say

$\mathfrak{M}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$

###  2.2. Symmetries

Assume that the input has symmetries. The simplest one could be permuting
coordinates, but other not so obvious symmetries could be present as well.

######  Example 1 (Symmetric polynomials).

Consider the following: $\mathbf{x}=(\alpha_{1},\ldots,\alpha_{n})$ and
$\mathbf{y}=(y_{0},\ldots,y_{n-1})$ where coordinates of $\mathbf{y}$ are
coefficients of the polynomial

$F(x):=\prod_{i=1}^{n}(x-\alpha_{i})$

Obviously permuting roots $\alpha_{1},\ldots,\alpha_{n}$ does not affect the
outcome here, which is the set of elementary symmetric polynomials
$\mathbf{y}=(s_{0},\ldots,s_{n_{1}})$ well known in algebra. In this case,

$\begin{split}s_{1}&=\sum_{i=1}^{n}\alpha_{i}=\alpha_{1}+\alpha_{2}+\cdots+\alpha_{n},\\\
s_{2}&=\sum_{i\neq j}^{n}\alpha_{i}\alpha_{j},\\\ \vdots\\\
s_{n}&=\prod_{i=1}^{n}\alpha_{i}=\alpha_{1}\cdots\alpha_{n}\end{split}$

We can generalize this concept by group actions, which is a well understood
concept from abstract algebra. The symmetric group $S_{n}$ acts on
$\\{\alpha_{1},\ldots,a_{n}\\}$ by permuting the roots. Notice that symmetric
polynomials $s_{0},\ldots,s_{n}$ are unchanged (invariant) under this action.

Can we use this idea for neural networks? In other words, if a model network
is given by
$\mathfrak{M}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$ and a
group $G$ acts on $\operatorname{\mathcal{X}}$ when can we use this action to
get a more efficient model? What about if we have a group $G$ acting not only
on the space of in-features $\operatorname{\mathcal{X}}$, but also on the
space of out-features $\operatorname{\mathcal{Y}}$? We will explore what
conditions have to be met by these actions and the model so that we can make
use of it. This lead to two interesting types of neural networks: invariant
networks and equivariant networks.

###  2.3. Groups acting on sets

Let $\operatorname{\mathcal{X}}$ be a set and $G$ a group. We say that the
group $G$ acts on $\operatorname{\mathcal{X}}$ if there is a function

$\blacktriangleright:\;G\times\operatorname{\mathcal{X}}\to\operatorname{\mathcal{X}}\quad\text{
such that }\quad(g,x)\to g\blacktriangleright x$

which satisfies the following properties:

  1. i)

$e\blacktriangleright x=x$ for every $x\in\operatorname{\mathcal{X}}$

  2. ii)

$g\blacktriangleright(h\blacktriangleright x)=(gh)\blacktriangleright x$, for
every $g,h\in G$.

The set $\operatorname{\mathcal{X}}$ is called a $G$-set. When there is no
confusion $g\blacktriangleright x$ is simply denoted by $gx$. Let $G$ acts on
$\operatorname{\mathcal{X}}$ and $x,y\in\operatorname{\mathcal{X}}$. We say
that $x$ and $y$ are  $G$-equivalent if there exists $g\in G$ such that
$gx=y$. If two elements are $G$-equivalent, we write $x\sim_{G}y$ or $x\sim
y$.

######  Proposition 1.

Let $\operatorname{\mathcal{X}}$ be a $G$ -set. Then, $G$-equivalent is an
equivalence relation in $\operatorname{\mathcal{X}}$.

The kernel of the action is the set of elements

$\ker(f)\,=\,\\{g\in G\,\mid\,gx=x,\text{ for all
}x\in\operatorname{\mathcal{X}}\\}$

For $x\in\operatorname{\mathcal{X}}$, the stabilizer of $x\in G$ is defined as

$\mbox{Stab}_{G}(x)=\\{g\in G\,|\,gx=x\\}$

sometimes denoted by $G_{x}$. The stabilizer $\mbox{Stab}_{G}(x)$ is a
subgroup of $G$.

######  Lemma 1.

Let $\operatorname{\mathcal{X}}$ be a $G$ -set and assume that $x\sim y$.
Then, the stabilizer $\mbox{Stab}_{G}(x)$ is isomorphic to the stabilizer
$\mbox{Stab}_{G}(y)$.

The action of $G$ on $\operatorname{\mathcal{X}}$ is called faithful if its
kernel is the identity. The orbit of $x\in\operatorname{\mathcal{X}}$ (or
$G$-orbit) is the set

$\operatorname{Orb}(x)=\\{gx\in\operatorname{\mathcal{X}}\,\mid\,g\in G\\}$

An action is called transitive if for every $x,x\in X$, there is $g\in G$ such
that $x=gy$.

######  Lemma 2.

Let $G$ act on a set $\operatorname{\mathcal{X}}$ and
$x\in\operatorname{\mathcal{X}}$. Then, the cardinality of the orbit
$\operatorname{Orb}(x)$ is the index of the stabilizer
$|\operatorname{Orb}(x)|=[G:\mbox{Stab}_{G}(x)]$.

A $G$-set is transitive if it has only one $G$-orbit. This is equivalent with
the above definition of the transitive. Let $\operatorname{\mathcal{X}}$ be a
finite $G$-set and $\operatorname{\mathcal{X}}_{G}$ the set of fixed points in
$\operatorname{\mathcal{X}}$ (sometimes set of invariants )

$\operatorname{\mathcal{X}}_{G}=\\{x\in\operatorname{\mathcal{X}}:gx=x\text{
for every }\;g\in G\\}.$

Since the orbits partition $\operatorname{\mathcal{X}}$ we have

$\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}\mspace{1.0mu}\right\rvert=\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}_{G}\mspace{1.0mu}\right\rvert+\sum_{i=k}^{n}\left\lvert\mspace{1.0mu}\operatorname{Orb}(x_{i})\mspace{1.0mu}\right\rvert,$

where $x_{k},\ldots,x_{n}$ are representative of distinct orbits of
$\operatorname{\mathcal{X}}$.

For any $g\in G$ the set of fixed points of $g$ in
$\operatorname{\mathcal{X}}$, which we denote with
$\operatorname{\mathcal{X}}_{g}$, is the set of all points
$x\in\operatorname{\mathcal{X}}$ such that $gx=x$. Thus,

$\operatorname{\mathcal{X}}_{g}=\\{x\in\operatorname{\mathcal{X}}\,|\,gx=x\\}$

######  Theorem 1 (Orbit counting theorem).

Let $G$ be a finite group acting on $\operatorname{\mathcal{X}}$. If $N$ is
number of orbits, then

$N\,=\,\frac{1}{\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert}\,\sum_{g\in
G}\left\lvert\mspace{1.0mu}X_{g}\mspace{1.0mu}\right\rvert.$

Hence, the number of orbits is equal to the average number of points fixed by
an element of $G$.

######  Corollary 1.

Let $G$ be a finite group and $\operatorname{\mathcal{X}}$ a finite set such
that
$\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}\mspace{1.0mu}\right\rvert>1$.
If $G$ acts on $\operatorname{\mathcal{X}}$ transitively then there exists
$\tau\in G$ with no fixed points.

###### Proof.

Let $\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert=n$. Since the action
is transitive then there is only one $G$-orbit. Form the above theorem we have
that

$\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert=F(1_{G})+F(g_{1})+\cdots
F(g_{n})=\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}\mspace{1.0mu}\right\rvert+\cdots$

If $F(\tau)\geq 1$ for all $\tau\in G$ then

$\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert=\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}\mspace{1.0mu}\right\rvert+\sum_{\sigma\in
G}F(\tau)\geq\left\lvert\mspace{1.0mu}\operatorname{\mathcal{X}}\mspace{1.0mu}\right\rvert+(n-1)$

Thus, $\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert>n$ which is a
contradiction. Hence, there must be some $\tau\in G$ such that $F(\tau)=0$. ∎

###  2.4. Invariant and equivariant maps

From now on $G$ acts on $\operatorname{\mathcal{X}}$ via
$\blacktriangleright:G\times\operatorname{\mathcal{X}}\to\operatorname{\mathcal{X}}$.
A function
$\operatorname{\mathcal{T}}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$
is called G-invariant if and only if,

$\operatorname{\mathcal{T}}(g\blacktriangleright
x)=\operatorname{\mathcal{T}}(x),\>\>\>\forall g\in G,\forall
x\in\operatorname{\mathcal{X}}$

In other words,

$\xymatrix{\operatorname{\mathcal{X}}\ar[r]^{\operatorname{\mathcal{T}}}\ar[d]_{\blacktriangleright}&\operatorname{\mathcal{Y}}\\\
\operatorname{\mathcal{X}}\ar[ur]_{\operatorname{\mathcal{T}}}\\\
}\qquad\qquad\xymatrix{x\ar[r]\ar[d]_{\blacktriangleright}&\operatorname{\mathcal{T}}(x)=\operatorname{\mathcal{T}}(g\blacktriangleright
x)\\\ g\blacktriangleright x\ar[ur]_{\operatorname{\mathcal{T}}}\\\ }$

Assume now that $G$ also acts on $\operatorname{\mathcal{Y}}$, say $G$ acts on
$\operatorname{\mathcal{Y}}$ as

$\bigstar:G\times\operatorname{\mathcal{Y}}\to\operatorname{\mathcal{Y}},\quad(g,y)\to
g\bigstar y$

Then,
$\operatorname{\mathcal{T}}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$
is called G-equivariant if

$\operatorname{\mathcal{T}}(g\blacktriangleright
x)=g\bigstar\operatorname{\mathcal{T}}(x)\>\>\>\forall g\in G,\forall
x\in\operatorname{\mathcal{X}}$
$\xymatrix{\operatorname{\mathcal{X}}\ar[r]^{\operatorname{\mathcal{T}}}\ar[d]_{\blacktriangleright}&\operatorname{\mathcal{Y}}\ar[d]^{\bigstar}\\\
\operatorname{\mathcal{X}}\ar[r]_{\operatorname{\mathcal{T}}}\operatorname{\mathcal{Y}}\\\
}\qquad\qquad\xymatrix{x\ar[r]^{\operatorname{\mathcal{T}}}\ar[d]_{\blacktriangleright}&\operatorname{\mathcal{T}}(x)\ar[d]^{\bigstar}\\\
g\blacktriangleright
x\ar[r]_{\operatorname{\mathcal{T}}}\operatorname{\mathcal{T}}(g\blacktriangleright
x)\\\ }$

###  2.5. Quotient spaces

The set of orbits (left) of $G$ acting on $\operatorname{\mathcal{X}}$ is
denoted by

$G\backslash\operatorname{\mathcal{X}}:=\\{\operatorname{Orb}(x)\mid
x\in\operatorname{\mathcal{X}}\\}$

and is called a quotient space. The corresponding quotient map is called the
map

$\pi:\operatorname{\mathcal{X}}\to G\backslash\operatorname{\mathcal{X}},\quad
x\to\operatorname{Orb}(x)$

Notice that in the case of right action, the symbol
$\operatorname{\mathcal{X}}/G$ is used for the quotient space.

###  2.6. Group representations

Let $V$ be a vector space (finite dimension) over a field $k$.
$\operatorname{GL}(V)$ the general linear group of $V$ (i.e., group of
invertible linear maps $L:V\to V$). Let $G$ a locally compact group (i.e.,
finite groups, compact groups, Lie groups are all locally compact) A linear
representation  of $G$ on $V$ is a tuple $(\rho,V)$ such that

$\rho:G\to\operatorname{GL}(V)$

is a group homomorphism. $V$ is called the representation space. Sometimes
$(\rho_{G},V)$ is used. If $V=k^{n}$ then $\forall g\in G$, we have
$\rho(g)\in\operatorname{GL}_{n}(k)$, so $\rho(g)$ is an $n\times n$
invertible matrix when a basis in $V$ is chosen.

Any $G$-representation $(\rho,V)$ defines an action

$\triangleright:G\times V\to V,\quad\quad(g,v)\to\rho(g)v$

Conversely, from any linear $G$ action $\triangleright:G\times V\to V$ we get
a representation

$\rho_{\triangleright}:\quad G\to\operatorname{GL}(V),\qquad g\to L_{g}$

where $L_{g}(v)=g\triangleright(v)$, for all $v\in V$. Hence, there is a one
to one correspondence between $G$-representations on $V$ and (linear)
$G$-group actions on $V$. Here are some common representations (we will skip
details)

  1. (i)

trivial representation ( $\rho(g)=\operatorname{id}_{V}$)

  2. (ii)

standart representation ($\rho(g)=g$)

  3. (iii)

tensor representation

  4. (iv)

regular representation

Let $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ be two given
$G$-representations. Let $V_{1}\oplus V_{2}$ be the direct sum and

$\begin{split}\alpha:\operatorname{GL}(V_{1})\times\operatorname{GL}(V_{2})&\to\operatorname{GL}(V_{1}\oplus
V_{2})\\\ (v_{1},v_{2})&\to v_{1}\oplus v_{2}\end{split}$

Then we can define the direct sum representation as given by
$\rho_{1}\oplus\rho_{2}=\alpha\circ(\rho_{1}\times\rho_{2})$ as

$\begin{split}G\stackrel{{\scriptstyle\rho_{1}\times\rho_{2}}}{{\longrightarrow}}\operatorname{GL}(V_{1})\times\operatorname{GL}(V_{2})&\stackrel{{\scriptstyle\alpha}}{{\longrightarrow}}\operatorname{GL}(V_{1}\oplus
V_{2})\\\ g\to(\rho_{1}(g),\rho_{2}(g))&\to\rho_{1}(g)\oplus\rho_{2}(g)\\\
\end{split}$

The matrix representation of it (when bases for $V_{1}$ and $V_{2}$ are
chosen) is

$(\rho_{1}\oplus\rho_{2})(g)=\begin{pmatrix}\rho_{1}(g)&0\\\ 0&\rho_{2}(g)\\\
\end{pmatrix}$

###  2.7. Quotient representation:

Let $W\subset V$ be a subspace and $V/W$ the quotient space. $G$ acts on $V/W$
via

$G\times W\to W,\quad(g,v+W)\to\rho(g)v+W$

With this action $V/W$ is called the quotient representation of $V$ under $W$

Let $(\rho,V)$ be a $G$-representation and consider a subspace $W\subset V$.
$W$ is called invariant if it is closed under the action of $\rho$, i.e.,
$\rho(g)w\in W$, for any $w\in W$ and $g\in G$. Hence the restriction of
$\rho$ in $W$ is a homomorphism:

$\rho_{W}:G\longrightarrow\operatorname{GL}(W)$

######  Definition. 2.

A representation $(\rho,V)$ is called irreducible representation (irrep) it it
has only the two trivial subrepresentations $W=V$ and $W=\\{0\\}$.

######  Example 2.

Of course the fact that $(\rho,V)$ is irreducible or not depends on the field
$k$. For example, let $G=\operatorname{SO}(2,\mathbb{R})$. Its real valued
irreducible representation are

$\rho_{m}^{G,\mathbb{R}}(\phi)=\begin{pmatrix}\cos(m\phi)&-\sin(m\phi)\\\
\sin(m\phi)&\cos(m\phi)\end{pmatrix},\quad m\in\mathbb{N}$

However, over $\mathbb{C}$

Let $(\rho_{1},V_{1})$ nd $(\rho_{2},V_{2})$ be $G$-representations. An
intertwiner between them is an equivariant linear map

$L:V_{1}\longrightarrow V_{2},\quad\text{which satisfies}\quad
L\circ\rho_{1}(g)=\rho_{2}(g)\circ L$

The space of intertwines is a vector space denoted by
$\operatorname{Hom}_{G}(V_{1},V_{2})$.

######  Example 3.

Convolutions are intertwiners

######  Definition. 3 (Equivalent (isomorphic) representations).

Two representations $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ are called
equivalent or isomorphic if there exists an isomorphism

$L:V_{1}\longrightarrow V_{2},\quad\text{ such that }\quad
L\circ\rho_{1}(g)=\rho_{2}(g)\circ L,\quad\text{ for all }\quad g\in G$

This is equivalent as matrix representations $\rho_{1}(g)$ and $\rho_{2}(g)$
are similar for every $g\in G$.

######  Definition. 4 (Endomorphisms).

Intertwines from $(\rho,V)$ to itself are called endomorphisms. In other
words, an endomorphism is a linear map $L:V\to V$ such that

$L\circ\rho(g)=\rho(g)\circ L,$

for all $g\in G$. The endomorphism space is denoted by
$\operatorname{End}_{G}(V)=\operatorname{Hom}_{G}(V,V)$

######  Lemma 3 (Schur’s lemma).

Let $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ be $G$-irreps over
$k=\mathbb{R}$ or $k=\mathbb{C}$. Then:

  1. (1)

If $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ are not isomorphic, then there is
no (non-trivial) intertwiner between them

  2. (2)

If $(\rho_{1},V_{1})=(\rho_{2},V_{2})=:(\rho,V)$ are identical, any
intertwiner is an isomorphism and

    1. (a)

If $k=\mathbb{C}$ then

$\rho=\lambda\operatorname{id}_{v},\quad\text{ for }\quad\lambda\in\mathbb{C}$

    2. (b)

If $k=\mathbb{R}$, then $\operatorname{End}_{G}(V)$ has dimension 1, 2, or 4
depending on whether $(\rho,V)$ is real, complex, or quaternionic type.

###  2.8. Tensor products

The tensor product $V\otimes_{k}W$ of two vector spaces $V$ and $W$ over a
field $k$ is the $k$-vector space based on elements $v\otimes w$, and with
relations for all $k\in\mathbb{C}$, $v\in V$, $w\in W$

$\begin{split}(v_{1}+v_{2})\otimes w&=v_{1}\otimes w+v_{2}\otimes w\\\
v\otimes(w_{1}+w_{2})&=v\otimes w_{1}+v\otimes w_{2}\\\ (k\cdot v)\otimes
w&=v\otimes(k\cdot w)=k\cdot(v\otimes w)\\\ \end{split}$

If $\\{v_{1},\ldots,v_{n}\\}$ is a basis for $V$ and
$\\{w_{1},\ldots,w_{m}\\}$ is a basis for $W$, then $\\{v_{i}\otimes w_{j}\\}$
is a basis for $V\otimes W$.

Let $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ be two representations of a
group $G$. The tensor product representation
$\left(\rho_{1}\otimes\rho_{2},V_{1}\otimes V_{2}\right)$ is defined as

$(\rho_{1}\otimes\rho_{2})(g)(v_{1}\otimes
v_{2}):=\rho_{1}(g)(v_{1})\otimes\rho_{2}(g)(v_{2})$

and extended to all vectors in $V\otimes W$ by linearity. It has dimension
$\dim(V_{1})\cdot\dim(V_{2})$.

If $\dim V$, $\dim W<\infty$, then there is a natural isomorphism of vector
spaces (preserving $G$-actions, if defined) from $W\otimes V$ to
$\operatorname{Hom}(V,W)$.

###  2.9. Topological groups

A topological group is a group which is also a topological space and for which
the group operation is continuous. It is called compact if it is so as a
topological space.

A representation of a topological group $G$ on a finite-dimensional vector
space $V$ is a continuous group homomorphism

$\rho:G\to\operatorname{GL}(V),$

with the topology of $\operatorname{GL}(V)$ inherited from the space
$\operatorname{End}(V)$ of linear self-maps. Notice that now, we can naturally
replace
$\frac{1}{\left\lvert\mspace{1.0mu}G\mspace{1.0mu}\right\rvert}\sum_{g\in G}$
with $\int_{G}\,dg$ (see Haar measure, Borel measure, etc).

###  2.10. Clebsch-Gordan decomposition

Let $(\rho_{1},V_{1})$ and $(\rho_{2},V_{2})$ be unitary irreducible
$G$-representations of a compact group $G$. Let $\widehat{G}$ denote the set
of isomorphisms classes f unitary irreducible representations of $V$

Their tensor product $\rho_{1}\otimes\rho_{2}$ is no necessary irreducible.
However, there exists an isomorphism

$\phi:V_{l}\otimes
V_{k}\longrightarrow\oplus_{j\in\widehat{G}}\oplus_{s=1}^{m_{j,lk}}V_{j}$

such that $V_{j}$ are irreducible, where $m_{j,lk}$ the multiplicity of
irreducible representation $j$ in the tensor product of irreducible
representations $l$ and $k$. This is called Clebsch-Gordan decomposition.

Consider a choice of basis for $V_{j}$, say

$\\{e_{j}^{1},e_{j}^{2},\ldots,e_{j}^{\dim V_{j}}\\}$

Hence, $e_{l}^{m}\otimes e_{k}^{n}$ are basis elements in $V_{l}\otimes V_{k}$
which are mapped to the basis elements of $\oplus_{s=1}^{m_{j,lk}}V_{j}$.
Hence we get a matrix associated to $\phi$, its elements are called Clebsch-
Gordan coefficients

A real-valued function $f:\mathbb{R}\to\mathbb{R}$ is called a square-
integrable function if

$\int_{-\infty}^{\infty}\left\lvert\mspace{1.0mu}f(x)\mspace{1.0mu}\right\rvert^{2}\,dx<\infty$

Let $L^{2}(\mathbb{R}$ denote the space of square-integrable functions.
$L^{2}(\mathbb{R})$ is a vector space and a Hilbert space.

######  Theorem 5 (Peter-Weyl).

The space of square integrable functions on $G$ is an Hilbert space, direct
sum over finite dimensional irreducible representations $V$

$L^{2}(G){\,\cong\,}\oplus\operatorname{End}(V)$

where

$f\to\int_{G}f(g)\cdot\rho_{V}(g)\,dg$

The inverse map sends $\phi\in\operatorname{End}(V)$ to the function

$g\to Tr_{V}(\rho_{V}(g)^{*}\phi)$

Let $G$ be a compact group and $L_{k}^{2}(G/H)$ as above. Denote by :
$\widehat{G}$ is the set of isomorphism classes of $G$ irreducible
representations, $\widehat{(\cdot)}$ is a topological closure, and by
$m_{j}\leq\dim V_{j}$ is the multiplicity of irreducible representation of
$V_{j}$ in $L_{k}^{2}(G/H)$.

######  Theorem 6.

The quotient representation $\left(\rho_{quot}^{G/H},L_{k}^{2}(G/H)\right)$
decomposes into irreducible subrepresentations

$L_{k}^{2}(G/H){\,\cong\,}\widehat{\oplus}_{j\in\widehat{G}}\,\oplus_{i=1}^{m_{j}}V_{j}$

Notice that if $k=\mathbb{C}$ and $H=\\{e\\}$ then $m_{j}=\dim V_{j}$.

##  3\. Equivariant Neural Networks

Let us see now how to construct some Equivariant Neural Networks. Let
$\operatorname{\mathcal{X}}$ be the space of input features and
$\operatorname{\mathcal{Y}}$ the space of output features. Let
$\mathfrak{M}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$ be a
model. Usually we want to approximate some target function

$\operatorname{\mathcal{T}}:\operatorname{\mathcal{X}}\to\operatorname{\mathcal{Y}}$

Let $\operatorname{\mathcal{H}}_{full}$ denote the space of all models under
consideration during the training, we call this the hypothesis space. Assume
$G$ acts on $\operatorname{\mathcal{X}}$ and $\operatorname{\mathcal{Y}}$ as

$G\times\operatorname{\mathcal{X}}\to\operatorname{\mathcal{X}},(g,x)\to
g\blacktriangleright x\quad\text{ and }\quad
G\times\operatorname{\mathcal{Y}}\to\operatorname{\mathcal{Y}},(g,y)\to
g\bigstar y$

We denote by $\operatorname{\mathcal{H}}_{inv}$ the space of invariant models
and by $\operatorname{\mathcal{H}}_{equiv}$ the space of equivariant models.
So we have

$\operatorname{\mathcal{H}}_{inv}\subset\operatorname{\mathcal{H}}_{equiv}\subset\operatorname{\mathcal{H}}_{full}$

Consider now instead of having a network sending
$\mathbf{x}\to\mathfrak{M}(\mathbf{x})$ we have one which sends
$\operatorname{Orb}(\mathbf{x})\to\mathfrak{M}(\mathbf{x})$.

$\xymatrix{\operatorname{\mathcal{X}}\ar[r]^{\mathfrak{M}}\ar[d]_{\pi}&\operatorname{\mathcal{Y}}\\\
G\backslash\operatorname{\mathcal{X}}\ar[ur]_{\mathfrak{M}_{inv}}\\\
}\qquad\qquad\xymatrix{\mathbf{x}\ar[r]^{\mathfrak{M}}\ar[d]_{\blacktriangleright}&\
{\mathfrak{M}}(\mathbf{x})\\\
\operatorname{Orb}(\mathbf{x})\ar[ur]_{\mathfrak{M}^{\downarrow}}\\\ }$

So $\mathfrak{M}^{\downarrow}$ is an invariant map and
$\operatorname{\mathcal{H}}_{inv}$ can be thought of the space of models
$\mathfrak{M}^{\downarrow}$.

###  3.1. Equivariant neural networks

A feed forward neural network is a sequence

$\operatorname{\mathcal{X}}_{0}\stackrel{{\scriptstyle\mathcal{L}_{1}}}{{\longrightarrow}}\operatorname{\mathcal{X}}_{1}\stackrel{{\scriptstyle\mathcal{L}_{2}}}{{\longrightarrow}}\operatorname{\mathcal{X}}_{2}\cdots\stackrel{{\scriptstyle\mathcal{L}_{N-1}}}{{\longrightarrow}}\operatorname{\mathcal{X}}_{N}$

of parametrization layers
$\mathcal{L}_{i}:\operatorname{\mathcal{X}}_{i-1}\to\operatorname{\mathcal{X}}_{i}$,
where $\operatorname{\mathcal{X}}_{i}$ is a feature space (vector space) and
$\mathcal{L}_{i}$ the $i$-th layer. Constructing equivariant networks
typically involves designing each layer to be individually equivariant.
Therefore, each feature space $\operatorname{\mathcal{X}}_{i}$ has it’s own
group action:

$\blacktriangleright_{i}:G\times\operatorname{\mathcal{X}}_{i}\to\operatorname{\mathcal{X}}_{i},\>\>\>\>(g,x)\to
g\blacktriangleright_{i}x.$

In the network the input $\operatorname{\mathcal{X}}_{0}$ and output $X_{N}$
actions $\blacktriangleright_{0}$ and $\blacktriangleright_{N}$ are determined
by learning task, while the intermediate actions are selected by the user.

For a layer ${\mathcal{L}_{i}}$ to be invariant it’s input and output actions
must satisfies the following,

${\mathcal{L}}_{i}(g\blacktriangleright_{i-1}x)=g\blacktriangleright_{i}{\mathcal{L}}_{i}(x),\>\>\forall
g\in G,x\in\operatorname{\mathcal{X}}_{i-1}$

The visualization of equivariant neural networks is given below

$\xymatrix{\operatorname{\mathcal{X}}_{0}\ar[r]^{\mathcal{L}_{1}}\ar[d]_{g\blacktriangleright_{0}}&\operatorname{\mathcal{X}}_{1}\ar[d]^{g\blacktriangleright_{1}}\ar[r]^{\mathcal{L}_{2}}\operatorname{\mathcal{X}}_{2}\ar[d]^{g\blacktriangleright_{2}}\ar[r]^{\mathcal{L}_{3}}\cdots\ar[r]^{\mathcal{L}_{N-1}}\operatorname{\mathcal{X}}_{N-1}\ar[d]^{g\blacktriangleright_{N-1}}\ar[r]^{\mathcal{L}_{N}}\operatorname{\mathcal{X}}_{N}\ar[d]^{g\blacktriangleright_{N}}\\\
\operatorname{\mathcal{X}}_{0}\ar[r]_{{\mathcal{L}_{1}}}\operatorname{\mathcal{X}}_{1}\ar[r]^{\mathcal{L}_{2}}\operatorname{\mathcal{X}}_{2}\ar[r]^{\mathcal{L}_{3}}\cdots\ar[r]^{\mathcal{L}_{N-1}}\operatorname{\mathcal{X}}_{N-1}\ar[r]^{\mathcal{L}_{N}}\operatorname{\mathcal{X}}_{N}\\\
}$

###  3.2. Convolution Neural Networks: translation equivariance

A Euclidean feature map in $d$ dimensions with $c$ channels is a function
$F:\mathbb{R}^{d}\to\mathbb{R}^{c}$ that assigns a $c$-dimensional feature
vector $F(x)$ for every point $\mathbf{x}\in\mathbb{R}^{d}$.

Let $\mathcal{E}_{(d,c)}$ be the set of all Euclidean feature maps
$\mathbb{R}^{d}\to\mathbb{R}^{c}$. A translation group is called the additive
group of the Euclidean space $V=\mathbb{R}^{d}$. It acts on $\mathbb{R}^{d}$
by shifting (or translation)

$(\mathbb{R}^{d},+)\times\mathbb{R}^{d}\to\mathbb{R}^{d},\quad(t,x)\to x+t$

It induces an action on $\mathcal{E}_{(d,c)}$ via

$(\mathbb{R}^{d},+)\times\mathcal{E}_{(d,c)}\to\mathcal{E}_{(d,c)},\quad
F\to(t\blacktriangleright F)(x)=F(x-t)$

This action is known as regular representation

The feature spaces of translation equivariant Euclidean Convolutional Neural
networks are vector spaces

$\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c}):=\left\\{\mathcal{E}_{d,c}:\mathbb{R}^{d}\to\mathbb{R}^{c}\>|\>\int_{\mathbb{R}^{d}}||F(x)||^{2}\,dx\right\\}$

And the translation group action on this space as described above.

A translation equivariant network between feature maps with $c_{in}$ inputs
channels and $c_{out}$ output channels l are functions:

$L:L^{2}(R^{d},\mathbb{R}^{c_{in}}\longrightarrow
L^{2}(R^{d},\mathbb{R}^{c_{out}}$

such that the following diagram commutes for $t\in(\mathbb{R}^{d},+)$.

$\xymatrix{\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{in}})\ar[r]^{L}\ar[d]_{t\blacktriangleright}&\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{out}})\ar[d]^{t\bigstar}\\\
\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{in}})\ar[r]_{L}\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{out}})\\\
}\qquad$

Linear translation equivariant functions mapping between feature maps are
essentially convolutions.

###  3.3. Integral transforms

Let

$\mathcal{I}_{\kappa}:\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{in}})\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{out}})$

be integral transform map that is parametrized by a square integrable two-
argument kernel $\kappa$

$\kappa:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}^{c_{in}\times
c_{out}},\>\>(x,y)\to\kappa(x,y)$

defined by

$\boxed{\mathcal{I}_{\kappa}(F)(x):=\int_{\mathbb{R}^{d}}\kappa(x,y)\,F(y)\,dy}$

Let

$\mathcal{K}:\mathbb{R}^{d}\to\mathbb{R}^{c_{in}\times c_{out}},\>\>\>\Delta
x\to\mathcal{K}(\Delta x)$

defined by $\mathcal{K}(\Delta_{x}):=\kappa(\Delta_{x},0)$.

######  Theorem 7 (Regular translations intertwiners are convolutions).

The integral transform $\mathcal{I}_{\kappa}$ is equivariant if and only if
the two-argument kernel $\kappa$ satisfies

$\kappa(x+\mathbf{t},y+\mathbf{t})=\kappa(\mathbf{x},\mathbf{y}),\quad\text{
for any }\mathbf{x},\mathbf{y},\mathbf{t}\in\mathbb{R}^{d}.$

Moreover, the integral trasform reduces to a convolution integral

$\mathcal{I}_{\kappa}(F)(\mathbf{x})=\int_{\mathbb{R}^{d}}K(\mathbf{x},\mathbf{y})F(\mathbf{y})\,dy$

###### Proof.

The integral transformation form $\mathcal{I}_{\kappa}$ is translation
equivariant if $\mathcal{I}_{\kappa}(t\blacktriangleright
F)=t\bigstar\mathcal{I}_{\kappa}(F)$, for all
$\mathbf{t}\in(\mathbb{R}^{d},+)$. Begin with the left-hand side of the
equality:

$\mathcal{I}_{\kappa}(\mathbf{t}\blacktriangleright
F)(x)=\int_{\mathbb{R}^{d}}\kappa(\mathbf{x},\mathbf{y})(t\blacktriangleright
F)(y)\,dy=\int_{\mathbb{R}^{d}}\kappa(\mathbf{x},\mathbf{y})\,F(y-\mathbf{t})\,dy=\int_{\mathbb{R}^{d}}\kappa(\mathbf{x},\tilde{\mathbf{y}}+\mathbf{t})\,F(\tilde{\mathbf{y}})\,d\tilde{\mathbf{y}},$

where $\mathbf{y}=\tilde{\mathbf{y}}+\mathbf{t}$. While the right-hand side is
given by

$\mathbf{t}\bigstar\mathcal{I}_{\kappa}(F)=\int_{\mathbb{R}^{d}}\kappa(\mathbf{x}-\mathbf{t},\mathbf{y})\,F(\mathbf{y})\,dy,\quad\forall\mathbf{t}\in(\mathbb{R}^{d},+),\forall
F\in\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{in}})$

implies the translation invariance constraint

$\kappa(\mathbf{x}+\mathbf{t},\mathbf{y}+\mathbf{t})=\kappa(\mathbf{x},\mathbf{y}),\>\>\>\forall\mathbf{x},\mathbf{y},\mathbf{t}\in\mathbb{R}^{d}$

of the neural connectivity (spatial weight sharing).

If we let $\mathbf{t}=-\mathbf{y}$ then,

$\kappa(\mathbf{x},\mathbf{y})=\kappa(\mathbf{x}-\mathbf{y},0)=\kappa(\Delta
x,0)=\mathcal{K}(\Delta x).$

Thus, this makes the integral transform a convolution. ∎

###  3.4. Translation equivariant bias summation

Let $\mathbf{b}:\mathbb{R}^{d}\to\mathbb{R}^{c_{in}}$ be a bias field. The
bias preserves the number of channels, so $c:=c_{in}=c_{out}$ Consider a bias
operation

$\begin{split}B_{\beta}:\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c})&\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c}),\\\
F&\to F+\mathbf{b}\end{split}$

be an unconstrained bias summation that is parametrized by a square integrable
bias field $\mathbf{b}:\mathbb{R}^{d}\to\mathbb{R}^{c}$

######  Theorem 8 (Translation equivariant bias summation).

The bias summation $B_{\mathbf{b}}$ is equivariant if and only if $\mathbf{b}$
is constant (i.e., $\mathbf{b}(x)=b$ for some $b\in\mathbb{R}^{c}$).

###### Proof.

The bias summation $B_{\mathbf{b}}$ is equivariant if
$B_{\mathbf{b}}(g\blacktriangleright F)=g\blacktriangleright
B_{\mathbf{b}}(F),\>\>\>\forall g\in(\mathbb{R}^{d},+)$.

Begin with the left-hand side of the equality:

$B_{\mathbf{b}}(g\blacktriangleright F)(x)=(g\blacktriangleright
F)(x)+\mathbf{b}(x)=F(x-g)+\mathbf{b}(x),$

While the right-hand side is given by

$[g\blacktriangleright
B_{\mathbf{b}}(F)](x)=[g\blacktriangleright(F+\mathbf{b})](x)=F(x-g)+\mathbf{b}(x-g),$

Hence

$\mathbf{b}(x)=\mathbf{b}(x-g)\>\>for\>\>\>arbitrary\>\>\>x,g\in\mathbb{R}^{d}.$

The bias field is required to be translation invariant. Thus,
$\mathbf{b}(x)=b\>\>\>for\>\>b\in\mathbb{R}^{d}.$ ∎

###  3.5. Translation equivariant local nonlinearities

Let

$S_{\sigma}:\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{in}})\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c_{out}}),\>\>\>\>F\to
S_{\sigma}(F)$

defined by $S_{\sigma}(F)(x)=:\sigma_{x}(F(x))$, where $\sigma$ is a spatially
dependent localized nonlinearity

$\sigma:\mathbb{R}^{d}\times\mathbb{R}^{c_{in}}\to\mathbb{R}^{c_{out}},\>\>\>\>(x,y)\to\sigma_{x}(y)$

######  Theorem 9 (Translation equivariant local nonlinearities).

The spatially dependent localized nonlinearity operation $S_{\sigma}$ is
translation equivariant if and only if $\sigma_{x}=s$ for some
$x\in\mathbb{R}^{d}$.

###### Proof.

The spatially dependent localized nonlinearity operation $S_{\sigma}$ is
translation equivariant if :

$S_{\sigma}(\mathbf{t}\blacktriangleright F)=\mathbf{t}\bigstar
S_{\sigma}(F),\>\>\>\forall\mathbf{t}\in(\mathbb{R}^{d},+)$

Begin with the left-hand side of the equality:

$S_{\sigma}[\mathbf{t}\blacktriangleright
F](\mathbf{x})=\sigma_{\mathbf{x}}[\mathbf{t}\blacktriangleright
F](\mathbf{x})=\sigma_{\mathbf{x}}[F(\mathbf{x}-g)]$

While the right-hand side is given by

$[\mathbf{t}\bigstar
S_{\sigma}(F)](\mathbf{x})=S_{\sigma}[F(\mathbf{x}-\mathbf{t})]=\sigma_{\mathbf{x}-\mathbf{t}}[F(\mathbf{x}-\mathbf{t})]$

Hence $s:=\sigma_{\mathbf{x}}=\sigma_{\mathbf{x}-\mathbf{t}}$ for an arbitrary
$\mathbf{x},\mathbf{t}\in\mathbb{R}^{d}$ ∎

###  3.6. Translation equivariant local pooling operations

Local max pooling is a nonlinear operation that generates a feature field
where the value at a point $x_{0}\in\mathbb{R}^{d}$ is determined by taking
the maximum feature value across channels within a defined pooling region
$R_{x_{0}}\subset R^{d}$ centered around $x_{0}$.

$\mathcal{P}:\;\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c})\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c}),\quad
F\to\max_{y\in R_{x}}{F(y)}$

######  Theorem 10 (Translation equivariance local max pooling).

The local max pooling operation. $\mathcal{P}$ is translation equivariant if
and only if $g^{-1}R_{x}=R_{x-g}$ for all $x\in\mathbb{R}^{d}$,
$g\in(\mathbb{R},+)$.

###### Proof.

The local max pooling operation $\mathcal{P}$ is translation equivariant if

$\mathcal{P}(g\blacktriangleright
F)=g\blacktriangleright\mathcal{P}(F),\quad\forall
x\in\mathbb{R}^{d},g\in(\mathbb{R}^{d},+)$

Begin with the left-hand side:

$\mathcal{P}(g\blacktriangleright F)(x)=\max_{y\in R_{x}}[g\blacktriangleright
F](y)=\max_{y\in R_{x}}[F(y-g)]=\max_{y\in g^{-1}R_{x}}{F(y)}$

And the right-hand side is given by

$[g\blacktriangleright\mathcal{P}(F)](x)=\mathcal{P}[F(x-g)]=\max_{y\in
R_{x-g}}{F(y)}$

∎

######  Definition. 11.

Local average pooling calculates the channel-wise average of the responses.

$\mathcal{P}_{\alpha}:\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c})\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c}),\quad
F\to\alpha\star{F}$

where $\alpha$ is a scalar weighting kernel:

$\alpha:\mathbb{R}^{d}\to\mathbb{R}.$

######  Theorem 12 (Translation equivariance of local average pooling).

The local average pooling operation $\mathcal{P}_{\alpha}$ is by construction
translation equivariant.

###### Proof.

The local max pooling operation $\mathcal{P}$ is translation equivariant if

$\mathcal{P}_{\alpha}(g\blacktriangleright
F)(x)=[g\blacktriangleright\mathcal{P}_{\alpha}(F)](x),\qquad\forall
x\in\mathbb{R}^{d},g\in(\mathbb{R}^{d},+)$
$\mathcal{P}_{\alpha}(g\blacktriangleright F)(x)=\max_{y\in
R_{x}}[g\blacktriangleright F](y)=\max_{y\in R_{x}}[F(y-g)]=\max_{y\in
g^{-1}R_{x}}{F(y)}$

And the right-hand side is given by

$[g\blacktriangleright\mathcal{P}(F)](x)=\mathcal{P}[F(x-g)]=\max_{y\in
R_{x-g}}{F(y)}.$

.∎

###  3.7. Affine group equivariance and steerable Euclidean CNNs

Let $G\leq\operatorname{GL}_{d}(\mathbb{R})$ be a given group. Affine groups
$\operatorname{Aff}(G)$ are semi-direct products of translations
$t\in(\mathbb{R}^{d},+)$ and $G$,
$\operatorname{Aff}(G):=(\mathbb{R}^{d},+)\rtimes G$. The affine group
$\operatorname{Aff}(G)$ acts on Euclidean spaces,

$\begin{split}\operatorname{Aff}(G)\times\mathbb{R}^{d}&\to\mathbb{R}^{2}\\\
(tg,x)&\to gx+t\end{split}$

Notice that

$\left((th)^{-1},x\right)\to g^{-1}(x-t)$

####  3.7.1. Euclidean feature fields and induced affine group representations

The feature spaces of $\operatorname{Aff}(G)$-equivariant Euclidean steerable
CNNs are vector spaces

$L^{2}(\mathbb{R}^{d},\mathbb{R}^{c}):=\\{\mathbb{R}^{d}\to\mathbb{R}^{c}\>|\int_{\mathbb{R}^{d}}||F(x)||^{2}\,dx\leq\infty\\}.$

of square integrable $c$-channel feature fields in $d$ spatial dimensions. The
are associated to some $\rho:G\to\operatorname{GL}_{c}(\mathbb{R})$. The
affine group acts via

$\begin{split}\blacktriangleright_{\rho}:\operatorname{Aff}(G)\times\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c})&\to\mathcal{L}^{2}(\mathbb{R}^{d},\mathbb{R}^{c}),\\\
(tg,F)&\to tg\blacktriangleright_{\rho}F:=\rho(g)F(tg)^{-1}\end{split}$

This action it corresponds to

$\begin{split}\operatorname{Ind}_{G}^{\operatorname{Aff}(G)}\rho:\operatorname{Aff}(G)&\to\operatorname{GL}\left(L^{2}(\mathbb{R}^{d},\mathbb{R}^{c})\right),\\\
tg&\to tg\blacktriangleright_{\rho}(\cdot)\end{split}$

known as induced representation that turns G-representations to
$\operatorname{Aff}(G)$-representations. Euclidean feature fields are elements
of induced affine group representation spaces
$\operatorname{Ind}_{G}^{\operatorname{Aff}(G)}$. Hence,
$\operatorname{Ind}_{G}^{\operatorname{Aff}(G)}\rho$ is a functor that turns
$G$-representations into $\operatorname{Aff}(G)$-representations. Our goal of
the next section is to explore wthether the same machinery can be constructed
when we replace the affine space with a graded vector space.

A full feature space of steerable convolutional neural networks (CNNs)
comprises multiple individual feature fields
$F_{i}:\mathbb{R}^{d}\textrightarrow\mathbb{R}^{c_{i}}$ of different types
$\rho_{i}$ and dimensionalities $c_{i}$. The composite field
$F=\oplus_{i}F_{i}$ transforms according to the direct sum

$\oplus_{i}\operatorname{Ind}_{G}^{\operatorname{Aff}(G)}\rho_{i}=\operatorname{Ind}_{G}^{\operatorname{Aff}(G)}\oplus_{i}\rho_{i}$

and can therefore be viewed as being of type $\oplus_{i}\rho_{i}$ The block
structure of the direct sum representation guarantees hereby that the
individual fields $f_{i}$ transform independently from each other, that is,
their channels do not mix under $G$-transformations. The following visual
illustration is [weiler-book, Fig. 4.4]

Figure 1. Decomposition of steerable CNNs; [weiler-book, Fig. 4.4]

The goal for the rest of this paper is to develop a similar theory of what is
described in this section for artificial neural networks over graded vector
spaces or even more generally over graded modules.

##  4\. Graded vector spaces

Here we give the bare minimum of the background graded vector spaces. The
interested reader can check details at [bourbaki], [roman], [kocul] among
other places.

A graded vector space is a vector space that has the extra structure of a
grading or gradation, which is a decomposition of the vector space into a
direct sum of vector subspaces, generally indexed by the integers. For the
purposes of this paper we will focus on graded vector spaces indexed by
integers, but we also give below the definition of such spaces for a general
index set $I$.

###  4.1. Integer gradation

Let $\mathbb{N}$ be the set of non-negative integers. An $\mathbb{N}$–graded
vector space, often called simply a graded vector space without the prefix
$\mathbb{N}$, is a vector space $V$ together with a decomposition into a
direct sum of the form

${\displaystyle V=\bigoplus_{n\in\mathbb{N}}V_{n}}$

where each $V_{n}$ is a vector space. For a given $n$ the elements of $V_{n}$
are then called homogeneous elements of degree $n$.

Graded vector spaces are common. For example the set of all polynomials in one
or several variables forms a graded vector space, where the homogeneous
elements of degree n are exactly the linear combinations of monomials of
degree $n$.

######  Example 4.

Let $k$ be a field and consider $\mathcal{V}_{(2,3)}$ the space of degree 2
and 3 homogenous polynomials in $k[x,y]$. It is decomposed as
$\mathcal{V}_{(2,3)}=\mathcal{V}_{2}\oplus\mathcal{V}_{3}$, whwre
$\mathcal{V}_{2}$ is the space of binary quadratics and $V_{3}$ the space of
binary cubics. Let $\mathbf{u}=[f,g]\in\mathcal{V}_{2}\oplus\mathcal{V}_{3}$.
Then the scalar multiplication works as

$\lambda\star\mathbf{u}=\lambda\star[f,g]=[\lambda^{2}\,f,\lambda^{3}\,g]$

We will use this example repeatedly for the rest of the paper. ∎

Next we give another example that was our main motivation for machine learning
models over graded vector spaces.

######  Example 5 (Moduli space of genus 2 curves).

Assume $\mbox{char }k\neq 2$ and $C$ a genus 2 curve defined over $k$. Then
$C$ has affine equation $y^{2}=f(x)$ where $f(x)$ is a degree 6 polynomial.
The ismorphism class of $C$ is determined by its invariants
$J_{2},J_{4},J_{6},J_{10}$, which are homogenous polynomials of degree 2, 4,
6, and 10 respectively in terms of the coefficients of $C$. The moduli space
of genus 2 curves defined over $k$ is isomorphic to the weighted projective
space $\mathbb{WP}_{(2,4,6,10),k}$.

###  4.2. General gradation

The subspaces of a graded vector space need not be indexed by the set of
natural numbers, and may be indexed by the elements of any set $I$. An
$I$-graded vector space $V$ is a vector space together with a decomposition
into a direct sum of subspaces indexed by elements $i$ of the set $I$:

${\displaystyle V=\bigoplus_{i\in I}V_{i}}$

The case where $I$ is the ring $\mathbb{Z}/2\mathbb{Z}$ (the elements 0 and 1)
is particularly important in physics. A ($\mathbb{Z}/2\mathbb{Z}$)-graded
vector space is known as a supervector space.

###  4.3. Graded linear maps

For general index sets $I$, a linear map between two $I$-graded vector spaces
$f:V\to W$ is called a graded linear map if it preserves the grading of
homogeneous elements,

$f(V_{i})\subseteq W_{i},\quad\text{ for all }\quad i\in I.$

A graded linear map is also called a homomorphism (or morphism) of graded
vector spaces, or homogeneous linear map.

When $I$ is a commutative monoid (such as $\mathbb{N}$), then one may more
generally define linear maps that are homogeneous of any degree $i$ in $I$ by
the property

$f(V_{j})\subseteq W_{i+j},\quad\text{for all }\quad j\in I$

where ”+” denotes the monoid operation. If moreover $I$ satisfies the
cancellation property so that it can be embedded into an abelian group $A$
that it generates (for instance the integers if $I$ is the natural numbers),
then one may also define linear maps that are homogeneous of degree $i$ in $A$
by the same property (but now ”+” denotes the group operation in $A$).
Specifically, for $i\in I$ a linear map will be homogeneous of degree $-i$ if

$f(V_{i+j})\subseteq W_{j},\;\text{ for all }\;j\in I,\;\text{ while
}\;f(V_{j})=0\;\text{ if }\;j-i\notin I$

Let us see a simple example of a graded linear map.

######  Example 6.

Consider $\mathcal{V}_{(2,3)}=V_{2}\oplus V_{3}$ as in Example 4. Then a
linear map $L:\mathcal{V}_{(2,3)}\to\mathcal{V}_{(2,3)}$ satisfies

$L([\lambda\star\mathbf{u}])=L([\lambda^{2}f,\lambda^{3}g])=[\lambda^{2}L(f),\lambda^{3}L(g)]=\lambda\star[L(f),L(g)]=\lambda\star
L(\mathbf{u})$

and

$\begin{split}&L\left([f,g]\oplus[f^{\prime},g^{\prime}]\right)=L\left([f+f^{\prime},g+g^{\prime}]\right)=[L(f)+L(f^{\prime}),L(g)+L(g^{\prime})]\\\
&=[L(f),L(g)]\oplus[L(f^{\prime}),L(g^{\prime})]=L([f,g])\oplus
L([f^{\prime},g^{\prime}])\end{split}$

We can see things more explicitly if we choose a basis for
$\mathcal{V}_{(2,3)}$. Since $V_{2}$ is the space of binary quadratics
$ax^{2}+bxy+cy^{2}$ we can pick the standard basis of monomials for $V_{2}$ as
$\mathcal{B}_{1}=\\{x^{2},xy,y^{2}\\}$. Similarly a standard basis for $V_{3}$
can be chosen as $\mathcal{B}_{2}=\\{x^{3},x^{2}y,xy^{2},y^{3}\\}$. Hence, a
basis for $\mathcal{V}_{(2,3)}$ can be picked as

$\mathcal{B}=\\{x^{2},xy,y^{2},x^{3},x^{2}y,xy^{2},y^{3}\\}$

For example, the polynomial
$F(x,y)=\left(x^{2}+xy+y^{2}\right)+\left(x^{3}+x^{2}y+xy^{2}+y^{3}\right)$
has coordinates $\mathbf{u}=[1,1,1,1,1,1,1]^{t}$ in this basis. ∎

Further details on isomorphisms of graded rings of linear transformations of
graded vector spaces can be found in [bourbaki], [balaba], [bondarenko], and
others.

Notice that the simplest graded linear map is the ”multiplication” by a
scalar, say $L(\mathbf{x})=\lambda\mathbf{x}$ which has matrix representation
the diagonal matrix

$\begin{bmatrix}\lambda^{q_{0}}&0&0&\dots\\\ 0&\lambda^{q_{1}}&0&\vdots\\\
\vdots&0&\ddots&0\\\ \dots&\dots&0&\lambda^{q_{n}}\\\ \end{bmatrix}$

###  4.4. Operations over graded vector spaces

Some operations on vector spaces can be defined for graded vector spaces as
well. For example, given two $I$-graded vector spaces $V$ and $W$, their
direct sum has underlying vector space $V\oplus W$ with gradation

$\left(V\oplus W\right)_{i}=V_{i}\oplus W_{i}$

If $I$ is a semigroup, then the tensor product of two $I$-graded vector spaces
$V$ and $W$ is another $I$-graded vector space,

$\left(V\otimes W\right)_{i}=\bigoplus_{(j,k):j+k=i}(V_{j}\otimes W_{k})$

We will come back back to tensor products of vectors spaces when more details
are needed.

###  4.5. Inner graded vector spaces

Consider now the case when each $V_{i}$, is a finite fdimensional inner space
and let $\langle\cdot,\cdot\rangle_{i}$ denote the corresponding inner
product. Then we can define an inner product on $V$ as follows. For
$\mathbf{u}=u_{1}+\ldots+u_{n}$ and $\mathbf{v}=v_{1}+\ldots+v_{n}$ we define

$\langle\mathbf{u},\mathbf{v}\rangle=\langle
u_{1},v_{2}\rangle_{1}+\ldots+\langle u_{n},v_{n}\rangle_{n}$

which is the standart product. Then the Euclidean norm is as expected

$\|\mathbf{u}\|=\sqrt{u_{1}^{2}+\ldots+u_{n}^{2}}$

If such $V_{i}$ are not necessary finite dimensional then we have to assume
that $V_{i}$ is a Hilbert space (i.e. a real or complex inner product space
that is also a complete metric space with respect to the distance function
induced by the inner product). This case of Hilbert spaces is especially
important in machine learning and artificial intelligence as pointed out by
Thm. 5.

Obviously having a norm on a graded vector space is important for machine
learning if we want to define a cost function of some type. The simpler case
of Euclidean vector spaces and their norms was considered in [moskowitz],
[songpon].

######  Example 7.

Let us continuw with the space $\mathcal{V}_{(2,3)}$ from Example 4. We
continue with with bases $\mathcal{B}_{1}=\\{x^{2},xy,y^{2}\\}$ and
$\mathcal{B}_{2}=\\{x^{3},x^{2}y,xy^{2},y^{3}\\}$ as in Example 6. Hence, a
basis for $\mathcal{V}_{(2,3)}$ can be picked as
$\mathcal{B}=\\{x^{2},xy,y^{2},x^{3},x^{2}y,xy^{2},y^{3}\\}$. Let us see how
we could define an inner product in $\mathcal{V}_{(2,3)}$. Let
$\mathbf{u},\mathbf{v}\in\mathcal{V}_{(2,3)}$ given by

$\begin{split}\mathbf{u}&=\mathbf{a}+\mathbf{b}=\left(u_{1}\,x^{2}+u_{2}\,xy+u_{3}\,y^{2}\right)+\left(u_{4}\,x^{3}+u_{5}\,x^{2}y+u_{6}\,xy^{2}+u_{7}\,y^{3}\right)\\\
\mathbf{v}&=\mathbf{a}^{\prime}+\mathbf{b}^{\prime}=\left(v_{1}\,x^{2}+v_{2}\,xy+v_{3}\,y^{2}\right)+\left(v_{4}\,x^{3}+v_{5}\,x^{2}y+v_{6}\,xy^{2}+v_{7}\,y^{3}\right)\end{split}$

Then

$\begin{split}\langle\mathbf{u},\mathbf{v}\rangle&=\langle\mathbf{a}+\mathbf{b},\mathbf{a}^{\prime}+\mathbf{b}^{\prime}\rangle=\langle\mathbf{a}+\mathbf{a}^{\prime}\rangle+\langle\mathbf{b}+\mathbf{b}^{\prime}\rangle\\\
&=u_{1}v_{2}+u_{2}v_{2}+u_{3}v_{3}+u_{4}v_{4}+u_{5}v_{5}+u_{6}v_{6}+u_{7}v_{7}\end{split}$

and the Euclidean norm is defined as expected
$\|\mathbf{u}\|=\sqrt{u_{1}^{2}+\ldots u_{7}^{2}}$. ∎

There are other ways to define a norm on graded spaces. Consider a Lie algebra
$\mathfrak{g}$. It is called graded if there is a finite family of subspaces
$V_{1},\ldots,V_{r}$ such that $\mathfrak{g}=V_{1}\oplus\dots\oplus V_{r}$ and
$[V_{i},V_{j}]\subset V_{i+j}$, where $[V_{i},V_{j}]$ is the Lie bracket. When
$\mathfrak{g}$ is graded we define for $t\in\mathbb{R}^{\times}$,
$\alpha_{t}:\mathfrak{g}\to\mathfrak{g}$ such that

$\alpha_{t}(v_{1},\ldots,v_{n})=(tv_{1},t^{2}v_{2},\ldots,t^{r}v_{r}).$

We define a homogenous norm on $\mathfrak{g}$ as

$\|\mathbf{v}\|=\|(v_{1},\ldots,v_{n})\|=\left(\|v_{1}\|_{1}^{2r}+\|v_{2}\|_{2}^{2r-2}+\dots+\|v_{r}\|_{r}^{2}\right)^{1/2r}$

where $\|\cdot\|_{i}$ is the Euclidean norm in $V_{i}$. For details see
[Moskowitz2, moskowitz]. It is shown in [songpon] that this norm satisfies the
triangle inequality.

A more general approach is considered in [SS] defining norms for line bundles
and using such norms in the definition of weighted heights on weighted
projective variaties.

##  5\. Artificial neural networks over graded vector spaces

Let us now try to design artificial neural networks over graded vector spaces.
Let $k$ be a field and for any integer $n\geq 1$ denote by
$\mathbb{A}_{k}^{n}$ (resp. ${\mathbb{P}}_{k}^{n}$) the affine (resp.
projective) space over $k$. When $k$ is an algebraically closed field, we will
drop the subscript. A fixed tuple of positive integers
$\mathbf{w}=(q_{0},\dots,q_{n})$ is called set of weights. The weight of
$\alpha\in k$ will be denoted by $\mathbf{wt}(\alpha)$. The set

$\mathcal{V}_{\mathbf{w}}^{n}(k):=\\{(x_{1},\ldots,x_{n})\in
k^{n}\;|\;\mathbf{wt}(x_{i})=q_{i},\,i=1,\ldots,n\\}$

is a graded vector space over $k$. From now on, when there is no confusion we
will simply use $\mathcal{V}_{\mathbf{w}}$ for a graded vector space.

We follow the analogy with the classical case of artificial neural networks. A
neuron on a graded vector space $\mathcal{V}_{\mathbf{w}}$ is a function
$f:\mathcal{V}_{\mathbf{w}}^{n}\to k$ such that

$\alpha_{\mathbf{w}}(\mathbf{x})=\sum_{i-0}^{n}w_{i}^{q_{i}}x_{i}+\mathbf{b},$

where $\mathbf{b}\in k$ is a constant called bias. We can generalize neurons
to tuples of neurons via

$\begin{split}\phi:=\mathcal{V}_{\mathbf{w}}^{n}(k)&\to\mathcal{V}_{\mathbf{w}}^{n}(k)\\\
\mathbf{x}&\to
g\left(\alpha_{0}(\mathbf{x}),\ldots,\alpha_{n}(\mathbf{x})\right)\end{split}$

for any gives set of weights $\mathbf{w}_{0},\ldots,\mathbf{w}_{n}$. Then
$\phi$ is a $k$-linear function with matrix written as

$\phi(\mathbf{x})=W\cdot\mathbf{x}+\mathbf{b},$

for some $\mathbf{b}\in k^{n+1}$ and $W$ an $n\times n$ matrix with integer
entries.

######  Remark 1.

There is a big confusion here when it comes to terminology. The elements
$w_{i}$ are called weights in classical neural networks, but these are
different from weights of the graded vector space $q_{i}$. The matrix $W$ is
called the matrix of weights since it is the matrix $W=[w_{i,j}]$, but again
those weights are not the same as weights $q_{0},\ldots,q_{n}$.

A non-linear function
$g:\mathcal{V}_{\mathbf{w}}^{n}\to\mathcal{V}_{\mathbf{w}}^{n}$ is called an
graded activation function. A graded network layer is a function

$\begin{split}\mathcal{V}_{\mathbf{w}}^{n}(k)&\to\mathcal{V}_{\mathbf{w}}^{n}(k)\\\
\mathbf{x}&\to g\left(W\cdot\mathbf{x}+\mathbf{b}\right)\end{split}$

for some some activation function $g$. A graded neural network is the
composition of many layers. The $l$-th layer

$\begin{split}\cdots\longrightarrow\mathcal{V}_{\mathbf{w}}^{n}(k)&\stackrel{{\scriptstyle\phi_{l}}}{{\longrightarrow}}\mathcal{V}_{\mathbf{w}}^{n}(k)\longrightarrow\cdots\\\
\mathbf{x}&\longrightarrow\phi_{l}(\mathbf{x})=g_{l}\left(W^{l}\mathbf{x}+\mathbf{b}^{l}\right),\end{split}$

where $g_{l}$, $W^{l}$, and $\mathbf{b}^{l}$ are the activation, matrix, and
bias corresponding to this layer. After $m$ layers the output (predicted
values) will be denoted by
$\hat{\mathbf{y}}=[\hat{y}_{1},\ldots,\hat{y_{n}}]^{t}$, where

$\hat{\mathbf{y}}=\phi_{m}\left(\phi_{m-1}\left(\ldots(\phi_{1}(\mathbf{x}))\ldots\right)\right),$

while the true values by
$\mathbf{y}=[y_{1},\ldots,y_{n}]^{t}\in\mathcal{V}_{\mathbf{w}}^{n}$.

The _relu activation_ function for graded neural networks can be defined
analougusly. Let $\mathbf{x}=[x_{0},\ldots,x_{n}]^{t}\in V_{\mathbf{w}}^{n}$.
Then for each coordinate $i=0,\ldots,n$ we define

$\operatorname{ReLu}_{i}(x_{i})=\max\\{0,\left\lvert\mspace{1.0mu}x_{i}\mspace{1.0mu}\right\rvert^{1/q_{i}}\\}$

Hence, is defined as

$\operatorname{ReLu}(\mathbf{x}):=[\operatorname{ReLu}_{0}(x_{0}),\ldots,\operatorname{ReLu}_{n}(x_{n})]$

######  Example 8.

Consider $\mathcal{V}_{(2,3)}$ as above. Let
$\mathbf{u}\in\mathcal{V}_{(2,3)}$. Then, $\mathbf{u}=f+g$ such that $f\in
V_{2}$ and $g\in V_{3}$. Assume

$f=2x^{2}-9xy+y^{2},\qquad\text{ and }\qquad g=x^{3}-2x^{2}y+xy^{2}+y^{3}$

The coordinated of $\mathbf{u}$ with respect to the basis $\mathcal{B}$ fixed
in Example 4 are $\mathbf{u}_{\mathcal{B}}=[2,-3,1,1,-2,11]^{t}$ and

$\operatorname{ReLu}(\mathbf{u})=[\sqrt{2},3,1,1,\sqrt{2},1,1]^{t}.$

It remains to be seen if this activation function or many others which can be
adopted in our settings will be efficient. Notice the similarity of this
definition with the weighted heights defined in [SS, vojta].

###  5.1. Artificial neural networks on weighted projective spaces

Our intention is to build a complete theory of equivariant nueral networks
over graded vector spaces and make it possible to design machine learning
models to study weighted projective spaces and weighted varieties among other
applications. It is unclear how such models would perform, however
mathematically there is every reason to believe that computations on weighted
projective varieties are more efficient than over classical projective
varieties.

In [sh-2024] we used current techniques of machine learning to study the
weighted projective space $\mathbb{WP}_{(2,4,6,10)}$ which is the moduli soace
of genus two curves. The input features were invariants $J_{2},J_{4},J_{6}$,
and $J_{10}$, which represent a point in a graded space. However, current
techniques are designed for classical vector spaces. Hence, while we got some
interesting results in [sh-2024] we weren’t sure what these results represent.
In other words, one can’t really understand what is happening in a graded
space unless the gradation becomes part of the training.

This paper is the first attempt in what we envision as a long project of
machine learning in graded vector spaces. Most of mathematical details have
yet to be worked out and one has to explore graded Lie algebras, graded
manifolds and also different gradings. Especially interesting is the case when
the ground field is not $\mathbb{R}$ or $\mathbb{C}$, but $\mathbb{Q}$ or a a
field of positive characteristic. Such tasks present challenges mathematically
and from the implementation point of view and their performance and efficiency
are still open questions.

## References