Using the total CNOT counts given in Fig. <ref>(d), we get a rough estimate of the time taken to apply a single block encoding step $\Pi_{\varphi_j}U$ by multiplying the CNOT count by the two-qubit gate time for each processor. The values calculated are shown in Table <ref> both as an absolute time, as well as a multiple of the for T1 and T2 times. This makes three assumptions, (1) when compiled to the particular hardware gate set, the number of native two-qubit gates is roughly equal to the number of CNOTs we have counted, (2) total runtime is dominated by two-qubit gate operations, and (3) CNOT gates are implemented in serial. In reality, many of these CNOTs could be run in parallel or even removed by recompilation; so this is likely to be an overestimate. We can instead assume that all gates are maximally parallelised such that two-qubit gates are applied in layers of $N_\text{qubits}/2$ acting at the same time to give a lower bound on the run time. These values are shown in Table <ref>.

Hardware information for range of superconducting (SC) and ion trap quantum processors. T1 and T2 correspond to qubit relaxation and coherence times and two qubit gate time refers to native two-qubit gate execution times. The Quantinuum H2-1 and IonQ Forte processors, T1, T2 and two qubit gate times were calculated as described in main text of this Appendix. T1 time for the Quantinuum H2-1 could not be found. Values for other processors are median values taken from the corresponding references.

Processor     Type    # Qubits     T1     T2     two qubit gate time

IBM Eagle r3 <cit.>     Superconducting     127     275 $\mu$s     117 $\mu$s     636 ns

Google Sycamore <cit.>     Superconducting     70     22.9 $\mu$s     15.5 $\mu$s    20 ns

Quantinuum H2-1 <cit.>     Ion trap     32     ?     1410 s     6.46 ms

IonQ Forte <cit.>     Ion trap     32     10–100 s     $\sim$1s    931 $\mu$s

Estimated block encoding runtime for different quantum processors assuming that two-qubit gates are executed in series for different system sizes. Runtimes are given in seconds as well as proportions of the T1 (where known) and T2 times for each processor using data from Table <ref>.

2*Processor     3cBlock encoding runtime (s)     3cRuntime as fraction of T1     3cRuntime as fraction of T2

$N$=100     $N$=1,000     $N$=10,000     $N$=100     $N$=1,000     $N$=10,000     $N$=100     $N$=1,000     $N$=10,000

Eagle r3     $5.14 \times 10^{-4}$     $4.47 \times 10^{-2}$     $5.92 \times 10^{0}$     $1.87 \times 10^{0}$     $1.63 \times 10^{2}$     $2.15 \times 10^{4}$     $4.40 \times 10^{0}$     $3.82 \times 10^{2}$     $5.06 \times 10^{4}$

Sycamore     $1.62 \times 10^{-2}$     $1.41 \times 10^{0}$     $1.86 \times 10^{2}$     $7.06 \times 10^{2}$     $6.14 \times 10^{4}$     $8.13 \times 10^{6}$     $1.04 \times 10^{3}$     $9.08 \times 10^{4}$     $1.20 \times 10^{7}$

H2-1     $5.22 \times 10^{3}$     $4.54 \times 10^{5}$     $6.01 \times 10^{7}$     -     -     -     $3.71 \times 10^{0}$     $3.22 \times 10^{2}$     $4.26 \times 10^{4}$

Forte     $7.53 \times 10^{2}$     $6.55 \times 10^{4}$     $8.66 \times 10^{6}$     $7.53 \times 10^{1}$     $6.55 \times 10^{3}$     $8.66 \times 10^{5}$     $7.53 \times 10^{2}$     $6.55 \times 10^{4}$     $8.66 \times 10^{6}$

Estimated block encoding run time for different quantum processors assuming that  two-qubit gates are maximally parallelised. Run times are given as proportions of the T1 (where known) and T2 times for each processor using data from Table <ref>.

2*Processor     3cBlock encoding runtime (s)      3cRuntime as fraction of T1     3cRuntime as fraction of T2

$N$=100     $N$=1,000     $N$=10,000     $N$=100     $N$=1,000     $N$=10,000     $N$=100     $N$=1,000     $N$=10,000

Eagle r3     $5.16 \times 10^{-7}$     $2.99 \times 10^{-6}$     $2.97 \times 10^{-5}$     $1.88 \times 10^{-3}$     $1.09 \times 10^{-2}$     $1.08 \times 10^{-1}$     $4.41 \times 10^{-3}$     $2.56 \times 10^{-2}$     $2.54 \times 10^{-1}$

Sycamore    $1.62 \times 10^{-5}$     $9.41 \times 10^{-5}$     $ 9.34 \times 10^{-4}$     $7.09 \times 10^{-1}$     $4.11 \times 10^{0}$     $4.08 \times 10^{1}$     $1.05 \times 10^{0}$     $6.07 \times 10^{0}$     $6.02 \times 10^{1}$

H2-1     $5.24 \times 10^{0}$     $3.04 \times 10^{-1}$     $3.02 \times 10^{-2}$     -     -     -     $3.72 \times 10^{-3}$     $2.16 \times 10^{-2}$     $2.14 \times 10^{-1}$

Forte     $7.55 \times 10^{-1}$     $4.38 \times 10^{0}$     $4.35 \times 10^{1}$     $7.55 \times 10^{-2}$     $4.38 \times 10^{-1}$     $4.35 \times 10^{0}$     $7.55 \times 10^{-1}$     $4.38 \times 10^{0}$     $4.35 \times 10^{1}$


11institutetext: Johannes Kepler University, Linz, Austria 22institutetext:
LIT AI Lab, Linz, Austria 33institutetext: University of Alberta, Edmonton,
Canada
33email<EMAIL_ADDRESS>
33email<EMAIL_ADDRESS>
# Efficiently Training Neural Networks for Imperfect Information Games
by Sampling Information Sets

Timo Bertram 11 Johannes Fürnkranz 1122 Martin Müller 33

###### Abstract

In imperfect information games, the evaluation of a game state not only
depends on the observable world but also relies on hidden parts of the
environment. As accessing the obstructed information trivialises state
evaluations, one approach to tackle such problems is to estimate the value of
the imperfect state as a combination of all states in the information set,
i.e., all possible states that are consistent with the current imperfect
information. In this work, the goal is to learn a function that maps from the
imperfect game information state to its expected value. However, constructing
a perfect training set, i.e. an enumeration of the whole information set for
numerous imperfect states, is often infeasible. To compute the expected values
for an imperfect information game like Reconnaissance Blind Chess, one would
need to evaluate thousands of chess positions just to obtain the training
target for a single state. Still, the expected value of a state can already be
approximated with appropriate accuracy from a much smaller set of evaluations.
Thus, in this paper, we empirically investigate how a budget of perfect
information game evaluations should be distributed among training samples to
maximise the return. Our results show that sampling a small number of states,
in our experiments roughly 3, for a larger number of separate positions is
preferable over repeatedly sampling a smaller quantity of states. Thus, we
find that in our case, the quantity of different samples seems to be more
important than higher target quality.

###### Keywords:

neural networks imperfect information games

##  1 Introduction

Imperfect information games, which are characterised by unobservable aspects,
are an important part of Game AI research. In recent years, they have received
increased attention due to the inherent complexity of managing uncertainty.
This category encompasses a wide array of games, spanning from classical card
games like Poker and Bridge to adaptions of traditional board games such as
Dark Hex and Reconnaissance Blind Chess and real-time video games like
Starcraft, Dota II and Counter-Strike. Thus, we see much interest —
commercially and scientifically — in mastering this category of games.
However, methods that conquered many classical perfect information games, e.g.
AlphaZero [20], do not easily carry over to imperfect information games [16].
In imperfect information settings, decisions are typically based on a fusion
of the public information and an implicitly learned or directly computed
expected value of the hidden information. While there are several approaches
to learning evaluations implicitly (see Section 3), we here focus on learning
them explicitly in a supervised fashion.

At every decision point of an imperfect information game, the set of all
possible states from one observer’s perspective is called an information set.
For sequential games such as Poker and Reconnaissance Blind Chess, it is
possible to enumerate the states in the information set, thus allowing for
approximation of the total evaluation of the imperfect information state as an
aggregation of the single evaluations in the set. However, a perfect
enumeration of the set can be vastly too costly. Thus, we aim to learn an
evaluator which encapsulates the relationship between an imperfect information
state and its aggregated target evaluation, enabling fast evaluation of
states. In training, however, the same problem persists; generating perfect
target values for states requires an unreasonable amount of evaluations. To
reduce this load, we only use parts of the information set for target
generation and investigate how many individual states are required for
training.

Figure 1: Estimating the value of an imperfect information position (left) as
the average of the perfect information evaluations of all positions in the
information set.

This setting is motivated by work in Reconnaissance Blind Chess (RBC), a chess
variant where parts of the board are not observable, and thus agents have to
evaluate imperfect information game states. Many leading programs approximate
these values by evaluating the states in the information set with a
conventional perfect information chess engine and averaging the results. While
these averages can serve as training signals for a model, evaluating every
single board state in each information set is unreasonable. In this work, we
question how we can best invest a fixed budget of total game state
evaluations; should they be spread out to obtain inaccurate estimates for more
training data or should they provide more reliable training signals for fewer
data points? We begin this work by outlining the problem more formally
(Section 2) and give a brief overview of related work and problems (Section
3). Subsequently, we empirically investigate the problem in two different
imperfect information games:

  * •

Heads-Up Poker: Here we evaluate a 2-card hand without knowledge about the
community cards or the opponent’s cards, sampling from information sets of
possible completions to estimate the win rate of a hand.

  * •

Reconnaissance Blind Chess (RBC) [7]: In this chess variant, the opponent’s
moves are often uncertain because of limited information. We receive an
estimated evaluation of the imperfect information game state based on possible
perfect game evaluations obtained from a conventional chess engine.

We summarise our results in Section 5 and give an outlook on potential future
extensions in Section 6.

##  2 Problem Statement

We formalise the problem as follows: Given is a dataset of examples
$\mathcal{D}=(\mathbf{x}_{i},y_{i})\subset X\times Y$, where each label
$y_{i}=f(\mathbf{x}_{i},\mathbf{h}_{i})$ is determined by a function $f$,
dependent not only on the observable information $\mathbf{x}_{i}$, but also on
the hidden information $\mathbf{h}_{i}$. Our goal is to find a function
$g(\mathbf{x})$ which approximates $f(\mathbf{x},\mathbf{h})$, such that
$\forall i\in\\{1,..,|D|\\}:g(\mathbf{x}_{i})\approx
f(\mathbf{x}_{i},\mathbf{h}_{i})$. This task is non-trivial, and such a
function $g$ does not generally exist, as the same observable
$\tilde{\mathbf{x}}$ can occur multiple times with different labels because,
in general, $f(\tilde{\mathbf{x}},\mathbf{h^{(1)}})\neq
f(\tilde{\mathbf{x}},\mathbf{h^{(2)}})$ for
$\mathbf{h^{(1)}}\neq\mathbf{h^{(2)}}$.

Our motivation for this problem originates from imperfect information games,
where the information set represents all possible game states given one
player’s information. In several such games, remarkable performance has been
achieved by basing the imperfect information gameplay, whether implicitly or
explicitly, on perfect information evaluations of states in an information set
[3, 1, 5]. For example, the value of a player’s hand in Poker can be estimated
as the expected value of the hand over all possible variations of the
community and opponent’s cards. Similarly, many strong RBC agents rely heavily
on chess engines for evaluating conventional chess positions [7, 15, 8] and
approximate the imperfect information state with the expected values of the
states in the information set. Figure 1 illustrates this situation with a
schematic $3\times 3$ chess board where only the white pieces are public and
the evaluation of this board is obtained by averaging the (hypothetical)
evaluations of all possible configurations of black pieces.

It is important to acknowledge the limitations of basing imperfect information
policy fully on perfect information evaluations, as it is trivial to construct
counterexamples where this fails. Despite its fundamental problems, the idea
of estimating the evaluations by sampling from information sets has been
successful in many algorithms. For example, using perfect information search
on sampled states has been used in games such as Bridge [9], Skat [12] and
Scrabble [19]. In particular, Blüml et al. [3] have noticed that perfect
information evaluations for imperfect information games perform surprisingly
strongly. On a smaller scale, one can also relate this concept to the central
idea of Monte-Carlo Tree Search. There, nodes in the game tree are evaluated
based on stochastic rollouts, which results in a noisy but still useful
signal. Similarly to the problem at hand, one can use more sophisticated, but
also slower, rollout strategies to receive more accurate signals or more
frequent fast and noisy rollouts.

Figure 2: Learning an imperfect information evaluation function from $n$
examples, for which the target evaluation is estimated from $k$ position,
using a constant budget of $N=n\cdot k$ perfect information evaluations.

The central objective of this work is to learn the function $g$ which receives
the public information of a state $\mathbf{x}$ and approximates the expected
value of that state, without the need for its (potentially expensive) explicit
computation by iterating over the information set:

$g(\mathbf{x})\approx\hat{y}=\sum_{\mathbf{h}}P(\mathbf{h}|\mathbf{x})\cdot
f(\mathbf{x},\mathbf{h})$ (1)

Here, $\mathbf{h}\in\mathcal{I}$ are all possible configurations of private
information that are part of the information set $\mathcal{I}$, $f$ is an
evaluator of a perfect information state and $P$ is a function which gives the
probability of each hidden state for the given configuration $\mathbf{x}$. In
practice, $P$ can be influenced by, among others, stochastic environments or
adversary’s (hidden) policies. In all cases, it is assumed to be an
unobservable and unalterable part of the domain. In our experiments, as in
Figure 1, we assume that all possible determinations are equally likely, i.e.
$P(\mathbf{h}|\mathbf{x})=\nicefrac{{1}}{{|\mathcal{I}(\mathbf{x})|}}$ but one
could also directly learn meaningful weights for the positions in the
information set from past behaviour or observations [2].

A simple strategy to learn $g$ is to collect samples of the form
$(\mathbf{x}_{i},\hat{y}_{i})$, i.e., to compute the exact value $\hat{y}_{i}$
as in Equation (1) for many training positions $\mathbf{x}_{i}$, and to use
supervised learning to learn the function $\hat{y}_{i}=g(\mathbf{x}_{i})$ from
these samples. However, this approach generally is too costly due to the
potentially large size of information sets, so obtaining a single
$\hat{y}_{i}$ can require thousands of evaluations. Alternatively, $\hat{y}$
can be approximated by randomly sampling only a few of the possible
$\mathbf{h}^{(j)}$, resulting in less accurate training signals $\dot{y}$ at a
lower computational cost.

In our work, we aim to answer a fundamental question: Given a fixed budget of
$N$ perfect information evaluations, how should we generate the training data
for the learner? Options range from generating $N$ different training examples
$\mathbf{x}_{i}$, each evaluated with one random sample, over using a fixed
number of $k$ evaluations to generate targets for $n=\nicefrac{{N}}{{k}}$
positions, up to exhausting the budget with exactly computing $\hat{y}$ for as
many examples as possible. This trade-off between the training set size $n$
(the number of distinct $\mathbf{x}_{i}$) and label quality (the number of
evaluations $k$ used to estimate the intended target values $\hat{y}_{i}$ for
each $\mathbf{x}_{i}$) forms the core focus of this paper.

##  3 Related Work

The problem formulated in Section 2 is multifaceted and occurs in several
learning paradigms, thus we can only give a brief overview of how it manifests
in practice.

Several learning settings may be viewed as special cases of this formulation.
Conventional supervised learning emerges when
$\mathbf{h}_{i}=\emptyset,\forall i$, i.e. when no hidden information
determines $y_{i}$. Similarly, learning from noisy labels can be formulated
with a single hidden variable $\mathbf{h}_{i}$, which determines whether the
original label remains intact or is corrupted. Knowledge of this hidden
information makes the underlying function $f$ deterministic, but $g$ does not
have access to the information about the corruption.

Our setting is also closely related to research in the area of noisy labels
[21, 11], which extends to crowdsourcing [18, 10] and aggregating labels from
different labellers. It also shares commonalities with active learning [17],
which in our case is a problem of deciding whether to re-sample an existing
example to improve label quality or to obtain a new sample to increase overall
training data quantity. Importantly, most of this research aims to improve
data distribution to the labellers or to reduce bias post-sampling, which
differs significantly from choosing a sampling frequency a priori. In
addition, noisy classification differs substantially from noisy regression.

In the context of imperfect information games, numerous approaches exist to,
explicitly or implicitly, evaluate an information set. Techniques such as
Perfect Information Monte Carlo [13, 6] combine evaluations of different
perfect information searches into a policy for the imperfect information state
and Information Set Monte Carlo Tree Search [22] operates on information sets.
Counterfactual Regret Minimization [23], as well as its successors and ReBeL
[4] learn the utility of individual information sets through self-play. Recent
work by Blüml et al.[3] samples individual world states and constructs
imperfect information policies based on their evaluations. In essence, most
techniques for solving imperfect information games involve estimating the
value of information sets, further motivating the importance of the question
which we aim to answer.

Finally, the general idea that sampling more states from information sets will
lead to a more accurate estimate of its overall value is simply an instance of
the law of large numbers and thus finds application in a variety of problems.
This trade-off between quantity and quality of evaluations is analogous to the
choice of rollout policy in classical Monte Carlo Tree Search [5], where one
has to decide between random rollouts (fast, and thus allowing a larger
quantity, but less informative) and more sophisticated rollout-policies
(slower, thus limiting their number, but better at approximating true
behaviour). However, one has to consider that MCTS differs slightly because
UCB will over time re-visit promising states, thus allowing the resampling of
important ones.

##  4 Experiments

Here, we present a series of experiments designed to investigate the trade-off
between target quality and training data quantity. The learner has a limited
budget of total evaluations $N$ and can decide how many evaluations $k$ should
be spent on each training example. Each evaluation yields one value, which are
aggregated into a training target by averaging.

###  4.1 Texas Hold’em Poker

The first experiment aims at a real-world setting where we have to balance the
label accuracy with the number of total training examples seen. Here, the
learner aims to estimate the win probability of a given 2-card hand of cards
in two-player heads-up poker. This is a direct implementation of the problem
outlined in Section 2. Pre-computed win-rate tables for any given hand exist,
so one should regard this as a proof-of-concept rather than a useful
application.

Figure 3: Monte-Carlo estimate of error in evaluating the change of winning
with a given hand pre-flop- in heads-up poker. Estimations are computed by
averaging over n samples of possible opponent hands and rivers.

####  4.1.1 Setup

In principle, for a given hand $\mathbf{x}$, $g(\mathbf{x})$ could be directly
computed as the average over all possible hidden contexts
$\mathbf{h}_{\mathbf{x}}$, but doing so requires a large amount of
computation. Without accounting for symmetry, a player can have
$\binom{52}{2}=1326$ unique Poker hands. One would need to compute all
possible arrangements of the remaining cards into two opponent cards and five
community cards, i.e.
$\binom{52}{2}\cdot\binom{50}{2}\cdot\binom{48}{5}=2{,}781{,}381{,}002{,}400$
total combinations. For each of these combinations, one needs to evaluate
which player won the game and average this for all configurations that pertain
to the same player’s hand to estimate the overall winning probability of that
hand. While public data for the win-chances of a hand exists, such data is
only available for the most popular games and computing them is much costlier
in other games with higher degrees of uncertainty or more expensive state
evaluations. Thus, we aim to decrease the computational cost by only sampling
parts of the information set instead of enumerating it entirely, and thus
allow for the extension of the concept to a larger variety of applications.

When training the learner, we sample $k$ different configurations of cards for
the given hand, evaluate the result of each configuration ($0$, $1$ or $0.5$),
and train the network to predict the mean of all $k$ samples. Sampling more
combinations leads to a smaller difference between the estimate and the ground
truth, but results in a smaller amount of total hands seen when equating by
the total number of evaluations.

####  4.1.2 Results

As a first estimate, Figure 3 shows the discrepancy between an estimated hand
strength through evaluations and the mathematical true win chance. Notable,
with only a single sampled configuration, it is impossible to exactly receive
the true win chance of most hands as the only possible results are $0$, $0.5$,
and $1$, thus resulting in three error clusters for one sample.

Figure 4: Average training curves of learning to evaluate a poker hand with
different numbers of evaluations per training example. The $x$-axis is
logarithmically scaled either by the total number of hand evaluations (top) or
by the total number of update steps made (bottom).

The training process (Figure 4) shows that training with fewer evaluations per
sample leads to much quicker progress when regarding the performance in
relation to the total number of evaluations, but when the examples have
higher-quality evaluations, each update is more meaningful. When comparing the
best versions (Figure 5), we see that even when equating for the total number
of evaluations, using a single evaluation leads to worse peak results than
using two, three, five, and ten sampled evaluations. As it should be, equating
training updates makes more evaluations perform strictly better than less. All
in all, these results suggest that multiple samples are useful for this
setting, but naturally, spending too much computation on a single example
degrades the overall performance as the total training size diminishes.

Figure 5: Average lowest received error for the different options of hand
validations, given a total budget of either 100M hand evaluations or 1M
training updates.

We extend our experiments in the next section, where we regard a similar
setting. For that, pre-computed evaluations do not exist and approximating
through sampling is a realistic setting.

###  4.2 Reconnaissance Blind Chess

Reconnaissance Blind Chess.111https://rbc.jhuapl.edu/ RBC is an imperfect
information adaption of chess, where players receive limited information about
the opponent’s moves. When training agents to play this game, it is highly
useful to be able to evaluate a specific situation (i.e. the received
observations at one point in time), and evaluation functions for regular chess
are readily available (e.g., from open-source programs such as
Stockfish222https://stockfishchess.org/. Thus, computing the average
evaluation of all states in an information set is an intuitive approach, but
doing so is largely unfeasible due to the information set size.

####  4.2.1 Setup

For this experiment, training data is created offline for each $k$, thus
generating a fixed training set for each setting. Each learner has a fixed
budget of 1 million state evaluations that can be arbitrarily distributed
among different information sets. Based on the previous results, sensible
values of $k$ were chosen as{1,2,3,5,10,25,50,100,1000}, thus resulting in
datasets of approximately {1M, 500k, 333k, 200k, 100k, 40k, 20k, 10k, 1k}
examples respectively.333The exact numbers vary slightly because the
information set can consist of fewer states than $k$, thus exhausting it
completely. Importantly, the number of potential public-information states to
get evaluated in this experiment is much higher than in the previous Section
4.1. For Poker, there are only 1326 unique 2-card hands. However, in RBC, the
number of potential observations which form one information set is estimated
to be $10^{139}$ [14], much larger than our training datasets, thus minimising
the probability of overlapping training example and increasing the importance
of meaningful target value estimations.

####  4.2.2 Results

Figure 6: Monte-Carlo estimate of error in evaluating the odds of winning for
a given observation. Estimations are computed by averaging over $k$ samples of
possible board states, true evaluation is defined as the average over the
whole information set. Note that the y-axis is in a logarithmic scale to
improve readability. Figure 7: Average lowest received error for training
datasets created with a total budget of 1M sampled boards. For each training
example, $k$ different boards are sampled from the information set, leading to
1M/$k$ different training examples.

Again approximating the sampling error through Monte-Carlo estimates (Figure
6), we receive the expected result: Sampling only a small number of states can
lead to a large discrepancy between the approximation and the ground truth,
but we see diminishing returns, such that sampling more than 50 positions only
leads to slight improvements in the approximation. We thus expect efficiency
increases with repeated sampling, but more than 50 states should lead to
meaningfully degrading performance.

Figure 8: Mean training progress of learning to evaluate imperfect information
states with different numbers of boards sampled per training example. The
x-axis is scaled by the total number of evaluations seen. Curves vary in
length because the Neural Networks are individually trained until no further
improvements are seen.

This observation is confirmed by the results in Figures 7 and 8. The training
curves of Figure 8 show that using singular samples results in poor overall
accuracy, but vast oversampling leads to too few total training examples. The
extreme choices of k (1 and 1000) perform poorly, but moderate sampling is
useful.

##  5 Summary and Conclusion

With this work, we provided experimental results on the influence of sampling
different numbers of states from an information set to learn an evaluation of
the whole set. For a given task, a total budget of $N$ evaluations is given,
which can be distributed across information sets. Thus, we investigated the
trade-off between the overall number of training samples generated and the
accuracy of their associated training targets.

Firstly, the trade-off is influenced by the cost of generating evaluations and
the cost of making an update to the learner, so the choice of $k$ needs to be
related to their balance. We find that in Heads Up Poker and Reconnaissance
Blind Chess (Sections 4.1 and 4.2), multiple evaluations consistently improve
the performance and efficiency of the learner. In both domains, sampling two
evaluations per training example leads to the overall best results, and only
using one sample did not perform well in comparison. As the results for both
tasks were similar, we speculate that these findings will translate to more
scenarios, but more work is required to validate this.

##  6 Future Work

We see multiple intriguing lines of further work based on these findings.
First, we here assumed no agency over the process of sampling from the
information sets and no online variation in sample numbers. Removing either of
those assumptions will likely lead to better results, and some strategies have
previously been outlined by Sheng[18] for categorical tasks. In addition,
while our general formulation holds for other distributions of states, we used
a uniform distribution of states for our experiments. This is a sensible
assumption for Poker, but information sets do not follow a uniform
distribution in RBC. Knowledge of this, or even access to a proxy of such a
distribution, would lead to more accurate estimations in real-world tasks.
Whether a non-uniform distribution changes the best choice of sampled
evaluations will be investigated in the future.

## References

  * [1] Bertram, T., Fürnkranz, J., Müller, M.: Supervised and reinforcement learning from observations in reconnaissance blind chess. In: Proceedings of the 2022 IEEE Conference on Games (CoG). pp. 608–611. IEEE (2022)
  * [2] Bertram, T., Fürnkranz, J., Müller, M.: Weighting information sets with Siamese neural networks in reconnaissance blind chess. In: Proceedings of the 2023 IEEE Conference on Games (CoG). IEEE (2023)
  * [3] Blüml, J., Czech, J., Kersting, K.: Alphaze**: Alphazero-like baselines for imperfect information games are surprisingly strong. Frontiers in Artificial Intelligence 6, 1014561 (2023)
  * [4] Brown, N., Bakhtin, A., Lerer, A., Gong, Q.: Combining deep reinforcement learning and search for imperfect-information games. Advances in Neural Information Processing Systems 33, 17057–17069 (2020)
  * [5] Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Cowling, P.I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., Colton, S.: A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games 4(1), 1–43 (2012)
  * [6] Furtak, T., Buro, M.: Recursive Monte Carlo search for imperfect information games. In: 2013 IEEE Conference on Computational Inteligence in Games (CIG). pp. 1–8. IEEE (2013)
  * [7] Gardner, R.W., Lowman, C., Richardson, C., Llorens, A.J., Markowitz, J., Drenkow, N., Newman, A., Clark, G., Perrotta, G., Perrotta, R., Highley, T., Shcherbina, V., Bernadoni, W., Jordan, M., Asenov, A.: The first international competition in machine reconnaissance blind chess. In: Escalante, H.J., Hadsell, R. (eds.) Proceedings of the NeurIPS 2019 Competition and Demonstration Track. vol. 123, pp. 121–130. PMLR, Vancouver, Canada (2019)
  * [8] Gardner, R.W., Perrotta, G., Shah, A., Kalyanakrishnan, S., Wang, K.A., Clark, G., Bertram, T., Fürnkranz, J., Müller, M., Garrison, B.P., et al.: The machine reconnaissance blind chess tournament of NeurIPS 2022. In: Proceedings of the NeurIPS 2022 Competitions and Demonstrations Track. pp. 119–132. PMLR (2023)
  * [9] Ginsberg, M.L.: GIB: imperfect information in a computationally challenging game. Journal of Artificial Intelligence Research 14, 303–358 (2001)
  * [10] Karger, D.R., Oh, S., Shah, D.: Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research 62(1), 1–24 (2014)
  * [11] Khetan, A., Lipton, Z.C., Anandkumar, A.: Learning from noisy singly-labeled data. In: Proceedings of the 6th International Conference on Learning Representations (ICLR). OpenReview.net, Vancouver, BC, Canada (2018)
  * [12] Long, J., Sturtevant, N., Buro, M., Furtak, T.: Understanding the success of perfect information Monte Carlo sampling in game tree search. In: Burgard, W., Roth, D. (eds.) Proceedings of the 25th AAAI Conference on Artificial Intelligence. pp. 134–140 (2010)
  * [13] Long, J., Sturtevant, N., Buro, M., Furtak, T.: Understanding the success of perfect information Monte Carlo sampling in game tree search. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 24, pp. 134–140 (2010)
  * [14] Markowitz, J., Gardner, R.W., Llorens, A.J.: On the complexity of reconnaissance blind chess. arXiv preprint arXiv:1811.03119 (2018)
  * [15] Perrotta, G., Gardner, R.W., Lowman, C., Taufeeque, M., Tongia, N., Kalyanakrishnan, S., Clark, G., Wang, K., Rothberg, E., Garrison, B.P., Dasgupta, P., Canavan, C., McCabe, L.: The second NeurIPS tournament of reconnaissance blind chess. In: Kiela, D., Ciccone, M., Caputo, B. (eds.) Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track. vol. 176, pp. 53–65. PMLR (2021)
  * [16] Schmid, M., Moravcik, M., Burch, N., Kadlec, R., Davidson, J., Waugh, K., Bard, N., Timbers, F., Lanctot, M., Holland, Z., Davoodi, E., Christianson, A., Bowling, M.: Player of games. arXiv preprint 2112.03178 (2021)
  * [17] Settles, B.: Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers (2012)
  * [18] Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 614–622 (2008)
  * [19] Sheppard, B.: Mastering Scrabble. IEEE Intelligent Systems 14(6), 15–16 (November/December 1999)
  * [20] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140–1144 (2018)
  * [21] Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. pp. 254–263 (2008)
  * [22] Whitehouse, D., Powley, E.J., Cowling, P.I.: Determinization and information set monte carlo tree search for the card game dou di zhu. In: 2011 IEEE Conference on Computational Intelligence and Games (CIG’11). pp. 87–94 (2011)
  * [23] Zinkevich, M., Johanson, M., Bowling, M., Piccione, C.: Regret minimization in games with incomplete information. Advances in Neural Information Processing Systems 20 (2007)


11institutetext: University of Ljubljana, Faculty of Computer and Information
Science,
Večna pot 113, 1000 Ljubljana, Slovenia
11email<EMAIL_ADDRESS>
# One model to rule them all: ranking Slovene summarizers

Aleš Žagar  Marko Robnik-Šikonja

###### Abstract

Text summarization is an essential task in natural language processing, and
researchers have developed various approaches over the years, ranging from
rule-based systems to neural networks. However, there is no single model or
approach that performs well on every type of text. We propose a system that
recommends the most suitable summarization model for a given text. The
proposed system employs a fully connected neural network that analyzes the
input content and predicts which summarizer should score the best in terms of
ROUGE score for a given input. The meta-model selects among four different
summarization models, developed for the Slovene language, using different
properties of the input, in particular its Doc2Vec document representation.
The four Slovene summarization models deal with different challenges
associated with text summarization in a less-resourced language. We evaluate
the proposed SloMetaSum model performance automatically and parts of it
manually. The results show that the system successfully automates the step of
manually selecting the best model.

###### Keywords:

Text summarization low-resource languages meta-model Slovene language.

##  1 Introduction

Text summarization identifies the essential information in a document or a
collection of documents and presents it in a concise and coherent manner. In
spite of the long efforts of natural language processing (NLP), text
summarization is still a challenging task. With the explosive growth of
digital information, summarizing large volumes of text into a shorter, more
manageable form is becoming increasingly important.

There are two main approaches to text summarization: extractive and
abstractive. Extractive summarization selects a subset of sentences or phrases
from the original text that best represents the content. The selected
sentences are combined to form a summary. Abstractive summarization, on the
other hand, generates new sentences that capture the meaning of the original
text. Extractive summarization is simpler and faster than abstractive
summarization, but it can result in summaries that contain redundant and
repetitive content. Abstractive summarization is more challenging and requires
more advanced natural language processing techniques, but it can produce
human-like summaries.

State-of-the-art technology for text summarization has seen a significant
shift in recent years with the rise of transformer neural network
architectures, such as T5 [13] and GPT-3 [1]. This resulted in the
summarization models whose summaries closely resemble those written by humans,
with few repetitions and inaccuracies. These models are also capable of
processing increasingly long content, enabling the creation of summaries for
larger volumes of text. Consequently, state-of-the-art automatic summaries can
be clear and easy to comprehend for end-users.

In the context of the less-resourced morphologically-rich Slovene language,
text summarization is even more challenging than in English, due to limited
availability of resources and data, as well as research. We produced four
Slovene summarization models with different properties and trained them on
different training data.111Within the scope of the RSDO project:
https://www.cjvt.si/rsdo/. Our four models encompass two extraction
summarizers (one based on a simple word frequency sentence selection, the
other being graph-based), an abstractive T5-based model, and a hybrid
extractive-abstractive model. In general, the T5-based transformer model works
best but may not generalize well for all types of input text. Therefore, we
address the problem of which summarization model is the most appropriate for a
given text, based on text length and genre.

We propose a novel Slovene summarization system (named SloMetaSum), consisting
of extractive, abstractive, and hybrid summarizers and a meta-model that
selects among them. The proposed meta-system consists of a fully connected
neural network that analyzes the input content and recommends the most
suitable summarization model for a given text. To achieve this, SloMetaSum
uses the Doc2Vec [7] numerical representation of documents and predicts the
ROUGE scores for each of the summarizers. By using a combination of
approaches, the system can effectively generate high-quality summaries that
are informative and easy to understand for many types of text, regardless of
their length and genre. 222The demo is available at
https://slovenscina.eu/en/povzemanje. The code repositories are available at
https://github.com/azagsam/metamodel and
https://github.com/clarinsi/SloSummarizer..

Our contributions are:

  * •

We have developed four summarization models that can effectively summarize
text of varying lengths and genres, making them versatile for a range of
applications.

  * •

We overcame the challenges of the low-resourced Slovene language, and created
high-performing models for summarizing Slovene text.

  * •

We have also created a meta-model that can recommend the best-suited
summarization model for a given text based on factors such as length,
complexity, level of abstraction, and intended use case.

The rest of the paper is organized as follows. We present related research in
Section 2. Section 3 describes the datasets. In Section 4, we describe
summarization systems and the meta-model. We present our experiments and
discuss the findings in Section 5. Section 6 concludes and recommends future
research.

##  2 Related work

Early approaches to text summarization relied on statistical frequencies of
words, sentence position, and sentences containing keywords [12]. These
approaches aimed to extract important sentences or phrases from a text and
generate a summary by concatenating them. Abstractive methods involved
deleting less important words from the text to create a summary [6].

Graph-based methods have been another popular approach to text summarization.
In this approach, the document is represented as a graph, where sentences are
nodes, and edges represent the relationships between them. The graph is then
used to generate a summary by selecting the most important sentences. This
method has been explored in several works [10], [3].

With the advent of neural networks, there has been an increasing interest in
developing abstractive summarization techniques. Early neural abstractive
systems used methods such as LSTM and other recurrent neural networks [14],
[11]. However, transformer-based architectures have emerged as state-of-the-
art models for abstractive text summarization [18], [9]. These models use
self-attention mechanisms to selectively focus on important parts of the text
and can generate more fluent and coherent summaries compared to earlier
methods.

While several approaches have been proposed for text summarization, many of
them are designed to handle specific genres or types of text. In this work,
our goal is to build a summarization system that can handle every type of text
and genre with every possible property that can appear in the real world. This
includes texts of varying lengths, topics, styles, and summaries that capture
the most important information in the text. Achieving this goal requires
developing a robust and adaptable model that can learn to summarize texts of
diverse types and produce high-quality summaries.

##  3 Datasets

In this section, we describe the datasets we used in our research. Below, we
provide a short description of the datasets, with their statistics contained
in Table 1.

The STA dataset (general news articles from the Slovenian Press Agency)
consists of 366,126 documents and the first paragraph of each article was used
as a proxy for summary since the dataset does not contain hand-written human
summaries. This is a common technique in text summarization, especially in
languages that do not have dedicated news article summarization datasets such
as English.

AutoSentiNews [2] is a similar dataset to STA, consisting of 256,567 articles
from the Slovenian news portals 24ur, Dnevnik, Finance, RTVSlo, and Žurnal24.
The summaries are produced from the first paragraph in the same way as they
are in the STA dataset.

The SURS dataset is a small financial news dataset from the Slovenian
statistical office and consists of 4,073 documents.

The KAS corpus of Slovene academic writing [16] consists of BSc/BA, MSc/MA,
and PhD theses written from 2000 - 2018 and gathered from the digital
libraries of Slovene higher education institutions via the Slovene Open
Science portal 333http://openscience.si/. The corpus contains human-written
abstracts of academic texts.

CNN/Daily Mail dataset [5] is for text summarization. It has human-generated
abstractive summary bullets from news stories on CNN and Daily Mail websites.
The corpus has 286,817 training pairs, 13,368 validation pairs, and 11,487
test pairs. The source documents have 766 words and the summaries consist of
53 words on average. We translated the dataset in Slovene using machine
translation [8].

Dataset | Number of documents
---|---
STA | 334,696
AutoSentiNews | 256,567
SURS | 4,073
KAS | 82,308
Total | 677,644
Table 1: Corpora and datasets used to train a Doc2vec document representation
model and the meta-model.

##  4 The summarization models and the meta-model

In this section, we describe the components of our SloMetaSum system which
consists of four summarization models, a technique for document
representation, and the meta-model.

###  4.1 Summarization models

We produced four summarization models, described below.

Sumbasic [12] uses a simple word frequency approach to select the most
informative sentences. The graph-based summarization model [17] was inspired
by the TextRank algorithm [10] and uses centrality scores of sentences to rank
them. Both models belong to extractive methods and can be used on documents of
any size. In contrast to the original TextRank, we used the transformer-based
LaBSE sentence encoder [4], to numerically represent sentences. The T5-article
abstractive summarization model uses a pre-trained Slovene T5 model [15] and
is fine-tuned on a machine-translated CNN/Daily Mail dataset [5] using the
Slovene machine translation system [8]. The hybrid-long summarization model is
a combination of the graph-based and the T5-article model. It first constructs
a short text by concatenating the most informative sentences (extractive
step). In the next, abstractive step, these sentences are summarized with the
T5-article summarizer.

###  4.2 Doc2Vec model representation

To select the most suitable summarization method for a given text, the meta-
model has to get information about different text properties. We apply the
Doc2Vec model for document representation and train it on the Slovene
documents presented in Table 1 (without abstracts). In the preprocessing step,
we removed high-frequency words that do not contribute to the meaning of a
document, such as pronouns, conjunctions, etc.; to further reduce the number
of different words, we lemmatized the whole dataset.

###  4.3 Meta-model

Our meta-model consists of a fully connected neural network, trained to
predict the ROUGE scores of the summarizers. For a training dataset, we
randomly selected 93,419 examples from the raw concatenated dataset. After
that, each of our four summarizers produced a summary for all examples. We
calculated ROUGE scores between the reference and generated summaries. ROUGE
(Recall-Oriented Understudy for Gisting Evaluation) is a metric most commonly
used for the evaluation of automatically generated text summaries. It measures
the quality of a summary by the number of overlapping units (n-grams,
sequences of texts, etc.) between summaries created by humans and summaries
created by summarization systems. ROUGE is not a single metric but a family of
metrics. The most commonly used are ROUGE-N and ROUGE-L. The first measures
the overlapping of n-grams (typically unigrams and bigrams), while the second
measures the longest common subsequence found in both summaries. As an input
to our meta-model, we use four ROUGE F1-scores (ROUGE-1, ROUGE-2, ROUGE-L,
ROUGE-LSum) that show how good the generated summaries are. We split data into
train, validation, and test sets in ratios of 90:5:5.

The sizes of both datasets are presented in Table 2. In Table 3, we present
the average ROUGE values of our summarizers on long and short texts.
Summarizers that are specialized for short texts achieve better results on
short texts and vice versa.

Model | Training size
---|---
Doc2Vec | 677,644
Meta-model | 93,419
Table 2: Number of training samples for each model. | t5-article | sumbasic | graph-based | hybrid-long
---|---|---|---|---
Short | 14,01 | 13,11 | 13,15 | 12,55
Long | 10,51 | 13,12 | 17,71 | 17,59
Table 3: Summarizers ROUGE scores for long and short texts. The best scores
for short and long texts are in bold.

##  5 Results

In this section, we present our results and evaluation. We report the
performance of the Doc2Vec model and Meta-model in each separate subsection.

###  5.1 Doc2Vec

We used the following hyperparameters for training the Doc2Vec document
representation model: the maximum allowed vocabulary size is 100,000, the size
of the vector used for word representation is 256, the window size of context
words is 5, the minimum frequency of a word to be included in the vocabulary
is 1, and the total number of epochs or iterations for training the model is
5.

We evaluated the Doc2Vec model using manual and automatic techniques. For
manual analysis, we inspected the top 3 most similar returned documents for
each of a few randomly chosen samples using the cosine similarity and observe
whether the topics of the documents overlap. The topics of the documents were
similar in most cases and based on that we concluded that the model works as
expected. The automatic evaluation was part of the whole pipeline, where the
model hyperparameters were tuned to optimize the loss of the meta-model.

###  5.2 Meta-model

Our final results are presented in Table 5. We compared the proposed meta-
model selection mechanism with three baselines. The _Mean-baseline_ model
simply takes the predictions for each summarization model and averages them.
The highest-scoring model is always selected. The _Tree_ uses a regression
tree; using the hyperparameter grid search, the minimum number of samples
required to split an internal node is 100. The _Forest_ method uses a random
forest; we experimented with similar values as for the Tree model and set the
number of tree estimators to 300.

Our best model is a neural network with two hidden layers. The hidden layers
contain 1024 neurons, and we used a validation split of 0.1 during the
training process. The activation function used for this model is the rectified
linear unit (ReLU). In addition, for the early stopping scheduling strategy,
we set the patience parameter to 2. The loss function utilized for this model
is the mean squared error.

Meta-model stopped learning after 7 epochs and performed almost 15 points
above Mean-baseline on the test set. We observed that choosing different
hyperparameters does not seem to significantly affect the results. We
experimented with different hidden layer sizes, numbers of units, and
activation functions. We also tried different max vocabulary and window sizes
of the Doc2Vec model. We report only the values of the best model.

Overall, this model was found to be the most effective among the meta-model
selection strategies we tested. The high number of neurons in the hidden layer
likely contributed to its superior performance, as it allows for a greater
degree of complexity in the model’s representation of the data.

We further experimented with two variations of the meta-model. Meta-model-
length adds another input neuron that explicitly encodes the input length. We
found that this does not improve the model and hypothesize that academic texts
are of different genres and the document embedding technique covers it well
already. We also tried to balance data since the original dataset contains a
1:5 ratio of long to short texts which rises a potential issue of overfitting
on short texts. We reduced the number of short texts in a training set to get
a balanced dataset of 16,932 samples for our Meta-model-balanced model. This
resulted in a worse-performing model but still better than Mean-baseline.

Table 4 shows the frequencies of how many times each model was recommended by
a meta-model out of 1000 samples from a test set. We can see that the
t5-article model was recommended the most, with a count of 595 out of 1000
samples. The hybrid-long model was recommended 254 times, followed by the
Sumbasic model, which was recommended 80 times. The graph-based model was
recommended the least, with a count of 71 out of 1000 samples.

Model | Count
---|---
t5-article | 595
hybrid-long | 254
sumbasic | 80
graph-based | 71
Total | 1000
Table 4: Frequencies of how many times each model was recommended by the meta-model out of 1000 samples from the test set. Model | Mean squared error |
---|---|---
Mean-baseline | 84.493 |
Tree | 81.631 |
Random forest | 74.975 |
Meta-model-baseline | 70.066 |
Meta-model-length | 70.146 |
Meta-model-balanced | 79.044 |
Table 5: Results of our four models on the test set. Meta-model-baseline showed significant improvement over Mean-baseline and tree methods. Encoding the length feature explicitly or balancing the dataset did not improve the results. Method | Precision | Recall | F1-score | Support
---|---|---|---|---
t5-article | 0.33 | 0.11 | 0.16 | 1069
hybrid-long | 0.25 | 0.34 | 0.29 | 817
sumbasic | 0.28 | 0.10 | 0.15 | 1196
graph-based | 0.38 | 0.67 | 0.48 | 1589
Table 6: Classification report. The table includes precision, recall, and
F1-score for each method, as well as the number of instances in the test set
(Support). The methods include t5-article, hybrid-long, sumbasic, and graph-
based. Test accuracy was 0.34.

According to Table 6, the graph-based method achieved the highest F1-score of
0.48, with a precision of 0.38 and recall of 0.67. The hybrid-long method
achieved F1-score of 0.29, with precision 0.25 and recall 0.34. The sumbasic
method produced F1-score of 0.15, precision 0.28, and recall 0.10. Finally,
the t5-article method achieved the lowest F1-score of 0.16, with precision of
0.33 and recall of 0.11. Overall, the test accuracy for all methods combined
was 0.34.

###  5.3 Meta-model vs. the rest

In Table 7, we present the final evaluation results obtained from our
experiments on the test set. It is noteworthy that the proposed Meta-model
outperformed all other models across all ROUGE scores. This result highlights
the effectiveness and superiority of the Meta-model in selecting the most
suitable summarization approach for a given text. This outcome showcases the
potential of our approach in automating the process of selecting the best
summarization model, eliminating the need for manual intervention.

Model | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
t5-article | 19.01 | 5.61 | 13.52
graph-based | 19.47 | 5.52 | 12.50
hybrid-long | 18.55 | 5.42 | 11.73
sumbasic | 18.86 | 5.04 | 12.25
Meta-model | 20.38 | 5.85 | 13.67
Table 7: Performance on the test set for all models. Meta-model achieves the
best results in all three categories.

##  6 Conclusion

In this paper, we proposed a novel system for extractive, abstractive, and
hybrid summarization tasks. Our system consists of a trained fully connected
neural network that analyzes the input content and recommends the most
suitable summarization model for a given text. This approach addresses the
problem of selecting the appropriate model for a new text, which can be short,
long, and of various genres, and can come from almost anywhere when used in
production. Our system provides a more effective and efficient way of
generating high-quality summaries for Slovene texts.

While the proposed SloMetaSum model presents an innovative solution to the
problem of selecting the most suitable summarization model for a given text,
it is not without its weaknesses. One major drawback is the reliance on the
ROUGE score as the sole criterion for model selection. While ROUGE is a
commonly used metric in the field of text summarization, it does not always
accurately reflect the quality of a summary or capture its coherence and
readability. Another potential weakness is the limited scope of the study,
which focuses exclusively on the Slovene language. While the four
summarization models developed for Slovene are an important contribution to
the field, they may not generalize well to other less-resourced languages
since it requieres a good automatic translation system.

Future work could involve extending this system to other languages. Another
area for future work could involve comparing the proposed system with recent
large language models. In addition to evaluating the technical performance of
the system, it would also be useful to conduct user studies to assess its
usefulness and effectiveness in real-world scenarios. For example, researchers
could design experiments to evaluate the system’s ability to summarize news
articles, academic papers, and other types of content that people encounter in
their daily lives.

## Acknowledgments

The work was partially supported by the Slovenian Research Agency (ARRS) core
research programme P6-0411, as well as projects J6-2581, J7-3159, and CRP
V5-2297.

## References

  * [1] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)
  * [2] Bučar, J.: Automatically sentiment annotated slovenian news corpus AutoSentiNews 1.0 (2017), http://hdl.handle.net/11356/1109, slovenian language resource repository CLARIN.SI
  * [3] Erkan, G., Radev, D.R.: Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research 22, 457–479 (2004)
  * [4] Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic BERT sentence embedding. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 878–891 (2022)
  * [5] Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching machines to read and comprehend. Advances in neural information processing systems 28 (2015)
  * [6] Knight, K., Marcu, D.: Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence 139(1), 91–107 (2002)
  * [7] Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International conference on machine learning. pp. 1188–1196. PMLR (2014)
  * [8] Lebar Bajec, I., Repar, A., Bajec, M., Bajec, Ž., Rizvič, M.: NeMo neural machine translation service RSDO-DS4-NMT-API 1.0 (2022), http://hdl.handle.net/11356/1739, slovenian language resource repository CLARIN.SI
  * [9] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880 (2020)
  * [10] Mihalcea, R., Tarau, P.: Textrank: Bringing order into text. In: Proceedings of the 2004 conference on empirical methods in natural language processing. pp. 404–411 (2004)
  * [11] Nallapati, R., Zhou, B., dos Santos, C., Gulçehre, Ç., Xiang, B.: Abstractive text summarization using sequence-to-sequence rnns and beyond. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. pp. 280–290 (2016)
  * [12] Nenkova, A., Vanderwende, L.: The impact of frequency on summarization. Tech. rep., Microsoft Research (2005)
  * [13] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21(140), 1–67 (2020)
  * [14] See, A., Liu, P.J., Manning, C.D.: Get to the point: Summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1073–1083 (2017)
  * [15] Ulčar, M., Robnik-Šikonja, M.: Sequence to sequence pretraining for a less-resourced slovenian language. arXiv preprint arXiv:2207.13988 (2022)
  * [16] Žagar, A., Kavaš, M., Robnik-Šikonja, M., Erjavec, T., Fišer, D., Ljubešić, N., Ferme, M., Borovič, M., Boškovič, B., Ojsteršek, M., Hrovat, G.: Corpus of academic slovene KAS 2.0 (2022), http://hdl.handle.net/11356/1448, slovenian language resource repository CLARIN.SI
  * [17] Žagar, A., Robnik-Šikonja, M.: Unsupervised approach to multilingual user comments summarization. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation. pp. 89–98. Association for Computational Linguistics (Apr 2021)
  * [18] Zhang, J., Zhao, Y., Saleh, M., Liu, P.: Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In: International Conference on Machine Learning. pp. 11328–11339. PMLR (2020)