# Quantization of Large Language Models
with an Overdetermined Basis

Daniil Merkulov
Moscow Institute of Physics and Technology,
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Daria Cherniuk*
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Alexander Rudikov
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Ivan Oseledets
Artificial Intelligence Research Institute
<EMAIL_ADDRESS>
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Ekaterina Muravleva
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
&Aleksandr Mikhalev
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
Boris Kashin
Steklov Mathematical Institute
of Russian Academy of Sciences
Moscow, Russia
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

In this paper, we introduce an algorithm for data quantization based on the
principles of Kashin representation. This approach hinges on decomposing any
given vector, matrix, or tensor into two factors. The first factor maintains a
small infinity norm, while the second exhibits a similarly constrained norm
when multiplied by an orthogonal matrix. Surprisingly, the entries of factors
after decomposition are well-concentrated around several peaks, which allows
us to efficiently replace them with corresponding centroids for quantization
purposes. We study the theoretical properties of the proposed approach and
rigorously evaluate our compression algorithm in the context of next-word
prediction tasks and on a set of downstream tasks for text classification. Our
findings demonstrate that Kashin Quantization achieves competitive or superior
quality in model performance while ensuring data compression, marking a
significant advancement in the field of data quantization.

_Keywords_ Quantization $\cdot$ LLMs $\cdot$ Kashin representation

##  1 Introduction

The realm of data processing and machine learning is increasingly confronting
challenges linked to the efficient storage and management of large volumes of
data. Traditional methods frequently result in elevated storage costs and
computational inefficiencies, particularly when handling modern Large Language
Models (LLMs). The availability of open source LLMs, such as OPT Zhang et al.
[2022], Llama 1 and 2 Touvron et al. [2023a, b], Falcon Almazrouei et al.
[2023] stimulates the need to run them efficiently on local hardware.

In response, we introduce Kashin Quantization, a novel approach for data
quantization that extends beyond the conventional use of the Kashin
representation algorithm. We studied the algorithm for constructing Kashin
representationsKashin and Romskii [2023], proposed a fast matrix version of
the algorithm, and suggested an approach to use it for the quantization of
large models.

The method re-imagines data representation by breaking down any data structure
into two key factors. The first factor is optimized for a minimal infinity
norm, while the second maintains a small infinity norm when post-multiplied by
an orthogonal matrix. This strategy is not merely theoretical; its linear
convergence makes it both effective and rapid in practical applications.

Addressing the challenge of working with resource-intensive orthogonal
matrices, Kashin Quantization integrates structured matrix types, such as
Householder, DCT, and Butterfly matrices. These matrices enable fast matvec
operations, boosting memory efficiency and accelerating computation, thereby
elevating the overall effectiveness of the algorithm.

A pivotal aspect of Kashin Quantization is its proficiency in concentrating
data values around defined peaks (Figure 6), substantially reducing the need
for expansive storage. By mapping these values to centroids of two-dimensional
distributions (one dimension for each factor), it remarkably reduces data
representation size, compressing traditional 32-bit floats to low-bit values.
This technique is notably advantageous in large-scale machine learning models,
where data size and efficiency are paramount.

Expanding its utility, Kashin Quantization is not only pertinent for
quantizing Large Neural Networks but also for compressing exchange
information, like gradients in Federated Learning Safaryan et al. [2022] and
distributed computations. This extension illustrates the versatility of our
approach in diverse machine learning environments.

Furthermore, we enhance Kashin Quantization by incorporating a matrix
reformulation of the Kashin algorithm, which substantially accelerates
computations. This advancement provides a more efficient methodology for
handling large-scale data and complex computations in neural networks.

To validate our approach’s efficacy, we apply Kashin Quantization to several
common benchmarks in natural language processing. Employing language models
like OPT, Bert, RoBerta we demonstrate that our method leads to competitive or
superior predictive quality on both next token prediction and text
classification tasks while maintaining level of quantization. This balance of
efficiency and performance highlights Kashin Quantization’s potential to
transform data quantization practices in machine learning and allied fields.

This paper makes the following contributions:

  1. 1.

We introduce the Matrix Decomposition Kashin Algorithm. The vanilla algorithm
for building Kashin representation requires vectorization of input data and
building an orthogonal matrix of possibly huge dimensions. For an $m\times n$
matrix to decompose, we need a matrix of $(mn)^{2}$ elements. Instead, we
propose to use a Kronecker-factored matrix, which leads to the memory
footprint of $m^{2}+n^{2}$. We benchmark the proposed approach and study its
limitations.

  2. 2.

We explore the theoretical properties of the convergence rate of the algorithm
we proposed. We observed, that the specific choice of basis vectors affects
the convergence. We establish the connection between the convergence rate of
the Vector Decomposition Kashin Algorithm and the Kolmogorov width,
corresponding to the specific choice of basis.

  3. 3.

We propose the Kashin Quantization algorithm for neural network weights
quantization, that utilizes the properties of the Kashin representations. We
benchmark the proposed approach empirically on the family of OPT models on the
next token prediction problem and compare perplexity with the baselines,
demonstrating the validity of the proposed approach. Additionally, we conduct
experiments on the GLUE benchmark, quantizing Bert and RoBerta models
finetuned on several downstream tasks and achieve superior results compared to
common quantization methods.

##  2 Method

#### Representing the vector in overdetermined basis.

Suppose we have a vector $x\in\mathbb{R}^{n}$. Its elements may have different
magnitudes (large ’spread’) and that can prohibit efficient quantization
(i.e., low-bit representation) of the elements. We aim to transform this
vector into another vector, which can be represented with fewer number of
bits. First, we introduce an overdetermined basis of size $2n$: instead of $n$
coefficients, we will represent the vector with $m=2n$ coefficients, but with
fewer bits per element. This basis will consist of the original standard basis
and the basis given by the columns of the orthogonal matrix $Q$. In vector
notation, this is equivalent to representing $x$ as a sum

$x\approx u+Qv,$ (1)

which admits good low-bit approximation (we will discuss this in detail in
Section 4). Fortunately, based on the results in Kashin [1977] and Kashin and
Romskii [2023] this representation can be computed efficiently for an
arbitrary $Q$ except for the set of small measure:

######  Theorem 1 (Kashin and Romskii [2023]).

For $\forall x\in B^{N}_{2}=\\{x\in\mathbb{R}^{N}:\|x\|_{2}\leq 1\\}$ a greedy
algorithm in $k$ steps builds vectors $u_{k}$ and $v_{k}$ such that

$\|x-u_{k}-v_{k}\|_{2}\leq\eta^{k}$
$\|u_{k}\|_{\infty}\leq\frac{c}{\sqrt{N}},\quad\|Qv_{k}\|_{\infty}\leq\frac{c}{\sqrt{N}}$

where $c>0$ is an absolute constant, $Q\in O^{N}$, where $O^{N}$ is a set of
orthogonal matrices in $\mathbb{R}^{N}$ with Haar measure $\mu_{H}$.

#### Computing the representation using a greedy approach.

Without loss of generality, we assume that the vector $x$ has a unit Euclidean
norm $\|x\|_{2}=1$ (otherwise it can be scaled). Given $x$, we can find the
closest vector $\hat{x}$ with elements $\hat{x}_{i}\in\\{-1,1\\}$ as
$\hat{x}_{i}=\mathrm{sign}(x_{i})$, note, that $\|\hat{x}\|_{2}=\sqrt{n}$. The
chosen vector serves as the direction for the first vector $u$ in the relation
(1). Which means, that the projection of $x$ onto $\hat{x}$ is

$\pi_{x}(\hat{x})=\lambda\hat{x}=\|x\|_{2}\dfrac{\langle
x,\hat{x}\rangle}{\|x\|_{2}\|\hat{x}\|_{2}}\dfrac{\hat{x}}{\|\hat{x}\|_{2}}=\dfrac{\|x\|_{1}}{n}\hat{x}.$

Figure 1: We select the vector to project on in a greedy manner.

and since three vectors $x$, $\lambda\hat{x}$ and $x-\lambda\hat{x}$ will
always form an orthogonal triangle by design the distance is given as

$\|x-\lambda\hat{x}\|_{2}^{2}=\|x\|_{2}^{2}-\frac{\|x\|_{1}^{2}}{n^{2}}\|\hat{x}\|_{2}^{2}=\|x\|_{2}^{2}-\frac{\|x\|_{1}^{2}}{n}.$
(2)

The second component $Qv$ from (1) will be formed similarly. Note, that
multiplying the vector $v$ with $\pm 1$ components by matrix $Q$ is equivalent
to the rotation of the whole space with the matrix $Q$, which is also the same
as the backward rotation of all vectors except $v$ with the matrix $Q^{T}$.
Indeed, from (1) $Q^{T}x\approx Q^{T}u+v$. Thus, the closest vector to $x$ of
the form $Qv$ with $v_{i}\in\\{-1,1\\}$ is given as

$\hat{x}=\lambda Q\mathrm{sign}(Q^{T}x),$

and the residual given by the Pythagorean theorem again is

$\|x-\hat{x}\|^{2}_{2}=\|x\|_{2}^{2}-\frac{\|Q^{T}x\|^{2}_{1}}{n}.$ (3)

In the greedy algorithm we start from $x_{0}=x$, then for $k=0,\ldots$ we
select a vector of form $\lambda\hat{x_{k}}$ or $\lambda Q\hat{x_{k}}$ (where
$\hat{x_{k}}$ has elements $\pm 1$) that is the closest to the vector $x_{k}$,
subtract the projection $\pi_{x_{k}}\left(\mathrm{sign}(x_{k})\right)$ or
$\pi_{x_{k}}\left(\mathrm{sign}(Q^{T}x_{k})\right))$ from it and repeat the
procedure until convergence. Thus, we decompose the initial vector decreasing
the norm of $x_{k}$ each iteration and increasing the norm of factors $u_{k}$
and $\hat{v_{k}}$. The idea is summarized in Algorithm 1.

Algorithm 1 Vector Decomposition Kashin Algorithm

Input: Vector $x\in\mathbb{R}^{n}$, Orthogonal matrix $Q$, Tolerance
$\varepsilon>0$

Output: Vectors $u,\hat{v}\in\mathbb{R}^{n}$ such that $x\approx
u+\hat{v}=u+Qv$, and both $u$ and $v$ have small infinity norm.

Initialize $u\leftarrow 0^{n},\hat{v}\leftarrow 0^{n}$

Define projection $\pi_{x}(y)\coloneqq\dfrac{x^{\top}y}{\|y\|_{2}^{2}}\cdot y$

while $\|x-u-\hat{v}\|\geq\varepsilon$ do

if $\|x\|_{1}>\|Q^{T}x\|_{1}$ then

$\pi\leftarrow\pi_{x}(\text{Sign}(x))$

$u\leftarrow u+\pi$

else

$\pi\leftarrow\pi_{x}(Q\text{Sign}(Q^{T}x))$

$\hat{v}\leftarrow\hat{v}+\pi$

end if

$x\leftarrow x-\pi$

end while

Return: $x,u,\hat{v}$

#### Convergence of the algorithm

Figure 2: Convergence criterion ($\|x_{0}-u_{k}-\hat{v}_{k}\|_{2}$) for
randomly generated vector $x$ of length $1000$. The semi-transparent area
shows the confidence interval.

The vector algorithm is guaranteed to converge for arbitrary orthogonal matrix
$Q$ except for a set of small measure Kashin [1977]. Our experiments
demonstrate, that the convergence speed varies when choosing from the classes
of structured orthogonal matrices. The demonstration is presented in Figure 2,
where we took 5 different orthogonal matrices $Q$ and generated $23$ random
vectors of the form $x\in\mathbb{R}^{1000}$. Householder reflection and Givens
rotation perform poorly, whereas dct, random orthogonal and butterfly matrices
perform on par.

Differences in convergence raise a natural question about the convergence
properties of Algorithm 1. On each iteration we have a vector as an input $x$,
from which we subtract a projection, resulting in a vector $x-\hat{x}$. Thus,
we consider the difference between them as a decreasing factor on the
iteration and any constructive approach to bound its norm with respect to the
matrix $Q$ properties is of our interest.

Figure 3: Convergence criterion ($\|x-\hat{x}\|_{2}$) of the matrix
decomposition Kashin algorithm for different layers of OPT-125m model. Left:
DCT matrix converges very slowly after several iterations. Right: DCT matrix
converges well for another layer.

Suppose on $k$-th iteration we have $x_{k}$ and the squared norm after a
single iteration of the algorithm $\|x_{k+1}\|_{2}^{2}$ could be either
$\|x\|_{2}^{2}-\frac{\|x\|_{1}^{2}}{n}$ or
$\|x\|_{2}^{2}-\frac{\|Q^{T}x\|^{2}_{1}}{n}$ from (2) and (3). Note also, that
the choice is made in favor of maximum $1$-norm
$\max(\|x\|_{1},\|Q^{T}x\|_{1})$. Overall, we decrease the residual norm at
each iteration, which ensures the convergence of the algorithm. One can
consider starting iteration, where $\|x_{0}\|_{2}=\|x\|_{2}=1$, which leads to
the

$\|x-\hat{x}\|^{2}\leq
1-\frac{1}{n}\left(\max(\|x\|_{1},\|Q^{T}x\|_{1})\right)^{2}$ (4)

Clearly, the specific choice of the matrix $Q$ affects the quantity
$\max(\|x\|_{1},\|Q^{T}x\|_{1})$. Moreover, this simple estimation immediately
poses another interesting question. How does the choice of an input vector $x$
affect the convergence? The answer could be obtained from studying the
following optimization problem, which is clearly related to the Kolmogorov
width Temlyakov [1998] estimation:

$\min_{\|x\|_{2}=1}\max(\|x\|_{1},\|Q^{T}x\|_{1})$ (5)

The problem (5) determines the worst-case convergence rate for a specific
matrix $Q$. To address the problem we considered several classes of orthogonal
matrices $Q$ (see more details in section 3) and performed a search across all
eigenvectors of the matrix $Q^{T}$ ($\|Q^{T}x\|_{1}=\|\lambda x\|_{1}$). The
results are presented in Table 1.

#### Matrix version of the algorithm.

As we delved into the nuances of matrix decomposition, leveraging the
pioneering Kashin vector decomposition algorithm 1 initially seemed a
straightforward extension: vectorize the matrices and proceed with the
decomposition. Yet, this simplistic approach soon revealed its limitations,
especially when faced with matrices of even moderate size, such as the
$768\times 3072$ matrices encountered in the fully connected layers of the
OPT-125m model. The challenge of managing large vectors soon made it clear
that we needed a new approach. This led to the development of the Matrix
Decomposition Kashin Algorithm (Algorithm 2), an improved version of the
earlier method, specifically created to tackle the difficulties involved in
breaking down matrices.

Algorithm 2 Matrix Decomposition Kashin Algorithm

Input: Matrix $X\in\mathbb{R}^{m\times n}$, Orthogonal matrices $Q_{1},Q_{2}$,
Tolerance $\varepsilon>0$

Output: Matrices $U,\hat{V}\in\mathbb{R}^{m\times n}$, such that $X\approx
U+\hat{V}=U+Q_{1}VQ_{2}^{T}$ and both $\text{Vec}(U)$ and $\text{Vec}(V)$ have
small infinity norm.

Initialize $U\leftarrow 0^{m\times n},\hat{V}\leftarrow 0^{m\times n}$

Define projection
$\pi_{X}(Y)\coloneqq\dfrac{\text{Vec}(X)^{\top}\text{Vec}(Y)}{\|\text{Vec}(Y)\|_{2}^{2}}\cdot
Y$

while $\|X-U-\hat{V}\|\geq\varepsilon$ do

$Y\leftarrow Q_{1}^{T}XQ_{2}$

if $\|\text{Vec}(X)\|_{1}>\|\text{Vec}(Y)\|_{1}$ then

$\pi\leftarrow\pi_{X}(\text{Sign}(X))$

$U\leftarrow U+\pi$

else

$\pi\leftarrow\pi_{X}(Q_{1}\text{Sign}(Y)Q_{2}^{\top})$

$\hat{V}\leftarrow\hat{V}+\pi$

end if

$X\leftarrow X-\pi$

end while

Return: $X,U,\hat{V}$

In the naive implementation in order to decompose a matrix of shape $m\times
n$ one needs to store matrix $Q$ of shape $mn\times mn$, while the matrix
version leads to memory reduction since we need to store $m\times m+n\times n$
for $Q_{1}$ and $Q_{2}$. The genesis of the proposed Algorithm 2 follows from
the simple observation. Let’s assume our orthogonal matrix $Q$ can be
represented as a Kronecker product of two smaller orthogonal matrices
$Q_{1}^{T}$ and $Q_{2}^{T}$:

$Q=Q_{2}^{T}\otimes Q_{1}^{T}$ (6)

From the properties of Kronker product, we get:

$(Q_{2}^{T}\otimes Q_{1}^{T})\text{vec}(X)=\text{vec}(Q_{1}^{T}XQ_{2})$ (7)

However, by assuming the special structure of $Q$ we might lose the robustness
guaranteed for random orthogonal matrices (as proven in Kashin [1977],
Lyubarskii and Vershynin [2010]). Indeed, as shown in Guedon et al. [2008],
some discrete orthonormal systems produce slightly worse bounds.

Figure 4 illustrates that the time needed for the same computations is notably
less for the matrix version of the algorithm than for the vector one.

Figure 4: Left: Matrix Decomposition Kashin Algorithm 2 performs significantly
faster for matrices, than vectorization and applying Vector Decomposition
Kashin Algorithm 1. Right: Time comparison of different orthogonal matrices
for Matrix Decomposition Kashin Algorithm

##  3 How orthogonal matrix $Q$ affects the algorithm

The choice of orthogonal matrix $Q$ in algorithms 1 and 2 can significantly
impact convergence and execution time. Figure 3 demonstrates the different
convergence properties of the algorithm for different input matrices.

In this work, we have tried several classes of orthogonal matrices:

###  3.1 Random Orthogonal Matrix

Random orthogonal matrix $Q$ is obtained from the QR decomposition of a matrix
generated from a normal distribution.

###  3.2 Householder Reflection

The Householder matrix is an orthogonal matrix that describes a reflection
across a hyperplane orthogonal to some unit vector $y$:

$Q=I-2yy^{*}$ (8)

###  3.3 Discrete Cosine Transform

Discrete Cosine Transform (DCT) is an orthogonal linear mapping similar to the
discrete Fourier transform (DFT), but using only the cosine functions. We
chose the most commonly used form, DCT-II:

$Q[i,j]=\begin{cases}\frac{1}{\sqrt{n}},~{}j=0\\\
\sqrt{\frac{2}{n}}\cos\Big{(}\frac{\pi(2i+1)j}{2n}\Big{)}\end{cases}\,$ (9)

###  3.4 Butterfly matrices

Butterfly matrices are a class of structured orthogonal matrices that allow
fast vector-by-matrix multiplication.

A butterfly matrix $Q$ of size $n=2^{m}$ can be represented as a product of
$m$ block diagonal matrices (called butterfly factor matrices). We denote each
butterfly factor matrix as $B_{k}$, where $k$ indicates block size:

$\begin{split}&Q=B_{n}B_{\frac{n}{2}}\dots B_{2}\\\
&B_{k}=\text{diag}(F_{1},F_{2},\dots,F_{\frac{n}{k}})\end{split}$ (10)

Each block $F_{i}$ (a butterfly factor) of size $k$ has a form
$\begin{bmatrix}D_{1}&D_{2}\\\ D_{3}&D_{4}\end{bmatrix}$ where $D_{i}$ is a
$\frac{k}{2}\times\frac{k}{2}$ diagonal matrix. Figure 5 shows an example of a
butterfly matrix of size 16.

Figure 5: Butterfly factor matrices from left to right: $B_{8}$, $B_{4}$,
$B_{2}$

Matrices like Householder Reflection (3.2), Discrete Cosine Transform (3.3),
and Butterfly matrices (3.4) support faster matvec multiplications (faster
than $O(n^{2})$) due to their special structure. For example, Householder
transformation boils down to a simple operation with two vectors:

$(I-2yy^{*})x=x-2\langle y,x\rangle y$ (11)

while DCT and Butterfly matrices allow for $O(n\log n)$ complexity of matrix-
vector multiplication.

###  3.5 Convergence properties

However, an obvious problem arises for Householder Reflexion considering the
convergence bound (4): when $\|x\|_{1}$ is small (for example, $x$ is very
sparse) and the dimension is high, vectors $y$ and $x$ might be orthogonal,
thus $Q^{T}x$ will be equal to $x$ ($x$ will lie on the reflection plane and
will be the eigenvector of $Q^{T}$). This keeps
$\max(\|x\|_{1},\|Q^{T}x\|_{1})$ small and, consequently, the convergence
bound - high.

We try to estimate potential minimums of $\max(\|x\|_{1},\|Q^{T}x\|_{1})$
expression in Table 1 by taking the real part of eigenvalues (only the
Householder matrix is symmetric and has real eigenvectors) of each orthogonal
matrix. These results allow us to hypothesize that qr and butterfly matrices
in the general case should converge best. These results coincide with the
empirical convergence of the algorithm for randomly generated vectors (Figure
2).

Table 1: Estimation for $x=\text{Re}[y]$, $y\in\text{eig}(Q)$. Mean over 100
generated Q matrices (except for DCT). Matrix size is $500\times 500$. All
results are divided by $\sqrt{n}$.
$Q$ | $\min_{\|x\|_{2}=1}\max\\{\|x\|_{1},\|Q^{T}x\|_{1}\\}$
---|---
Random | 0.548
Householder | 0.076
DCT | 0.330
Butterfly | 0.547

##  4 Quantization approach

Figure 6: Factors $U$ and $V$ are well quantized after computing Kashin
decomposition.

As soon as we have the decomposition $X=U+Q_{1}VQ_{2}^{T}$, where $u$ and $v$
have small infinity norm, we may hope, that the entries distribution of these
vectors may be quantized well. The idea behind the quantization approach is
fairly simple. We replace the values in the factors with the nearest centroids
values, calculated with the clustering procedure. For illustration, we
consider matrix $X\in\mathbb{R}^{500\times 200}$ with entries from standard
distribution and apply Matrix Decomposition Kashin Algorithm 2 to it. Figure 6
clearly illustrates that values of factors $U$ and $V$ are well quantized.

The red lines and dots on the graph represent the centroids of corresponding
clusters of values. The number of clusters is defined by the number of bits
for quantization. Figure 6 contains $4$ clusters per factor, which means, that
we need to store the values of $16=2^{4}$ points or $4$ bits. From the same
figure, one can conclude, that $16$ clusters per factor leads to the $16\times
16=256=2^{8}$ possible pairs of clusters.

To measure the quality of the proposed quantization procedure we decided to
consider weights of OPT models Zhang et al. [2022] and decompose all layers
except the embedding layer in the model in a similar manner. So, we applied
Algorithm 2 for each weight matrix $X_{i}$ and recieved $U_{i},\hat{V}_{i}$
for layer $i$. After that, we applied the clustering procedure to the set of
2-dimensional vectors obtained from stacking $U_{i}$ values and
$V_{i}=Q_{1}^{T}\hat{V}_{i}Q_{2}$ values with a selected number of clusters
($32$ for $5$ bits, $64$ for $6$ bits, etc.). After obtaining centroids, we
replace each entry in the matrices $U_{i},V_{i}$ with its closest centroid
value. As a result, we have quantized factors $U_{i}^{q}$ and $V_{i}^{q}$ with
relatively small infinity error. And all we have to do is to replace the old
forward pass with matrix $X_{i}(\cdot)$ with the new forward pass
$\left[U_{i}^{q}+Q_{1}V_{i}^{q}Q_{2}^{T}\right](\cdot)$. It is important to
highlight, that we do not need to store the matrices $Q_{1}$ and $Q_{2}$,
their action is replaced with effective DCT or Butterfly matvec procedures.

However, we observed, that not all layers could be compressed equally well
with Algorithm 2. This is clearly distinguished either visually (see Figure 7)
or from monitoring convergence speed. The reasons behind such behavior of the
method are yet to be described. For practical reasons, we decided not to
decompose such layers. The results are presented in Table 2.

Figure 7: Left: Slow convergence of the Matrix Decomposition Kashin algorithm
for a particular weight matrix from OPT-125m leads to poor concentration of
values and high quantization error. Right: another layer weight matrix could
be easily decomposed and quantized. Table 2: Perplexity for OPT family of
models with different quantization schemes. Part of the transformer layers are
quantized with Kashin M-bit quantization (those, for which Kashin
decomposition has converged) and others are quantized with N-bit. The
"Quantized Layers" column specifies how many layers were quantized with each
method (first fraction for N-bit and second fraction for kM-bit).
Quantization | Perplexity | Quantized layers
---|---|---
opt-125m
FP32 | 42.01 |
8-bit | 42.19 | 72/72
8-bit + k6-bit | 43.38 | 15/72 + 57/72
8-bit + k5-bit | 45.74 | 15/72 + 57/72
4-bit | 48.92 | 72/72
4-bit + k6-bit | 44.88 | 15/72 + 57/72
4-bit + k5-bit | 47.25 | 15/72 + 57/72
opt-350m
FP32 | 36.64 |
8-bit | 36.77 | 146/146
8-bit + k6-bit | 37.59 | 25/146 + 121/146
8-bit + k5-bit | 40.01 | 25/146 + 121/146
4-bit | 44.46 | 146/146
4-bit + k6-bit | 37.61 | 25/146 + 121/146
4-bit + k5-bit | 40.05 | 25/146 + 121/146
Table 3: Experiments on GLUE benchmark. We compare Kashin Quantization to
common quantization methods such as Kmeans and Uniform Quantization. We
quantize all linear layers except for embedding and lmhead.
Quantization | COLA | SST-2 | QQP | QNLI | MNLI | RTE | STS-B | MRPC
---|---|---|---|---|---|---|---|---
roberta-base
FP32 | 59.06 | 93.8 | 91.24/88.36 | 92.62 | 88.1/87.43 | 67.87 | 89.65/89.49 | 87.75/91.2
Uniform 4bit | 0.0 | 49.08 | 36.82/53.82 | 49.46 | 31.82/31.82 | 52.71 | 10.08/9.37 | 68.38/81.22
Kmeans 4bit | 46.97 | 92.77 |  88.77/87.39 | 89.27 | 83.98/82.85 | 55.23 | 79.65/80.47 | 71.32/75.77
Kashin 4bit (ours) | 52.09 | 90.37 |  89.53/86.73 | 91.14 | 86.41/85.51 | 60.28 | 87.29/87.26 | 83.08/87.10
bert-base
FP32 | 59.31 | 91.74 | 90.66/87.39 | 90.74 | 83.96/84.24 | 64.98 | 88.94/88.77 | 84.31/88.81
Uniform 4bit | 1.24 | 49.66 | 38.09/52.75 | 49.22 | 32.24/33.34 | 50.18 | -0.21/-0.25 | 63.97/76.02
Kmeans 4bit | 54.43 | 91.51 | 88.87/85.33 | 88.01 | 78.63/78.76 | 55.23 | 85.06/85.0 | 34.55/8.87
Kashin 4bit (ours) | 59.65 | 91.63 | 90.28/87.33 | 90.06 | 83.88/84.01 | 63.53 | 88.76/88.56 | 84.80/89.31

##  5 Experiments

Figure 8: Scheme of post-training quantization approach.

To prove the efficacy of our method, we use the Open Pre-trained Transformers
(OPT Zhang et al. [2022]) language models and the LAMBADA dataset Paperno et
al. [2016].

First, we try to compare Kashin Decomposition with different orthogonal
matrices for several layers of the opt-125m language model. Unlike randomly
generated matrices, the convergence of Kashin decomposition for network
weights can be quite unpredictable. Figure 3 depicts the value of the
convergence criterion for two different layers of the first transformer block.
For the fc1 layer, the algorithm fails to converge in the maximum number of
steps and dct converges slower than butterfly and qr. For the q_proj layer,
dct performs on par with qr and slightly better than butterfly. Kashin
Decomposition with householder reflection performs poorly in both cases. In
all further experiments, we use either qr or butterfly orthogonal matrices.

Next, we apply Kashin Quantization and measure LLM’s perplexity. As seen in
Figures 3 and 7, Kashin Decomposition doesn’t always converge for LLM weights,
which leads to high quantization error. In our experiments with the OPT family
of models, we encountered about 0.2 fractions of poorly converged layers. To
compensate for these layers, we leave their weights with the default
quantization scheme (8-bit or 4-bit versions of weights from the huggingface
hub). Results can be seen in Table 2.

Kashin Quantization achieves a trade-off between low bit weights, inference
speed, and LLM prediction quality. For example, full 4-bit quantization
significantly raises the model’s perplexity, whereas 4-bit + Kashin 6-bit
quantization almost restores the quality of the 8-bit model. That, in
conjunction with the ability to perform lower-bit operations and faster matrix
multiplication (with the right orthogonal matrix choice, for example,
butterfly matrices) can significantly reduce LLM’s inference speed and memory
requirements.

Additionaly, we conduct experiments on text-classification with a set of
downstream tasks (GLUE benchmark). We have finetuned Bert and RoBerta models,
measured their accuracy as a baseline, then quantized linear layers in
transformer blocks with three quantization methods: uniform, kmeans and kashin
quantization (ours). Kashin quantization demonstrates superior quality on
almost all GLUE datasets for both models (Figure 3).

##  6 Related Work

Large Language Models (LLMs) nowadays are ubiquitous and used in all
imaginable applications. However, their complexity heavily depends on several
parameters. For example, the flagship of LLAMA-2 Touvron et al. [2023a] family
of models has over 70 billion parameters. This impedes model speed and
sometimes makes training or even inference impossible due to a lack of GPU
memory.

Quantization Nagel et al. [2021] is a popular technique to reduce the model
size and inference time. Naturally, it is often applied to LLMs. For instance,
QLORA Dettmers et al. [2023] paper used quantized weights in conjunction with
Low-Rank Adapters to fine-tune language models on downstream tasks, and
LLM.int8 Dettmers et al. [2022] has introduced a new Int8 matrix
multiplication procedure for feed-forward and attention projection layers in
transformers. Still, the quantization of transformers often proves to be a
tricky task, especially the quantization of activations. It happens due to the
presence of huge outliers in activations that originate from the attention
mechanism (Bondarenko et al. [2021], Bondarenko et al. [2023]).

Best accuracy rsults are usually obtained with quantization-aware training.
Another approach, taken in van Baalen et al. [2020], is to learn bit-width and
quantization parameters simultaneously with model weights. However, both
techniques require dataset and resources to train the model with additional
step of quantization on each weight update.

To avoid training expenses, methods that allow for no training/fine-tuning
have been of particular interest in several papers: authors in Eldad et al.
[2019] scale the weights of consecutive layers which allows them achieve
smaller quality drop on a wide variety of networks; AdaRound Nagel et al.
[2020] proves that rounding to the nearest node of the quantization grid is
not the best strategy and propose to choose rounding by optimization process
which requires only a few thousand samples of data, researches in Hubara et
al. [2020] further develop the idea of AdaRound, complementing it with integer
programming to determine the best bit-width for each layer.

It is important to note that there are other ways to reduce memory footprint
when using large models, such as automatic mixed precision Micikevicius et al.
[2017], operations approximation Bershatsky et al. [2022], Novikov et al.
[2023], and checkpointing Chen et al. [2016], Gusak et al. [2022].

In our work we use Kashin Representations, introduced in Kashin [1977]. It’s
an expansion of vector $x$ in a redundant orthogonal basis which guarantees
that the maximum among absolute values of coefficients (the infinity norm) is
upper bounded. It is a source for representation in Eq. 1 and Algorithm 1.
Garnaev and Gluskin [1984] discusses the redundancy level in Kashin
representations and Lyubarskii and Vershynin [2010] investigates the
applicability of Kashin representations to vector quantization using tight
orthogonal frames that satisfy uncertainty principle.

##  7 Conclusion and further research

We have proposed a matrix case generalization of the Kashin Representation
algorithm and applied it to the weights of transformer language models. We
have compared different choices of orthogonal matrices and theorized on
efficacy of each based on both empirical results and metrics in Eq. 5
estimation among eigenvectors of $Q$.

Kashin Quantization backed up by the conventional uniform quantization
achieves a good compromise between model perplexity and memory requirements.
Additionally, Kashin representation allows to use of faster matrix
multiplication due to the choice of $Q$.

There is much yet to be determined about the properties of convergence of
Algorithm 2. As noted in Section 5, the proposed decomposition for matrices of
model weights doesn’t always converge in a restricted number of iterations.
The proper investigation on both failed matrix weights and special structure
of orthonormal basis $Q$ is a question of further research.

We are looking forward to applying Kashin Decomposition to the quantization of
activations. As shown in Bondarenko et al. [2021] and Bondarenko et al.
[2023], quantization of transformer activations often suffers from huge
outliers. Since Kashin Representation guarantees an upper bound on infinity
norms (maximum absolute value) of decomposition factors, it can be assumed
that Kashin Quantization should be particularly beneficial to LLM activations
quantization.

## References

  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al.  The falcon series of open language models.  _arXiv preprint arXiv:2311.16867_ , 2023.
  * Bershatsky et al. [2022] Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa, Julia Gusak, Daniil Merkulov, and Ivan Oseledets.  Memory-efficient backpropagation through large linear layers.  _arXiv preprint arXiv:2201.13195_ , 2022.
  * Bondarenko et al. [2021] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort.  Understanding and overcoming the challenges of efficient transformer quantization, 2021.
  * Bondarenko et al. [2023] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort.  Quantizable transformers: Removing outliers by helping attention heads do nothing, 2023.
  * Chen et al. [2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ , 2016.
  * Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.
  * Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * Eldad et al. [2019] Meller Eldad, Finkelstein Alexander, Almog Uri, and Grobman Mark.  Same, same but different: Recovering neural network quantization error through weight factorization.  volume 2019-June, 2019.
  * Garnaev and Gluskin [1984] A. Yu. Garnaev and E. D. Gluskin.  On widths of the Euclidean ball.  _Sov. Math., Dokl._ , 30:200–204, 1984.  ISSN 0197-6788.
  * Guedon et al. [2008] Olivier Guedon, Shahar Mendelson, Alain Pajor, and Nicole Tomczak-Jaegermann.  Majorizing measures and proportional subsets of bounded orthonormal systems, 2008.
  * Gusak et al. [2022] Julia Gusak, Daria Cherniuk, Alena Shilova, Alexander Katrutsa, Daniel Bershatsky, Xunyi Zhao, Lionel Eyraud-Dubois, Oleg Shlyazhko, Denis Dimitrov, Ivan Oseledets, et al.  Survey on large scale neural network training.  _arXiv preprint arXiv:2202.10435_ , 2022.
  * Hubara et al. [2020] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.  Improving post training neural quantization: Layer-wise calibration and integer programming.  6 2020.  doi:10.48550/arxiv.2006.10518.  URL https://arxiv.org/abs/2006.10518.
  * Kashin [1977] Boris Sergeevich Kashin.  Diameters of some finite-dimensional sets and classes of smooth functions.  _Izvestiya Rossiiskoi Akademii Nauk. Seriya Matematicheskaya_ , 41(2):334–351, 1977.
  * Kashin and Romskii [2023] Boris Sergeevich Kashin and DG Romskii.  An efficient algorithm for decomposing a vector into two vectors with a small uniform norm.  _Matematicheskie Zametki_ , 114(6):945–948, 2023.
  * Lyubarskii and Vershynin [2010] Yurii Lyubarskii and Roman Vershynin.  Uncertainty principles and vector quantization.  _IEEE Transactions on Information Theory_ , 56(7):3491–3501, July 2010.  ISSN 1557-9654.  doi:10.1109/tit.2010.2048458.  URL http://dx.doi.org/10.1109/TIT.2010.2048458.
  * Micikevicius et al. [2017] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  _arXiv preprint arXiv:1710.03740_ , 2017.
  * Nagel et al. [2020] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In _International Conference on Machine Learning_ , pages 7197–7206. PMLR, 2020.
  * Nagel et al. [2021] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort.  A white paper on neural network quantization, 2021.
  * Novikov et al. [2023] Georgii Sergeevich Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Valerievich Dimitrov, and Ivan Oseledets.  Few-bit backward: Quantized gradients of activation functions for memory footprint reduction.  In _International Conference on Machine Learning_ , pages 26363–26381. PMLR, 2023.
  * Paperno et al. [2016] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez.  The lambada dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, 2016.  doi:10.18653/v1/p16-1144.  URL http://dx.doi.org/10.18653/v1/P16-1144.
  * Safaryan et al. [2022] Mher Safaryan, Egor Shulgin, and Peter Richtárik.  Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor.  _Information and Inference: A Journal of the IMA_ , 11(2):557–580, 2022.
  * Temlyakov [1998] Vladimir Nikolaevich Temlyakov.  Nonlinear kolmogorov widths.  _Mathematical Notes_ , 63(6):785–795, 1998.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * van Baalen et al. [2020] Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling.  Bayesian bits: Unifying quantization and pruning.  volume 2020-December, 2020.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.


# An algebraic approach to the hyperbolicity of moduli stacks of Calabi-Yau
varieties

Yohan Brunebarbe  Institut de Mathématiques de Bordeaux, Université de
Bordeaux, 351 cours de la Libération, F-33405 Talence
<EMAIL_ADDRESS>
###### Abstract.

The moduli stacks of Calabi-Yau varieties are known to enjoy several
hyperbolicity properties. The best results have so far been proven using
sophisticated analytic tools such as complex Hodge theory. Although the
situation is very different in positive characteristic (e.g. the moduli stack
of principally polarized abelian varieties of dimension at least 2 contains
rational curves), we explain in this note how one can prove many hyperbolicity
results by reduction to positive characteristic, relying ultimately on the
nonabelian Hodge theory in positive characteristic developed by Ogus and
Vologodsky.

##  1\. Introduction

###  1.1. Main result

The goal of this note is to provide an algebraic proof of the following
result.

######  Theorem 1.1.

Let $S$ be a smooth proper connected complex variety. Let $f:X\rightarrow S$
be a smooth projective morphism with connected fibres. Assume that the
relative canonical bundle $\omega_{X/S}$ is relatively trivial and that the
Hodge line bundle $f_{\ast}(\omega_{X/S})$ is big. Then:

  1. (1)

the canonical bundle of $S$ is big (in other words $S$ is of general type),

  2. (2)

the cotangent bundle of $S$ is big in the sense of Hartshorne,

  3. (3)

the cotangent bundle of $S$ is weakly positive.

The definitions of bigness and weak-positivity are recalled in section 2.
Observe that the assumptions of the theorem are satisfied for example if
$X\rightarrow S$ is an abelian scheme of maximal variation. Our proof of
Theorem 1.1 easily extends to the logarithmic setting.

Theorem 1.1 (and its extension to the logarithmic setting) is not new: it can
easily be deduced from general results on variations of Hodge structures
[Gri70, Zuo00, BKT13, Bru18, BC20, Bru17], using that Calabi-Yau manifolds
satisfy the infinitesimal Torelli property (cf. section 3). The goal of this
note is to present a proof which does not use any differential geometric
notions, such as metrics and curvature computations, and relies ultimately on
the nonabelian Hodge theory in positive characteristic developed by Ogus and
Vologodsky [OV07].

We give also an algebraic proof of the following higher-dimensional version of
the Arakelov inequality [Ara71, Fal83, Pet90, Kim98, JZ02, Pet00, Vie09,
Zuo00].

######  Theorem 1.2 (Arakelov inequality).

Same assumptions as Theorem 1.1. If $n$ is the dimension of $S$ and $d$ is the
relative dimension of $f$, then

$[f_{\ast}(\omega_{X/S})]\leq\frac{1}{n+d}\cdot\dbinom{n+d}{n+1}\cdot[\omega_{S}].$

In the theorem, $[f_{\ast}(\omega_{X/S})]$ and $[\omega_{S}]$ are the divisors
classes associated to the line bundles $f_{\ast}(\omega_{X/S})$ and
$\omega_{S}$ respectively, and for any two $\mathbb{Q}$-divisors classes $A$
and $B$ the notation $A\leq B$ means that any line bundle representing (up to
a positive rational number) the difference $B-A$ is weakly-positive.

In a previous paper, we proved with analytic techniques [Bru20a, Theorem 1.6]
that the preceding inequality holds with $\frac{d}{2}$ instead of
$\frac{1}{n+d}\cdot\dbinom{n+d}{n+1}$, but we were unable to prove it
algebraically when the base is not a curve. Note that, in particular, the
constant should only depend on $d$ but not on $n$.

###  1.2. Examples

Let $\mathcal{A}_{g}$ be the moduli stack of principally polarized complex
abelian varieties of dimension $g$, and
$\pi:\mathbb{A}_{g}\rightarrow\mathcal{A}_{g}$ be the universal family. Since
the Hodge line bundle $\pi_{\ast}\omega_{\mathbb{A}_{g}/\mathcal{A}_{g}}$ is
known to be ample on (the coarse moduli space of) $\mathcal{A}_{g}$ (see
[MB85] for an algebraic proof), it follows from Theorem 1.1 that any smooth
proper connected complex variety that admits a generically finite morphism to
$\mathcal{A}_{g}$ is of general type and has a cotangent bundle which is
weakly-positive and big in the sense of Hartshorne. Thanks to the Torelli
Theorem, this applies in particular to any smooth proper complex variety that
admits a generically finite morphism to the moduli stack $\mathcal{M}_{g}$ of
smooth projective curve of genus $g>1$.

Similarly, for any polynomial $P\in\mathbb{Q}[T]$, let $\mathcal{M}_{P}$ be
the stack that classifies pairs $(X,\mathcal{L})$ consisting in a smooth
projective complex variety $X$ with a trivial canonical bundle and an ample
line bundle $\mathcal{L}$ on $X$ whose Hilbert polynomial equals $P$, up to
the equivalence relation
$(X,\mathcal{L})\equiv(X^{\prime},\mathcal{L}^{\prime})$ if there exists an
isomorphism $\mu:X\rightarrow X^{\prime}$ such that
$\mu^{\ast}\mathcal{L}^{\prime}$ is numerically equivalent to $\mathcal{L}$.
Then $\mathcal{M}_{P}$ is a separated Deligne-Mumford stack of finite type
over $\mathbb{C}$ and its coarse moduli space is quasi-projective [Vie95]. If
$\pi:\mathcal{X}\rightarrow\mathcal{M}_{P}$ denote the universal family, then
the Hodge line bundle $\pi_{\ast}\omega_{\mathcal{X}/\mathcal{M}_{P}}$ is
known to be ample on (the coarse moduli space of) $\mathcal{M}_{P}$, cf.
[BBT19, Theorem 7.8] (however, besides the case of abelian varieties and K3
surfaces [Mau14], it seems that there is no algebraic proof of this fact).
Therefore, any smooth proper connected complex variety that admits a
generically finite morphism to the stack $\mathcal{M}_{P}$ satisfies the
conclusions of Theorem 1.1.

###  1.3. Semi-positivity from Higgs bundles of geometric origin

Let $S$ be a smooth complex scheme. Let $\mathrm{HIG}(S)$ denote the Abelian
category of Higgs bundles $(\mathcal{E},\theta)$ on $S$, where $\mathcal{E}$
is a quasicoherent $\mathcal{O}_{S}$-modules and
$\theta:\mathcal{E}\rightarrow\mathcal{E}\otimes_{\mathcal{O}_{S}}\Omega_{S}^{1}$
is a $\mathcal{O}_{S}$-linear morphism such that $\theta\wedge\theta=0$. For
any smooth proper morphism $f:X\rightarrow S$ and any integer $k$, we write
$\mathrm{H}^{k}_{Dol}(X/S)$ for the locally-free coherent
$\mathcal{O}_{S}$-module
$\oplus_{p+q=k}\mathrm{R}^{q}f_{\ast}(\Omega_{X/S}^{p})$ endowed with the
Gauss-Manin Higgs field. We say that a Higgs bundle on $S$ is of geometric
origin if it belongs to the full subcategory of $\mathrm{HIG}(S)$ which
contains the direct summands of the $\mathrm{H}^{k}_{Dol}(X/S)$ for any smooth
proper $S$-scheme $X$, and any extension of these factors.

The true mathematical content of this note consists in fact in explaining how
one can reduce the proofs of Theorem 1.1 and Theorem 1.2 to the following
result, using arguments from complex algebraic geometry.

######  Theorem 1.3 (Beilinson-Deligne, Simpson).

For any smooth proper morphism $f:X\rightarrow S$ with $S$ a smooth projective
complex curve, and any integer $k$, the Higgs bundle
$\mathrm{H}^{k}_{Dol}(X/S)$ is semistable.

This result was first proved by Beilinson-Deligne (unpublished) and later, in
a greater generality, by Simpson using transcendental tools. An algebraic
proof was then given by Ogus and Vologodsky, as a consequence of their
nonabelian Hodge theory in positive characteristic, cf. [OV07, Theorem 4.21].

A key ingredient of our proof of Theorem 1.1 is given by the following
technical result.

######  Theorem 1.4.

Le $S$ be a smooth proper complex variety and $(\mathcal{E},\theta)$ be a
Higgs bundle on $S$ of geometric origin. If
$(\mathcal{F},\theta)\subset(\mathcal{E},\theta)$ is a Higgs subsheaf, then
the line bundle $\det\mathcal{F}^{\vee}$ is weakly positive. If moreover
$\mathcal{F}$ is a locally split $\mathcal{O}_{S}$-submodule of $\mathcal{E}$,
then the line bundle $\det\mathcal{F}^{\vee}$ is nef.

Theorem 1.4 can be proved with analytic methods using the special curvature
properties of the Hodge metric. We offer an alternative proof which is
algebraic and arguably simpler, with Theorem 1.3 as the main input. As a
direct corollary, we obtain an algebraic proof of the following special case
of a result due to Zuo [Zuo00] and the author [Bru18, Bru17].

######  Corollary 1.5.

Le $S$ be a smooth proper complex variety $(\mathcal{E},\theta)$ be a Higgs
bundle on $S$ of geometric origin. If $\mathcal{F}\subset\mathcal{E}$ is a
$\mathcal{O}_{S}$-submodule of $\mathcal{E}$ contained in the kernel of
$\theta$, then its dual $\mathcal{F}^{\vee}$ is weakly positive. If moreover
$\mathcal{F}$ is a locally split $\mathcal{O}_{S}$-submodule of $\mathcal{E}$,
then $\mathcal{F}^{\vee}$ is nef.

As a direct application of Corollary 1.5, we obtain yet another proof of the
following well-known result of Griffiths:

######  Theorem 1.6.

Let $A\rightarrow S$ be an abelian scheme over a smooth proper complex variety
$S$. Then the vector bundle $\Omega^{1}_{A/S}$ is nef.

######  Remark 1.7.

Note that thanks to Lefschetz principle all the preceding results are true
over any field of characteristic zero (in fact all the proofs in this paper
work in this generality). However, they are no longer true in characteristic
$p>2$, since Moret-Bailly has constructed non-isotrivial families of abelian
surfaces $f:X\rightarrow\mathbb{P}^{1}$ such that
$f_{*}\omega_{X/{\mathbb{P}^{1}}}=\mathcal{O}_{\mathbb{P}^{1}}(p)\bigoplus\mathcal{O}_{\mathbb{P}^{1}}(-1)$,
cf. [MB81].

##  2\. Positivity of torsion-free sheaves

We recall for the reader convenience the definitions and properties of the
different notions of positivity for torsion-free sheaves on smooth proper
varieties that are used in this paper.

We start with the notions of weak positivity and bigness introduced by
Viehweg, cf. [Vie83] and [Vie95, p.59-67]. Recall that a coherent sheaf
$\mathcal{F}$ on a scheme $X$ is globally generated at a point $x\in X$ if the
natural map
$\mathrm{H}^{0}(X,\mathcal{F})\otimes_{\mathbb{C}}\mathcal{O}_{X}\rightarrow\mathcal{F}$
is surjective at $x$.

######  Definition 2.1 (Viehweg).

Let $X$ be a smooth projective variety and $\mathcal{F}$ a torsion-free sheaf
on $X$. Let $i:V\hookrightarrow X$ denotes the inclusion of the biggest open
subset on which $\mathcal{F}$ is locally free.

  1. (1)

We say that $\mathcal{F}$ is weakly positive over the dense open subset
$U\subset V$ if for every ample invertible sheaf $\mathcal{H}$ on $X$ and
every positive integer $\alpha>0$ there exists an integer $\beta>0$ such that
$\widehat{S}^{\alpha\cdot\beta}\mathcal{F}\otimes_{\mathcal{O}_{X}}{\mathcal{H}}^{\beta}$
is globally generated over $U$.

  2. (2)

We say that $\mathcal{F}$ is Viehweg-big over the dense open subset $U\subset
V$ if for any line bundle $\mathcal{H}$ there exists $\gamma>0$ such that
$\widehat{S}^{\gamma}\mathcal{F}\otimes\mathcal{H}^{-1}$ is weakly positive
over $U$.

(Here the notation $\widehat{S}^{k}\mathcal{F}$ stands for the reflexive hull
of the sheaf $S^{k}\mathcal{F}$, i.e.
$\widehat{S}^{k}\mathcal{F}=i_{*}(S^{k}i^{*}\mathcal{F})$.) We say that
$\mathcal{F}$ is weakly positive (resp. Viehweg-big) if there exists a dense
open subset $U\subset V$ such that $\mathcal{F}$ is weakly positive (resp.
Viehweg-big) over $U$.

######  Remark 2.2.

If $X$ is a smooth projective variety and $\mathcal{F}$ is a locally free
$\mathcal{O}_{X}$-module, then $\mathcal{F}$ is nef if and only if it is
weakly positive over $X$, and $\mathcal{F}$ is ample if and only if it is
Viehweg-big over $X$.

######  Lemma 2.3 (Viehweg).

Let $\mathcal{F}$ and $\mathcal{G}$ be torsion-free sheaves on a smooth
projective variety $X$.

  1. (1)

If $\mathcal{F}\rightarrow\mathcal{G}$ is a morphism, surjective over $U$, and
if $\mathcal{F}$ is weakly positive (resp. Viehweg-big) over $U$, then
$\mathcal{G}$ is weakly positive (resp. Viehweg-big) over $U$.

  2. (2)

Let $f:Y\rightarrow X$ be a morphism between two smooth projective varieties.
If $\mathcal{F}$ is weakly positive over $U\subset X$ and $f^{-1}(U)$ is dense
in $Y$, then $f^{*}\mathcal{F}/(f^{*}\mathcal{F})_{tors}$ is weakly positive
over $f^{-1}(U)$.

  3. (3)

If $f:X^{\prime}\rightarrow X$ is a birational morphism between two smooth
projective varieties which is an isomorphism over $U\subset X$, then
$\mathcal{F}$ is weakly positive (resp. Viehweg-big) over $U$ if and only if
$f^{*}\mathcal{F}/(f^{*}\mathcal{F})_{tors}$ is weakly positive (resp.
Viehweg-big) over $f^{-1}(U)$.

  4. (4)

If $\mathcal{F}$ is weakly positive and $\mathcal{H}$ is a Viehweg-big line
bundle, then $\mathcal{F}\otimes\mathcal{H}$ is Viehweg-big.

If $\mathcal{F}$ is a torsion-free sheaf on a smooth proper (but non-
necessarily projective) variety $X$, we say that $\mathcal{F}$ is weakly
positive (resp. Viehweg-big) if for any birational morphism
$f:X^{\prime}\rightarrow X$ from a smooth projective variety, the torsion-free
sheaf $f^{*}\mathcal{F}/(f^{*}\mathcal{F})_{tors}$ is weakly positive (resp.
Viehweg-big). This definition extends the definition for torsion-free sheaves
on projective varieties thanks to the preceding lemma.

We introduce now a weaker notion of bigness. If $\mathcal{E}$ is a vector
bundle on a smooth proper variety $X$, we let
$\pi:\mathbb{P}(\mathcal{E}):=\mathrm{Proj}_{\mathcal{O}_{X}}(\mathrm{Sym}\,\mathcal{E})\rightarrow
X$ be the projective bundle of one-dimensional quotients of $\mathcal{E}$ and
$\mathcal{O}_{\mathcal{E}}(1)$ be the tautological line bundle which fits in
an exact sequence
$\pi^{\ast}\mathcal{E}\rightarrow\mathcal{O}_{\mathcal{E}}(1)\rightarrow 0$.

######  Definition 2.4 (cf. [Bru20b, Definition 2.6] and [Bru20b, Lemma 2.5]).

A vector bundle $\mathcal{E}$ on a smooth proper variety $X$ is called big (in
the sense of Hartshorne) if it satisfies one of the following equivalent
conditions:

  1. (1)

The line bundle $\mathcal{O}_{\mathcal{E}}(1)$ is Viehweg-big.

  2. (2)

For some (resp. any) Viehweg-big line bundle $\mathcal{H}$, there exists an
injective map $0\rightarrow\mathcal{H}\rightarrow S^{k}\mathcal{E}$ for some
$k>0$.

  3. (3)

For some (resp. any) Viehweg-big torsion-free sheaf $\mathcal{F}$, there
exists a non-zero map $\mathcal{F}\rightarrow S^{k}\mathcal{E}$ for some
$k>0$.

A Viehweg-big vector bundle is big in the sense of Hartshorne, but the
converse is not true (consider for example the rank $2$ vector bundle
$\mathcal{O}(1)\oplus\mathcal{O}(-1)$ on $\mathbb{P}^{1}$). Note however that
the two notions coincide for line bundles.

##  3\. Families of varieties with trivial canonical bundle

According to Griffiths, the following observation is due to Andreotti.

######  Lemma 3.1.

Let $X$ be a smooth proper complex variety of dimension $d$. If the canonical
bundle $\omega_{X}$ of $X$ is trivial, then the canonical $\mathbb{C}$-linear
map

$\mathrm{H}^{1}(X,T_{X})\otimes_{\mathbb{C}}\mathrm{H}^{0}(X,\omega_{X})\rightarrow\mathrm{H}^{1}(X,\Omega_{X}^{d-1})$

is an isomorphism.

In particular, any smooth proper variety with a trivial canonical bundle
satisfies the infinitesimal Torelli property.

The following result relates the positivity of the Hodge line bundle of a
family of projective Calabi-Yau varieties to the injectivity of the Kodaira-
Spencer map.

######  Proposition 3.2.

Let $S$ be a smooth proper connected complex variety. Let $f:X\rightarrow S$
be a smooth projective morphism with connected fibres. Assume that the
relative canonical bundle $\omega_{X/S}$ is relatively trivial. If the Hodge
line bundle $f_{\ast}(\omega_{X/S})$ is big, then the Kodaira-Spencer map
$T_{S}\rightarrow R^{1}T_{X/S}$ is an injective morphism of
$\mathcal{O}_{S}$-modules.

Using analytic techniques, it is in fact possible to prove that for any family
of smooth proper complex varieties with trivial canonical bundle, the bigness
of the Hodge line bundle is equivalent to the injectivity of the Kodaira-
Spencer map. However, we do not know an algebraic proof of this fact.

###### Proof.

Let $\mathcal{L}$ be a line bundle on $X$ which is relatively ample with
respect to $f$. Since the Hilbert polynomial $P$ of
$(X_{s},\mathcal{L}_{|X_{s}})$ is independent of the point $s\in S$, we get a
morphism $S\rightarrow\mathcal{M}_{P}$ with values in the corresponding moduli
stack of polarized pairs modulo numerical equivalence. It is a separated
Deligne-Mumford stack of finite type over $\mathbb{C}$ [Vie95]. We denote by
$M_{P}$ the associated coarse moduli space. The Hodge line bundle
$f_{\ast}(\omega_{X/S})$ is the pull-back of a line bundle on
$\mathcal{M}_{P}$, hence some positive power of the Hodge line bundle is the
pullback of a line bundle on $M_{P}$. Since the Hodge line bundle on $S$ is
big by assumption, the morphism $S\rightarrow M_{P}$ is generically finite.
But the Kodaira-Spencer map of the universal family on $\mathcal{M}_{P}$ is an
injective morphism of $\mathcal{O}_{\mathcal{M}_{P}}$-modules. It follows that
the Kodaira-Spencer map $T_{S}\rightarrow R^{1}T_{X/S}$ is a generically
injective morphism of $\mathcal{O}_{S}$-modules, and even injective everywhere
since $T_{S}$ is locally-free hence torsion-free.

∎

##  4\. Semi-positivity from Higgs bundles

######  Definition 4.1.

A Higgs bundle $(\mathcal{E},\theta)$ on a smooth proper curve is said
semistable if it has no Higgs subbundle $(\mathcal{F},\theta)$ with
$\deg(\mathcal{F})>\deg(\mathcal{E})$. More generally, a Higgs bundle
$(\mathcal{E},\theta)$ on a smooth proper variety $S$ is said semistable if
for every smooth proper curve $C$ equipped with a morphism $f:C\rightarrow S$
the Higgs bundle $f^{\ast}(\mathcal{E},\theta)$ is semistable.

Note that by the very definition, the pull-back of a semistable Higgs bundle
is semistable (in particular, this definition of semistability is stronger
than the usual definition of semistability with respect to an ample divisor
class). The following result is the main input of this note (see below for the
proof).

######  Theorem 4.2.

Any Higgs bundle of geometric origin on a smooth proper complex variety is
semistable and its first rational Chern class is zero.

One can prove that all the higher rational Chern classes are zero too, but we
won’t need it. In view of the preceding result, Theorem 1.4 and Corollary 1.5
are consequences of the following general results.

######  Proposition 4.3.

Le $S$ be a smooth proper complex variety. Let $(\mathcal{E},\theta)$ be a
semistable Higgs bundle on $S$ with a vanishing first rational Chern class. If
$(\mathcal{F},\theta)\subset(\mathcal{E},\theta)$ is a Higgs subsheaf, then
the line bundle $\det\mathcal{F}^{\vee}$ is weakly positive. If moreover
$\mathcal{F}$ is a locally split $\mathcal{O}_{S}$-submodule of $\mathcal{E}$,
then the line bundle $\det\mathcal{F}^{\vee}$ is nef.

######  Corollary 4.4.

Le $S$ be a smooth proper complex variety. Let $(\mathcal{E},\theta)$ be a
semistable Higgs bundle on $S$ with a vanishing first rational Chern class. If
$\mathcal{F}\subset\mathcal{E}$ is a $\mathcal{O}_{S}$-submodule of
$\mathcal{E}$ contained in the kernel of $\theta$, then its dual
$\mathcal{F}^{\vee}$ is weakly positive. If moreover $\mathcal{F}$ is a
locally split $\mathcal{O}_{S}$-submodule of $\mathcal{E}$, then
$\mathcal{F}^{\vee}$ is nef.

###### Proof of Proposition 4.3.

Let $(\mathcal{E},\theta)$ be a semistable Higgs bundle with a vanishing first
rational Chern class and $(\mathcal{F},\theta)\subset(\mathcal{E},\theta)$ be
a Higgs subsheaf. By definition, for any smooth projective curve $C$ equipped
with a morphism $f:C\rightarrow S$, the pulled-back Higgs bundle
$f^{\ast}(\mathcal{E},\theta)$ is semistable. If $\mathcal{F}$ is a locally
split $\mathcal{O}_{S}$-submodule of $\mathcal{E}$, then
$f^{\ast}(\mathcal{F},\theta)$ is a Higgs subsheaf of
$f^{\ast}(\mathcal{E},\theta)$ and by semistability
$f^{\ast}\det\mathcal{F}=\det f^{\ast}\mathcal{F}\leq\det
f^{\ast}\mathcal{E}=0$. Hence $\det\mathcal{F}^{\vee}$ is nef. In general, a
standard argument using resolution of singularities implies the existence of a
birational morphism $\nu:S^{\prime}\rightarrow S$ from a smooth projective
variety $S^{\prime}$ such that $\nu^{\ast}\mathcal{F}$ has a morphism to a
locally split $\mathcal{O}_{S^{\prime}}$-submodule $\mathcal{G}$ of
$\nu^{\ast}\mathcal{E}$, which is generically an isomorphism. The Higgs field
of $\nu^{\ast}(\mathcal{E},\theta)$ stabilizes $\mathcal{G}$, hence the line
bundle $\det\mathcal{G}^{\vee}$ is nef. Since there is a morphism
$\det\mathcal{G}^{\vee}\rightarrow\det(\nu^{\ast}\mathcal{F})^{\vee}$ which is
generically an isomorphism, it follows that
$\det(\nu^{\ast}\mathcal{F})^{\vee}=\nu^{\ast}(\det\mathcal{F}^{\vee})$ and
$\det\mathcal{F}^{\vee}$ are weakly positive thanks to Lemma 2.3. ∎

###### Proof of Corollary 4.4.

As in the proof of Proposition 4.3, it is sufficient to consider the case
where $\mathcal{F}$ is a locally split $\mathcal{O}_{S}$-submodule of
$\mathcal{E}$ which is contained in the kernel of the Higgs field $\theta$.
Let
$\pi:\mathbb{P}(\mathcal{F}^{\vee}):=\mathrm{Proj}_{\mathcal{O}_{S}}(\mathrm{Sym}\,\mathcal{F}^{\vee})\rightarrow
S$ be the projective bundle of one-dimensional subspace of $\mathcal{F}$ and
$\mathcal{O}_{\mathcal{F}^{\vee}}(-1)$ be the tautological line bundle which
fits in an exact sequence
$0\rightarrow\mathcal{O}_{\mathcal{F}^{\vee}}(-1)\rightarrow\pi^{\ast}\mathcal{F}$.
Then $\pi^{\ast}\mathcal{E}$ equipped with the induced Higgs field
$\pi^{\ast}\theta$ is a semistable Higgs bundle and $\pi^{\ast}\mathcal{F}$ is
contained in the kernel of $\pi^{\ast}\theta$. A fortiori,
$\mathcal{O}_{\mathcal{F}^{\vee}}(-1)$ is contained in the kernel of
$\pi^{\ast}\theta$, so that $(\mathcal{O}_{\mathcal{F}^{\vee}}(-1),0)$ can be
viewed as a Higgs subsheaf of $(\pi^{\ast}\mathcal{E},\pi^{\ast}\theta)$.
Since $\mathcal{O}_{\mathcal{F}^{\vee}}(-1)$ is a locally split
$\mathcal{O}_{\mathbb{P}(\mathcal{F}^{\vee})}$-submodule of
$\pi^{\ast}\mathcal{E}$, we get from Proposition 4.3 that
$\mathcal{O}_{\mathcal{F}^{\vee}}(-1)^{\vee}=\mathcal{O}_{\mathcal{F}^{\vee}}(1)$
is a nef line bundle, so that $\mathcal{F}^{\vee}$ is nef. ∎

######  Remark 4.5.

The conclusions of Proposition 4.3 and Corollary 4.4 hold more generally for
any Higgs bundle with zero rational Chern numbers $c_{1}(E)=c_{2}(E)=0$ which
is semistable with respect to an ample divisor. The proof is similar, once we
know that these properties are preserved by pull-back (see [Lan15, Theorem 12]
for an algebraic proof of this fact).

###### Proof of Theorem 4.2.

Let $(\mathcal{E},\theta)$ be a Higgs bundle of geometric origin on a smooth
proper variety $S$. Thanks to the base-change formula, the pull-back of a
Higgs bundle of geometric origin is also of geometric origin. On the other
hand, a locally free coherent $\mathcal{O}_{S}$-module has a vanishing first
rational Chern class if it has degree zero in restriction to any curve.
Therefore it is sufficient to consider the case where the base $S$ is a smooth
proper curve.

Given an extension
$0\rightarrow(\mathcal{E}_{1},\theta_{1})\rightarrow(\mathcal{E},\theta)\rightarrow(\mathcal{E}_{2},\theta_{2})\rightarrow
0$ of Higgs bundles on a smooth proper curve, if both
$(\mathcal{E}_{1},\theta_{1})$ and $(\mathcal{E}_{2},\theta_{2})$ are
semistable of degree zero, then $(\mathcal{E},\theta)$ is semistable of degre
zero, and the converse holds if the extension splits. Therefore, one is
reduced to consider the case where
$(\mathcal{E},\theta)=\mathrm{H}^{i}_{Dol}(X/S)$ for a smooth proper morphism
$f:X\rightarrow S$ and an integer $i$. The underlying $\mathcal{O}_{S}$-module
has degree zero, since it is isomorphic to the graded object associated to the
relative De Rham cohomology $\mathrm{H}^{i}_{DR}(X/S)$ endowed with its Hodge
filtration, and recalling that the relative De Rham cohomology is naturally
equipped with a flat connection (the Gauss-Manin connection) so that all its
rational Chern classes are zero. Finally, the semistability of the Higgs
bundle $\mathrm{H}^{i}_{Dol}(X/S)$ can either be seen as an easy consequence
of the computation of the curvature of the Hodge metric by Griffiths [Gri70],
or it can also be proved algebraically with the help of the nonabelian Hodge
theory in positive characteristic of Ogus and Vologodsky, cf. [OV07, Theorem
4.21]. ∎

##  5\. Hyperbolicity from Higgs bundles

###  5.1. Weak-positivity of the cotangent bundle

Let $S$ be a smooth proper complex variety, and let $(\mathcal{E},\theta)$ be
a Higgs bundle on $S$. The Higgs field
$\theta:\mathcal{E}\rightarrow\Omega^{1}_{S}\otimes_{\mathcal{O}_{S}}\mathcal{E}$
is equivalently seen as a $\mathcal{O}_{S}$-linear map of
$\mathcal{O}_{S}$-modules $\phi:T_{S}\rightarrow\mathrm{End}(\mathcal{E})$.

######  Proposition 5.1.

Assume that the Higgs bundle $(\mathcal{E},\theta)$ is semistable with a
vanishing first rational Chern class and that the associated
$\mathcal{O}_{S}$-linear map $\phi:T_{S}\rightarrow\mathrm{End}(\mathcal{E})$
is injective. Then the cotangent bundle $\Omega_{S}^{1}$ of $S$ is weakly-
positive.

In view of the following lemma, the proposition is a direct consequence of
Corollary 4.4 applied to the Higgs bundle $\mathrm{End}(\mathcal{E},\theta)$
and its $\mathcal{O}_{S}$-submodule $T_{S}$.

######  Lemma 5.2.

If
$\Theta:\mathrm{End}(\mathcal{E})\rightarrow\Omega_{S}^{1}\otimes_{\mathcal{O}_{S}}\mathrm{End}(\mathcal{E})$
denotes the induced Higgs field on $\mathrm{End}(\mathcal{E})$, then the
composition of $\phi:T_{S}\rightarrow\mathrm{End}(\mathcal{E})$ with $\Theta$
is zero.

###### Proof of the Lemma.

The induced Higgs field $\Theta$ on $\mathrm{End}(\mathcal{E})$ satisfies:

$(\Theta_{s}(\Psi))(v)=\theta_{s}(\Psi(v))-\Psi(\theta_{s}(v))$

for $\Psi$ a section of $\mathrm{End}(\mathcal{E})$, $v$ a section of
$\mathcal{E}$ and $s$ a section of $T_{S}$. Letting $\Psi=\theta_{t}$ for a
section $t$ of $T_{S}$, we obtain

$(\Theta_{s}(\theta_{t}))(v)=\theta_{s}(\theta_{t}(v))-\theta_{t}(\theta_{s}(v))=0$

since $\theta\wedge\theta=0$. ∎

###  5.2. Bigness of the cotangent bundle

######  Proposition 5.3.

Let $S$ be a smooth proper complex variety, and let $(\mathcal{E},\theta)$ be
a Higgs bundle of geometric origin on $S$. Assume that $\mathcal{E}$ contains
a big line bundle (or more generally a $\mathcal{O}_{S}$-submodule whose
determinant is big). Then the cotangent bundle of $S$ is big in the sense of
Hartshorne (hence a fortiori $S$ is of general type).

######  Remark 5.4.

The same proof works more generally for any semistable Higgs bundle with a
vanishing first rational Chern classe and whose Higgs field is nilpotent.

###### Proof.

Let $(\mathcal{E},\theta)$ be a Higgs bundle of geometric origin on $S$ and
assume that $\mathcal{E}$ contains a $\mathcal{O}_{S}$-submodule $\mathcal{F}$
of rank $r$ such that the line bundle $\det\mathcal{F}$ is big. Since the
$r$-th tensor power of exterior of $(\mathcal{E},\theta)$ is also of geometric
origin and contains the big line bundle $\det\mathcal{F}$, one can assume from
the beginning that $\mathcal{F}$ is a line bundle.

By iterating the Higgs field $\theta$, we get for every positive integer $k$ a
morphism of $\mathcal{O}_{S}$-modules
$\phi_{k}:\mathrm{Sym}^{k}T_{S}\otimes\mathcal{F}\rightarrow\mathcal{E}$.
Since the Higgs field is nilpotent, the $\phi_{k}$ are zero if $k\gg 1$. On
the other hand, the morphism $\phi_{1}$ is nontrivial. Indeed otherwise
$\mathcal{F}$ would be contained in the kernel of the Higgs field, hence its
dual $\mathcal{F}^{\vee}$ would be weakly-positive thanks to Corollary 1.5, in
contradiction with the assumption that $\mathcal{F}$ is big. If $k$ is the
biggest (positive) integer for which $\phi_{k}$ is nonzero, then its image
$\mathcal{N}_{1}$ is a $\mathcal{O}_{S}$-submodule of $\mathcal{E}$ contained
in the kernel of the Higgs field. Therefore, its dual $\mathcal{N}_{1}^{\vee}$
is weakly-positive thanks to Corollary 1.5. Since there is a non-zero morphism
$\mathcal{N}_{1}^{\vee}\otimes\mathcal{F}\rightarrow\mathrm{Sym}^{k}\Omega^{1}_{S}$
and the $\mathcal{O}_{S}$-submodule $\mathcal{N}_{1}^{\vee}\otimes\mathcal{F}$
is big in the sense of Viehweg, it follows that $\Omega^{1}_{S}$ is big in the
sense of Hartshorne, cf. Definition 2.4. ∎

##  6\. Proofs of Theorem 1.1 and Theorem 1.2

Let $S$ be a smooth proper connected complex variety of dimension $n$. Let
$f:X\rightarrow S$ be a smooth projective morphism with connected fibres of
relative dimension $d$ such that the relative canonical bundle $\omega_{X/S}$
is relatively trivial. Consider the Higgs bundle
$(\mathcal{E},\theta):=\mathrm{H}^{d}_{Dol}(X/S)$. Therefore
$\mathcal{E}=\bigoplus_{k=0}^{d}\mathcal{E}^{k}$ with
$\mathcal{E}^{k}:=\mathrm{R}^{d-k}f_{\ast}(\Omega_{X/S}^{k})$, so that in
particular $\mathcal{E}^{d}$ is the Hodge line bundle
$f_{\ast}(\omega_{X/S})$, and
$\theta(\mathcal{E}^{k})\subset\mathcal{E}^{k-1}\otimes_{\mathcal{O}_{S}}\Omega_{S}^{1}$
for every integer $k$.

From now on, we assume that the Hodge line bundle
$\mathcal{E}^{d}=f_{\ast}(\omega_{X/S})$ is big. In particular, it follows
from Lemma 3.1 and Proposition 3.2 that the morphism of
$\mathcal{O}_{S}$-modules
$T_{S}\otimes_{\mathcal{O}_{S}}\mathcal{E}^{d}\rightarrow\mathcal{E}^{d-1}$ is
injective. A fortiori, the $\mathcal{O}_{S}$-linear map
$\phi:T_{S}\rightarrow\mathrm{End}(\mathcal{E})$ associated to $\theta$ is
injective, hence the cotangent bundle $\Omega_{S}^{1}$ of $S$ is weakly-
positive thanks to Proposition 5.1.

Since $\mathcal{E}$ contains the big line bundle $\mathcal{E}^{d}$, we know
from Proposition 5.3 that $\Omega_{S}^{1}$ is big in the sense of Hartshorne.
A fortiori, $S$ is of general type thanks to a general result of Campana-Păun
[CP15]. But in our situation this can also be seen in an elementary way as
follows. As in the proof of Proposition 5.3, by iterating the Higgs field, we
get for every positive integer $k$ a morphism of $\mathcal{O}_{S}$-modules
$\phi_{k}:\mathrm{Sym}^{k}T_{S}\otimes_{\mathcal{O}_{S}}\mathcal{E}^{d}\rightarrow\mathcal{E}^{d-k}$.
If we take for $k$ the biggest integer for which $\phi_{k}$ is nonzero, then
its image $\mathcal{N}_{1}$ is a $\mathcal{O}_{S}$-submodule of $\mathcal{E}$
contained in the kernel of the Higgs field (note that $k\geq 1$ since
$\phi_{1}$ is injective). Therefore, its dual $\mathcal{N}_{1}^{\vee}$ is
weakly-positive thanks to Corollary 1.5. Since there is a non-zero morphism
$\mathcal{N}_{1}^{\vee}\otimes_{\mathcal{O}_{S}}\mathcal{E}^{d}\rightarrow\mathrm{Sym}^{k}\Omega^{1}_{S}$
and the $\mathcal{O}_{S}$-submodule
$\mathcal{N}_{1}^{\vee}\otimes_{\mathcal{O}_{S}}\mathcal{E}^{d}$ is big in the
sense of Viehweg, we recover that $\Omega^{1}_{S}$ is big in the sense of
Hartshorne. Consider the exact sequence of coherent $\mathcal{O}_{S}$-modules:

$0\rightarrow\mathcal{N}_{2}\rightarrow\mathrm{Sym}^{k}T_{S}\rightarrow\mathcal{N}_{1}\otimes_{\mathcal{O}_{S}}(\mathcal{E}^{d})^{\vee}\rightarrow
0.$

Taking determinants we get that

$\det(\mathrm{Sym}^{k}\Omega^{1}_{S})=(\mathcal{E}^{d})^{\otimes\mathrm{rk}\,\mathcal{N}_{1}}\otimes_{\mathcal{O}_{S}}(\det\mathcal{N}_{1}\otimes\det\mathcal{N}_{2})^{\vee}.$

Note that $\mathcal{N}_{2}^{\vee}$ is weakly-positive, since $\Omega_{S}^{1}$
is weakly-positive and there is a generically surjective
$\mathcal{O}_{S}$-linear map
$\mathrm{Sym}^{k}\Omega^{1}_{S}\rightarrow\mathcal{N}_{2}^{\vee}$. A fortiori,
the line bundle $(\det\mathcal{N}_{2})^{\vee}$ is weakly-positive. We have
also seen that $\mathcal{N}_{1}^{\vee}$ is weakly-positive, therefore
$(\det\mathcal{N}_{1})^{\vee}$ is weakly-positive too. Finally, using that
$\det(\mathrm{Sym}^{k}\Omega^{1}_{S})=(\omega_{S})^{\otimes C_{k}}$ with
$C_{k}=\dbinom{n+k-1}{n}$, we obtain that

$(\omega_{S})^{\otimes
C_{k}}\geq(\mathcal{E}^{d})^{\otimes\mathrm{rk}\,\mathcal{N}_{1}}.$

(Given two line bundles $A$ and $B$ on $S$, the notation $A\geq B$ means that
the line bundle $A\otimes_{\mathcal{O}_{S}}B^{\vee}$ is weakly-positive.)
Using that $\mathrm{rk}\,\mathcal{N}_{1}\geq 1$ and
$\dbinom{n+k-1}{n}\leq\dbinom{n+d-1}{n}$, we have thus proved the following
Arakelov inequality

$\mathcal{E}^{d}\leq(\omega_{S})^{\otimes\dbinom{n+d-1}{n}}.$

In particular $\omega_{S}$ is a big line bundle. This finishes the proof of
Theorem 1.1.

To obtain the slightly better constant in the Arakelov inequality of Theorem
1.2, we can argue as follows. For every integer $k\geq 1$, let
$\mathcal{G}^{d-k}$ denote the image of
$\phi_{k}:\mathrm{Sym}^{k}T_{S}\otimes_{\mathcal{O}_{S}}\mathcal{E}^{d}\rightarrow\mathcal{E}^{d-k}$.
If we set $\mathcal{G}^{d}:=\mathcal{E}^{d}$ and
$\mathcal{G}:=\bigoplus_{k=0}^{d}\mathcal{G}^{k}$, then $(\mathcal{G},\theta)$
is by definition a Higgs subsheaf of $(\mathcal{E},\theta)$ (and it is the
smallest Higgs subsheaf of $(\mathcal{E},\theta)$ containing
$\mathcal{E}^{d}$). Let as before $r$ denote the biggest integer $k$ such that
$\phi_{k}$ is not zero, so that $\mathcal{G}^{d-k}=0$ if $k>r$. Define the
$\mathcal{K}^{i}$’s so that the following sequence

$0\rightarrow\mathcal{K}^{d-k}\rightarrow\mathrm{Sym}^{k}T_{S}\rightarrow\mathcal{G}^{d-k}\otimes_{\mathcal{O}_{S}}(\mathcal{E}^{d})^{\vee}\rightarrow
0$

is exact for every non-negative integer $k$. We know thanks to Theorem 1.1
that the cotangent bundle of $S$ is weakly positive. Therefore the dual of
$\det\mathcal{K}^{d-k}$ is weakly-positive for every positive $k$, and we get

$\det\mathrm{Sym}^{k}T_{S}\leq\det\left(\mathcal{G}^{d-k}\otimes_{\mathcal{O}_{S}}(\mathcal{E}^{d})^{\vee}\right)=\det(\mathcal{G}^{d-k})\otimes_{\mathcal{O}_{S}}\left((\mathcal{E}^{d})^{\vee}\right)^{\otimes\mathrm{rk}\,(\mathcal{G}^{d-k})}.$

Since that this holds also trivially for $k=0$, we get that

$\otimes_{k=0}^{r}\left(\det(\mathrm{Sym}^{k}T_{S})\otimes_{\mathcal{O}_{S}}(\mathcal{E}^{d})^{\otimes\mathrm{rk}\,(\mathcal{G}^{d-k})}\right)\leq\otimes_{k=0}^{r}\det(\mathcal{G}^{d-k})=\det\mathcal{G}.$

Recall that $\det(\mathrm{Sym}^{k}\Omega^{1}_{S})=(\omega_{S})^{\otimes
C_{k}}$ with $C_{k}=\dbinom{n+k-1}{n}$, and that

$\sum_{k=0}^{r}\dbinom{n+k-1}{n}=\dbinom{n+r}{n+1}.$

Since $\det\mathcal{G}\leq 0$ thanks to Proposition 4.3, it follows that

$(\mathcal{E}^{d})^{\otimes\mathrm{rk}\,(\mathcal{G})}\leq(\omega_{S})^{\otimes\dbinom{n+r}{n+1}}.$

Note that $\mathrm{rk}\,(\mathcal{G}^{d})=1$,
$\mathrm{rk}\,(\mathcal{G}^{d-1})=n$ (since $\phi_{1}$ is injective) and
$\mathrm{rk}\,(\mathcal{G}^{d-k})\geq 1$ for any $2\leq k\leq r$, so that
$\mathrm{rk}\,(\mathcal{G})\geq 1+n+(r-1)=n+r$. Since the quantity
$\frac{1}{n+r}\cdot\dbinom{n+r}{n+1}$ grows with $r$, we finally get that

$(\mathcal{E}^{d})^{d+1}\leq(\omega_{S})^{\otimes\dbinom{n+d}{n+1}}.$

##  7\. A few questions

### 7.1.

For any smooth proper morphism $f:X\rightarrow S$ with $S$ a smooth projective
complex curve, and any integer $k$, the Higgs bundle
$\mathrm{H}^{k}_{Dol}(X/S)$ is semistable. In fact, using harmonic metrics,
Simpson proved that they are even polystable. Find an algebraic proof of this
fact.

### 7.2.

Find an algebraic proof that the Hodge line bundle is ample on any moduli
stack of polarized Calabi-Yau varieties (up to numerical equivalence).

### 7.3.

Find an algebraic proof that the Arakelov inequality in Theorem 1.2 holds with
$C=\frac{d}{2}$.

## References

  * [Ara71] S. Ju. Arakelov.  Families of algebraic curves with fixed degeneracies.  Izv. Akad. Nauk SSSR Ser. Mat., 35:1269–1293, 1971.
  * [BBT19] Benjamin Bakker, Yohan Brunebarbe, and Jacob Tsimerman.  o-minimal GAGA and a conjecture of Griffiths, 2019.
  * [BC20] Yohan Brunebarbe and Benoît Cadorel.  Hyperbolicity of Varieties Supporting a Variation of Hodge Structure.  Int. Math. Res. Not. IMRN, (6):1601–1609, 2020.
  * [BKT13] Yohan Brunebarbe, Bruno Klingler, and Burt Totaro.  Symmetric differentials and the fundamental group.  Duke Math. J., 162(14):2797–2813, 2013.
  * [Bru17] Yohan Brunebarbe.  Semi-positivity from Higgs bundles.  Preprint, 2017.
  * [Bru18] Yohan Brunebarbe.  Symmetric differentials and variations of Hodge structures.  J. Reine Angew. Math., 743:133–161, 2018.
  * [Bru20a] Yohan Brunebarbe.  Increasing hyperbolicity of varieties supporting a variation of Hodge structures with level structures, 2020.
  * [Bru20b] Yohan Brunebarbe.  A strong hyperbolicity property of locally symmetric varieties.  Ann. Sci. Éc. Norm. Supér. (4), 53(6):1545–1560, 2020.
  * [CP15] Frédéric Campana and Mihai Păun.  Orbifold generic semi-positivity: an application to families of canonically polarized manifolds.  Ann. Inst. Fourier (Grenoble), 65(2):835–861, 2015.
  * [Fal83] G. Faltings.  Arakelov’s theorem for abelian varieties.  Invent. Math., 73(3):337–347, 1983.
  * [Gri70] Phillip A. Griffiths.  Periods of integrals on algebraic manifolds. III. Some global differential-geometric properties of the period mapping.  Inst. Hautes Études Sci. Publ. Math., (38):125–180, 1970.
  * [JZ02] Jürgen Jost and Kang Zuo.  Arakelov type inequalities for Hodge bundles over algebraic varieties. I. Hodge bundles over algebraic curves.  J. Algebraic Geom., 11(3):535–546, 2002.
  * [Kim98] Minhyong Kim.  $ABC$ inequalities for some moduli spaces of log-general type.  Math. Res. Lett., 5(4):517–522, 1998.
  * [Lan15] Adrian Langer.  Bogomolov’s inequality for Higgs sheaves in positive characteristic.  Invent. Math., 199(3):889–920, 2015.
  * [Mau14] Davesh Maulik.  Supersingular K3 surfaces for large primes.  Duke Math. J., 163(13):2357–2425, 2014.  With an appendix by Andrew Snowden.
  * [MB81] Laurent Moret-Bailly.  Familles de courbes et de variétés abéliennes sur ${\mathbb{P}}^{1}$. II. Exemples.  Number 86, pages 125–140. 1981.  Seminar on Pencils of Curves of Genus at Least Two.
  * [MB85] Laurent Moret-Bailly.  Pinceaux de variétés abéliennes.  Astérisque, (129):266, 1985.
  * [OV07] A. Ogus and V. Vologodsky.  Nonabelian Hodge theory in characteristic $p$.  Publ. Math. Inst. Hautes Études Sci., (106):1–138, 2007.
  * [Pet90] C. A. M. Peters.  Rigidity for variations of Hodge structure and Arakelov-type finiteness theorems.  Compositio Math., 75(1):113–126, 1990.
  * [Pet00] Chris Peters.  Arakelov-type inequalities for Hodge bundles, 2000.
  * [Vie83] Eckart Viehweg.  Weak positivity and the additivity of the Kodaira dimension for certain fibre spaces.  In Algebraic varieties and analytic varieties (Tokyo, 1981), volume 1 of Adv. Stud. Pure Math., pages 329–353. North-Holland, Amsterdam, 1983.
  * [Vie95] Eckart Viehweg.  Quasi-projective moduli for polarized manifolds, volume 30 of Ergebnisse der Mathematik und ihrer Grenzgebiete (3) [Results in Mathematics and Related Areas (3)].  Springer-Verlag, Berlin, 1995.
  * [Vie09] Eckart Viehweg.  Arakelov inequalities.  In Surveys in differential geometry. Vol. XIII. Geometry, analysis, and algebraic geometry: forty years of the Journal of Differential Geometry, volume 13 of Surv. Differ. Geom., pages 245–275. Int. Press, Somerville, MA, 2009.
  * [Zuo00] Kang Zuo.  On the negativity of kernels of Kodaira-Spencer maps on Hodge bundles and applications.  volume 4, pages 279–301. 2000.  Kodaira’s issue.