# Homomorphisms of algebraic groups: representability and rigidity

Michel Brion  Université Grenoble Alpes, Institut Fourier, CS 40700, 38058
Grenoble Cedex 9, France

###### Abstract.

Given two algebraic groups $G$, $H$ over a field $k$, we investigate the
representability of the functor of morphisms (of schemes) $\mathbf{Hom}(G,H)$
and the subfunctor of homomorphisms (of algebraic groups) $\mathbf{Hom}_{{\rm
gp}}(G,H)$. We show that $\mathbf{Hom}(G,H)$ is represented by a group scheme,
locally of finite type, if the $k$-vector space ${\mathcal{O}}(G)$ is finite-
dimensional; the converse holds if $H$ is not étale. When $G$ is linearly
reductive and $H$ is smooth, we show that $\mathbf{Hom}_{{\rm gp}}(G,H)$ is
represented by a smooth scheme $M$; moreover, every orbit of $H$ acting by
conjugation on $M$ is open.

##  1\. Introduction

The starting point of this article is the classification problem for actions
of an algebraic group $G$ on an algebraic variety $X$. When $X$ is proper over
the ground field $k$, its automorphism functor is represented by a locally
algebraic group ${\rm Aut}_{X}$ (i.e., a group scheme, locally of finite
type), see [MO67, Thm. 3.7]. Then the $G$-actions on $X$ correspond
bijectively to the homomorphisms $f:G\to{\rm Aut}_{X}$, and the above problem
is equivalent to classifying these homomorphisms up to conjugation by ${\rm
Aut}(X)={\rm Aut}_{X}(k)$. This motivates the following questions:

  * •

Given an algebraic group $G$ and a locally algebraic group $H$, is the functor
of homomorphisms $\mathbf{Hom}_{{\rm gp}}(G,H)$ represented by a scheme $M$?

  * •

If so, $M$ is equipped with an action of $H$ by conjugation on the target; how
to describe the orbits?

When $G$ is of multiplicative type and $H$ is smooth and affine, the
representability of $\mathbf{Hom}_{{\rm gp}}(G,H)$ is due to Grothendieck; he
showed in addition that the representing scheme $M$ is smooth and the morphism

(1.1) $H\times M\longrightarrow M\times M,\quad(h,f)\longmapsto(hfh^{-1},f)$

is smooth as well (see [SGA3, Exp. XI, Thm. 4.2, Cor. 5.1]; these results are
obtained over an arbitrary base). As a consequence, for any field extension
$K/k$ and any $f\in M(K)=\operatorname{Hom}_{K-{\rm gp}}(G_{K},H_{K})$, the
orbit map $H_{K}\to M_{K}$, $h\mapsto hfh^{-1}$ is smooth, and hence every
orbit is open. This may be viewed as a rigidity property for actions of group
schemes of multiplicative type: the only way to deform such an action is via
conjugation on the target.

For $G$ reductive and $H$ smooth affine, the representability of
$\mathbf{Hom}_{{\rm gp}}(G,H)$ was obtained by Demazure; in characteristic
$0$, he also showed that the representing scheme $M$ is smooth (see [SGA3,
Exp. XXIV, Cor. 7.2.3, Prop. 7.3.1]; these results hold again over an
arbitrary base). Further rigidity properties of $M$ were obtained by Margaux
when $G$ is linearly reductive (see [Mar09] and Remark 6.4).

Note that the above scheme $M$ is not necessarily of finite type; for example,
if $G=H={\mathbb{G}}_{m}$ then $M$ is the constant scheme ${\mathbb{Z}}_{k}$.
Much more elaborate examples occur in recent work of Lesieutre (see [Les18,
Lem. 12, Cor. 15]) and Dinh & Oguiso (see [DO19, Lem. 4.5]): they constructed
smooth complex projective varieties $X$ such that ${\rm Aut}(X)$ is discrete
and has infinitely many conjugacy classes of involutions.

Yet if $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme $M$, then $M$
is locally of finite type. Indeed, when viewed as a functor from $k$-algebras
to sets, $\mathbf{Hom}_{{\rm gp}}(G,H)$ commutes with direct limits as a
consequence of [EGA, IV.8.8.2]; thus, the assertion follows from the
characterization of schemes locally of finite type in terms of their functors
of points obtained in [EGA, IV.8.14.2]. Likewise, if the functor of morphisms
(of schemes) $\mathbf{Hom}(G,H)$ is represented by a scheme $N$, then $N$ is
locally of finite type.

The functors $\mathbf{Hom}(G,H)$ and $\mathbf{Hom}_{{\rm gp}}(G,H)$ usually
contain more information than their sets of $K$-valued points for all field
extensions $K/k$. For example, every morphism
$f:{\mathbb{G}}_{a,K}\to{\mathbb{G}}_{m,K}$ is constant, but
$\mathbf{Hom}({\mathbb{G}}_{a},{\mathbb{G}}_{m})$ is not representable, nor is
$\mathbf{Hom}_{{\rm gp}}({\mathbb{G}}_{a},{\mathbb{G}}_{m})$ (see e.g. [Mil17,
Exercise 1.1]). Also, if $G$ and $H$ are linear algebraic groups over an
algebraically closed field $k$ of characteristic $0$, then the set of
homomorphisms $\operatorname{Hom}_{{\rm gp}}(G,H)$ has a natural structure of
affine ind-variety of finite dimension; if in addition $G$ is unipotent, then
one even gets an affine variety (by results of Furter & Kraft, see [FK18, Lem.
8.2.1, Prop. 8.4.1]). But in the latter case, $\mathbf{Hom}_{{\rm gp}}(G,H)$
is representable if and only if $H$ is an extension of a finite group by a
unipotent one.

With these motivations and examples in mind, we consider in this article the
issues of representability of $\mathbf{Hom}(G,H)$ and $\mathbf{Hom}_{{\rm
gp}}(G,H)$, where again $G$ is an algebraic group, and $H$ a locally algebraic
group. Observe that $\mathbf{Hom}(G,H)$ is a group functor relative to
pointwise multiplication in $H$, and is the semi-direct product of the normal
subgroup functor $\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ of pointed
homomorphisms by the group $H$ of constant morphisms (see Lemma 2.6 for
details). Also, it is easy to show that $\mathbf{Hom}_{{\rm gp}}(G,H)$ is a
closed subfunctor of $\mathbf{Hom}(G,H)$ (Lemma 2.1).

The case of an étale group scheme $H$ is easy as well: then
$\mathbf{Hom}(G,H)$ is represented by an étale scheme (Proposition 4.1). Thus,
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by an étale scheme as well. So
we may exclude this case in our first main result, which handles the
representability of $\mathbf{Hom}(G,H)$:

######  Theorem 1.

Let $G$ be an algebraic group, and $H$ a locally algebraic group. Assume that
$H$ is not étale. Then $\mathbf{Hom}(G,H)$ is represented by a locally
algebraic group if and only if the vector space ${\mathcal{O}}(G)$ is finite-
dimensional.

This can be reformulated by using the affinization theorem (see [DG70,
III.3.8.2]): for any algebraic group $G$, the affine scheme $G^{{\rm
aff}}=\operatorname{Spec}{\mathcal{O}}(G)$ is an algebraic group and the
canonical morphism $G\to G^{{\rm aff}}$ is a faithfully flat homomorphism.
Moreover, its kernel $N$ is smooth, connected, central in $G^{0}$ (in
particular, commutative) and satisfies ${\mathcal{O}}(N)=k$; we say that $N$
is anti-affine. As a consequence, $N$ is the largest anti-affine subgroup of
$G$; we denote it by $G_{{\rm ant}}$.

Thus, Theorem 1 asserts that $\mathbf{Hom}(G,H)$ is represented by a locally
algebraic group if and only if $G$ is an extension of a finite group scheme by
an anti-affine one. The proof begins with a reduction to the case where $G$ is
anti-affine; we then show that $\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ is equal
to $\mathbf{Hom}_{{\rm gp}}(G,H)$ and is represented by a form of some
constant group scheme ${\mathbb{Z}}^{n}_{k}$ (Proposition 5.3). For this, we
use a rigidity lemma for anti-affine schemes (Lemma 5.2), a version of a
result of C. and F. Sancho de Salas (see [SS09, Thm. 1.7]).

Our second main result gives a sufficient condition for the homomorphism
functor to be representable. To state it, we introduce a variant of the
classical notion of linear reductivity. We say that an algebraic group $G$
(possibly non-affine) is _linearly reductive_ if every $G$-module is semi-
simple; this is equivalent to the affinization $G^{{\rm aff}}$ being linearly
reductive. Examples of linearly reductive groups include group schemes of
multiplicative type and semi-abelian varieties; see the beginning of Section 6
for more on this notion.

We may now state our second main result in a simplified version; see Theorem
6.3 for the full, more technical statement.

######  Theorem 2.

Let $G$ be a linearly reductive algebraic group, and $H$ a locally algebraic
group. Then $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a smooth scheme
$M$. Moreover, the morphism (1.1) is smooth.

Conversely, if the assertions of Theorem 2 hold for an algebraic group $G$ and
all affine algebraic groups $H$, then $G$ is linearly reductive; see Remark
6.5. So this theorem yields a version of Grothendieck’s representability and
rigidity results mentioned above, which is close to optimal for group schemes
over a field. We refer to [Ro21, Thm. 2] for a generalization of the
representability theorem in its original setting of group schemes of
multiplicative type over an arbitrary base.

This article is organized as follows. Section 2 contains preliminary results
on functors of (homo)morphisms; some of them are obtained in [SGA3, Exp. I] in
a much greater generality. The tangent spaces to these functors are described
in Section 3 by using constructions and results from [SGA3, Exp. II]. Section
4 collects representability results for these functors, when restricted to
various classes of group schemes. In Section 5, we first prove the rigidity
lemma mentioned above, and then deduce Theorem 1. Theorem 6.3 is stated and
proved in the final Section 6, after some preliminary results on linear
reductivity and a closely related notion of semi-reductivity.

Notation and conventions. We consider schemes over a field $k$ of
characteristic $p\geq 0$, with separable closure $k_{s}$ and algebraic closure
$\bar{k}$. Morphisms and products are understood to be over $k$ unless
otherwise stated. Schemes are assumed to be separated and locally of finite
type throughout.

The structural morphism of a scheme $X$ is denoted by
$\pi_{X}:X\to\operatorname{Spec}(k)$. Given a field extension $K/k$, we denote
by $X_{K}$ the $K$-scheme obtained from $X$ by the base change
$\operatorname{Spec}(K)\to\operatorname{Spec}(k)$.

Group schemes are assumed to be locally algebraic in view of our convention on
schemes. Morphisms of group schemes will also be called homomorphisms. The
neutral element of a group scheme $G$ is denoted by $e_{G}$. An algebraic
group is a group scheme of finite type.

We will freely use some of the functorial language of algebraic geometry
developed in [DG70, I.1, I.2] (see also [EH00, Chap. VI]). We identify every
scheme $S$ with its functor of points $h_{S}$.

##  2\. Hom functors

We first recall some basic notions and results from [SGA3, Exp. I, §7] in our
special setting. Given two schemes $X$, $Y$, we denote by $\mathbf{Hom}(X,Y)$
the contravariant functor from schemes to sets which sends every scheme $S$ to
$\operatorname{Hom}_{S}(X\times S,Y\times S)$, and every morphism of schemes
$u:S^{\prime}\to S$ to the pullback map

$\operatorname{Hom}_{S}(X\times S,Y\times
S)\longrightarrow\operatorname{Hom}_{S^{\prime}}(X\times S^{\prime},Y\times
S^{\prime}).$

We may identify $\mathbf{Hom}(X,Y)(S)$ with $\operatorname{Hom}(X\times S,Y)$
by sending every morphism $f:X\times S\to Y$ to $(f,{\rm pr}_{S}):X\times S\to
Y\times S$. This identifies $\mathbf{Hom}(X,Y)(u)$ with the map

$\operatorname{Hom}(X\times S,Y)\longrightarrow\operatorname{Hom}(X\times
S^{\prime},Y),\quad f\longmapsto f\circ({\rm id},u)$

for any $u$ as above. As a consequence,
$\mathbf{Hom}(\operatorname{Spec}(k),Y)$ may be identified with $Y$.

The formation of $\mathbf{Hom}(X,Y)$ commutes with base change by field
extensions $K/k$. Also, every morphism of schemes $\varphi:X^{\prime}\to X$
induces a morphism of functors

$\varphi^{*}:\mathbf{Hom}(X,Y)\longrightarrow\mathbf{Hom}(X^{\prime},Y)$

via precomposition with $\varphi$. Likewise, every morphism of schemes
$\psi:Y\to Y^{\prime}$ induces a morphism of functors

$\psi_{*}:\mathbf{Hom}(X,Y)\longrightarrow\mathbf{Hom}(X,Y^{\prime})$

via postcomposition with $\psi$. For any family of schemes $(X_{i})_{i\in I}$,
the inclusions $X_{i}\to\coprod_{j\in I}X_{j}$ yield an isomorphism of
functors

$\mathbf{Hom}(\coprod_{i\in
I}X_{i},Y)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\prod_{i\in
I}\mathbf{Hom}(X_{i},Y).$

Likewise, for any family of schemes $(Y_{i})_{i\in I}$, the projections ${\rm
pr}_{i}:\prod_{j\in I}Y_{j}\to Y_{i}$ yield an isomorphism of functors

$\mathbf{Hom}(X,\prod_{i\in
I}Y_{i})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\mathbf{Hom}(X,\prod_{i\in
I}Y_{i}).$

We will also freely use the canonical isomorphism of functors

$\mathbf{Hom}(X\times
Y,Z)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\mathbf{Hom}(X,\mathbf{Hom}(Y,Z))$

for any schemes $X$, $Y$, $Z$ (see [SGA3, Exp. I, Prop. 1.7.1]). This
identifies $\mathbf{Hom}(Y,Z)$ with the Weil restriction functor
$\mathbf{R}_{Y/k}(Z)$.

Next, recall the following result (a special case of [DG70, I.2.7.5]):

######  Lemma 2.1.

Let $\psi:Y\to Y^{\prime}$ be a closed immersion of schemes. Then the morphism
of functors $\psi_{*}:\mathbf{Hom}(X,Y)\to\mathbf{Hom}(X,Y^{\prime})$ is a
closed immersion.

We now consider two morphisms of schemes
$\varphi_{1},\varphi_{2}:X^{\prime}\to X$, and their equalizer
$\mathbf{Ker}(\varphi_{1}^{*},\varphi_{2}^{*})$, i.e., the subfunctor of
$\mathbf{Hom}(X,Y)$ such that for any scheme $S$, the set
$\mathbf{Ker}(\varphi_{1}^{*},\varphi_{2}^{*})(S)$ consists of the morphisms
$f:X\times S\to Y$ such that $f\circ(\varphi_{1},{\rm
id})=f\circ(\varphi_{2},{\rm id})$.

######  Lemma 2.2.

With the above notation, $\mathbf{Ker}(\varphi_{1}^{*},\varphi_{2}^{*})$ is a
closed subfunctor of $\mathbf{Hom}(X,Y)$.

###### Proof.

We have a cartesian diagram of functors

$\textstyle{\mathbf{Ker}(\varphi_{1}^{*},\varphi_{2}^{*})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathbf{Hom}(X,Y)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(\varphi_{1}^{*},\varphi_{2}^{*})}$$\textstyle{\mathbf{Hom}(X,Y)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\Delta_{*}}$$\textstyle{\mathbf{Hom}(X,Y\times
Y),}$

where $\Delta:Y\to Y\times Y$ denotes the diagonal, and
$\mathbf{Hom}(X,Y\times Y)$ is identified with
$\mathbf{Hom}(X,Y)\times\mathbf{Hom}(X,Y)$. Moreover, $\Delta_{*}$ is a closed
immersion by Lemma 2.1; this yields the assertion. ∎

######  Lemma 2.3.

Let $\varphi:X^{\prime}\to X$ be a faithfully flat morphism of schemes, and
${\rm pr}_{1},{\rm pr}_{2}:X^{\prime}\times_{X}X^{\prime}\to X^{\prime}$ the
projections.

  1. (i)

$\varphi^{*}$ identifies $\mathbf{Hom}(X,Y)$ with the equalizer
$\mathbf{Ker}({\rm pr}_{1}^{*},{\rm pr}_{2}^{*})$.

  2. (ii)

$\varphi^{*}$ is a closed immersion.

###### Proof.

(i) This holds by descent theory, see e.g. [Vis05, Thm. 2.55] (note that
$\varphi$ is locally of finite presentation in view of our standing assumption
on schemes).

(ii) This follows from (i) together with Lemma 2.2. ∎

In particular, we obtain:

######  Corollary 2.4.

The structural morphism of $X$ yields a closed immersion

$\pi_{X}^{*}:Y=\mathbf{Hom}(\operatorname{Spec}(k),Y)\longrightarrow\mathbf{Hom}(X,Y).$

We may thus see $Y$ as the closed subfunctor of $\mathbf{Hom}(X,Y)$ consisting
of constant morphisms.

As a further direct consequence of Lemma 2.3, we record:

######  Corollary 2.5.

Let $G$ be a group scheme, and $\varphi:X^{\prime}\to X$ a $G$-torsor for the
fpqc topology. Then $\varphi^{*}$ identifies $\mathbf{Hom}(X,Y)$ with the
closed subfunctor of $\mathbf{Hom}(X^{\prime},Y)$ consisting of $G$-invariant
morphisms.

We now assume that $X$ (resp. $Y$) is equipped with a $k$-rational point $x$
(resp. $y$). This yields a subfunctor $\mathbf{Hom}(X,Y;x\mapsto y)$ of
$\mathbf{Hom}(X,Y)$, such that for any scheme $S$, the set
$\mathbf{Hom}(X,Y;x\mapsto y)(S)$ consists of the morphisms $f:X\times S\to Y$
which satisfy $f(x,s)=y$ identically on $S$. In view of the cartesian diagram
of functors

$\textstyle{\mathbf{Hom}(X,Y;x\mapsto
y)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathbf{Hom}(X,Y)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{x^{*}}$$\textstyle{\operatorname{Spec}(k)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{y}$$\textstyle{Y,}$

we see that $\mathbf{Hom}(X,Y;x\mapsto y)$ is a closed subfunctor of
$\mathbf{Hom}(X,Y)$.

In particular, for any group scheme $H$, we obtain a closed subfunctor
$\mathbf{Hom}(X,H;x\mapsto e_{H})$ of $\mathbf{Hom}(X,H)$. Also, note that
$\mathbf{Hom}(X,H)$ is a group functor relative to pointwise multiplication,
and $\mathbf{Hom}(X,H;x\mapsto e_{H})$ is a normal subgroup functor. We also
have the closed subfunctor $H$ of constant morphisms (Corollary 2.4); this is
a subgroup functor as well.

######  Lemma 2.6.

For any scheme $X$ equipped with a $k$-rational point $x$ and for any group
scheme $H$, we have an isomorphism of group functors

$\mathbf{Hom}(X,H)\simeq\mathbf{Hom}(X,H;x\mapsto e_{H})\rtimes H.$

###### Proof.

Let $S$ be a scheme, and $f\in\operatorname{Hom}(X\times S,H)$. Then we have
$f=gh$, where $g\in\operatorname{Hom}(X\times S,H)$ sends $x\times S$ to
$e_{H}$, and $h\in\operatorname{Hom}(S,H)$: just take $h(s)=f(x,s)$ and
$g=fh^{-1}$. Moreover, such a decomposition of $f$ is clearly unique. This
yields the statement. ∎

Next, consider an exact sequence of group schemes

$1\longrightarrow N\stackrel{{\scriptstyle
i}}{{\longrightarrow}}H\stackrel{{\scriptstyle q}}{{\longrightarrow}}Q,$

i.e., $i$, $q$ are homomorphisms, $i$ is a closed immersion, and its schematic
image is the kernel of $q$. Then we readily obtain:

######  Lemma 2.7.

With the above notation, the sequence of group functors

$1\longrightarrow\mathbf{Hom}(X,N)\stackrel{{\scriptstyle
i_{*}}}{{\longrightarrow}}\mathbf{Hom}(X,H)\stackrel{{\scriptstyle
q_{*}}}{{\longrightarrow}}\mathbf{Hom}(X,Q)$

is exact. If $X$ is equipped with a $k$-rational point $x$, then the sequence
of group functors

$1\to\mathbf{Hom}(X,N;x\mapsto e_{N})\stackrel{{\scriptstyle
i_{*}}}{{\to}}\mathbf{Hom}(X,H;x\mapsto e_{H})\stackrel{{\scriptstyle
q_{*}}}{{\to}}\mathbf{Hom}(X,Q;x\mapsto e_{Q})$

is exact as well.

Given two group schemes $G$, $H$, we denote by $\mathbf{Hom}_{{\rm gp}}(G,H)$
the subfunctor of $\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ consisting of
homomorphisms. Clearly, the $H$-action on $\mathbf{Hom}(G,H;e_{G}\mapsto
e_{H})$ by conjugation on the target normalizes $\mathbf{Hom}_{{\rm
gp}}(G,H)$. If $H$ is commutative, then $\mathbf{Hom}_{{\rm gp}}(G,H)$ is a
subgroup functor of the commutative group functor $\mathbf{Hom}(G,H)$.

######  Lemma 2.8.

For any group schemes $G$, $H$, the subfunctor $\mathbf{Hom}_{{\rm gp}}(G,H)$
is closed in $\mathbf{Hom}(G,H)$.

###### Proof.

We adapt the argument of the proof of Lemma 2.2. Let $S$ be a scheme, and
$f\in\operatorname{Hom}(G\times S,H)$. Then $f$ is a homomorphism if and only
if the diagram

$\textstyle{G\times G\times
S\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(\mu,{\rm
id})}$$\scriptstyle{(f,f,{\rm id})}$$\textstyle{G\times
S\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(f,{\rm
id})}$$\textstyle{H\times H\times
S\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(\nu,{\rm
id})}$$\textstyle{H\times S}$

commutes, where $\mu$ (resp. $\nu$) denotes the multiplication in $G$ (resp.
$H$). This yields a cartesian diagram of functors

$\textstyle{\mathbf{Hom}_{{\rm
gp}}(G,H)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mu^{*}}$$\textstyle{\mathbf{Hom}(G,H)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{(\mu^{*},\nu_{*}\circ\Delta)}$$\textstyle{\mathbf{Hom}(G\times
G,H)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\mathbf{Hom}(G\times
G,H\times H),}$

where

$\Delta=(\Delta_{H})_{*}:\mathbf{Hom}(G,H)\to\mathbf{Hom}(G,H\times
H)=\mathbf{Hom}(G,H)\times\mathbf{Hom}(G,H)$

denotes the diagonal. Since $\Delta_{H}$ is a closed immersion, this yields
the statement in view of Lemma 2.1. ∎

######  Lemma 2.9.

Let $G_{1}$, $G_{2}$, $H$ be group schemes, and $\alpha:G_{1}\times G_{2}\to
G_{2}$ an action of $G_{1}$ on $G_{2}$ by automorphisms. Consider the semi-
direct product $G=G_{1}\ltimes G_{2}$. Then the product of restriction
functors

$\mathbf{Hom}_{{\rm gp}}(G,H)\longrightarrow\mathbf{Hom}_{{\rm
gp}}(G_{1},H)\times\mathbf{Hom}_{{\rm gp}}(G_{2},H)$

is a closed immersion.

###### Proof.

Let $S$ be a scheme, and $f:G\times S\to H$ a homomorphism. Denote by $f_{1}$
(resp. $f_{2}$) the restriction of $f$ to $G_{1}\times S$ (resp. $G_{2}\times
S$). Then we have identically on $G_{1}\times G_{2}\times S$

$f(g_{1}g_{2},s)=f_{1}(g_{1},s)f_{2}(g_{2},s),\quad
f_{2}(\alpha(g_{1},g_{2}),s)=f_{1}(g_{1},s)f_{2}(g_{2},s)f_{1}(g_{1},s)^{-1}$

by the definition of the semi-direct product. Conversely, every pair of
homomorphisms $(f_{1},f_{2})$ satisfying the latter equality defines a unique
homomorphism $f:G\times S\to H$, where the scheme $G$ is identified with
$G_{1}\times G_{2}$. This yields the assertion by arguing as in the proof of
Lemma 2.2. ∎

##  3\. Tangent spaces

We first recall the notion of tangent space for a functor $\mathbf{F}$ from
$k$-algebras to sets (see e.g. [EH00, VI.1.3]). Denote by $D=k[t]/(t^{2})$ the
algebra of dual numbers, so that $D=k\oplus k\varepsilon$ where
$\varepsilon^{2}=0$. The algebra homomorphism

$\sigma:D\longrightarrow k,\quad\varepsilon\longmapsto 0$

yields a map $\mathbf{F}(\sigma):\mathbf{F}(D)\to\mathbf{F}(k)$. The fiber of
this map at $f\in\mathbf{F}(k)$ is the tangent space $T_{f}(\mathbf{F})$; it
is equipped with an action of the multiplicative group $k^{\times}$ (the
automorphism group of the $k$-algebra $D$).

More generally, each vector space $V$ yields a $k$-algebra
$\mathbf{D}(V)=k\oplus\varepsilon V$, equipped with the projection
$\sigma_{V}:\mathbf{D}(V)\to k$ with kernel the ideal $\varepsilon V$ of
square $0$. This defines a functor $\mathbf{D}$ from vector spaces to
algebras, satisfying $\mathbf{D}(k)=D$. For any two vector spaces $V$, $W$, we
obtain a fiber product of algebras

(3.1) $\textstyle{\mathbf{D}(V\oplus
W)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\mathbf{D}({\rm
pr}_{V})}$$\scriptstyle{\mathbf{D}({\rm
pr}_{W})}$$\textstyle{\mathbf{D}(V)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\sigma_{V}}$$\textstyle{\mathbf{D}(W)\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\sigma_{W}}$$\textstyle{k.}$

If $\mathbf{F}$ commutes with such fiber products, then $T_{f}(\mathbf{F})$
has a natural structure of $k$-vector space.

Given two functors $\mathbf{F}_{1}$, $\mathbf{F}_{2}$ as above and a morphism
of functors $u:\mathbf{F}_{1}\to\mathbf{F}_{2}$, the induced map
$u(D):\mathbf{F}_{1}(D)\to\mathbf{F}_{2}(D)$ yields the differential

$T_{f}(u):T_{f}(\mathbf{F}_{1})\longrightarrow T_{u(f)}(\mathbf{F}_{2})$

for any $f\in\mathbf{F}_{1}(k)$. If $\mathbf{F}_{1}$, $\mathbf{F}_{2}$ commute
with the fiber products (3.1), then $T_{f}(u)$ is $k$-linear.

These notions may be applied to any contravariant functor from schemes to
sets, and hence to $\mathbf{Hom}(X,Y)$ where $X$, $Y$ are schemes. The
resulting functor from algebras to sets commutes with the fiber products (3.1)
in view of [SGA3, Exp. II, Cor. 3.11.2]. Moreover, for any
$f\in\operatorname{Hom}(X,Y)$, we have a canonical isomorphism of vector
spaces

(3.2)
$T_{f}\mathbf{Hom}(X,Y)\simeq\operatorname{Hom}_{Y}(X,{\mathbb{V}}(\Omega^{1}_{Y})),$

where $\Omega^{1}_{Y}$ denotes the ${\mathcal{O}}_{Y}$-module of Kähler
differentials of $Y$ over $k$, and
${\mathbb{V}}(\Omega^{1}_{Y})=\operatorname{Spec}_{Y}\operatorname{Sym}_{{\mathcal{O}}_{Y}}(\Omega^{1}_{Y})$
(see [SGA3, Exp. II, Prop. 3.3, Cor. 3.11.3]). Equivalently, we have canonical
isomorphisms of vector spaces

$T_{f}\mathbf{Hom}(X,Y)\simeq\operatorname{Hom}_{{\mathcal{O}}_{Y}}(\Omega^{1}_{Y},f_{*}({\mathcal{O}}_{X}))\simeq\operatorname{Hom}_{{\mathcal{O}}_{X}}(f^{*}(\Omega^{1}_{Y}),{\mathcal{O}}_{X}).$

Next, consider a group scheme $H$. By [SGA3, Exp. I, Prop. 6.8.6], the
${\mathcal{O}}_{H}$-module $\Omega^{1}_{H}$ is $H\times H$-equivariant, where
$H\times H$ acts on $H$ by left and right multiplication. In particular,
$\Omega^{1}_{H}$ is equivariant relative to the right $H$-action. Thus, there
is a canonical isomorphism
$\Omega^{1}_{H}\simeq{\mathcal{O}}_{H}\otimes\Omega^{1}_{H}(e_{H})$ in view of
[SGA3, Exp. I, Prop. 6.8.1]. Moreover, we have canonical isomorphisms
$\Omega^{1}_{H}(e_{H})\simeq\mathfrak{m}/\mathfrak{m}^{2}\simeq\operatorname{Lie}(H)^{*}$,
where $\mathfrak{m}$ denotes the maximal ideal of the local ring
${\mathcal{O}}_{H,e_{H}}$, and $\operatorname{Lie}(H)$ stands for the Lie
algebra. This yields a canonical isomorphism
$\Omega^{1}_{H}\simeq{\mathcal{O}}_{H}\otimes\operatorname{Lie}(H)^{*}$. In
view of the isomorphism (3.2), this yields in turn:

######  Lemma 3.1.

Let $X$ be a scheme, $H$ a group scheme, and $f:X\to H$ a morphism. Then there
is a canonical isomorphism of vector spaces

(3.3)
$i:T_{f}\mathbf{Hom}(X,H)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}{\mathcal{O}}(X)\otimes\operatorname{Lie}(H).$

######  Remark 3.2.

(i) With the above notation, we may view $\operatorname{Lie}(H)$ as the affine
space
$\operatorname{Spec}\,\operatorname{Sym}(\mathfrak{m}/\mathfrak{m}^{2})$; this
identifies ${\mathcal{O}}(X)\otimes\operatorname{Lie}(H)$ with
$\operatorname{Hom}(X,\operatorname{Lie}(H))$.

If $X$ is equipped with a $k$-rational point $x$ and $f(x)=e_{H}$, then $i$
restricts to an isomorphism

(3.4) $j:T_{f}\mathbf{Hom}(X,H;x\mapsto
e_{H})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\operatorname{Hom}(X,\operatorname{Lie}(H);x\mapsto
0).$

(ii) By [SGA3, Exp. II, §3.11], the isomorphism (3.3) may be interpreted as
follows: consider

$\varphi\in
T_{f}\mathbf{Hom}(X,H)=\operatorname{Hom}_{Y}(X,{\mathbb{V}}(\Omega^{1}_{H}))=H^{0}(X,f^{*}(T_{H})),$

where $T_{H}$ denotes the tangent bundle. Let $S$ be a scheme, and $x\in
X(S)$; then $f(x)\in H(S)$. Thus, we may view $\varphi(x)$ as an $S$-point of
$T_{H}$ above $f(x)$. Also, $T_{H}$ is equipped with a group scheme structure,
the semi-direct product $\operatorname{Lie}(H)\rtimes H$ (see [SGA3, Exp. II,
§4.1]). Thus, $\varphi(x)f(x)^{-1}$ is an $S$-point of the affine space
$\operatorname{Lie}(H)$, and the assignment $x\mapsto\varphi(x)f(x)^{-1}$
gives back the isomorphism $i$.

Next, consider a homomorphism of group schemes $f:G\to H$, that is,
$f\in\mathbf{Hom}_{{\rm gp}}(G,H)(k)$. Then the $H$-action on
$\mathbf{Hom}_{{\rm gp}}(G,H)$ by conjugation yields a morphism of functors

$\gamma_{f}:H\longrightarrow\mathbf{Hom}_{{\rm gp}}(G,H),\quad
h\longmapsto(g\mapsto hf(g)h^{-1})$

that we may see as the orbit map associated with $f$. Since
$\gamma_{f}(e_{H})=f$, we have the differential

$T_{e_{H}}(\gamma_{f}):T_{e_{H}}(H)\longrightarrow T_{f}\mathbf{Hom}_{{\rm
gp}}(G,H)\subset T_{f}\mathbf{Hom}(G,H)$

that we will view as a map
$\operatorname{Lie}(H)\to\operatorname{Hom}(G,\operatorname{Lie}(H))$ by using
the isomorphism (3.3).

######  Lemma 3.3.

Keep the above notation and assumptions.

  1. (i)

The tangent space $T_{f}\mathbf{Hom}_{{\rm gp}}(G,H)$ is the subspace
$Z^{1}(G,\operatorname{Lie}(H))$ of
$\operatorname{Hom}(G,\operatorname{Lie}(H))$ consisting of the morphisms
$\varphi:G\to\operatorname{Lie}(H)$ such that
$\varphi(g_{1}g_{2})=\varphi(g_{1})+\operatorname{Ad}(f(g_{1}))\varphi(g_{2})$
identically on $G\times G$.

  2. (ii)

The image of the differential $T_{e_{H}}(\gamma_{f})$ is the subspace
$B^{1}(G,\operatorname{Lie}(H))$ of
$\operatorname{Hom}(G,\operatorname{Lie}(H))$ consisting of the morphisms
$g\mapsto\operatorname{Ad}(f(g))x-x$, where $x\in\operatorname{Lie}(H)$.

###### Proof.

(i) This follows from [SGA3, Exp. II, Prop. 4.2].

(ii) We have $\gamma_{f}(h)f(g)^{-1}=hf(g)h^{-1}f(g)^{-1}$ identically on
$G\times H$. In view of Remark 3.2, it follows that
$T_{e_{H}}(\gamma_{f})(x)(g)=x-\operatorname{Ad}(f(g))x$ for any
$x\in\operatorname{Lie}(H)$ and any schematic point $g$ of $G$.

∎

######  Remark 3.4.

(i) The space $Z^{1}(G,\operatorname{Lie}(H))$ consists of the $1$-cocycles of
the Hochschild complex $C^{*}(G,\operatorname{Lie}(H))$, where
$\operatorname{Lie}(H)$ is a $G$-module via $\operatorname{Ad}\circ f$;
moreover, the subspace $B^{1}(G,\operatorname{Lie}(H))$ consists of the
$1$-coboundaries (see [SGA3, Exp. I, §5.1] or [DG70, II.3]). Thus, we have

$Z^{1}(G,\operatorname{Lie}(H))/B^{1}(G,\operatorname{Lie}(H))=H^{1}(G,\operatorname{Lie}(H)),$

the first cohomology group of this module.

With the notation and assumptions of Lemma 3.3, it follows that the map

$T_{e_{H}}(\gamma_{f}):T_{e_{H}}(H)\longrightarrow T_{f}\mathbf{Hom}_{{\rm
gp}}(G,H)$

is surjective if and only if $H^{1}(G,\operatorname{Lie}(H))=0$.

(ii) Most results on cohomology groups of a group scheme $G$ are obtained
under the assumption that $G$ is affine. When $G$ is an algebraic group, this
entails no loss of generality in view of the affinization theorem recalled in
the introduction. Indeed, the pullback map ${\mathcal{O}}(G^{{\rm
aff}})\to{\mathcal{O}}(G)$ is an isomorphism. Moreover, the representation
$\operatorname{Ad}\circ f:G\to\operatorname{GL}(\operatorname{Lie}(H))$
factors uniquely through a representation of $G^{{\rm aff}}$. Therefore,
$\operatorname{Lie}(H)$ is a $G^{{\rm aff}}$-module, and the pullback maps

$Z^{1}(G^{{\rm aff}},\operatorname{Lie}(H))\to
Z^{1}(G,\operatorname{Lie}(H)),\;B^{1}(G^{{\rm aff}},\operatorname{Lie}(H))\to
B^{1}(G,\operatorname{Lie}(H))$

are isomorphisms. So we obtain an isomorphism

$H^{1}(G^{{\rm
aff}},\operatorname{Lie}(H))\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}H^{1}(G,\operatorname{Lie}(H))$

(which extends to all cohomology groups of all $G$-modules).

Next, we obtain a key smoothness result:

######  Lemma 3.5.

Let $G$ be an algebraic group, and $H$ a smooth group scheme. Assume that
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme $M$. Then the
following conditions are equivalent:

  1. (i)

The morphism (1.1)

$\gamma:H\times M\longrightarrow M\times M,\quad(h,f)\longmapsto(hfh^{-1},f)$

is smooth.

  2. (ii)

For any field extension $K/k$ and any $f\in M(K)=\operatorname{Hom}_{K-{\rm
gp}}(G_{K},H_{K})$, the morphism

$\gamma_{f}:H_{K}\longrightarrow M_{K},\quad h\longmapsto(g\mapsto
hf(g)h^{-1})$

is smooth.

  3. (iii)

For any field extension $K/k$ and any $f\in M(K)$, we have

$H^{1}(G_{K},\operatorname{Lie}(H_{K}))=0,$

where $\operatorname{Lie}(H_{K})$ is a $G_{K}$-module via
$\operatorname{Ad}\circ f$.

These conditions hold whenever $G^{{\rm aff}}$ is linearly reductive.

###### Proof.

(i)$\Rightarrow$(ii) Just observe that $\gamma$ is a morphism of $M$-schemes
relative to the second projections; moreover, $\gamma_{f}$ is obtained from
$\gamma$ by base change via $f:\operatorname{Spec}(K)\to M$.

(ii)$\Rightarrow$(i) This follows from the above observation together with the
fiberwise criterion for smoothness (see [EGA, IV.17.8.2]).

(ii)$\Rightarrow$(iii) Since $\gamma_{f}$ is smooth at $e_{H}$, its
differential at this point is surjective. This yields the assertion in view of
Lemma 3.3.

(iii)$\Rightarrow$(ii) We may assume that $K=k$ is algebraically closed. Then
$H$ is a disjoint union of $H(k)$-cosets of the neutral component $H^{0}$, and
hence we may further assume that $H$ is a smooth connected algebraic group.
Also, $f\in M(k)$ and the morphism $\gamma_{f}:H\to M$ has a surjective
differential at $e_{H}$ by Lemma 3.3 again. We now adapt to this setting some
standard considerations for actions of smooth algebraic groups on schemes of
finite type (recall that $M$ is locally of finite type).

Consider the schematic image $X$ of $\gamma_{f}$; this is a closed integral
subscheme of $M$, stable under $H(k)$ and hence under $H$ (see [DG70,
II.5.3.2]). Moreover, $\gamma_{f}$ factors uniquely through a dominant
morphism $\varphi:H\to X$, equivariant for the action of $H$ by left
multiplication on itself. By generic flatness (see [EGA, IV.6.9.1]), there
exists a dense open subscheme $U$ of $X$ such that the pullback
$\varphi^{-1}(U)\to U$ is flat. Since the $H(k)$-translates of
$\varphi^{-1}(U)$ cover $H$, it follows that $\varphi$ is flat.

The fiber of $\varphi$ at $e_{H}$ is the isotropy subgroup scheme ${\rm
Stab}_{H}(f)$, which is smooth in view of the vanishing of
$H^{1}(G,\operatorname{Lie}(H))$ (see [DG70, II.5.2.8]). Thus, $\varphi$ is
smooth at $e_{H}$ (see [EGA, IV.17.5.1]) and hence everywhere by equivariance.
So the image of $\varphi$ (the orbit $H\cdot f$) is open in $X$ and smooth
(see [EGA, IV.2.4.6, IV.6.8.3]); in particular, $H\cdot f$ is locally closed
in $M$. Also, $T_{f}(H\cdot f)=T_{f}(M)$ by Lemma 3.3 again.

We now claim that the natural homomorphism of local rings

$u:{\mathcal{O}}_{M,f}\longrightarrow{\mathcal{O}}_{H\cdot f,f}$

is an isomorphism. Indeed, $u$ is clearly surjective. Consider the associated
graded homomorphism

${\rm gr}(u):{\rm gr}({\mathcal{O}}_{M,f})\longrightarrow{\rm
gr}({\mathcal{O}}_{H\cdot f,f}).$

The right-hand side is a polynomial ring in $n$ generators of degree $1$,
where $n=\dim(H\cdot f)$. Also, ${\rm gr}(u)$ induces an isomorphism on
subspaces of degree $1$. As the left-hand side is generated in degree $1$, it
follows that ${\rm gr}(u)$ is an isomorphism. As a consequence, $u$ is
injective, proving the claim.

By this claim, $H\cdot f$ contains an open neighborhood of $f$ in $M$. Using
equivariance again, it follows that $H\cdot f$ is open in $M$. Since the
morphism $H\to H\cdot f$ is smooth, so is $\gamma_{f}$.

Finally, if $G^{{\rm aff}}$ is linearly reductive, then Remark 3.4(ii) and
[DG70, II.3.3.7] yield the vanishing of
$H^{1}(G_{K},\operatorname{Lie}(H_{K}))$ for any field extension $K/k$. ∎

##  4\. Representability: first steps

###  4.1. Morphisms to étale group schemes

For any group scheme $G$, we denote by $G^{0}$ its neutral component, i.e.,
the connected component of $e_{G}$. Recall that $G^{0}$ is an algebraic group,
and is the kernel of the homomorphism

$\gamma:G\longrightarrow\pi_{0}(G),$

where $\pi_{0}(G)$ is the étale group scheme of connected components.
Moreover, $\gamma$ is faithfully flat (see [DG70, II.5.1.8]), and hence a
$G^{0}$-torsor.

######  Proposition 4.1.

Let $G$ be an algebraic group, and $H$ an étale group scheme.

  1. (i)

The pullback $\gamma^{*}:\mathbf{Hom}(\pi_{0}(G),H)\to\mathbf{Hom}(G,H)$ is an
isomorphism.

  2. (ii)

The functor $\mathbf{Hom}(G,H)$ is represented by an étale group scheme $N$.
If $H$ is finite, then $N$ is finite as well.

###### Proof.

(i) By Corollary 2.5, $\gamma^{*}$ identifies $\mathbf{Hom}(\pi_{0}(G),H)$
with the subfunctor of $G^{0}$-invariants in $\mathbf{Hom}(G,H)$. So it
suffices to show that for any scheme $S$, every morphism $G\times S\to H$ is
invariant under $G^{0}$. For this, we may assume $k$ algebraically closed by
descent. Then the group schemes $\pi_{0}(G)$ and $H$ are constant; moreover,
$G=\coprod_{i\in I}g_{i}G^{0}$ for some family $(g_{i})_{i\in I}$ of $G(k)$,
where $I=\pi_{0}(G)(k)$. So we may further assume that $G$ is connected. Then
it suffices to show that every morphism $f:G\times S\to H$ that sends
$e_{G}\times S$ to $e_{H}$ is constant (Lemma 2.6). We may assume that $S$ is
connected; then $G\times S$ is connected as well (see e.g. [SGA3, Exp. V, Lem.
2.1.2]). The schematic fiber of $f$ at $e_{H}$ is open and closed in $G\times
S$, and contains $e_{G}\times S$; so this fiber is the whole $G\times S$ as
desired.

(ii) In view of (i), we may replace $G$ with $\pi_{0}(G)$, and hence assume
that $G$ is finite and étale. Using Galois descent, we may further assume that
$G$ is constant. Then $\mathbf{Hom}(G,H)$ is the constant group scheme
associated with the group of maps $G(k)\to H(k)$. ∎

######  Corollary 4.2.

Let $G$ be a connected algebraic group, and $H$ a group scheme. Then the
inclusion of $H^{0}$ in $H$ induces isomorphisms

$\mathbf{Hom}(G,H^{0};e_{G}\mapsto
e_{H})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\mathbf{Hom}(G,H;e_{G}\mapsto
e_{H}),$ $\mathbf{Hom}_{{\rm
gp}}(G,H^{0})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\mathbf{Hom}_{{\rm
gp}}(G,H).$

###### Proof.

By Proposition 4.1, the natural morphism
$\pi_{0}(H)\to\mathbf{Hom}(G,\pi_{0}(H))$ is an isomorphism. Thus,
$\mathbf{Hom}(G,\pi_{0}(H);e_{G}\mapsto
e_{\pi_{0}(H)})=\operatorname{Spec}(k)$. Together with the exact sequence
$1\to H^{0}\to H\to\pi_{0}(H)$ and with Lemma 2.7, this yields the assertions.
∎

###  4.2. Two finiteness notions

We introduce finiteness notions on schemes, which will be very convenient for
proving Theorem 6.3.

We say that a scheme $X$ satisfies (FT) (resp. (AFT)) if every connected
component of $X$ is of finite type (resp. affine of finite type). Since $X$ is
locally of finite type, its connected components are open (see [EGA, I.Cor.
6.1.9]). Thus, (FT) (resp. (AFT)) is equivalent to $X$ being a sum (in the
sense of [EGA, I.3.1]) of schemes of finite type (resp. affine of finite
type).

We now record some basic properties of these notions, with no attempt for
exhaustivity.

######  Lemma 4.3.

Consider a group scheme $G$ and two schemes $X$, $Y$.

  1. (i)

$G$ satisfies (FT). Also, $G$ satisfies (AFT) if and only if $G^{0}$ is
affine.

  2. (ii)

If $X$ is étale, then it satisfies (AFT).

  3. (iii)

If $X$ and $Y$ satisfy (FT) (resp. (AFT)), then so does $X\times Y$.

  4. (iv)

If $X$ satisfies (FT) (resp. (AFT)), then so does the $K$-scheme $X_{K}$ for
any field extension $K/k$.

  5. (v)

If there exists a finite Galois extension $K/k$ such that $X_{K}$ satisfies
(FT) (resp. (AFT)), then so does $X$.

  6. (vi)

If $Y$ is a closed subscheme of $X$, and $X$ satisfies (FT) (resp. (AFT)),
then so does $Y$.

###### Proof.

(i) The assertion on (FT) follows from the “theorem of the neutral component”
(see [DG70, II.5.1.1] or [SGA3, Exp. VIA, Cor. 2.4.1]).

If $G$ satisfies (AFT), then $G^{0}$ is clearly affine. Conversely, assume
that $G^{0}$ is affine, and consider a connected component $C$ of $G$. Then
$C_{\bar{k}}$ is the sum of finitely many translates of $G^{0}_{\bar{k}}$ and
hence is affine. By descent, it follows that $C$ is affine as well.

(ii) Just recall that every connected component of $X$ is of the form
$\operatorname{Spec}(K)$ for some finite (separable) field extension $K/k$.

(iii) This follows from the fact that finite products commute with sums (see
[EGA, I.3.2.8]), and preserve the properties of being of finite type (resp.
affine of finite type); see [EGA, I.Prop. 6.3.4].

(iv) This is checked by a similar argument.

(v) Assume that $X_{K}$ satisfies (FT) for $K/k$ finite Galois with group
$\Gamma$. Then $\Gamma$ acts on $X_{K}$, and permutes its connected
components. Thus, $X_{K}$ is a sum of $\Gamma$-stable schemes of finite type.
By Galois descent for subschemes, it follows that $X$ is a sum of schemes of
finite type, i.e., it satisfies (FT). The same argument works for (AFT).

(vi) Let $Y^{\prime}$ be a connected component of $Y$. Then $Y^{\prime}$ is a
closed subscheme of a unique connected component $X^{\prime}$ of $X$. So the
assertion follows from [EGA, I.Prop. 6.3.4] again. ∎

###  4.3. Morphisms from finite group schemes

We first record an easy observation:

######  Lemma 4.4.

Let $X$ be a finite scheme, and $H$ a group scheme. Then the functor
$\mathbf{Hom}(X,H)$ is represented by a group scheme.

###### Proof.

Recall the isomorphism $\mathbf{Hom}(X,H)\simeq\mathbf{R}_{X/k}(H)$, where
$\mathbf{R}_{X/k}$ denotes the Weil restriction functor. By Lemma 4.3, every
finite subset of the underlying topological space of $H$ is contained in an
open affine subscheme of finite type. The representability of
$\mathbf{R}_{X/k}(H)$ by a scheme (locally of finite type) follows from this
by [DG70, I.1.6.6] and its proof. ∎

Next, let $G$, $H$ be group schemes, where $G$ is finite. Combining Lemmas
2.8, 4.3 and 4.4, we see that $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by
a scheme $M$ satisfying (FT). If $H$ is algebraic, then $M$ is of finite type.

Also, recall from [DG70, II.4.7.1] that a group scheme $G$ is _infinitesimal_
if $G$ is finite and $e_{G}$ is its unique point.

######  Proposition 4.5.

With the above notation and assumptions, the scheme $M$ is affine of finite
type under either of the following conditions:

  1. (i)

$G$ is infinitesimal.

  2. (ii)

$H$ is affine.

  3. (iii)

$H$ is connected.

###### Proof.

(i) We may assume that ${\rm char}(k)=p>0$. For any scheme $X$, we denote by
$F_{X}:X\to X^{(p)}$ the relative Frobenius morphism and by

$F^{n}_{X}:X\longrightarrow X^{(p^{n})}$

its $n$th iterate, where $n$ is a positive integer. If $X$ is a group scheme,
then $F^{n}_{X}$ is a homomorphism; we denote by $X_{n}$ its kernel (the $n$th
Frobenius kernel).

By assumption, we have $G=G_{n}$ for $n\gg 0$. For any scheme $S$ and any
homomorphism $f:G\times S\to H$, we have a commutative diagram

$\textstyle{G\times
S\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\scriptstyle{(\pi_{G},F^{n}_{S})}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{F^{n}_{H}}$$\textstyle{S^{(p^{n})}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f^{(n)}}$$\textstyle{H^{(p^{n})}.}$

Thus, we have identically on $G\times S$

$F^{n}_{H}(f(g,s))=f^{(n)}(F^{n}_{S}(s))=F^{n}_{H}f(e_{G},s)=e_{H^{(p^{n})}}.$

So $f$ factors uniquely through $H_{n}$. Thus, $\mathbf{Hom}_{{\rm
gp}}(G,H_{n})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\mathbf{Hom}_{{\rm
gp}}(G,H)$. As $G$ and $H_{n}$ are finite, we may view $\mathbf{Hom}_{{\rm
gp}}(G,H_{n})$ as the functor of Hopf algebra homomorphisms
${\mathcal{O}}(H_{n})\to{\mathcal{O}}(G)$. This is a closed subfunctor of the
functor of morphisms of $k$-modules ${\mathcal{O}}(H_{n})\to{\mathcal{O}}(G)$,
and the latter functor is represented by an affine space.

(ii) Since Hom functors commute with base change, we may assume $k$
algebraically closed. Then $G\simeq I\rtimes F$, where $I=G^{0}$ is
infinitesimal, and $F\simeq\pi_{0}(G)$ is the constant group scheme associated
with $G(k)$ (see e.g. [DG70, II.5.2.4]). In view of Lemma 2.9, we may thus
assume in addition that $G$ is either infinitesimal or constant. In the former
case, we conclude by (i). In the latter case, $\mathbf{Hom}(G,H)\simeq H^{n}$,
where $n$ denotes the order of $G$. Thus, $\mathbf{Hom}(G,H)$ is affine of
finite type, and hence so is $\mathbf{Hom}_{{\rm gp}}(G,H)$.

(iii) By arguing as in the proof of (ii), we reduce to the case where $k$ is
algebraically closed and $G$ is constant of order $n$. Then $g^{n}=e_{G}$
identically on $G$. Thus, for any scheme $S$, every homomorphism $f:G\times
S\to H$ factors uniquely through the schematic fiber at $e_{H}$ of the $n$th
power map of $H$. Denoting this fiber by $H[n]$, it follows that
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is isomorphic to a closed subscheme of
$\mathbf{Hom}(G,H[n])\simeq H[n]^{n}$. So it suffices to show that $H[n]$ is
affine.

As $H$ is connected, there exists an exact sequence of algebraic groups

(4.1) $1\longrightarrow N\longrightarrow H\stackrel{{\scriptstyle
q}}{{\longrightarrow}}A\longrightarrow 1,$

where $N$ is affine and $A$ is an abelian variety (see [Ray70, Lem. IX.2.7]).
Thus, $H[n]$ is a closed subscheme of the pullback $q^{-1}(A[n])$. Since
$A[n]$ is a finite group scheme and $q$ is affine, this yields the desired
statement. ∎

######  Example 4.6.

Let $G$ be the constant group scheme of order $2$, and $H=E\rtimes G$, where
$E$ is an elliptic curve on which $G$ acts by $\pm 1$. Then $H$ is a smooth
proper non-connected algebraic group, and one readily checks that
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by $H[2]\simeq E[2]\coprod E$.
So $\mathbf{Hom}_{{\rm gp}}(G,H)$ is not affine.

######  Example 4.7.

Assume that $p>0$ and consider the infinitesimal group scheme $\alpha_{p}$,
the kernel of the Frobenius endomorphism of ${\mathbb{G}}_{a}$. For any group
scheme $H$, the functor $\mathbf{Hom}_{{\rm gp}}(\alpha_{p},H)$ is represented
by the fiber at $0$ of the $p$th power map of $\operatorname{Lie}(H)$, as
follows from [SGA3, Exp. VII, Thm. 7.2] or [DG70, II.7.4.2]. For example,
$\mathbf{Hom}_{{\rm gp}}(\alpha_{p},{\mathbb{G}}_{a})$ is represented by the
affine line.

###  4.4. Homomorphisms from tori

The following result is a version of Grothendieck’s representability theorem
stated in the introduction (see [SGA3, Exp. XI, Thm. 4.2]):

######  Proposition 4.8.

Let $G$ be a torus, and $H$ a group scheme. Then the functor
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme satisfying (AFT).

###### Proof.

We first consider the case where $H$ is an abelian variety. We claim that for
any scheme $S$, every homomorphism $f:G\times S\to H$ is constant.

To show this, we adapt a classical rigidity argument. By descent, we may
assume that $k$ is algebraically closed. Also, we may assume that $S$ is
connected. Choose a positive integer $n$ prime to $p$. Then the $n$-torsion
subgroups $G[n]$, $H[n]$ are finite and constant. For any $g\in G[n](k)$, the
morphism $S\to H[n]$, $s\mapsto f(g,s)$ is constant. Choose $s_{0}\in S(k)$,
then we have $f(g,s)=f(g,s_{0})$ identically on $G[n]\times S$. Since the
family of the $G[n]$ for $n$ as above is schematically dense in $G$, the
family of the $G[n]\times S$ is schematically dense in $G\times S$ (see [EGA,
IV.11.10.6]). Thus, $f(g,s)=f(g,s_{0})$ identically on $G\times S$. Also, the
morphism $G\to H$, $g\mapsto f(g,s_{0})$ is a homomorphism; it follows that
$f(g,s_{0})=e_{H}$ identically on $G$, since $G$ is a torus, and $H$ an
abelian variety. This yields the claim.

We now consider the general case. By Corollary 4.2, we may assume that $H$ is
a connected algebraic group. Thus, $H$ lies in an exact sequence of the form
(4.1). Using the above claim together with Lemma 2.7, we may further assume
that $H$ is affine. Then $H$ is isomorphic to a closed subgroup scheme of some
general linear group $\operatorname{GL}_{n}$. In view of Lemmas 2.1, 2.8 and
4.3, we may thus reduce to the case where $H=\operatorname{GL}_{n}$. Also,
since the torus $G$ is split by a finite Galois extension of $k$, we may
assume that $G\simeq{\mathbb{G}}_{m}^{r}$ by using Lemma 4.3 again. For any
homomorphism $f:G\times S\to\operatorname{GL}_{n}$, the
${\mathcal{O}}_{S}$-module ${\mathcal{O}}_{S}^{n}$ is a $G$-module via
$(f,{\rm id})$, and hence is the direct sum of its weight submodules (see
[SGA3, Exp. I, Prop. 4.7.3]). It follows that $\mathbf{Hom}_{{\rm gp}}(G,H)$
is represented by the scheme

$\coprod_{(n_{1},\ldots,n_{s})}S_{s}\times\operatorname{GL}_{n}/\operatorname{GL}_{n_{1}}\times\cdots\times\operatorname{GL}_{n_{s}},$

where $(n_{1},\ldots,n_{s})$ runs over the tuples of positive integers with
sum $n$, and $S_{s}\subset({\mathbb{Z}}^{r})^{s}$ denotes the set of
$s$-tuples of pairwise distinct weights (viewed as a constant scheme). Since
every homogeneous space
$\operatorname{GL}_{n}/\operatorname{GL}_{n_{1}}\times\cdots\times\operatorname{GL}_{n_{s}}$
is affine of finite type, this completes the proof. ∎

######  Corollary 4.9.

Let $G$ be a reductive algebraic group, and $H$ a group scheme. Then the
functor $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme $M$
satisfying (FT). If $H$ is affine, then $M$ satisfies (AFT).

###### Proof.

By Corollary 4.2, we may assume that $H$ is a connected algebraic group. Also,
we may choose a maximal torus $T$ of $G$. Then $\mathbf{Hom}_{{\rm gp}}(T,H)$
is representable by Proposition 4.8; moreover, the morphism of functors

$u:\mathbf{Hom}_{{\rm gp}}(G,H)\longrightarrow\mathbf{Hom}_{{\rm gp}}(T,H)$

is relatively representable by a morphism locally of finite presentation, as a
consequence of [SGA3, Exp. XXIV, Prop. 7.2.1]. Thus, $\mathbf{Hom}_{{\rm
gp}}(G,H)$ is representable by a scheme $M$ (locally of finite type).

To show that $M$ satisfies (FT), recall that $T_{K}$ is split for some finite
Galois field extension $K/k$. Using Galois descent and Lemma 4.3 (v), we may
therefore assume that $T$ is split. We now use [SGA3, Exp. XXIV, Cor. 7.1.9],
which asserts that the above morphism $u$ satisfies every property
${\mathcal{P}}$ of morphisms which is stable by composition and base change,
and holds for closed immersions and for the structural morphism $\pi_{H}$. In
view of the assumption on $H$ and [EGA, I.Prop. 6.3.4], we may take for
${\mathcal{P}}$ the property of being of finite type. Also,
$\mathbf{Hom}_{{\rm gp}}(T,H)$ satisfies (FT) by Proposition 4.8 again. It
follows readily that $\mathbf{Hom}_{{\rm gp}}(G,H)$ satisfies (FT). The proof
for (AFT) is obtained similarly by taking for ${\mathcal{P}}$ the property of
being affine of finite type. ∎

###  4.5. Morphisms from abelian varieties

The following result is a consequence of the rigidity lemma proved in the next
section (Lemma 5.2). We provide a short and direct proof.

######  Proposition 4.10.

Let $G$ be an abelian variety, and $H$ a group scheme.

  1. (i)

$H$ has a largest abelian subvariety $H_{{\rm ab}}$.

  2. (ii)

We have

$\mathbf{Hom}_{{\rm gp}}(G,H_{{\rm ab}})=\mathbf{Hom}(G,H_{{\rm
ab}};e_{G}\mapsto e_{H})=\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$

and this functor is represented by a commutative étale group scheme.

###### Proof.

(i) Clearly, we may assume that $H$ is connected, and hence an algebraic
group. Let $A$, $B$ be abelian subvarieties of $H$, and assume that $A$ has
maximal dimension among all such subvarieties. Then $A$, $B$ are both
contained in $H_{{\rm ant}}$, and hence centralize each other (since every
anti-affine group is commutative). Thus, the morphism $A\times B\to H$,
$(a,b)\mapsto ab$ is a homomorphism, and its image is an abelian subvariety
$C$ of $H$. By maximality, we have $C=A$, and hence $B\subset A$.

(ii) By Corollary 4.2, we may again assume that $H$ is a connected algebraic
group. Then the scheme $H$ is quasi-projective (see [Ray70, Cor. VI.2.6]);
also, $G$ is projective. As a consequence, the functor $\mathbf{Hom}(G,H)$ is
represented by a quasi-projective scheme $M$ (see [Gro61, p. 268]). Thus, the
closed subfunctor $\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ is represented by a
closed subscheme $N$ of $M$. Moreover, the formation of $N$ commutes with base
change by field extensions.

We now show that $N$ is étale. For this, we may assume that $k$ is
algebraically closed in view of [EGA, IV.17.7.3]. Since $N$ is locally of
finite type, it suffices to show that $T_{f}(N)=0$ for any $f\in N(k)$. But
this follows from the isomorphism (3.4), since every morphism
$G\to\operatorname{Lie}(H)$ is constant.

We have a chain of closed subfunctors

$\mathbf{Hom}_{{\rm gp}}(G,H_{{\rm ab}})\subset\mathbf{Hom}(G,H_{{\rm
ab}};e_{G}\mapsto e_{H})\subset\mathbf{Hom}(G,H;e_{G}\mapsto e_{H}).$

Moreover, the resulting inclusions of sets of $K$-points are equalities for
any algebraically closed field $K$, in view of (i) and [Mum08, §4, Cor. 1].
Since $\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ is represented by an étale
scheme, these inclusions of subfunctors are equalities. As $\mathbf{Hom}_{{\rm
gp}}(G,H_{{\rm ab}})$ is a commutative group functor, this yields the
assertion. ∎

It is shown in [LS21, Thm. 7.2] that the formation of $H_{{\rm ab}}$ commutes
with base change by field extensions; we will not need this fact.

##  5\. Proof of Theorem 1

We begin with an easy observation:

######  Lemma 5.1.

Let $G$ be a group scheme.

  1. (i)

If $\mathbf{Hom}(G,H)$ is representable for some non-étale group scheme $H$,
then the vector space ${\mathcal{O}}(G)$ is finite-dimensional.

  2. (ii)

If $\mathbf{Hom}(G,{\mathbb{G}}_{a};e_{G}\mapsto 0)=\mathbf{Hom}_{{\rm
gp}}(G,{\mathbb{G}}_{a})$, then $G$ is anti-affine.

###### Proof.

(i) Consider the constant morphism $f:G\to H$ with image $e_{H}$. Then
$T_{f}\mathbf{Hom}(G,H)\simeq{\mathcal{O}}(G)\otimes\operatorname{Lie}(H)$ by
Lemma 3.1; also, $\operatorname{Lie}(H)\neq 0$ as $H$ is not étale. If
$\mathbf{Hom}(G,H)$ is representable, then its tangent space at $f$ is finite-
dimensional, hence the assertion.

(ii) Every $f\in{\mathcal{O}}(G)$ such that $f(e_{G})=0$ satisfies
$f(g_{1}g_{2})=f(g_{1})+f(g_{2})$ identically on $G\times G$. Applying this to
$f^{2}$, we obtain that $2f(g_{1})f(g_{2})=0$ identically. If $p\neq 2$, it
follows that $f=0$; thus, ${\mathcal{O}}(G)=k$. If $p=2$ we consider $f^{3}$
and argue similarly. ∎

Next, we obtain a key rigidity result:

######  Lemma 5.2.

Let $X$ be a geometrically reduced scheme of finite type such that
${\mathcal{O}}(X)=k$. Let $Y$ be a geometrically connected scheme, and
$f:X\times Y\to Z$ a morphism of schemes. Assume that there exist $x_{0}\in
X(\bar{k})$ and $y_{0}\in Y(\bar{k})$ such that $f(x,y_{0})=f(x_{0},y_{0})$
for all $x\in X(\bar{k})$. Then $f$ factors through the projection ${\rm
pr}_{Y}:X\times Y\to Y$.

###### Proof.

By fpqc descent, it suffices to show that the base change $f_{\bar{k}}$
factors through $({\rm pr}_{Y})_{\bar{k}}$. Thus, we may assume that $k$ is
algebraically closed.

Let $W$ be the pullback of ${\rm diag}(Z)$ under the morphism

$X\times Y\longrightarrow Z\times Z,\quad(x,y)\longmapsto(f(x,y),f(x_{0},y)).$

Then the equalizer $W$ is a closed subscheme of $X\times Y$.

Consider the subset $Y^{\prime}$ of $Y$ consisting of those $y$ such that
$X\times\\{y\\}\subset W$ as sets; equivalently, $X\times\\{y\\}\subset W$ as
schemes, since $X$ is reduced. We claim that $Y^{\prime}$ is closed in $Y$.
Indeed, $X\times Y^{\prime}\subset W$ as sets, and hence $\overline{X\times
Y^{\prime}}\subset W$. Since the projection $X\times Y\to Y$ is open, we have
$\overline{X\times Y^{\prime}}=X\times\overline{Y^{\prime}}$. Thus,
$\overline{Y^{\prime}}=Y^{\prime}$, proving the claim.

By this claim and the connectedness of $Y$, it suffices to show that every
$y\in Y^{\prime}(k)$ admits an open neighborhood $U=U(y)$ such that $X\times
U\subset W$ as schemes.

Let $z=f(x_{0},y)$; then $z\in Z(k)$, and $f$ induces a morphism

$f_{n}:X\times Y_{n}\longrightarrow Z_{n}$

for any positive integer $n$, where $Y_{n}$ (resp. $Z_{n}$) denotes the $n$th
infinitesimal neighborhood of $y$ in $Y$ (resp. of $z$ in $Z$). Since $Z_{n}$
is finite, $f_{n}$ is given by an algebra homomorphism

$f_{n}^{\\#}:{\mathcal{O}}(Z_{n})\longrightarrow{\mathcal{O}}(X\times Y_{n}).$

But we have
${\mathcal{O}}(Y_{n})\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}{\mathcal{O}}(X\times
Y_{n})$ since $X$ is of finite type, ${\mathcal{O}}(X)=k$ and $Y_{n}$ is
finite (see [DG70, I.2.2.6]). Thus, $f_{n}$ factors through ${\rm
pr}_{Y_{n}}:X\times Y_{n}\to Y_{n}$. So $f(x,y)=f(x_{0},y)$ identically on
$X\times Y_{n}$, i.e., $X\times Y_{n}\subset W$. By Krull’s intersection
theorem, the family $(Y_{n})_{n\geq 1}$ is schematically dense in an open
neighborhood $U$ of $y$ in $Y$. Thus, the family $(X\times Y_{n})_{n\geq 1}$
is schematically dense in $X\times U$ in view of [EGA, IV.11.10.6]. Since $W$
is a closed subscheme of $X\times Y$, we obtain that $X\times U\subset W$ as
desired. ∎

As mentioned in the introduction, the above result is a version of [SS09, Thm.
1.7]. The proof presented there (and again in [BSU13, §4.3] and [Bri17, Lem.
3.3.3]) requires some minor corrections, e.g., there is a confusion in the
final step between density and schematic density.

We may now obtain the following result, a version of [BSU13, Prop. 5.1.4]:

######  Proposition 5.3.

Let $G$ be an anti-affine algebraic group, and $H$ a group scheme. Then
$\mathbf{Hom}_{{\rm gp}}(G,H)=\mathbf{Hom}(G,H;e_{G}\mapsto e_{H})$ and this
functor is represented by a form of ${\mathbb{Z}}^{n}_{k}$ for some integer
$n\geq 0$.

###### Proof.

To show the equality of functors, we may assume that $k$ is algebraically
closed. We now adapt a classical argument (see [Mum08, p. 43]). Let $S$ be a
connected scheme, and $f:G\times S\to H$ a morphism that sends $e_{G}\times S$
to $e_{H}$. Consider the morphism

$F:G\times G\times S\longrightarrow H,\quad(x,y,s)\longmapsto
f(xy,s)f(y,s)^{-1}f(x,s)^{-1}.$

Then $F(x,e_{G},s)=F(e_{G},x,s)=e_{H}$ for all $x\in G$ and $s\in S$.
Moreover, $G\times S$ is connected. By Lemma 5.2, it follows that
$F(x,y,s)=e_{H}$ identically on $G\times G\times S$, i.e., $f$ is a
homomorphism.

It remains to show that $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a
form of some ${\mathbb{Z}}^{n}_{k}$. We first treat the case where $k$ is
separably closed. Consider again a connected scheme $S$, and let

$f:G\times S\longrightarrow H$

be a homomorphism. Choose an $s_{0}\in S(\bar{k})$; then the morphism of
$\bar{k}$-schemes

$G_{\bar{k}}\times_{\bar{k}}S_{\bar{k}}\longrightarrow
H_{\bar{k}},\quad(x,s)\longmapsto f(x,s)f(x,s_{0})^{-1}$

sends $G_{\bar{k}}\times_{\bar{k}}s_{0}$ to $e_{H}$, and hence factors through
the projection $G_{\bar{k}}\times_{\bar{k}}S_{\bar{k}}\to S_{\bar{k}}$ by
Lemma 5.2. Since $f(e_{G},s)=e_{H}$ identically on $S$, it follows that
$f(g,s)=f(g,s_{0})$ identically on $G_{\bar{k}}\times_{\bar{k}}S_{\bar{k}}$.
Thus, $f_{\bar{k}}$ factors through the projection
$G_{\bar{k}}\times_{\bar{k}}S_{\bar{k}}\to G_{\bar{k}}$. By fpqc descent, it
follows that $f$ factors through the projection $G\times S\to G$. This shows
that $\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by the constant scheme
$\operatorname{Hom}_{{\rm gp}}(G,H)_{k}$. Since $\operatorname{Hom}_{{\rm
gp}}(G,H^{0}_{{\rm
ant}})\stackrel{{\scriptstyle\sim}}{{\to}}\operatorname{Hom}_{{\rm gp}}(G,H)$,
this yields the assertion in view of [Bri09, Lem. 1.5(ii)].

The case of an arbitrary field $k$ follows by Galois descent. Indeed, for any
$f\in\operatorname{Hom}_{k_{s}-{\rm gp}}(G_{k_{s}},H_{k_{s}})$, there exist a
finite Galois extension $K/k$ and a homomorphism of $K$-group schemes
$\varphi:G_{K}\to H_{K}$ such that $f=\varphi_{k_{s}}$, since $f$ factors
through the algebraic group $H^{0}_{k_{s}}$. ∎

Completion of the proof of Theorem 1. If $\mathbf{Hom}(G,H)$ is representable,
then the vector space ${\mathcal{O}}(G)$ is finite-dimensional by Lemma 5.1.
Conversely, assume that ${\mathcal{O}}(G)$ is finite-dimensional. Then we have
an exact sequence of algebraic groups

$1\longrightarrow G_{{\rm ant}}\longrightarrow G\longrightarrow
F\longrightarrow 1,$

where $F$ is finite. By [Bri15, Thm. 1.1], there exists a finite subgroup
scheme $F^{\prime}\subset G$ such that $G=G_{{\rm ant}}F^{\prime}$. This
yields an exact sequence of algebraic groups

$1\longrightarrow F^{\prime\prime}\longrightarrow G_{{\rm ant}}\rtimes
F^{\prime}\longrightarrow G\longrightarrow 1,$

where $F^{\prime\prime}$ is finite as well. In view of Corollary 2.5, we may
thus assume that $G=G_{{\rm ant}}\rtimes F^{\prime}$. Then $G\simeq G_{{\rm
ant}}\times F^{\prime}$ as schemes, and hence

$\mathbf{Hom}(G,H)\simeq\mathbf{Hom}(G_{{\rm
ant}},\mathbf{Hom}(F^{\prime},H)).$

Since $\mathbf{Hom}(F^{\prime},H)$ is represented by a group scheme (Lemma
4.4), we may further assume that $G$ is anti-affine. Then the assertion
follows from Lemma 2.6 and Proposition 5.3.

##  6\. Proof of Theorem 2

We begin with some observations and structure results on the class of linearly
reductive groups. We will also consider the class of _semi-reductive_ groups:
we say that an algebraic group is semi-reductive if its affinization $G^{{\rm
aff}}$ is an extension of a finite group scheme by a reductive group scheme.
Both classes turn out to be closely related.

The _affine_ linearly reductive groups are well-understood: if $p=0$, they are
exactly the extensions of finite group schemes by reductive group schemes,
i.e., the affine semi-reductive groups (see e.g. [DG70, IV.3.3.3]). If $p>0$,
then the affine linearly reductive groups are exactly the extensions

$1\longrightarrow H\longrightarrow G\longrightarrow F\longrightarrow 1,$

where $F$ is a finite group scheme of order prime to $p$ and $H$ is a
connected group scheme of multiplicative type (see [DG70, IV.3.3.6]).
Moreover, $H$ has a largest subtorus $T$, with Cartier dual being the quotient
of the Cartier dual of $H$ by its torsion subgroup (see [DG70, IV.1.3]). As a
consequence, $T$ is characteristic in $H$, and hence normal in $G$. So the
affine linearly reductive groups are exactly the extensions of finite linearly
reductive groups by tori. Thus, every affine linearly reductive group is semi-
reductive, but the converse fails (if $p>0$ again).

As a consequence, _every linearly reductive group is semi-reductive; the
converse holds if and only if $p=0$._ If $p>0$ then the linearly reductive
groups are exactly the extensions of finite linearly reductive groups by semi-
abelian varieties (as follows from the fact that every anti-affine group is a
semi-abelian variety, see [Bri09, Prop. 2.2]). In particular, the smooth
connected linearly reductive groups are exactly the semi-abelian varieties.

We now discuss the behavior of both classes under base change by a field
extension $K/k$. By [Mar09, Prop. 3.2], an affine algebraic group $G$ is
linearly reductive if and only if so is $G_{K}$. Clearly, this invariance
property also holds for anti-affine algebraic groups. In view of the
affinization theorem, it follows that _an algebraic group $G$ is linearly
reductive if and only if so is $G_{K}$._

Also, _if an algebraic group $G$ is semi-reductive, then so is $G_{K}$._ The
converse holds if $K/k$ is separable algebraic (by Galois descent), but fails
for purely inseparable extensions. For example, if $k$ is separably closed but
not algebraically closed, then there exists a non-split extension of
${\mathbb{G}}_{m}$ by the infinitesimal group scheme $\alpha_{p}$, see [SGA3,
Exp. XVII, 5.9 c)]. As every such extension splits over $\bar{k}$, this yields
an example of an affine algebraic group $G$ such that $G_{\bar{k}}$ is semi-
reductive, but $G$ is not.

Next, we discuss the behavior of both classes under taking quotients, normal
subgroup schemes and extensions. Clearly, _every quotient of a linearly
reductive group is linearly reductive. The class of semi-reductive groups is
also stable under quotients,_ since so are the classes of anti-affine,
reductive and finite group schemes.

By [Mar09, Prop. 3.4], the affine linearly reductive groups are also stable by
normal subgroup schemes and extensions. But the class of linearly reductive
groups is not stable under normal subgroup schemes. To see this if $p=0$,
consider a non-trivial extension $G$ of an elliptic curve $E$ by
${\mathbb{G}}_{a}$; then $G$ is anti-affine (see [Bri09, Prop. 2.3]), but of
course ${\mathbb{G}}_{a}$ is not. If $p>0$, let
$H=\alpha_{p}\rtimes{\mathbb{G}}_{m}$, where ${\mathbb{G}}_{m}$ acts on
$\alpha_{p}$ as its automorphism group. Also, let $E$ be a supersingular
elliptic curve; then $\alpha_{p}$ is isomorphic to a subgroup of $E$. Consider
the pushout

$\textstyle{1\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\alpha_{p}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{{\mathbb{G}}_{m}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{1}$$\textstyle{1\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{E\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{G\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{{\mathbb{G}}_{m}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{1.}$

Then $G$ is linearly reductive, but $H$ is not. These examples also show that
the class of semi-reductive groups is not stable under normal subgroup
schemes.

_The class of linearly reductive groups is stable by extensions_. Indeed, by a
standard argument, an algebraic group $G$ is linearly reductive if and only if
the fixed point functor $V\mapsto V^{G}$ (from finite-dimensional $G$-modules
to vector spaces) is exact. Moreover, for any exact sequence of algebraic
groups $1\to N\to G\to Q\to 1$ and any $G$-module $V$, we have
$V^{G}=(V^{N})^{Q}$.

In particular, if $p=0$ then the class of semi-reductive groups is stable
under extensions. This fails if $p>0$ in view of the following:

######  Example 6.1.

Let $V$ be a vector space of finite dimension $n\geq 1$. Consider the relative
Frobenius morphism $F_{V/k}:V\to V^{(p)}$ and denote by $U$ its kernel; then
$U$ is an infinitesimal unipotent subgroup scheme of $V$, normalized by the
natural action of $\operatorname{GL}(V)$. Form the semi-direct product
$G=U\rtimes\operatorname{GL}(V)$; then $G$ is clearly an extension of a
reductive group scheme by a finite group scheme. But there is no exact
sequence

$1\longrightarrow R\longrightarrow G\longrightarrow F\longrightarrow 1,$

where $R$ is reductive and $F$ is finite. Otherwise, $R$ is the reduced
subscheme $G^{0}_{{\rm red}}$, and hence $R=\operatorname{GL}(V)$. As
$R\triangleleft G$, it follows that $\operatorname{GL}(V)$ centralizes $U$, a
contradiction.

Next, we obtain several criteria for semi-reductivity:

######  Proposition 6.2.

Let $G$ be an algebraic group. Consider the conditions:

  1. (i)

$G$ is semi-reductive.

  2. (ii)

There exists an exact sequence of algebraic groups

(6.1) $1\longrightarrow F_{1}\longrightarrow G_{1}\times G_{2}\longrightarrow
G\longrightarrow F_{2}\longrightarrow 1,$

where $F_{1}$, $F_{2}$ are finite, $G_{1}$ is anti-affine, and $G_{2}$ is
reductive.

  3. (iii)

The affinization $G^{{\rm aff}}$ is linearly reductive.

  4. (iv)

$G$ is smooth and $G_{\bar{k}}$ has no non-trivial smooth connected unipotent
normal subgroup.

Then (iii)$\Rightarrow$(i)$\Leftrightarrow$(ii) and (iv)$\Rightarrow$(i). If
$p=0$ then (i), (ii), (iii) are equivalent. If $p>0$ then (i), (ii), (iv) are
equivalent for $G$ smooth.

###### Proof.

(ii)$\Rightarrow$(i) Cut the long exact sequence (6.1) in two short exact
sequences

(6.2) $1\longrightarrow F_{1}\longrightarrow G_{1}\times G_{2}\longrightarrow
G_{0}\longrightarrow 1,\quad 1\longrightarrow G_{0}\longrightarrow
G\longrightarrow F_{2}\longrightarrow 1,$

where $G_{0}$ is the largest smooth connected normal subgroup scheme of $G$.
Denote by $N$ the schematic image of $G_{1}$ in $G_{0}$; then $N=(G_{0})_{{\rm
ant}}$ in view of [Bri17, Lem. 3.3.6]. Thus, $N=G_{{\rm ant}}$ is normal in
$G$, and $G/N=G^{{\rm aff}}$ is an extension of the finite group scheme
$F_{2}$ by the schematic image of $G_{2}$ in $G_{0}$; this image is a
reductive group scheme. This yields the assertion.

(iii)$\Rightarrow$(i) Just recall that every linearly reductive group is semi-
reductive.

(iv)$\Rightarrow$(i) It suffices to show that $G^{0}$ is semi-reductive. We
claim that $G^{0}$ satisfies (iv). Indeed, $G^{0}$ is smooth since so is $G$.
Consider the largest smooth connected unipotent normal subgroup $U$ of
$G^{0}_{\bar{k}}$; then $U$ is normalized by $G(\bar{k})$, and hence
$U\triangleleft G_{\bar{k}}$. So $U$ is trivial, proving the claim.

Thus, we may assume that $G$ is connected. By the Rosenlicht decomposition
(see e.g. [Bri17, Thm. 5.5.1]), there exists a smooth connected affine
algebraic $\bar{k}$-group $H\triangleleft G_{\bar{k}}$ such that
$G_{\bar{k}}=(G_{\bar{k}})_{{\rm ant}}H$. Since $(G_{\bar{k}})_{{\rm ant}}$ is
central in $G_{\bar{k}}$, we see that the unipotent radical of $H$ is trivial,
i.e., $H$ is reductive. So $G_{\bar{k}}/(G_{\bar{k}})_{{\rm ant}}$ is
reductive as well. Since the formation of $G_{{\rm ant}}$ commutes with field
extensions, it follows that $G/G_{{\rm ant}}$ is reductive.

For the remaining implications, we treat the cases $p=0$ and $p>0$ separately
as we use the structure of anti-affine groups, which differs in both cases
(see [Bri09, §2]).

Assume that $p=0$. Then (i)$\Rightarrow$(iii) follows from the linear
reductivity of affine semi-reductive groups. We now show (i)$\Rightarrow$(ii).
By assumption, we have an exact sequence

(6.3) $1\longrightarrow G_{{\rm ant}}\longrightarrow G^{0}\longrightarrow
R\longrightarrow 1,$

where $R$ is reductive. Consider again the Rosenlicht decomposition
$G^{0}=G_{{\rm ant}}G^{0}_{{\rm aff}}$. By the main result of [Mo56], there
exists a Levi decomposition $G^{0}_{{\rm aff}}=R_{u}(G^{0}_{{\rm aff}})\rtimes
L$, where $L$ is reductive, and $R_{u}$ denotes the unipotent radical. In view
of (6.3), it follows that $R_{u}(G^{0}_{{\rm aff}})$ is the largest unipotent
subgroup of $G_{{\rm ant}}$, and $G^{0}=G_{{\rm ant}}L$. This yields an exact
sequence

$1\longrightarrow M\longrightarrow L\longrightarrow R\longrightarrow 1,$

where $M=G_{{\rm ant}}\cap L$ is central in $L$, and hence of multiplicative
type. So there exists a reductive subgroup scheme $L^{\prime}$ of $L$ such
that $L=ML^{\prime}$ and $M\cap L^{\prime}$ is finite. Thus, $G^{0}=G_{{\rm
ant}}L^{\prime}$ and $G_{{\rm ant}}\cap L^{\prime}$ is finite. Equivalently,
we have an exact sequence

$1\longrightarrow G_{{\rm ant}}\cap L^{\prime}\longrightarrow G_{{\rm
ant}}\times L^{\prime}\longrightarrow G^{0}\longrightarrow 1,$

which yields the assertion.

Next, we assume that $p>0$, and show that (i)$\Rightarrow$(ii). Recall that
$G_{{\rm ant}}$ is a semi-abelian variety (see [Bri09, Prop. 2.1]). We may
assume that there is an exact sequence $1\longrightarrow G_{{\rm
ant}}\longrightarrow G\longrightarrow R\longrightarrow 1$, where $R$ is
reductive. In particular, $G$ is smooth and connected; hence its derived
subgroup ${\rm D}(G)$ is smooth, connected and affine. Also, $R=T{\rm D}(R)$
for some central torus $T$. Thus, the pullback of $T$ in $G$ is the largest
normal semi-abelian variety $G_{{\rm sab}}$; moreover, $G=G_{{\rm sab}}{\rm
D}(G)$, and ${\rm D}(G)$ is reductive. It follows that $G=G_{{\rm ant}}L$ for
some reductive subgroup scheme $L$, and we conclude as above.

Still assuming that $p>0$, we show that (i)$\Rightarrow$(iv) when $G$ is
smooth. We may assume that $k$ is algebraically closed. Let $U$ be a smooth
connected unipotent normal subgroup of $G$; then $U\cap G_{{\rm ant}}$ is
finite. As the unipotent radical of $G/G_{{\rm ant}}=G^{{\rm aff}}$ is
trivial, this yields the assertion. ∎

(As already mentioned, the implication (i)$\Rightarrow$(iii) fails if $p>0$.
Also, the implication (i)$\Rightarrow$(iv) fails if $p=0$, as shown again by
the example of a non-trivial extension of an elliptic curve by the additive
group.)

We now obtain a version of Theorem 2 in arbitrary characteristic:

######  Theorem 6.3.

Let $G$ be a semi-reductive algebraic group, and $H$ a group scheme. Then
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme $M$ satisfying (FT).
If $H$ is affine, then $M$ satisfies (AFT). Also, the morphism (1.1) is smooth
if $G$ is linearly reductive and $H$ is smooth.

###### Proof.

If $G$ is finite, then the first assertion follows from Lemmas 2.8, 4.3 and
4.4. For an arbitrary $G$, consider again the two exact sequences (6.2), where
$G_{1}$ is anti-affine, $G_{2}$ is reductive, and $F_{1}$, $F_{2}$ are finite.
Thus, there exists a finite subgroup scheme $F\subset G$ such that $G=G_{0}F$
(see [Bri15, Thm. 1]). By arguing as in the proof of Theorem 1 and using
Corollary 2.5 and Lemma 2.9, we may thus assume that $G=G_{0}$. In particular,
$G$ is connected. In view of Corollary 4.2, we may further assume that $H$ is
connected.

We now use again Corollary 2.5 and Lemma 2.9 to reduce to the case where $G$
is either anti-affine or reductive. In the latter case, we conclude by
Corollary 4.9; in the former case, we use Proposition 5.3.

This shows the assertion about Property (FT); the proof of the assertion about
(AFT) is similar and left to the reader. The final assertion follows from
Lemma 3.5. ∎

######  Remark 6.4.

Assume that $k$ is algebraically closed of characteristic $0$. Consider a
semi-reductive group $G$ and a group scheme $H$. By Theorem 6.3, for any $m\in
M(k)$, the orbit $H^{0}\cdot m$ is open in the connected component of $m$ in
$M$. Since every such orbit is connected, it follows that the connected
components of $M$ are exactly the $H^{0}$-orbits of $k$-rational points. So
the set of $H$-orbits in $M$ is in one-to-one correspondence with the set of
$k$-rational points of the quotient $\pi_{0}(M)/\pi_{0}(H)$, where
$\pi_{0}(M)$ denotes the (constant) scheme of connected components. Also,
since $k=\bar{k}$, the above set of $H$-orbits in $M$ may be identified with
the orbit space $M(k)/H(k)=\operatorname{Hom}_{{\rm gp}}(G,H)/H(k)$. As a
consequence, we have

(6.4) $\operatorname{Hom}_{{\rm
gp}}(G/H)/H(k)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}\operatorname{Hom}_{K-{\rm
gp}}(G_{K},H_{K})/H(K)$

for any algebraically closed field extension $K/k$.

Next, assume that $k$ is algebraically closed of characteristic $p>0$, and
consider a linearly reductive group $G$ and a smooth group scheme $H$. Then
all the above results hold without any change, in view of Lemma 3.5 and
Theorem 6.3 again. The bijection (6.4) gives back the main result of [Mar09]
(Theorem 1.1; see also [Vin96, Prop. 10]).

######  Remark 6.5.

Theorem 6.3 has a partial converse: let $G$ be an algebraic group and assume
that for any smooth affine algebraic group $H$, the functor
$\mathbf{Hom}_{{\rm gp}}(G,H)$ is represented by a scheme $M$ such that the
morphism (1.1) is smooth. Then $G$ is linearly reductive.

Indeed, we have $H^{1}(G,\operatorname{Lie}(H))=0$ for any
$f\in\operatorname{Hom}_{{\rm gp}}(G,H)$, where $G$ acts on
$\operatorname{Lie}(H)$ via $\operatorname{Ad}\circ f$ (Lemma 3.5). As a
consequence, $H^{1}(G,\operatorname{End}(M))=0$ for any finite-dimensional
representation $f:G\to\operatorname{GL}(M)$. But every finite-dimensional
$G$-module $V$ is a summand of some $\operatorname{End}(M)$: just take
$M=V\oplus k$, where $G$ acts trivially on $k$. Thus, $H^{1}(G,V)=0$ for any
such $V$; this yields the assertion by [DG70, II.3.3.7] together with Remark
3.4.

Acknowledgments. Many thanks to Mathieu Florence, Matthieu Romagny and Antoine
Vézier for their careful reading of preliminary versions and for very helpful
comments. Example 6.1 was suggested by Mathieu Florence. I thank Matthieu
Romagny and Peng Du for pointing out a confusion between density and schematic
density in the original proof of the rigidity lemma. Also, thanks to Philippe
Gille for asking a number of stimulating questions and for drawing my
attention on the rigidity result of Margaux. Finally, I thank the referee for
valuable remarks and comments.

## References

  * [Bri09] M. Brion: Anti-affine algebraic groups, J. Algebra 321 (2009), 934–952.
  * [Bri15] M. Brion: On extensions of algebraic groups with finite quotients, Pacific J. Math. 279 (2015), 135–153.
  * [Bri17] M. Brion: Some structure theorems for algebraic groups, Proc. Symp. Pure Math. 94 (2017), 53–125.
  * [BSU13] M. Brion, P. Samuel, V. Uma: Lectures on the structure of algebraic groups and geometric applications, Hindustan Book Agency, New Dehli, 2013; available at https://www-fourier.univ-grenoble-alpes.fr/$\;\widetilde{}\;$mbrion/chennai.pdf
  * [DG70] M. Demazure, P. Gabriel: Groupes algébriques, Masson, Paris, 1970.
  * [DO19] T.-C. Dinh, K. Oguiso: A surface with discrete and non-finitely generated automorphism group, Duke Math. J. 168 (2019), 941–966.
  * [EGA] A. Grothendieck: Éléments de géométrie algébrique (rédigés avec la collaboration de J. Dieudonné), Pub. Math. I.H.É.S. 4, 8, 11, 17, 20, 24, 28, 32 (1961–1967).
  * [EH00] D. Eisenbud, J. Harris: The geometry of schemes, Grad. Text Math. 197, Springer, 2000.
  * [FK18] J.-P. Furter, H. Kraft: On the geometry of the automorphism groups of affine varieties, arXiv: 1809.04175.
  * [Gro61] A. Grothendieck: Techniques de construction et théorèmes d’existence en géométrie algébrique IV : les schémas de Hilbert, Sém. Bourbaki, Vol. 6 (1960–1961), Exp. 221, 249–276.
  * [LS21] B. Laurent, S. Schröer: Para-abelian varieties and Albanese maps, preprint, arXiv: 2101:10829.
  * [Les18] J. Lesieutre: A projective variety with discrete, non-finitely generated automorphism group, Inventiones Math. 212 (2018), no. 1, 189–211.
  * [Mar09] B. Margaux: Vanishing of Hochschild cohomology for affine group schemes and rigidity of homomorphisms between algebraic groups, Documenta Math. 14 (2009), 653–672.
  * [MO67] H. Matsumura, F. Oort: Representability of group functors, and automorphisms of algebraic schemes, Invent. math. 4 (1967), 1–25.
  * [Mil17] J. S. Milne: Algebraic groups. The theory of group schemes of finite type over a field, Cambridge Stud. Adv. Math. 126, Cambridge Univ. Press, 2017.
  * [Mo56] G. D. Mostow, Fully reducible subgroups of algebraic groups, Amer. J. Math. 78 (1956), 200–221.
  * [Mum08] D. Mumford: Abelian varieties. With appendices by C. P. Ramanujam and Yuri Manin. Corrected reprint of the 2nd ed. 1974, Hindustan Book Agency, New Dehli, 2008.
  * [Ray70] M. Raynaud: Faisceaux amples sur les schémas en groupes et les espaces homogènes, Lecture Note Math. 119, Springer, 1970.
  * [Ro21] M. Romagny: Fixed point stacks under groups of multiplicative type, preprint, arXiv: 2101.02450.
  * [SGA3] M. Demazure, A. Grothendieck: Schémas en groupes (SGA3), Tome I. Propriétés générales des schémas en groupes; Tome III. Structure des schémas en groupes réductifs, Revised version edited by P. Gille and P. Polo, Doc. Math. 7, 8, Soc. Math. France, Paris, 2011.
  * [SS09] C. Sancho de Salas, F. Sancho de Salas: Principal bundles, quasi-abelian varieties and structure of algebraic groups, J. Algebra 322 (2009), 2751–2772.
  * [Vin96] E. B. Vinberg: On invariants of a set of matrices, J. Lie Theory 6 (1996), 249–269.
  * [Vis05] A. Vistoli, Notes on Grothendieck topologies, fiber categories and descent theory, in: Fundamental algebraic geometry: Grothendieck’s FGA explained, Math. Surveys Monogr. 123, Amer. Math. Soc., Providence, RI, 2005.