Further author information: (Send correspondence to Tsukasa Fukusato)
Zhengyu Huang: E-mail<EMAIL_ADDRESS>
Haoran Xie: E-mail<EMAIL_ADDRESS>
Tsukasa Fukusato: E-mail<EMAIL_ADDRESS>Telephone:
81-3-5841-4109

# Interactive 3D Character Modeling from 2D Orthogonal Drawings with
Annotations

Zhengyu Huang  Japan Advanced Institute of Science and Technology, Ishikawa,
Japan  Haoran Xie  Japan Advanced Institute of Science and Technology,
Ishikawa, Japan  Tsukasa Fukusato  The University of Tokyo, Tokyo, Japan

###### Abstract

We propose an interactive 3D character modeling approach from orthographic
drawings (e.g., front and side views) based on 2D-space annotations. First,
the system builds partial correspondences between the input drawings and
generates a base mesh with sweeping splines according to edge information in
2D images. Next, users annotates the desired parts on the input drawings
(e.g., the eyes and mouth) by using two type of strokes, called addition and
erosion, and the system re-optimizes the shape of the base mesh. By repeating
the 2D-space operations (i.e., revising and modifying the annotations), users
can design a desired character model. To validate the efficiency and quality
of our system, we verified the generated results with state-of-the-art
methods.

###### keywords:

Interactive modeling, sketch-based modeling, user interface

##  1 INTRODUCTION

In the animation and game industries, when modeling new 3D characters (or
objects), artists first draw orthographic views of them. However, it is
cumbersome and time consuming converting 2D drawings into 3D models manually
because 3D modeling with specialized tools (e.g., Maya and 3DMAX) require
professional knowledge and those user interfaces are not as intuitive as 2D
drawing.

Although several sketch-based modeling methods have been proposed for 3D
content creation [1] , it is still a challenging issue to represent
characteristics of character drawings — there is a gap between the generated
results and professional 3D modelings. To solve this issue, we proposed a user
interface to easily and efficiently design such characteristics on 3D shapes
with the help of 2D annotations. Leveraging orthogonal views, our system can
faithfully reconstruct 3D models from drawings. The main contribution of this
paper is to provide a novel user-friendly workflow for designing 3D models
from 2D drawings with sketch-like annotations, which eliminates the need for
complex 3D operations.

Figure 1: Overview of the proposed system.

##  2 Related Work

Sketching is a form of artistic expression that is highly abstracted from the
real world that has been used in various graphics applications, such as normal
map editing [2], flow design [3], and portrait drawing [4]. As input for 3D
modeling, 2D sketch has ambiguity problem in free-form drawing. To solve this
issue, Teddy [5] was proposed as one of the earliest free-form sketch modeling
user interface. Once 3D viewpoints are determined by users, smooth surfaces
are generated by interpolating the curves extracted from users’ sketches.
Several interpolation functions for sketch modeling [6, 7, 8] were proposed to
improve the results. However, these approaches usual tended to get over-
smoothed surfaces. Some approaches, such as BendSketch [9] offered a solution
for this issue.

Another popular approach to sketch-based modeling is data-based learning. By
analyzing a massive number of sketch-model pairs, methods of this type can
generate an accurate 3D model from a user’s simple sketch. Smirnov et al. [10]
applied Coons patches to learn shape surfaces, but their method is limited to
generating smooth shapes. Sketch2CAD allows users to create objects
incrementally with sketches, which were inferred to CAD instructions by
convolutional neural networks [11]. SimpModeling provides a sketching system
for animalmorphic-head modeling which can generate details of a head from
sketches by pixel-aligned implicit learning.

SketchModeling [12] is very relevant to our work, which is also attempting to
reconstruct character models from multi-views of sketch images, though it is
an automatic approach. With an encoder-decoder U-Net architecture,
SketchModeling can get depth maps and corresponding normal maps from input
sketches, optimize the point cloud by merging these views, and obtain complete
3D models. Inspired by structured annotations [13], which is a generalized-
cylinder-based modeling for a single view, we propose a user interface to
easily and efficiently design characteristics on 3D shapes with fewer types of
annotations leveraging the orthogonal views.

##  3 User interface

In this section, we describe how users interact with the proposed two-stage
user interface (see Fig. 2) to model a character with annotations. The tool
bar consists of four parts: (1) Annotation mode, including local mode
(alignment annotation addition and edge/background marking for corresponding
alignment annotation), addition annotation boundary addition(B), and erosion
annotation addition (E) from left to right; (2) View mode, including 2D front
view (V1), 2D side view (V2), 3D view (V3D) and selected-annotation-only mode;
(3) Rendering mode for annotations, including drawing as segments, drawing as
curve and drawing as generated cylinder; (4) Other options. including
relocation a selected annotation from V1 to V2 (and V2 to V1), a lock button
and a unlock button shows whether or not adopting epipolar constraint from the
other view as a reference when relocating.

###  3.1 Annotation Tool

Since input 2D orthogonal drawings often do not provide complete information
for 3D modeling, our system allows the user to freely draw annotations that
are not limited by edge information. The user can draw brief strokes in either
front view or side view by inserting key points on the canvas with a mouse-
click operation, and each stroke can be labelled as alignment, addition, or
erosion. Then, the system automatically generates corresponding strokes in the
other view with the same label. The user is allowed to edit strokes in editing
mode to calculate correct 3D coordinates of strokes in the 3D view. In
contrast, with the eraser tool, the user clicks on a stroke, and the system
deletes the selected stroke. Moreover, the undo shortcut (“Z” key) can delete
the last stroke from the stroke list. Note that our system can also load or
export the user-drawn annotations by pushing down the “Load”(“L”) or
“Save”(“S”) key on the keyboard.

Figure 2:  User interface in the proposed system. A user models each part of
the character by annotating on both views and viewing the results in 3D view.

###  3.2 Editing Tool

In order to facilitate repeated modification and find the appropriate
correspondence between annotations in two views, the system provides an
editing function. In 2D-view mode (V1 or V2), any vertex position of the
corresponding annotation can be modified by selecting any visible curve. In
editing mode, the system allows the user to generate polar constraints using
the corresponding annotations of another view, reducing the 2D editing of the
vertices to 1D editing.

##  4 Overview

Figure 1 shows an overview of our sketch-based character modeling system.
After a character design is input, users can draw corresponding annotations
one by one in both front and side views under the epipolar constraint. Users
can also add extra edge information that is invisible in the original images
by sketching for each part of the model with the help of our region-based
boundary extraction. Once the user’s annotations are completed, the
corresponding lines from alignment annotations (blue strokes) are extracted as
hard constraints, and those lines marked with the other two annotations are
excluded to revise relationships between the two views of the sketches and
calculate more precise coordinates of 3D points for base mesh. After the base
mesh is determined, cloud points will be sampled from edge information
according to the addition annotations (orange strokes) and erosion annotations
(green strokes) as constrains and optimization-based surface fitting is
conducted to generate a smooth surface.

###  4.1 Alignment-Based Global Modeling (Base-Mesh Generation)

Figure 3:  Generalized cylinders. Candidate boundaries which would be
projected into 2D views are colorized in right column. Three type boundaries
are orthogonal from one to each others.

In the first step, generalized cylinders are generated as a base mesh
according to corresponding alignment annotations in two views. Each part of
the character $P$ is modelled separately. Here, any annotation $A$ is restored
as a series of ordered $n$ key points $\\{\bm{p_{0},p_{1},...,p_{n-1}}\\}$,
belonging to a single part and represented as a Hermitian curve. In this
system, alignment annotations are mainly used to represent the skeleton or
center of gravity of modeling parts and to correct the 3D position of a
specific curve in some cases.

Primitives. A typical generalized cylinder is shown in Fig. 3, consisting of a
skeleton curve $\bm{s}(t)(t\in[0,1])$, and a cross-section radial distance
function $\bm{r}(t)$. Here, different types of candidate boundaries which may
be projected onto two views are marked with different colors.

Camera model and epipolar constraint. In $\mathbb{R}^{3}$ space, extrinsic
parameters of a camera for 3D reconstruction can be denoted by a translation
vector $T$ and a $3\times 3$ rotation matrix
$\bm{R}=[\bm{R}_{0},\bm{R}_{1},\bm{R}_{2}]^{T}$. Here $f$ denotes the focal
length of the camera. Given a set of 3D points $P$, the set of points
projected onto the front view is $Q_{1}$, and the set of points projected onto
the side-view is $Q_{2}$. If $\bm{p}\in P$, $\bm{q}_{1}\in Q_{1}$ and
$\bm{q}_{2}\in Q_{2}$ correspond to each other and the camera parameters of
two views are ($\bm{R}_{1},\bm{T}_{1},f_{1}$) and
($\bm{R}_{2},\bm{T}_{2},f_{2}$), respectively, then, $\bm{q}_{1}$ and
$\bm{q}_{2}$ can be expressed with following equations:

$\left\\{\begin{matrix}\bm{q}_{1x}&=f_{1}\frac{\bm{R}_{10}\cdot\bm{v}+T_{1y}}{\bm{R}_{12}\cdot\bm{v}+T_{1z}}\\\
\bm{q}_{1y}&=f_{1}\frac{\bm{R}_{11}\cdot\bm{v}+T_{1y}}{\bm{R}_{12}\cdot\bm{v}+T_{1z}}\\\
\bm{q}_{2x}&=f_{2}\frac{\bm{R}_{20}\cdot\bm{v}+T_{2y}}{\bm{R}_{22}\cdot\bm{v}+T_{2z}}\\\
\bm{q}_{2y}&=f_{2}\frac{\bm{R}_{21}\cdot\bm{v}+T_{2y}}{\bm{R}_{22}\cdot\bm{v}+T_{2z}}\\\
\end{matrix}\right.$ (1)

Since the two views are orthogonal to the y-axis, $\bm{q}_{1}$ and
$\bm{q}_{2}$ are located exactly at orthogonal planes. We also have
$f_{1}=f_{2}$, $T_{1z}=T_{2z}=0.0$, and $T_{1y}=T_{2y}$. Then, the epipolar
constraint derived from Equation. 1 can be simplified as:

$\bm{q}_{1y}=\bm{q}_{2y}$ (2)

Thus, the corresponding 3D positions can be calculated correctly. For
instance, support a point $\bm{p}$ with coordinates $(x_{1},y_{1})$ in the
front view and $(x_{2},y_{2})$ in the side view. Since $y_{1}=y_{2}$ epipolar
line constraint, its 3D coordinates are $(x_{1},y_{1},-x_{2})$. Although we
are using only this special case, the polar constraints can be naturally
extended to reduce the workload of a multi-view alignment.

Base mesh generation. Our system uses the edge information of the image and
the relative position of alignment annotations to automatically calculate the
preliminary boundary of 2D generalized cylinders of for each view. For any
point on the skeleton curve $\bm{s}(t)$ of an alignment annotation where
$t=t^{\prime}$, the nearest intersection points in two directions between its
vertical line ends and edges can be found as an initial boundary $\bm{b}(t)$
of cross-section $\bm{r}(t)$ . If edges $E$ denotes a set of world coordinates
converted from edge pixels in input images and function $D(E,\bm{b}(t))$
denotes the distance between $E$ and $\bm{b}(t)$, the formula can be described
as follows:

$\left\\{\begin{matrix}\bm{b}(t)=\mathop{\rm
arg~{}min}\limits(D(E,\bm{b}(t)))\\\ D(E,\bm{b}(t))=min_{\bm{e}\in
E}\\{\left\|\bm{b}(t)+\bm{s}(t)-\bm{e}\right\|_{2}+\left\|\bm{b}(t)\right\|_{2}\\}\end{matrix}\right.$
(3)

Note that the mesh generated by this method tends to fail for edges that are
close to parallel. The main reason for this is the need to discretize the mesh
when generating it, so parameter $t$ for $s(t)$ has a certain interval. This
weakness will be overcome in the following local refinement step.

###  4.2 Local Refinement with Annotation Constrains in Two Orthogonal Views

The next step is refining the base mesh with optimization. Here, we introduce
addition annotations and erosion annotations, as shown in Fig. 1, to realize
this objective: addition annotations $A_{a,t}$ define boundaries for cross-
sections along a skeleton curve, while erosion annotations $A_{e,t}$ modify
shapes of end-caps of generalized cylinders. Note both annotations only work
when they attached to an alignment annotation specified by the user.

These annotations should be converted to $\bm{b}(t)_{k}$, denoting a boundary
function with a type of $k$ ($k\in K$). $K$ is a set of types of boundary, and
in our case K=0, 1, 2 denoting a blue curve (cross-section contour), orange
curve, and green curve in the right column of Fig. 3, respectively. All
reconstructed parts of character should have minimized errors between the
visible contours and annotations constraints when project back to 2D views.
The objective function $F(B,E,A)$ in this step can be summarized as:

$\left\\{\begin{matrix}B=\mathop{\rm arg~{}min}\limits F(E,A)\\\
minF(E,A)=\sum_{\bm{b}(t)\in A,k\in K}{D(E,\bm{b}(t)_{k})}\end{matrix}\right.$
(4)

where $B$ is a set of boundaries extracted from annotations; edges $E$ and
function $D(E,\bm{b}(t))$ are the same as the one in Equation. 3 described.
Once $B$ is determined, the base mesh in the global step can be refined as
follows:

Cross-section modeling vertical to the skeleton direction of generalized
cylinder. When $\bm{b}(t)_{0}\notin B$, there are only at most four constraint
points for each cross-section. In this case, the system would fit an ellipse
by regarding these constraint points as its poles. Specially, only one
constraint point in the cross-section means this cross-section is a circle.

If $\exists t=t^{\prime}$ let $\bm{b}(t)_{0}\in B$, cross-sections between
these addition annotation boundaries will be calculated with cubic B-spline
interpolation.As the cross-sections along to the skeleton direction of
generalized cylinder have been calculated, the side surface of the generalized
cylinder matching with input and user-defined boundaries can be generated.

End-caps of generalized cylinder.  If there is an erosion annotation for a
generalized cylinder, the end-caps surfaces will be deformed with Laplacian-
based editing [14] according to the annotation’s shape. Otherwise, surfaces at
the ends of the generalized cylinder are planes.

The main idea of this step is to use two kinds of annotations to obtain
constraints for a generalized cylinder. Therefore, this step can not only
perform boundary refinement, but also improve the effectiveness of boundary
classification, which in turn improves modeling efficiency and accuracy.

##  5 Result and Discussion

(a) Input images

251mins

221mins

(b) SketchModeling

8-parts, 6mins

6-parts, 8mins

(c) Ours

Figure 4: The models created in the comparison study with SketchModeling[12]
and our proposed system.

In our implementation, the system was programmed in C$++$ as a real-time
drawing application on the Windows 10 platform. A workstation with Intel Core
i5-8400, 2.80GHz 2.80GHz, NVIDIA RTX2070 GPU, and 16GB RAM was used as the
testing computing environment. Figure 4 shows 3D modelling results with our
system comparing with SketchModeling [12]. Both the final cloud points and
mesh of the first-row character do not match well with the input sketch after
hours of calculations, which means this learning-based method failed to
predict the position and depth-map of some parts of the characters, while
models generated with our system were more faithful to the original design
drawings in less time by modelling several independent parts.

Unlike the template-based method, the proposed system enables users to make
character models without careful parameter tuning. By comparing our results
with results of state-of-the-art methods, we verified that the proposed system
could improve the quality of 3D character models with simpler but more
intuitive operations. The current system focuses mainly on the 3D shapes
reconstruction process, so texture mapping and complex structure modeling
might be a good topic for future research.

###### Acknowledgements.

This research was supported by the Kayamori Foundation of Informational
Science Advancement, JSPS KAKENHI JP20K19845, and JP19K20316.

## References

  * [1] Bhattacharjee, S. and Chaudhuri, P., “A survey on sketch based content creation: from the desktop to virtual and augmented reality,” Computer Graphics Forum 39(2), 757–780 (2020).
  * [2] He, Y., Xie, H., Zhang, C., Yang, X., and Miyata, K., “Sketch-based normal map generation with geometric sampling,” in [International Workshop on Advanced Imaging Technology (IWAIT) 2021 ], 11766, 261 – 266, International Society for Optics and Photonics, SPIE (2021).
  * [3] Hu, Z., Xie, H., Fukusato, T., Sato, T., and Igarashi, T., “Sketch2vf: Sketch-based flow design with conditional generative adversarial network,” Computer Animation and Virtual Worlds 30(3-4), e1889 (2019).  e1889 cav.1889.
  * [4] Huang, Z., Peng, Y., Hibino, T., Zhao, C., Xie, H., Fukusato, T., and Miyata, K., “dualface: Two-stage drawing guidance for freehand portrait sketching,” Computational Visual Media 8, 63–77 (2022).
  * [5] Igarashi, T., Matsuoka, S., and Tanaka, H., “Teddy: A sketching interface for 3d freeform design,” in [Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) ], 409–416, ACM (1999).
  * [6] Karpenko, O. A. and Hughes, J. F., “Implementation details of smoothsketch: 3d free-form shapes from complex sketches,” in [ACM SIGGRAPH 2006 Sketches ], 51–es, ACM (2006).
  * [7] Joshi, P. and Carr, N. A., “Repoussé: Automatic inflation of 2d artwork,” in [Proceedings of Eurographics Workshop on Sketch-Based Interfaces and Modeling (SBIM) ], 49–55, Eurographics Association (2008).
  * [8] Bernhardt, A., Pihuit, A., Cani, M., and Barthe, L., “Matisse: Painting 2d regions for modeling free-form shapes,” in [Proceedings of the Fifth Eurographics Conference on Sketch-Based Interfaces and Modeling (SBM) ], 57–64, Eurographics Association (2008).
  * [9] Li, C., Pan, H., Liu, Y., Tong, X., Sheffer, A., and Wang, W., “Bendsketch: modeling freeform surfaces through 2d sketching,” ACM Trans. Graph. 36(4), 125:1–125:14 (2017).
  * [10] Smirnov, D., Bessmeltsev, M., and Solomon, J., “Learning manifold patch-based representations of man-made shapes,” in [International Conference on Learning Representations (ICLR) ], 1–24 (2021).
  * [11] Li, C., Pan, H., Bousseau, A., and Mitra, N. J., “Sketch2CAD: Sequential CAD modeling by sketching in context,” ACM Transactions on Graphics (TOG) 39(6), 164:1–164:14 (2020).
  * [12] Lun, Z., Gadelha, M., Kalogerakis, E., Maji, S., and Wang, R., “3D shape reconstruction from sketches via multi-view convolutional networks,” in [Proceedings of International Conference on 3D Vision (3DV) ], 67–77, IEEE (2017).
  * [13] Gingold, Y. I., Igarashi, T., and Zorin, D., “Structured annotations for 2d-to-3d modeling,” ACM Trans. Graph. 28(5), 148 (2009).
  * [14] Au, O. K., Tai, C., Liu, L., and Fu, H., “Dual laplacian editing for meshes,” IEEE Transactions on Visualization and Computer Graphics 12(3), 386–395 (2006).


# LoQT: Low Rank Adapters for Quantized Training

Sebastian Loeschcke
University of Copenhagen
IT University of Copenhagen
<EMAIL_ADDRESS>
&Mads Toftrup∗
Aarhus University
<EMAIL_ADDRESS>
Michael J. Kastoryano
University of Copenhagen
<EMAIL_ADDRESS>
&Serge Belongie
University of Copenhagen
<EMAIL_ADDRESS>
&Vésteinn Snæbjarnarson
University of Copenhagen
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Training of large neural networks requires significant computational
resources. Despite advances using low-rank adapters and quantization,
pretraining of models such as LLMs on consumer hardware has not been possible
without model sharding, offloading during training, or per-layer gradient
updates. To address these limitations, we propose LoQT, a method for
efficiently training quantized models. LoQT uses gradient-based tensor
factorization to initialize low-rank trainable weight matrices that are
periodically merged into quantized full-rank weight matrices. Our approach is
suitable for both pretraining and fine-tuning models, which we demonstrate
experimentally for language modeling and downstream task adaptation. We find
that LoQT enables efficient training of models up to 7B parameters on a
consumer-grade 24GB GPU. We also demonstrate the feasibility of training a 13B
parameter model using per-layer gradient updates on the same hardware.

https://github.com/sebulo/LoQT

##  1 Introduction

Figure 1: Memory usage of Llama 13B, rank 1024. PL: per-layer gradient
updates. A8bit: Adam 8-bit

Training large neural networks requires substantial hardware and energy
resources. Reducing these requirements is thus important for cost efficiency
and environmental sustainability, while also lowering the entry barrier for
researchers and practitioners. The main barriers in training large models are
the compute operations required, as well as the memory needed to store those
computations, in this paper we focus on the latter. Memory use during training
comes primarily from the weights of the model itself as well as the optimizer
states used to train the model. To target the weights, variations on low-rank
adaptation (LoRA) [16, 12, 6, 22, 23] have been suggested to decrease the
number of trainable parameters, in combination with the use of low precision
representations.To target the optimizer, low-rank approaches for projecting
gradients to a lower rank have been used [44]. Finally, various applications
of quantization [10, 26, 6] have been used to decrease memory requirements. In
this work, we combine these approaches into a highly memory-efficient training
configuration.

In typical training setups, the optimizer states take up larger space than the
model itself, as methods such as Adam [18] need to keep track of two
parameters for each weight of the model. GaLore [44] significantly reduces the
number of parameters needed for storing the optimizer states by only keeping
track of the optimizer state in a low-rank projection which is then projected
up to be applied to the model weights. Combining this method with quantization
would further shrink the footprint of the model. However, updating the weights
of a highly quantized model directly in low-precision space has not been shown
to work. This is mainly due to the higher-precision gradient updates being too
small to have an impact on the lower-precision quantized states. Lastly, while
LoRA is memory efficient for parameter-efficient fine-tuning of pre-trained
models, it does not work as a pretraining method by itself [22].

Figure 2: Overview of LoQT. (1) Low-rank factors $P$ and $B$ are periodically
initialized from the gradient of the dequantized model weights $\nabla W$, (2)
then only $B$ is trained while $P_{q}$ and $W_{q}$ are kept quantized and
frozen, over an exponentially increasing interval until $T_{i}$, (3) the low-
rank factors are merged back into the quantized model. The process is repeated
until training halts.

To address these shortcomings, we propose a new method, LoQT. LoQT initializes
two low-rank factors for each weight matrix $W$: 1) $P$, using a projection of
$W$’s gradients into a low-rank subspace, and 2) $B$, initialized to minimize
the error of quantizing and represents the only low-rank matrix being trained.
Only training $B$ means that the optimizer state can be shrunk significantly.
The product $PB$ is periodically merged into the full rank matrix $W$ with
exponentially increasing scheduling. As $W$ and $P$ do not receive gradient
updates, they can be quantized, thus optimizing memory usage even further. We
stress that it is the large accumulated updates that make it possible to
update a quantized model as the addition of smaller changes would not register
in the quantized state. A high-level overview is given in Fig. 2.

We show that LoQT works well with and without a quantized model, enabling not
only a lower memory footprint in the optimizer state but also over the model
parameters. Our results show that we get competitive performance to prior
methods using significantly less memory, in particular when using quantization
of the model weights in an application such as training an LLM. We also
demonstrate superior performance when fine-tuning pre-trained models, by
training and evaluating on the GLUE [36] benchmark for natural language
understanding. Finally, we ablate several properties of the suggested approach
and we find that an exponentially increasing projection gap is beneficial, not
only to our work but also for GaLore. This is particularly crucial for the
training of quantized models. LoQT enables efficient training of 7B models on
consumer-grade hardware with 24GB of memory, and makes it feasible to train
models with up to 13 billion parameters without model parallel by making use
of per-layer gradient updates [25] on the same hardware as shown in Fig. 1.

##  2 Related Work and Background

###  2.1 Neural Network Quantization and NF4

Quantization compresses neural networks by converting high-precision values
into lower-precision formats, significantly reducing storage requirements [43,
33, 1, 4]. The process involves taking a datatype of high precision, such as
32-bit, requiring 4 bytes of memory, and converting it into a representation
with increasing rounding errors but lower memory cost. In this work, we use
NF4 quantization [6], since it is a 4-bit code it only contains $2^{4}$
different values. NF4 works by first normalizing values onto the interval
$[-1:1]$, these are then discretized onto quantiles of the normal
distribution, $(q_{i})_{i=1}^{16}$ (see [6] for details). The elements of a
layer are divided into blocks of 64 weights. Each block $\beta$ has a scaling
factor $\mathcal{M}_{\beta}=\text{max}_{w\in\beta}|w_{32}|$.

$\displaystyle w_{\text{NF4}}$
$\displaystyle=q_{\text{NF4}}(w,\mathcal{M}_{\beta})\coloneqq\text{argmin}_{q_{i}}|w/\mathcal{M}_{\beta}-q_{i}|,$
(1) $\displaystyle w$
$\displaystyle=q_{\text{NF4}}^{-1}(w_{\text{NF4}},\mathcal{M}_{\beta})\coloneqq\mathcal{M}_{\beta}\cdot
w_{\text{NF4}}.$ (2)

We provide an overview of different categories of quantization techniques, and
how they relate to LoQT, in Appendix A. Compared to prior approaches, LoQT
retains the benefits of reduced memory usage while minimizing accuracy loss,
using high-precision updates on a low-rank representation. This allows for
efficient model updates without the overhead of full matrix storage and re-
quantization.

###  2.2 Adaptation of Pretrained Networks

Low-Rank Adaptation (LoRA) [16] enables fine-tuning of pre-trained models
using low-rank adaptors, effectively reducing the memory footprint by only
training weight adaptors for targeted layers. However, simple low-rank
training using LoRA factor matrices has not been shown to work for pre-
training[22].

LoRA employs trainable low-rank matrices $A$ and $B$ that are used to update
$W$ following $W_{t}=W_{t-1}+AB$, where $W_{t-1}$ is frozen to enable precise
adjustments within a low-rank framework. Since LoRA only trains $A$ and $B$
and keeps $W$ fixed, QLoRA [16] explore quantizing $W$. They fine-tune a
quantized model $q(W)=W_{q}$ with 4-bit precision using randomly initialized
16-bit precision factors $A$ and $B$. To address quantization errors
$\mathcal{E}=|W_{q}-W|$, low-rank factors of the quantization error
$\mathcal{E}$ have been used [21].

LoQT extends LoRA to both pretraining and fine-tuning. Unlike traditional
LoRA, LoQT uses $A$ and $B$ to refine $W$ throughout training, with $A$
initialized from $W$’s gradient projection and $B$ trained along this gradient
path. LoQT also incorporates quantization and targeted optimization iterations
similar in spirit to LoftQ [21], correcting for quantization errors in
$W_{q}$, thus better aligning it with the original non-quantized $W$.

###  2.3 Memory Efficient Optimization

#### Optimizer memory consumption

A significant portion of the memory needed to train neural networks is
typically consumed by optimizer states. Notably, Adam [18], one of the most
widely used optimizers, uses double the amount of memory as the gradient
matrix to maintain first and second-order gradient statistics. Efforts to
reduce this overhead have led to the development of adaptive optimization
algorithms like Adafactor [32], which achieves sub-linear memory costs by
factorizing the second-order statistics into a row-column outer product.
GaLore [44] expands on this concept by using low-rank factorization and
projecting low-rank gradients up to a full-rank when updating model weights.

#### Periodic updating of weight matrices

ReLoRA [22] combines low-rank updates with initial full-rank training. They
find that doing one-third of the training in full-rank, and the subsequent
two-thirds in low-rank (see §2.2) results in comparable performance to
standard training methods.

#### Low-rank gradients

GaLore [44], focuses on the structure of the gradients, projecting them into a
low-rank space using factors $P$ and $Q$, which are derived from a truncated
singular value decomposition (SVD) of the weight matrix gradient,
$G_{W}\approx P_{r}\Sigma_{r}Q_{r}$. This reduces memory costs associated with
storing the optimizer states and aligns with findings from recent studies
which suggest that learning primarily occurs within a low-dimensional subspace
at a given time [19, 11]. This can be further combined with applying per-layer
gradient updates, reducing the memory needed for storing the gradients for the
full model at once [25].

LoQT builds on GaLore’s gradient projection (§3.1) to initialize LoRA factors
while updating the full matrix following a schedule inspired by ReLora, while
only training one low-rank matrix per layer. We achieve comparable quality to
GaLore and better performance than ReLoRA while reducing tunable parameters
and memory usage compared to both approaches.

##  3 Efficient Pretraining With LoQT

LoQT works by initializing and training low-rank adapters obtained by taking
the SVD of a given layer’s gradients. Let $W$ indicate the full weights matrix
of a given layer, $P$ be the left factor constructed from the SVD
decomposition of the gradients matrix: $\nabla W=U\Sigma V^{\top}$; i.e. $P$
consists of the first $r$ columns of $U$ corresponding to the singular vectors
with the $r$ largest singular values of $W$. The update rule for an interval
$[T_{i-1},T_{i}]$ is then given by $W_{T_{i}}=W_{T_{i-1}}+PB$, where only the
weights of $B$ are updated. $P$ and $W_{T_{i-1}}$ are kept constant over the
time interval. We describe this in more detail below, followed by a discussion
on periodic updating of the factor $P$, the enabling of quantized pre-
training, error compensation, and exponential update intervals. Pseudo-code
for LoQT is shown in Fig. 3.

###  3.1 Background: GaLore

Zhao et al. [44] show that gradients exhibit a low-rank structure during
training. They exploit this insight by projecting the gradient to a low-rank
subspace and applying the Adam optimizer before projecting back to the
original dimensions. By doing this, the memory-intensive optimizer states
required by Adam are shrunk significantly for low enough ranks.

######  Definition 3.1 (Gradient Low-rank Projection, def. 3.4 in [44]).

Gradient low-rank projection (GaLore) denotes the following gradient update
rules, where $\eta$ is the learning rate, $\rho$ is the Adam optimizer, and
$W\in R^{m\times n}$ is the weight matrix being trained, and $T$ represents
the total number of training iterations between recomputing the projection
matrices:

$W_{T}=W_{0}+\eta\sum_{t=0}^{T-1}\tilde{G}_{t},\text{ where}\hskip
10.00002pt\tilde{G}_{t}=P_{t}\rho_{t}(P_{t}^{\top}G_{t}Q_{t})Q_{t}^{\top},$

where $P_{t}\in R^{m\times r}$ and $Q_{t}\in R^{n\times r}$ are are the
top-$r$ singular vectors from the SVD decomposition of the gradient matrix at
each iteration $t$. In practice, this can be approximated by only applying a
one-sided projection, as in

$W_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}P_{t}\rho_{t}(P_{t}^{\top}G_{t})\hskip
5.0pt\text{ or }\hskip
5.0ptW_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}\rho_{t}(G_{t}Q_{t})Q_{t}^{\top}.$

GaLore demonstrates that it is sufficient to keep the projection matrix fixed
and only update it once every $T$ iterations, which we use in the following.

###  3.2 Low-rank Gradients as Adapters

We now describe the process by which we initialize the parameters we optimize
in LoQT. We adopt the memory-performance trade-off of using only a one-sided
projection. We compute $P^{\top}G$ if $m\leq n$ and $GQ$ otherwise. We want to
achieve a separation between trainable weights and static weights, which we
achieve by rewriting GaLore in terms of low-rank adaptors. Assume without loss
of generality that $m\leq n$. Using the fact that $P_{t}$ is fixed in the
interval $[0,T]$ we have that

$\displaystyle W_{T}$
$\displaystyle=W_{0}+\eta\sum_{t=0}^{T-1}P\rho_{t}(P^{\top}G_{t})$ (3)
$\displaystyle=W_{0}+\eta\underbrace{P\vphantom{\sum_{t=0}^{T-1}}}_{\in\mathbb{R}^{m\times
r}}\\!\underbrace{\sum_{t=0}^{T-1}\rho(P^{\top}G_{t})}_{B\in\mathbb{R}^{r\times
n}}$ (4)

From (4) it is clear that we can keep track of low-rank updates using rank-$r$
low-rank adaptors. We note that in the interval $[0,T]$ only $B$ is updated,
creating the desired separation. If implemented directly, we would need to
compute the gradient with respect to $W$ and then project it down using
$P^{\top}G_{t}$. We find that this step is unnecessary; it is sufficient to
train $B$ using standard gradient descent.

We now show that training the $B$ matrix using gradient descent is equivalent
to training w.r.t. $W_{t}$ as in definition 3.1 Let $G^{W}$ indicate the
gradient of the loss with respect to $W$, and $G^{B}$ for the gradient of the
loss with respect to $B$. Given a weight matrix $W$, an factor $P$ and a
matrix $B$, when computing the forward pass $y=xW+xPB$, the gradient of a loss
function $\mathcal{L}$ w.r.t. $B$ is $G^{B}=P^{\top}G^{W}$. This can be seen
by applying the chain rule to get
$G^{W}=x^{\top}\frac{\partial\mathcal{L}}{\partial y}$. The vector multiplied
onto $B$ is $xP$ giving $G^{B}=(xP)^{\top}\frac{\partial\mathcal{L}}{\partial
y}=P^{\top}x^{\top}\frac{\partial L}{\partial y}=P_{t}^{\top}G^{w}$. This
shows that calculating the gradient w.r.t $B$ gives the same as projecting the
gradient w.r.t $W$. It is thus clear that GaLore’s low-rank gradient updates
should be the same as those obtained using backpropagation through LoRA.

###  3.3 Enabling pretraining with LoRA

Previous work has shown that training low-rank weight matrices works well for
fine-tuning pre-trained weights. However, it has been shown that training low-
rank factors, and periodically merging them into frozen $W$, does not work
when starting with a randomly initialized matrix [22]. Here we address this
shortcoming to enable full training using low-rank weight matrices.

Inspired by prior work [22, 44], we periodically update a given layer
$W_{T+1}=W_{T}+P_{T}B_{T}$ at fixed steps $T\in\mathcal{T}$. This approach
allows $W$ to evolve as a sum of low-rank matrices aligning with GaLore’s
strategy of updating the gradient subspace during training:

$W_{t}=W_{0}+\Delta W_{T_{1}}+\Delta W_{T_{2}}+\ldots+\Delta W_{T_{n}},$

where $t=\sum_{i=1}^{|\mathcal{T}|}T_{i}$ and $\Delta
W_{T_{i}}=P_{T_{i}}B_{T_{i}}$ represents the product of the learning from $B$
during the interval $T_{i}-T_{i-1}$ scaled by the learning rate $\eta$ and
modulated by the gradient projection matrix $P_{T_{i}}$. After each update at
iteration $T_{i}\in\mathcal{T}$, we reinitialize the low-rank factors $P_{T}$
and $B_{T}$. As in [44], we compute the gradient of $W_{T}$ over a single
batch, focusing only on $\nabla W_{T}$ without needing full optimizer states.
Not requiring optimizer states reduces the memory increase compared to full-
rank training.

With the updated $W_{t}$ and reinitialized $P_{t}$ and $B_{t}$, a new gradient
subspace is established for exploring the next $T_{i+1}-T_{i}$ steps. Our
method treats $W_{t}$ as the full-rank repository of accumulated updates.
Although it is periodically updated, $W_{t}$ is not part of the optimizer
state computations and the gradients during the single forward pass are
offloaded to cpu. Since the SVD calculations are done layerwise only the
current layer is needed on GPU, or the SVD can be calculated on CPU. $P_{t}$
defines the general gradient subspace and trajectory for the upcoming
$T_{i+1}-T_{i}$ steps, and $B_{t}$ is adjusted to navigate optimally within
the direction set by $P_{t}$. As only $B_{t}$ is trained, the number of
parameters needing optimizer states is drastically reduced.

###  3.4 Quantized Training

Given that $B$ is the only matrix accumulating gradients and undergoing
changes, the other matrices $W$ and $P$ can be kept quantized. This approach
allows storing the weights in NF4 precision without requiring high-precision
gradient and weights to update $W$ and $P$. To the best of our knowledge, we
are the first to enable efficient 4-bit quantized training using gradient
descent without storing the weights in full precision.

We quantize weights $q_{\text{NF4}}(W)=W_{q}$ and $q_{\text{NF4}}(P)=P_{q}$ as
described in §2.1. During periodic updates at interval time steps
$(\sum_{i=1}^{n}T_{i})_{n=1}^{\text{max}}$, $P_{q}$ and $W_{q}$ are
dequantized using the inverse function,
$P_{\text{BF16}}=q_{\text{NF4}}^{-1}(P_{\text{NF4}})$ and
$W_{BF16}=q_{\text{NF4}}^{-1}(W_{\text{NF4}})$. After this,
$W_{T_{i}}=W_{T_{i-1}}+P_{T_{i-1}}B_{T_{i-1}}$ is computed and quantized. The
quantization and dequantization processes are applied layer by layer, ensuring
that not all layers are simultaneously in a non-quantized state to reduce
memory usage. Moreover, the quantization state itself is re-quantized for
further efficiency following [6]. We implement LoQT using weight-only
quantization, meaning the quantized weights are loaded into memory and then
dequantized before computing the matrix multiplications.

Algorithm 1 LoQT: Low Rank Adapters for Quantized Training

0: $W$: Weight, $T$: Update steps, $\eta$: LR, $r$: rank, $q_{N}(\cdot)$ and
$deq_{N}(\cdot)$: N-bit quant and dequant functions.

1: $G_{W}\leftarrow\nabla_{W}\mathcal{L}(W)$

2: $W_{Q},P_{Q},B\leftarrow\text{Initialize}(W,G_{W})$

3: for each $t$ in training steps do

4: if $t\in T$ then

5: $W\leftarrow W_{Q}+s\cdot P_{Q}\cdot B_{t}$

6: $G^{W}\leftarrow\nabla_{W}\mathcal{L}(W)$

7: $W_{Q},P_{Q},B_{t}\leftarrow\text{Initialize}(W,G^{W})$

8: else

9: $B_{t+1}\leftarrow B_{t}-\rho(G_{t}^{B})$

10: return $\theta$

Algorithm 2 Initialization Procedure

1: Initialize$(W,G^{W})$:

2: $U,S,V^{T}\leftarrow\text{SVD}(G^{W})$

3: $P\leftarrow U[:,:r]$ {First $r$ singular vectors}

4: $P_{q}\leftarrow q_{N}(P)$

5: $B\leftarrow 0$

6: $\hat{W}\leftarrow W$

7: for each $c$ in compensation steps $C$ do

8: $Q_{c}\leftarrow q_{N}(\hat{W})$

9: $B\leftarrow P^{+}(\hat{W}-Q_{c})$

10: $\hat{W}\leftarrow W-PB$

11: return $Q_{c},B,P_{q}$

Figure 3: Pseudo-code for LoQT.

###  3.5 Compensating for Errors Introduced by Quantization

As the quantization process inevitably leads to a discrepancy between the non-
quantized and quantized versions of $W$ we wish to reduce this effect as much
as possible. While compensating for quantization errors has been done before
[21], we need a tailored solution for LoQT.

During the merging phase, we first dequantize to obtain $W_{T-1}$ and
$P_{T-1}$, and then compute the update $W_{T}=W_{T-1}+P_{T-1}B_{T-1}$. This is
immediately followed by re-quantizing to get $Q_{T}=q_{\text{NF4}}(W_{T})$.
Our goal is thus to minimize the quantization error
$\|(Q_{T}+P_{T}B_{T})-W_{T}\|$. To achieve this, we solve for $B_{T}$ in the
merging step, initializing $B_{T}$ as $B_{T}:=P_{T}^{+}(Q_{T}-W_{T})$, where
$P_{T}^{+}$ is the Moore-Penrose pseudo-inverse. This approach avoids
initializing $B_{T}$ as zeros and instead uses it for minimizing the
quantization error $\|Q_{T}-W_{T}\|$. We then iteratively refine $B_{T}$,
recomputing $Q_{T}=q_{\text{NF4}}(W_{T}-P_{T}B_{T})$, improving the alignment
between the full-precision $W$ and its quantized state.

As training advances and the learning rate decays, the magnitude of the update
$B_{T-1}$ to form $W_{T}$ decreases. This leads to negligible differences
between $\left|q\left(Q_{t}+P_{t}B_{t}\right)-Q_{t}\right|$, which results in
the weights plateauing early, as depicted in Fig. 4(a). To address this, we
implement an exponentially increasing scheduler for updating $W$. Drawing from
GaLore’s observation on the exponential decay of gradient rank (Lemma 3.1
[44]), we start with a frequency gap $\tau$ and progressively increase the
update intervals by a factor of $\psi$. The sequence of updates is then given
by $(T_{i})_{i=0}^{\infty}=(\tau+\psi^{i})_{i=0}^{\infty}$ Each $T_{i}$ marks
a training step $t$ when $W$ is updated. This scheduling ensures more frequent
updates earlier in training and more well-spaced adjustments later, allowing
more accumulated gradients before each update.

##  4 Experiments and Results

Table 1: Comparison of low-rank pre-training methods for LLaMA2-style language
models on the C4 dataset. The table shows validation perplexity, memory
estimates, and quantization states for LoQT. The rank ratio $r/d_{model}$ is
relative to the largest weight matrix dimension. Perplexity values are
averaged over three seeds showing mean and standard error. (*) Denotes results
from GaLore [44]. Only one seed was used for the 1B experiment due to compute
constraints.

| 60M | 130M | 350M | 1B
---|---|---|---|---
Full | 33.32 $\pm$ 0.22 (0.36G) | 24.51 $\pm$ 0.03 (0.76G) | 18.87 $\pm$ 0.18 (2.06G) | 15.56* (7.80G)
LoQT (Ours) | 33.98 $\pm$ 0.15 (0.23G) | 24.57 $\pm$ 0.01 (0.49G) | 19.12 $\pm$ 0.01 (0.98G) | 15.55 (3.16G)
LoQT-nq (No quant.) | 33.55 $\pm$ 0.03 (0.28G) | 24.37 $\pm$ 0.02 (0.63G) | 18.85 $\pm$ 0.01 (1.47G) | 15.20 (5.11G)
GaLore | 34.15 $\pm$ 0.24 (0.24G) | 24.81 $\pm$ 0.04 (0.52G) | 19.47 $\pm$ 0.01 (1.22G) | 15.64* (4.38G)
LoRA | 34.99* (0.36G) | 33.92* (0.80G) | 25.58* (1.76G) | 19.21* (6.17G)
ReLoRA | 37.04* (0.36G) | 29.37* (0.80G) | 29.08* (1.76G) | 18.33* (6.17G)
$r/d_{model}$ | 128 / 256 | 256 / 768 | 256 / 1024 | 512 / 2048
Training Tokens | 1.1B | 2.2B | 6.4B | 13.1B

###  4.1 Experimental Setup

We evaluate LoQT by training LLaMA-based[34] language models on the C4 dataset
[30], a collection of processed text in English that was scraped from the
internet [30]. We train models of sizes of 60M, 130M, 350M, and 1B parameters,
adhering to single-epoch training cycles determined by Chinchilla Scaling Laws
[15]. While LoQT is capable of training models up to 13 billion parameters on
consumer GPUs, compute limits prevent us from training to convergence for
sizes above 1B. We also benchmark LoQT on the GLUE test-suite for natural
language understanding [37]. Runs were conducted on up to 4x 40GB NVIDIA A100s
2x 80GB NVIDIA H100s, or a single 24GB NVIDIA RTX 3090. The longest run was
the training of the 1B models, taking approximately four days on the four
A100s. The 3090 was used for throughput and to empirically verify memory
claims.

Hyperparameters are consistent across model sizes, with experiments in BF16
format for memory efficiency. All models use a maximum sequence length of
$256$, a total token batch size of 131K tokens, and a learning rate warmup for
the first 10% of the training steps, followed by cosine annealing to 10% of
the initial learning rate. Full experimental details, including the specific
hyperparameters for each task, are provided in Appendix B.

#### Baselines

For pre-training, we compare LoQT against LoRA [16], ReLoRA [22], GaLore [44],
and non-quantized version of LoQT, LoQT-nq. In our experiments, LoQT, LoRA,
and ReLoRA modify attention and fully connected layers while maintaining full-
rank embeddings and normalization layers. This contrasts with GaLore, which
keeps weights full-rank but projects gradients to low-rank, and standard full-
rank training. For fine-tuning, we benchmark LoQT against GaLore, LoftQ [21],
LoRA and LoQT-nq. All models use identical update frequencies for GaLore,
ReLoRA, LoQT-nq, and LoQT, starting with an update frequency of $T=100$ and
then with exponentially increasing update frequencies. This means that we do
more frequent updates early and fewer as the model stabilizes (see Section
4(b) for more details). All models are trained using the Adam optimizer,
except GaLore which uses GaLoreAdam for gradient projection.

###  4.2 Pre-training of Generative Language Models

Results and details for pretraining of language models of sizes 60M, 130M,
350M and 1B parameters are shown in Table 1. Model sizes are calculated based
on the full models without any low-rank methods. We see that LoQT and LoQT-nq
both perform very close to full rank pretraining and GaLore, while using
significantly less memory by keeping most of the model weights in a quantized
state. For the 60M model, full training is only slightly better than LoQT,
while we see results improve or be within the standard error for the other
sizes. We also notice a slight drop in performance from quantizing the
original weight matrix, comparing LoQT and LoQT-nq. The key difference between
the approaches are the theoretical memory estimaes, e.g. where LoQT requires
59% less memory for the 1B model in full precision and 28% less memory than
GaLore.

Table 2: Results with LoQT, LoQT-nq, and GaLore of DeBERTaV3-base models on
the GLUE development set. We report mean and standard error over three seeds.
The best results on each dataset are shown in bold.

Rank | Method | MNLI | QNLI | RTE | SST | MRPC | CoLA | QQP | STSB | Average
---|---|---|---|---|---|---|---|---|---|---
|  | Acc | Acc | Acc | Acc | f1 | Matt | f1 | PCorr |
32 |  LoQT-nq | 90.0$\pm$0.10 | 94.2$\pm$0.06 | 84.8$\pm$0.75 | 95.9$\pm$0.06 | 94.1$\pm$0.25 | 72.5$\pm$0.41 | 90.0$\pm$0.06 | 91.5$\pm$0.07 | 89.1
32 | LoQT | 90.0$\pm$0.09 | 94.3$\pm$0.04 | 84.1$\pm$0.91 | 95.5$\pm$0.10 | 94.4$\pm$0.20 | 70.5$\pm$0.35 | 89.2$\pm$0.02 | 91.5$\pm$0.13 | 88.7
32 | LoRA | 89.9$\pm$0.03 | 94.0$\pm$0.09 | 83.6$\pm$0.12 | 95.7$\pm$0.15 | 93.5$\pm$0.26 | 69.3$\pm$0.47 | 89.8$\pm$0.11 | 90.7$\pm$0.22 | 88.3
32 | LoftQ | 90.4$\pm$0.09 | 93.2$\pm$0.02 | 83.8$\pm$0.63 | 94.7* | 93.2$\pm$0.14 | 71.1$\pm$0.28 | 89.6$\pm$0.12 | 91.0$\pm$0.09 | 88.4
32 | GaLore | 90.3$\pm$0.07 | 94.0$\pm$0.04 | 83.7$\pm$0.79 | 95.6$\pm$0.07 | 93.4$\pm$0.38 | 70.7$\pm$0.24 | 89.8$\pm$0.05 | 90.6$\pm$0.01 | 88.5

###  4.3 Memory-efficient finetuning

We fine-tune the pre-trained DeBERTa-V3-base111From
https://huggingface.co/microsoft/deberta-v3-base. [13] model on GLUE tasks
using LoQT and compare its performance with a full fine-tuning baseline, LoRA,
LoftQ, and GaLore. See Appendix 5 for details on hyperparameters. Results are
given in Table 2.

We find that both LoQT-nq and LoQT perform well. And somewhat surprisingly, it
sometimes surpasses GaLore, LoftQ, and LoRA.This may indicate that
initializing the LoRA factors with information about the gradient of $W$ could
be a beneficial starting point compared to standard initialization methods. As
the goal of this work is to limit memory consumption, we leave out further
comparisons that could verify these findings to future work.

###  4.4 Memory and Throughput

Table 3: Comparison of memory usage between GaLore, LoRA, and LoQT. $W\in\mathbb{R}^{m\times n}$ ($m\leq n$), rank $r$. | GaLore | LoRA |  LoQT (Ours)
---|---|---|---
Weights | $mn$ | $mn+mr+nr$ | $mn+mr+nr$
Optimizer States | $mr+2nr$ | $2mr+2nr$ | $2nr$
Gradients | $mn$ | $mr+nr$ | $nr$
Pretraining | Yes | No | Yes
Fine-Tuning | Yes | Yes | Yes
Quantizeable | No | Yes | Yes

#### Memory usage

An overview of memory usage for GaLore, LoRA and LoQT is given in Table 3. We
see that LoQT makes use of the same number of trainable parameters as LoRA for
a given rank while using less memory for the optimizer states and gradients
than in both LoRA and GaLore.

We compare the LoQT to the closest in memory performance approach, GaLore, for
13B in Figure 1, and for other model-sizes in Figure 6. We compare three
different use cases, using the approaches directly, combining them with an
8-bit Adam optimizer [5], and using per-layer weight updates with offloading
(while still using 8-bit Adam). We see from the figures that LoQT
significantly shrinks both the number of trainable parameters and optimizer
states compared to GaLore.

Per-layer weight update is essential for GaLore; without it, an additional
$\sim\\!\\!12$ GB of VRAM is needed for gradients in a 7B model, making full-
parameter fine-tuning impossible on a 24GB GPU. Additionally, the per-layer
gradient updates may not work well with DDP (Distributed Data Parallel) and
gradient accumulation. With our method, we can get a lower memory than GaLore
even when they use per-layer gradient updates. When not using per-layer
gradient updates, this difference becomes even bigger as seen for the 7B model
in Figure 6.

Moreover, our method supports training 7B models without per-layer
computations on 24GB GPU. This makes it possible to use multi-GPU training, a
capability not possible with the current GaLore approach. Our memory advantage
allows for a batch size of $1280$ tokens compared to GaLore’s $256$ for the 7B
model on the 24GB RTX3090. With per-layer gradient updates, LoQT can train a
13B model on a single GPU, pushing the limits of hardware efficiency.

#### Throughput

We evaluate the throughput with a sample batch size of 16 with a total batch
size of 512 using gradient accumulation, which is the largest power of 2 that
fits on the GPU. We update the projection matrix $P$ for every 200 iterations.
The per-layer gradient update algorithms apply a weight update for every mini-
batch as they do not support gradient accumulation. For the evaluation, we use
a 1B parameter model with rank 512. We find that LoQT can process 16% fewer
tokens per second than only using AdamW, at 3996 tokens/s compared to 4782
tokens/s on the RTX3090.

##  5 Ablations

(a) EC: Error compensation, EF: Exp. decreasing update frequency.

(b) Ablation for different update frequencies, with exponentially increasing
at the bottom

Figure 4: Ablation results for update frequency, error-compensation,
quantization, model size 130m, and rank $256$.

#### Quantization Error Compensation and Initialization

To assess the impact of quantization error compensation, we analyze the
validation loss curves for a 130 million parameter model. Figure 4(a) shows
that quantizing $W$ or both $W$ and $P$ without error compensation, or
exponential frequency updates, causes the loss to stagnate early. We also note
that quantizing $P$ has a much smaller effect on the loss compared to
quantizing $W$. Error compensation significantly improves the model’s
performance, resulting in approximately 3.5 points better perplexity. Adding
exponentially increasing update frequency improves perplexity by an additional
1.5 points, achieving performance close to that of models without
quantization.

Without the quantization error compensation detailed in §3.5, LoQT’s
performance stagnates earlier and diverges more from the other models. This
demonstrates the effectiveness of our compensation approach in mitigating the
quantization errors introduced during the $W$ update with $AB$ and subsequent
quantization steps.

#### Projection update frequency

Our scheduling approach ensures more frequent updates early in training to
facilitate substantial weight adjustments. As training progresses, the update
frequency decreases, allowing for the accumulation of larger updates to
compensate for smaller updates that might be canceled out by quantization
errors. Figure 4(b) presents an ablation study on our method of progressively
increasing update frequency starting at $100$ and increasing by a factor of
$1.2^{T}$ up to $2500$. We show the validation loss curves for fixed update
frequencies $200$, $400$, $500$, and $1000$.

The results show that exponentially increasing the update gap is particularly
beneficial for models employing quantization, enabling them to achieve the
same perplexity as those without quantization while making use of GaLore.
Conversely, the performance gains are more subtle for models that do not use
quantization and rely solely on GaLore. We hypothesize that even these models
might benefit from the larger projection gap intervals. This could be due to
the reduction in the accumulation of errors from frequent updates of the
projection factor $P$, as the influence of outdated optimizer statistics
becomes less prevalent. Finally, an ablation on the ranks used for $P$ and $B$
is given in Figure 5 in the Appendix.

##  6 Discussion and Conclusion

We present LoQT, a method for memory-efficient pretraining and adaptation of
quantized models. The key insights behind the approach are the benefits of
initializing low-rank factors using the gradient of the weight matrix and
using exponentially increasing update gaps that make updating of a quantized
model possible. While our initial goal was to lower memory usage to facilitate
the training of models such as LLMs on consumer-grade hardware, we are
cautiously excited about the results sometimes being better than the
baselines. These evaluations will be explored in more detail in future work.

Our method is general and opens up new ways of decreasing memory use as well
as improving the training throughput. This could be done by implementing
kernel fusion and using other quantization methods such as NF2 [6] or
quantization of activations, making it possible to do the matrix
multiplications using modern tensor core formats such as FP8 or INT4.

##  7 Impact and Limitations

Our work has the potential to have a significant impact on those working in
hardware-constrained settings by enabling more efficient training on consumer
hardware. We are particularly excited to see the method being applied in
single GPU settings. We validate LoQT on several model sizes, by training over
many steps and by fine-tuning on a standard benchmark for natural language
understanding. While we are confident in our results, further exploration of
training duration, data diversity, and hyper-parameter tuning might lead to
different results in those settings.

##  8 Acknowledgements

This work is supported by the Danish Data Science Academy, which is funded by
the Novo Nordisk Foundation (NNF21SA0069429) and VILLUM FONDEN (40516). Serge
Belongie and Vésteinn Snæbjarnarson are supported by the Pioneer Centre for
AI, DNRF grant number P1. MJK acknowledges support from the Carlsberg
Foundation and the Novo Nordisk Foundation.

## References

  * [1] Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R. Lyu.  Towards efficient post-training quantization of pre-trained language models, 2021.
  * [2] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry.  Scalable methods for 8-bit training of neural networks, 2018.
  * [3] Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry.  Logarithmic unbiased quantization: Simple 4-bit training in deep learning, 2022.
  * [4] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022\.
  * [5] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.  8-bit optimizers via block-wise quantization, 2022.
  * [6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * [7] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh.  Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023.
  * [8] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh.  Extreme compression of large language models via additive quantization, 2024.
  * [9] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.
  * [10] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.  A survey of quantization methods for efficient neural network inference.  CoRR, abs/2103.13630, 2021.
  * [11] Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer.  Gradient descent happens in a tiny subspace, 2018.
  * [12] Soufiane Hayou, Nikhil Ghosh, and Bin Yu.  Lora+: Efficient low rank adaptation of large models.  arXiv preprint arXiv:2402.12354, 2024.
  * [13] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2023.
  * [14] Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee.  Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models, 2024.
  * [15] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [16] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * [17] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss, 2018.
  * [18] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * [19] Brett W. Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli.  How many degrees of freedom do we need to train deep networks: a loss landscape perspective, 2022.
  * [20] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee.  FlexRound: Learnable rounding based on element-wise division for post-training quantization.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 18913–18939. PMLR, 23–29 Jul 2023.
  * [21] Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao.  Loftq: Lora-fine-tuning-aware quantization for large language models, 2023\.
  * [22] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky.  Relora: High-rank training through low-rank updates, 2023.
  * [23] Baohao Liao and Christof Monz.  Apiq: Finetuning of 2-bit quantized large language model, 2024.
  * [24] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.  Llm-qat: Data-free quantization aware training for large language models.  arXiv preprint arXiv:2305.17888, 2023.
  * [25] Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu.  Full parameter fine-tuning for large language models with limited resources, 2023.
  * [26] Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei.  The era of 1-bit llms: All large language models are in 1.58 bits, 2024\.
  * [27] Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee.  Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, 2024.
  * [28] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng.  Fp8-lm: Training fp8 large language models, 2023.
  * [29] Sergio P. Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon.  Training and inference of large language models using 8-bit floating point, 2023.
  * [30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  arXiv e-prints, 2019.
  * [31] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo.  Omniquant: Omnidirectionally calibrated quantization for large language models, 2024.
  * [32] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
  * [33] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer.  Q-bert: Hessian based ultra low precision quantization of bert, 2019.
  * [34] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [35] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa.  Quip : Even better llm quantization with hadamard incoherence and lattice codebooks, 2024.
  * [36] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * [37] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
  * [38] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei.  Bitnet: Scaling 1-bit transformers for large language models, 2023.
  * [39] Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan.  Training deep neural networks with 8-bit floating point numbers, 2018\.
  * [40] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt.  Stable and low-precision training for large-scale vision-language models, 2023.
  * [41] Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu.  Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization, 2024.
  * [42] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models, 2024.
  * [43] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat.  Q8bert: Quantized 8bit bert.  In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS). IEEE, December 2019.
  * [44] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian.  Galore: Memory-efficient llm training by gradient low-rank projection, 2024.

##  Appendix A Quantization Methods

Quantization methods can be broadly categorized into Quantization-Aware
Training (QAT), Post-Training Quantization (PTQ), and Fully Quantized Training
(FQT).

#### Quantization-Aware Training (QAT)

QAT [24, 17, 8, 38, 26] integrates quantization in the training process by
emulating inference time quantization where the model weights are quantized.
By maintaining high precision gradients and optimizer states, QAT allows the
model to adapt to quantized weights while preserving accuracy. These methods
predominantly focus on weight-only quantization approaches, which involve
converting weight matrices into low-precision formats and then upcasting them
just before computation [38, 26]. This allows the main computation to occur at
high precision, effectively preserving model accuracy while significantly
compressing the model [9]. However, QAT can require significant computational
resources due to the need for full precision gradient calculations and large
optimization states [6].

#### Post-Training Quantization (PTQ)

PQT [9, 7, 35, 42, 27, 4, 20, 14, 31] involves converting a pre-trained high-
precision model into a lower precision format. This can be done directly or by
using a subset of the training data to calibrate the quantization process or
fine-tune the quantized weights to adapt the model to the quantization.
However, PTQ often results in reduced accuracy compared to QAT because the
model does not learn to adjust to quantized weights during training [9, 42].

#### Fully Quantized Training (FQT)

FQT aims to minimize memory and accelerate the training process by quantizing
both forward and backward passes [39, 3, 2, 29, 40]. These methods often
require specialized hardware [28, 41] but are promising for efficient
training, and current approaches cannot maintain accuracy [41].

LoQT is a form of QAT that gets close to FQT. As we perform a variant of LoRA
(see §2.2), we factor the layers $W$ into two matrices $P$ and $B$. We
quantize the $W$ and $P$ with NF4, but keep $B$ in 16-bit precision. We
periodically update the $W$ matrices using the product of the fixed $P$ and
the updated $B$s without ever dequantizing it all at once, only layerwise when
merging in $PB$. This approach retains the benefits of reduced memory usage
while minimizing accuracy loss, focusing high-precision updates on a low-rank
representation, and allowing efficient model updates without the overhead of
full matrix re-quantization.

##  Appendix B Hyperparamters

###  B.1 Pre-training

For the pre-training results shown in Table 1, we adopted configurations from
GaLore [44] and tested pre-training methods on different LLaMA 2 model sizes
using the C4 dataset. Training was conducted with optimizer states in BF16
precision, and NF4 precision quantization was used for LoQT. The model rank
was adapted based on the largest layer with specific parameters.

Table 1 shows the ratio $r/d_{model}$, which denotes the rank relative to the
largest weight matrix dimension. All experiments used a maximum sequence
length of 256, learning rate warmup for the first 10% of training steps, and
cosine annealing for the learning rate schedule, decaying to 10% of the
initial rate.

Galore, LoQT-nq, and LoQT used exponentially increasing update frequencies
starting at 100 and increasing by $100+\psi^{i}$, where $\psi$ is 1.2 and $i$
is the update counter (see Section C.1 for more details).

We tested learning rates of $0.01$, $0.005$, $0.001$, and $0.0005$ across
different model sizes. For models ranging from 60M to 350M parameters, a
learning rate of $0.01$ yielded the best performance. In contrast, full-rank
models required smaller learning rates: $0.001$ for 60M to 350M models and
$0.0005$ for the 1B model. To scale the learning rates for LoQT, LoQT-nq, and
Galore, we employed a scale parameter $s$ set to $0.5$ and $0.25$ for the 1B
model. This parameter functions similarly to the LoRA alpha parameter,
determining the weight on the learned factors for LoQT and LoQT-nq. For
Galore, our experiments indicated that $s=0.5$ was more effective than the
$0.25$ reported in [44]. This scaling approach effectively adjusts the
learning rate, resulting in an actual rate of $0.005$ for the multi-head
attention and feed-forward layers in LLaMA models, which is relatively large
compared to the $0.001$ used for full-rank models. Higher learning rates led
to spikes in the training loss for both full-rank and LoQT models.

Table 4: Pre-training hyperparameters of LLaMA models for evaluation. (-)
Indicates we did not train such a model.
Model Size | Hidden/Intermediate | Attention Heads | Layers | Steps | Data Amount | Rank
---|---|---|---|---|---|---
60M | 512 / 1376 | 8 | 8 | 10K | $1.3\mathrm{B}$ | 128
130M | 768 / 2048 | 12 | 12 | 20K | $2.6\mathrm{B}$ | 256
350M | 1024 / 2736 | 16 | 24 | 60K | $7.8\mathrm{B}$ | 256
1B | 2048 / 5461 | 24 | 32 | 100K | $13.1\mathrm{B}$ | 512
7B | 4096/11008 | 32 | 32 | - | - | 1024
13B | 5120/13824 | 40 | 40 | - | - | 1536

###  B.2 Fine-tuning

We test learning rates in the range of $1\times 10^{-5}$ to $5\times 10^{-4}$.
For LoQT qLoFT, we employed normal float NF4 quantization and performed 5
iterations of optimizing the error of quantization. We used a batch size of 32
and a maximum sequence length of 256. Table 5 summarizes the detailed
hyperparameters for tasks in GLUE using the DeBERTaV3-base model. We use a
fixed projection gap of 2400 for all runs.

Table 5: Hyperparameter setup for LoQT-nq, LoQT, LoftQ[21], LoRA[21], and
Galore across various tasks on the GLUE benchmark.

Method | Hyper-parameter | MNLI | RTE | QNLI | MRPC | QQP | SST-2 | CoLA | STS-B
---|---|---|---|---|---|---|---|---|---
LoQT, LoFTQ | # Epochs | 5 | 20 | 10 | 60 | 10 | 10 | 20 | 60
Learning Rate | $1\times 10^{-4}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $1\times 10^{-4}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $1\times 10^{-4}$ | $5\times 10^{-5}$
LoRA, Galore | # Epochs | 10 | 30 | 30 | 30 | 30 | 30 | 30 | 30
Learning Rate | $1\times 10^{-5}$ | $2\times 10^{-5}$ | $1\times 10^{-5}$ | $2\times 10^{-5}$ | $1\times 10^{-5}$ | $2\times 10^{-5}$ | $2\times 10^{-5}$ | $3\times 10^{-5}$

##  Appendix C Rank Ablation

Figure 5 shows the validation perplexity versus training steps for various
ranks using LoQT-nq and LoQT on a $130$ million parameter model over 20,000
iterations. All models employ an exponentially increasing update frequency
starting at $100$, with a factor of $1.2^{T_{i}}$. The results demonstrate
that both the quantized (LoQT) and non-quantized (LoQT-nq) models follow a
similar trajectory for ranks ranging from $64$ to $512$. However, for the
smaller rank of $64$, there is a slight divergence between LoQT-nq and LoQT,
indicating a limit to how low the rank can be while maintaining accuracy with
quantization. This plot highlights the tradeoff between rank and perplexity,
suggesting that while our method supports low-rank training, there is a
minimum rank threshold needed to achieve results comparable to regular pre-
training.

Figure 5: Rank ablation for LoQT and LoQT-nq showing perplexity as a function
of steps.

###  C.1 Memory Measurements

Figure 6 demonstrates that LoQT requires less memory than GaLore and Adam,
even without using per-layer gradients [25] or Adam 8-bit [5]. The gap between
LoQT and the baselines increases with larger model sizes. The configurations
and ranks for each model are shown in Table 4. With LoQT and Adam 8-bit, it is
possible to pre-train a 13B model with rank 1024 on a GPU with 24GB of VRAM.
This enables training with LoQT on consumer GPUs, such as the NVIDIA 4090,
using a small per-GPU token batch size 256. Figure 1 in the main text provides
a detailed breakdown of each memory component for the 13B model. Maximum
memory allocated is measured using nvitop
(https://github.com/XuehaiPan/nvitop).

Figure 6: Memory usage for LoQT vs baselines for different model sizes. LW
means per-layer gradient updates as per [25], and A8bit mean with Adam 8-bit.
We evaluate using a token batch size of 256.