# SCRAMBLE-CFI: Mitigating Fault-Induced Control-Flow
Attacks on OpenTitan

Pascal Nasahl  Graz University of TechnologyGrazAustria
<EMAIL_ADDRESS>and  Stefan Mangard  Graz University of
TechnologyGrazAustria<EMAIL_ADDRESS>
(2023)

###### Abstract.

Secure elements physically exposed to adversaries are frequently targeted by
fault attacks. These attacks can be utilized to hijack the control-flow of
software allowing the attacker to bypass security measures, extract sensitive
data, or gain full code execution.

In this paper, we systematically analyze the threat vector of fault-induced
control-flow manipulations on the open-source OpenTitan secure element. Our
thorough analysis reveals that current countermeasures of this chip either
induce large area overheads or still cannot prevent the attacker from
exploiting the identified threats. In this context, we introduce SCRAMBLE-CFI,
an encryption-based control-flow integrity scheme utilizing existing hardware
features of OpenTitan. SCRAMBLE-CFI confines, with minimal hardware overhead,
the impact of fault-induced control-flow attacks by encrypting each function
with a different encryption tweak at load-time. At runtime, code only can be
successfully decrypted when the correct decryption tweak is active. We open-
source our hardware changes and release our LLVM toolchain automatically
protecting programs. Our analysis shows that SCRAMBLE-CFI complementarily
enhances security guarantees of OpenTitan with a negligible hardware overhead
of less than 3.97 % and a runtime overhead of 7.02 % for the Embench-IoT
benchmarks.

secure element, fault attacks, cryptographic control-flow integrity

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference:  Proceedings of the Great Lakes Symposium on
VLSI; June 05–07, 2023; Knoxville, TN, USA††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Security and privacy Hardware attacks and
countermeasures

##  1\. Introduction

In a fault attack, the adversary injects one or multiple bit errors into a
chip by using non-invasive methods, such as voltage or clock glitching, or
semi-invasive techniques that include shooting with a laser into the silicon
of a decapsulated chip (Bar-El et al., 2006). The effects of these bit errors
can be exploited and enable the attacker to manipulate the control-flow of
software executed on the system (Karaklajic et al., 2013; Vasselle et al.,
2020; Timmers et al., 2016; Nasahl and Timmers, 2019).

Secure elements, such as OpenTitan (Johnson et al., 2018), are used in
smartphones and computers as a secure root of trust. As these chips run
security-critical programs, such as a key storage and authentication services,
they are lucrative fault targets. In addition, these devices are easily
accessible for fault attackers as they are typically deployed in the wild.
Hence, secure elements need to provide dedicated hardware- and software-based
countermeasures protecting the execution of software against faults.

### Contribution

In this paper, we first identify threat vectors enabling the adversary to
hijack the control-flow of software by inducing faults into OpenTitan.
Subsequently, we thoroughly analyze existing countermeasures aiming to
mitigate these attacks.

Our analysis shows that existing countermeasures either induce large area
overheads or inadequately reduce the attack surface to address control-flow
manipulations. SCRAMBLE-CFI significantly confines the effect of these attacks
with a minimal hardware overhead by introducing a cryptographically enforced
control-flow integrity scheme. In SCRAMBLE-CFI, each function is encrypted
with a different encryption tweak before the execution of the program. During
runtime, the instrumented program automatically updates the decryption tweak
in a CPU register. Only when the current execution context and the decryption
tweak in the register match, the code can be successfully decrypted and
executed. On control-flow redirections to functions outside of the call graph,
the tweak mismatches and garbled instructions are decoded, triggering an
alert. As SCRAMBLE-CFI utilizes the already existing scrambling unit of
OpenTitan for the encryption, our countermeasure only requires minimal
hardware changes, yielding an area overhead of less than 3.97 %. Furthermore,
our performance analysis shows a small runtime overhead of 7.02 % for the
Embench-IoT benchmarks.

In summary, our contributions are:

  * •

We provide a systematic analysis of threat vectors on OpenTitan allowing an
adversary to perform fault-induced control-flow manipulations.

  * •

We discuss existing hardware- and software-based fault countermeasures
integrated into OpenTitan.

  * •

We introduce and open-
source111https://extgit.iaik.tugraz.at/sesys/otcfiSCRAMBLE-CFI, an encryption-
based control-flow integrity scheme utilizing hardware features of OpenTitan.

  * •

Finally, we discuss how SCRAMBLE-CFI, complementarily to existing
countermeasures, enhances the resilience of OpenTitan against faults with a
small runtime and area overhead.

##  2\. Background

This section provides background on the OpenTitan chip and control-flow
integrity.

###  2.1. OpenTitan

OpenTitan is a secure element developed by Google and lowRISC and the entire
design, including silicon as well as the firmware, is open-source. The goal of
the project is to develop a chip that acts as a secure root of trust in
different computing systems. OpenTitan contains a rich set of hardware- and
software IP, including a key storage and an AES accelerator.

Figure 1. High-level overview of the OpenTitan chip.

Figure 1 highlights the main architectural building blocks of the OpenTitan
secure element. The Ibex 32-bit RISC-V processor is connected over the
TileLink bus to the program memory (flash), the data memory (SRAM), and
several other IP blocks. To provide data confidentiality, external data stored
in flash or SRAM is encrypted by the OpenTitan scrambling unit. This engine
encrypts all data using a round-reduced version of the PRINCE (Borghoff et
al., 2012) cipher.

###  2.2. Control-Flow Integrity

Control-flow integrity schemes aim to detect fault-induced control-flow
deviations from the intended control-flow. Here, these schemes (Oh et al.,
2002; Reis et al., 2005; Schilling et al., 2022) assign certain points in the
program a unique control-flow signature during compile-time. At runtime,
instrumented programs automatically derive the control-flow signature and
compare the derived to the predefined signature. On a mismatch, a control-flow
manipulation is detected. However, as the signature comparison induces
performance overhead, the signature checks are only conducted at a coarse
granularity, e.g., at the function or program end. Hence, an adversary might
still be able to execute security-sensitive code before the control-flow
manipulation is detected by CFI.

##  3\. Threat Model

Our threat model comprises an attacker with physical access to the OpenTitan
chip performing fault attacks. This attacker is capable of injecting single or
multiple faults into the secure element by using clock, voltage, or EM
glitching techniques or by performing laser fault injection (Bar-El et al.,
2006). We assume that these faults cause single or multiple bit-flips
(Verbauwhede et al., 2011) in the system. The goal of the attacker is to
redirect the control-flow of software executed on the chip.

##  4\. Analysis

In this section, we discuss potential attack vectors within the presumed
threat model (cf. Section 3) and systematically analyze existing hardware- and
software-based countermeasures of OpenTitan aiming to address the identified
threats.

###  4.1. Attack Vectors

The control-flow of software can be redirected at different control-flow
manipulation (CFM) granularities:
CFM1: The attacker redirects the control-flow to a function that cannot be
reached from the current execution context, i.e., escaping the call graph of
the program.
CFM2: The control-flow is redirected from one branch target to the other when
manipulating conditional branches.
CFM3: The attacker arbitrarily hijacks the control-flow within a function,
i.e., to any basic-block.

The attack surface (A) comprises all elements of OpenTitan (cf. Figure 1).
More specifically, a fault can be injected into CPU internal registers (AR),
the instruction cache (AIC), the bus infrastructure (AB), or the memory (AM).
Furthermore, the attacker can also target the core (AC), i.e., the instruction
fetch, decode, and execute pipeline stages. For example, when targeting AC,
the attacker can change the behavior of executed instructions when inducing
bit flips into the instruction decoder or influence the comparison for a
conditional branch in the ALU.

To manipulate the control-flow, we define three potential data targets (DT):
DT1: Control-flow related data comprises relative (DT1.1) and absolute (DT1.2)
addresses as well as the program counter (DT1.3). To flip bits in relative
addresses (DT1.1) used by unconditional and conditional branches, the attacker
can inject faults into the immediate field of instructions stored in the
program memory (AM) or the instruction cache (AIC) or transferred by the
instruction bus (AB) (Karaklajic et al., 2013). Indirect calls can be
manipulated by targeting absolute addresses (DT1.2a) stored in registers (AR)
or the data memory (AM) or transferred by the data bus (AR). Moreover, returns
can be redirected by flipping bits in return addresses (DT1.2b). Finally, the
attacker also can directly inject faults into the program counter (DT1.3)(AR).
Targeting DT1 enables the adversary to arbitrarily manipulate the control-
flow, i.e., CFM1, CFM2, and CFM3.
DT2: Non-control-flow related data, i.e., general purpose data, can be faulted
by targeting the SRAM (AM) or the data bus (AB). When the faulted data is used
by conditional branches (Vasselle et al., 2020), the attacker can influence
their execution (CFM2).
DT3: Instructions, stored in the flash (AM) or iCache (AIC), transferred by
the instruction bus (AB), and processed by the core (AC), can be manipulated
by inducing bit flips into the opcode or the operands (Timmers et al., 2016;
Nasahl and Timmers, 2019). Here, one possible attack would be to skip an
instruction by flipping the opcode from a jump (jalr) to a nop. Similarly, by
manipulating the operand, e.g., flipping jalr 0(x5) to jalr 0(x6), the
control-flow can also be arbitrarily redirected.

###  4.2. OpenTitan Countermeasures

##### Hardware-based Countermeasures

OpenTitan protects security-critical data throughout most of its life cycle
using an error correction code (ECC). More specifically, the integrity of data
is protected in the data memory (AM), the data bus (AB), as well as in the
register file (AR) of Ibex. An integrity error allows OpenTitan to detect bit-
flips in DT2 and DT1.2a. Additionally, ECC is used in the instruction cache
(AIC), the program memory (AM), as well as the instruction bus (AB) to detect
fault-induced bit-flips, protecting DT3 and DT1.1. To prevent manipulations of
the program counter (DT1.3), Ibex recalculates the derived program counter and
triggers an error on a mismatch.

Finally, OpenTitan also provides the possibility of instantiating the Ibex
core twice in a lockstep mode. Here, the execution of the second core is
delayed by some cycles and the outputs of the core, i.e., the data and
instruction interface outputs, are compared. Although this strategy provides
strong protection against faults induced into the pipeline (AC), it also more
than doubles the area of the CPU.

##### Software-based Countermeasures

In addition to the hardware-based countermeasures, the OpenTitan project also
provides software-based fault protection mechanisms that are integrated into a
modified LLVM toolchain. Programs compiled with this toolchain automatically
store the return address into a shadow stack. In the function epilogue, before
the return, the current return address is compared to the return address
stored in the shadow stack, mitigating bit-flips in return addresses (DT1.2b).
Furthermore, after each indirect branch and return, the compiler inserts an
illegal instruction that triggers an exception. This strategy hinders the
adversary from redirecting the control-flow by skipping these instructions
(DT1.3, DT3, and AC).

##  5\. SCRAMBLE-CFI

As shown in the previous section, current OpenTitan countermeasures either
induce large area overheads, i.e., the dual core lockstep approach, or only
provide limited protection against fault-induced control-flow manipulations
when targeting the core (AC). To that end, in this section, we introduce
SCRAMBLE-CFI, our control-flow integrity scheme that enhances the resilience
of OpenTitan against these attacks with a minimal area and runtime overhead.
Afterwards, in Section 6, we then highlight security guarantees and compare
SCRAMBLE-CFI to existing countermeasures.

###  5.1. Overview

Figure 2. Encrypted call graph.

In SCRAMBLE-CFI, each function is assigned an encryption tweak during program
compilation. Before execution, when loading the program into flash, the code
blocks are encrypted with the corresponding tweak. At runtime, before each
function call, the decryption tweak for the call target is placed into a CPU
register. As the modified OpenTitan scramble engine incorporates the content
of this register into the decryption, the instructions only can be decrypted
when the active tweak matches the tweak determined at compile-time. Figure 2
shows the SCRAMBLE-CFI’s encrypted call graph, i.e., a graph comprising all
valid transactions from one function to another. When redirecting the control-
flow from the current execution context to another function encrypted with a
different tweak, garbled instructions are fetched and the decoding fails with
a high probability. More specifically, as SCRAMBLE-CFI assigns each function,
for programs without indirect branches, a unique tweak, any cross-function
control-flow manipulation fails. For programs containing indirect branches,
SCRAMBLE-CFI guarantees that the attacker cannot redirect the control-flow
outside of the call graph. Summarized, the processor only can execute code
blocks when the corresponding decryption tweak is active.

###  5.2. Program Instrumentation

In order to decrypt the code of a called function, the corresponding
decryption tweak needs to be loaded into the tweak register before the
control-flow edge, i.e., direct and indirect branches. This program
instrumentation is done fully automatically in SCRAMBLE-CFI by a modified LLVM
(Lattner and Adve, 2004) RISC-V compiler.

Our custom compiler consists of an analysis and instrumentation pass operating
in the backend of the toolchain. This pass first performs a program analysis
to construct the call graph of the program to protect. Here, we scan each
function for direct and indirect calls and determine the call target. While
LLVM already provides this information for direct calls, indirect calls
require a points-to analysis (Nasahl et al., 2021) to reveal the set of
potential called functions.

After the extraction of the call graph, we assign each function a unique
encryption tweak. Depending on the number of functions, this tweak is either a
$5\text{\,}\mathrm{bit}$ or $20\text{\,}\mathrm{bit}$ random number.

Figure 3. Instrumentation of direct and indirect calls. The different colors
highlight code blocks encrypted with different tweaks.

Figure 3 shows the instrumentation of direct and indirect calls conducted by
the SCRAMBLE-CFI compiler. For the direct call from function A to B, we set
the tweak to the tweak of function B, i.e., $T_{B}$. When returning from this
function, the tweak is set back to the tweak used by function A, i.e.,
$T_{A}$. Similarly, for indirect branches, e.g., from C to B, we also set and
reset the tweak before and after the call. However, instead of using the same
tweak as for the direct call, i.e., $T_{B}$, we add an additional entry point
to the function, which is encrypted with a different tweak, i.e., $T_{BE}$. In
this entry point, we again update the tweak to the tweak for the function to
$T_{B}$. Moreover, we add a second exit point which is taken when the function
was called by an indirect branch. In this exit point, we set back the tweak to
the tweak of the entry point, i.e., $T_{BE}$. The compiler rewrites all
addresses used by direct branches to point to the instruction after the added
entry point. Furthermore, in the entry point, we set a flag indicating whether
the function returns with the default or the added exit point.

Adding entry points to the program is necessary because indirect calls can
have multiple call targets that need to be encrypted with the same tweak.
Without this additional entry point, also direct calls would need to be
encrypted with this tweak, allowing the adversary to redirect a direct branch
to another function which is called by an indirect branch and is outside of
the call graph.

Listing 1: Tweak update instruction sequence.

⬇

#5 bit tweak:

csrrwi x0, csr_tweak,
tweak${}_{{\color[rgb]{0,0.40625,0.7109375}5\,\textrm{bit}}}$

#20 bit tweak:

lui x28, tweak${}_{{\color[rgb]{0,0.40625,0.7109375}20\,\textrm{bit}}}$

csrrw x0, csr_tweak, x28

Listing LABEL:lst:otcfi:callinstrumentation shows the instruction sequence
used to set the $5\text{\,}\mathrm{bit}$ or $20\text{\,}\mathrm{bit}$ tweak
into the tweak control and status register (CSR). For programs with 32 or
fewer functions, a single csrrwi instruction loading the
$5\text{\,}\mathrm{bit}$ tweak from the immediate field into the CSR csr_tweak
is sufficient.

####  5.2.1. Alignment to Encryption Granular

As indicated in Figure 3, the next instruction after updating the tweak is
already decrypted with this tweak. Since the decryption granularity of the
underlying cipher, i.e., $64\text{\,}\mathrm{bit}$ for PRINCE, does not match
the natural RISC-V instruction alignment of 16 or $32\text{\,}\mathrm{bit}$,
we need to align the set tweak instruction sequence to the decryption
granularity. We conduct this alignment by padding these instructions with nops
to $64\text{\,}\mathrm{bit}$.

####  5.2.2. Metadata Section

The tweak for each code block is stored in a custom ELF section generated by
our toolchain in the AsmPrinter stage. Here, the compiler emits the start
address of each function and the corresponding number of basic-blocks.
Moreover, the offset of each basic-block and the corresponding decryption
tweak is emitted into the metadata section by the compiler. This section is
then processed during program deployment.

###  5.3. Program Deployment

Figure 4. Deployment of protected programs. Code blocks in flash memory
encrypted with different SCRAMBLE-CFI tweaks are highlighted with different
colors.

Figure 4 shows the deployment of a program protected with SCRAMBLE-CFI on
OpenTitan. Our toolchain first compiles the C source code of the program to an
instrumented ELF binary. Then, the flash utility program converts this binary
into a VMEM file, which is loaded into flash memory.

As at the time of writing this paper, the OpenTitan project only provides
hardware support for flash scrambling but not the necessary software support
and, therefore, disables the scrambling, we extended the flash utility program
to encrypt code with the PRINCE cipher. Here, we encrypt each
$64\text{\,}\mathrm{bit}$ word with the PRINCE cipher using the flash scramble
key and the SCRAMBLE-CFI tweak. Note that the flash scramble key is stored
inside the one-time programmable (OTP) memory. To retrieve the SCRAMBLE-CFI
tweak for each code block, the flash utility tool parses the custom metadata
section emitted by our toolchain. Afterwards, the flash is initialized with
the encrypted VMEM file and OpenTitan starts the execution of the encrypted
code.

###  5.4. Hardware Changes

To realize SCRAMBLE-CFI on OpenTitan, minimal-intrusive hardware changes are
required: First (i), additional control and status registers (CSRs) need to be
added to the Ibex processor. We implement these registers by using the shadow
register primitives provided by the OpenTitan project. As these shadow
registers duplicate the registers and compare the stored values, the content
of these CSRs are protected from faults. By writing to the CSRs, software,
i.e., binaries compiled with our custom toolchain, can set the current active
decryption tweak as well as the lower and upper address bound. The address
range registers comprise the lowest and highest address of program code, which
needs to be protected by SCRAMBLE-CFI. Enabling the tweak only for a certain
address range is required to access data, such as globals, also stored in the
flash memory. Second (ii), the tweak needs to be incorporated into the
encryption primitive. One possibility would be to extend the PRINCE cipher to
a tweakable block cipher with the TWEAKEY (Jean et al., 2014) framework.
However, the required TWEAKEY key schedule logic would increase the complexity
and area consumption of the cipher. Moreover, as SCRAMBLE-CFI does not need
cryptographic strength for control-flow integrity, we inject the tweak into
the key using a XOR operation. This exclusive or is conducted when the address
sent to the flash controller is between the lower and upper address stored in
the added CSRs. Otherwise, the tweak is set to $0$ and the PRINCE cipher uses
the default key provided by the OTP controller. Third (iii), when the
decryption tweak is changed by writing to the CSR, we flush the instruction
cache to avoid that scrambled cached instructions are executed.

##  6\. Security Analysis & Comparison

When the tweak register contains the tweak of the current function, the
execution of code fails when the control-flow is redirected to any other
function encrypted with a different encryption tweak. Before a function call,
SCRAMBLE-CFI updates the tweak register with the tweak of the called function.
Then, the control-flow can only be redirected to this function or functions
encrypted with the same tweak. In SCRAMBLE-CFI, the entry points (cf. Section
5.2) of functions that an indirect branch can call share the same encryption
tweak. This is inevitable as, at compile-time, the toolchain cannot determine
which function gets called by an indirect call at runtime. However, indirect
calls are rare in typical programs and the attacker only can redirect the
control-flow to functions that can be reached with this indirect call, i.e.,
are within the call graph. For programs without indirect calls, SCRAMBLE-CFI
assigns all functions a unique encryption tweak. Summarized, for any control-
flow redirection outside of the call graph (CFM1) the wrong decryption tweak
is deterministically used for programs that contain less than $2^{20}$
functions. For programs that contain more functions than the available tweak
space, tweak collisions can occur.

When fetching instructions from program memory with a wrong decryption tweak,
garbled instructions are retrieved. With a high probability, the decoding of
these instructions in the instruction decoder pipeline stage fail and an
exception is triggered. However, it could be possible that a garbled
instruction again forms a valid instruction. Nevertheless, (i) the probability
that the subsequent instructions are also valid is low and (ii) usually, the
attacker aims to execute a certain instruction in the function and not any
that does not trigger an exception.

Note that SCRAMBLE-CFI cannot prevent an adversary from manipulating
conditional branches (CFM2) or from redirecting the control-flow within a
function from one basic-block to another (CFM3). However, SCRAMBLE-CFI could
be extended to provide fine-granular protection for highly security-critical
code blocks by encrypting code blocks within a function with different
encryption keys.

In the current prototype of SCRAMBLE-CFI, we incorporate the tweak into the
encryption by XORing it to the encryption key. This XOR creates a dependency
of key and tweak, allowing an attacker to potentially learn about the key when
the tweak is known. However, the binary (including the tweaks) is inaccessible
to an adversary as it is stored in the encrypted flash.

###  6.1. Security Comparison

Table 1 highlights protection guarantees of different hardware- and software-
based OpenTitan countermeasures and SCRAMBLE-CFI against faults into different
attack targets.

Table 1. Protection guarantees of different countermeasure when targeting
different attack surfaces.

| Attack Targets
---|---
Countermeasure | AR | AIC | AB | AM | AC
Data memory ECC | - | - | - | ✔ | -
Program memory ECC | - | - | - | ✔ | -
Data bus ECC | - | - | ✔ | - | -
Instruction bus ECC | - | - | ✔ | - | -
Register file ECC | ✔ | - | - | - | -
iCache ECC | - | ✔ | - | - | -
PC protection | ✔ | - | - | - | ✔
SW-based defense |  | - | - | - |
Dual core lockstep | ✔ | ✔ | - | - | ✔
SCRAMBLE-CFI |  |  |  |  |

✔ Full  Partial - No Protection

The ECC-based countermeasures provide full protection when inducing faults
into their protection domain. For example, bit-flips induced into instructions
(DT3) stored in the instruction cache (AIC) or the program memory (AM) can be
detected reliably. As the program counter (PC) protection recalculates and
compares the PC to the current PC, also faults into the core (AC) can be
detected. The software-based defense integrated into the custom LLVM toolchain
only provides partial protection, i.e., only control-flow manipulations aiming
to manipulate return addresses (DT1.2b) or skipping certain instructions (DT3)
can be detected. As shown in Table 1, from the existing countermeasures, only
the lockstep approach can provide strong protection against faults induced
into the core (AC). However, this strategy also induces a high area overhead
as the Ibex core needs to be instantiated twice and an error detection logic
needs to be added. In comparison, SCRAMBLE-CFI provides strong protection
against control-flow manipulations for all attack targets, even when targeting
the core, with minimal hardware overhead. Especially when combined with the
other existing countermeasures, SCRAMBLE-CFI significantly minimizes the
attack surface and allows OpenTitan to withstand most of the control-flow
attacks. Combined, OpenTitan is protected against arbitrary control-flow
manipulations (CFM1, CFM2, and CFM3) when targeting all attack targets except
the core (AC). When injecting faults into AC, SCRAMBLE-CFI fills the
protection gap of hindering an adversary from redirecting the control-flow
outside of the call graph (CFM1). However, when also CFM2 and CFM3 need to be
mitigated, SCRAMBLE-CFI needs to be deployed on a finer granularity (cf.
Section 6) or the lockstep approach needs to be installed. Summarized, we
argue that SCRAMBLE-CFI provides a good area-security tradeoff allowing
OpenTitan to withstand a multitude of different fault attacks.

##  7\. Performance Overhead

To evaluate the performance and code size overhead, we compiled the Embench-
IoT (Patterson et al., 2023) benchmarks with our custom toolchain. We excluded
benchmarks requiring libraries, such as math and string, currently not
provided by the OpenTitan framework.

To measure the code size overhead, we compared the protected binaries to the
unprotected baseline with the GNU size utility. Our analysis shows a code size
overhead between 0.74 % and 8.88 % and a geometric mean of 1.69 %. This
overhead comprises the (i) inserted tweak switch instructions, the (ii)
alignment of these instructions to the encryption granular, and the (iii)
metadata binary section. As this section only is needed by the flash utility
program to encrypt code blocks with different encryption tweaks (cf. Section
5.3), this metadata is not stored in the flash memory.

Figure 5. Runtime overhead for the Embench-IoT benchmarks.

In order to analyze the performance impact of binaries instrumented with
SCRAMBLE-CFI, we measured the CPU cycles by reading the mcycle CSR of Ibex.
Here, we executed the protected and unprotected binaries on a cycle-accurate
Verilator model of OpenTitan. As shown in Figure 5, we measured a runtime
overhead between 0.22 % and 143.35 % and a geometric mean of 7.02 %.
Benchmarks, such as crc32 or tarfind, frequently calling small functions
induce larger runtime overheads than benchmarks only performing a small number
of function calls.

##  8\. Area Overhead

The public OpenTitan hardware design flow currently only allows to synthesize
the Ibex core with open-source synthesis tools. Therefore, we synthesized the
Ibex processor with the Yosys open synthesis suite and the Nangate
$45\text{\,}\mathrm{n}\mathrm{m}$ standard cell library to analyze the area
overhead introduced by our hardware changes. Here, we measured an area
increase from 26.48 kGE to 27.53 kGE (3.97 %). These hardware changes comprise
the additional CSRs as well as the address range comparison. Note that the XOR
of the tweak with the key is conducted in the flash controller and is
currently not reflected in the hardware overhead number. According to the
synthesis logs created with the proprietary design flow published by the
OpenTitan project (OpenTitan, 2023), the Ibex occupies 3.4 % of the overall
chip area. Hence, the hardware overhead induced by SCRAMBLE-CFI to the overall
area is negligible.

##  9\. Related Work

Currently, encryption-based control-flow integrity schemes, such as SOFIA (de
Clercq et al., 2017) or SCFP (Werner et al., 2018), require intrusive hardware
changes in the processor’s pipeline to realize their protection. Hence, when
integrating these changes into a chip, an extensive re-verification of the
entire processor is needed. Although EC-CFI (Nasahl et al., 2023) provides
cryptographic CFI on Intel hardware without hardware modifications, the
measured runtime overheads are high. In comparison, SCRAMBLE-CFI induces small
runtime overheads with minimal hardware changes that allow designers to re-
verify their design with minimal effort.

##  10\. Conclusion

In this paper, we thoroughly analyzed fault threat vectors allowing an
adversary to manipulate the control-flow of software executed on OpenTitan. We
provided an overview of current OpenTitan countermeasures and discussed their
protection capabilities. Furthermore, we introduced SCRAMBLE-CFI, which
mitigates fault attacks aiming to redirect the control-flow outside of the
call graph. We showcased that SCRAMBLE-CFI is a strong security addition to
existing countermeasures inducing minimal runtime and area overheads.

##  11\. Acknowledgments

This project has received funding from the Austrian Research Promotion Agency
(FFG) via the AWARE project (grant number 891092).

## References

  * (1)
  * Bar-El et al. (2006) Hagai Bar-El, Hamid Choukri, David Naccache, Michael Tunstall, and Claire Whelan. 2006\.  The Sorcerer’s Apprentice Guide to Fault Attacks.  _Proc. IEEE_ 94 (2006), 370–382.
  * Borghoff et al. (2012) Julia Borghoff, Anne Canteaut, Tim Güneysu, Elif Bilge Kavun, Miroslav Knezevic, Lars R. Knudsen, Gregor Leander, Ventzislav Nikov, Christof Paar, Christian Rechberger, Peter Rombouts, Søren S. Thomsen, and Tolga Yalçin. 2012\.  PRINCE - A Low-Latency Block Cipher for Pervasive Computing Applications - Extended Abstract. In _ASIACRYPT_ _(LNCS, Vol. 7658)_. 208–225.
  * de Clercq et al. (2017) Ruan de Clercq, Johannes Götzfried, David Übler, Pieter Maene, and Ingrid Verbauwhede. 2017\.  SOFIA: Software and control flow integrity architecture.  _Comput. Secur._ 68 (2017), 16–35.
  * Jean et al. (2014) Jérémy Jean, Ivica Nikolic, and Thomas Peyrin. 2014\.  Tweaks and Keys for Block Ciphers: The TWEAKEY Framework. In _ASIACRYPT_ _(LNCS, Vol. 8874)_. 274–288.
  * Johnson et al. (2018) Scott Johnson, Dominic Rizzo, Parthasarathy Ranganathan, Jon McCune, and Richard Ho. 2018\.  Titan: enabling a transparent silicon root of trust for cloud. In _Hot Chips: A Symposium on High Performance Chips_ , Vol. 194.
  * Karaklajic et al. (2013) Dusko Karaklajic, Jörn-Marc Schmidt, and Ingrid Verbauwhede. 2013.  Hardware Designer’s Guide to Fault Attacks.  _IEEE Trans. Very Large Scale Integr. Syst._ 21 (2013), 2295–2306.
  * Lattner and Adve (2004) Chris Lattner and Vikram S. Adve. 2004.  LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In _CGO_. 75–88.
  * Nasahl et al. (2021) Pascal Nasahl, Robert Schilling, and Stefan Mangard. 2021\.  Protecting Indirect Branches Against Fault Attacks Using ARM Pointer Authentication. In _HOST_. 68–79.
  * Nasahl et al. (2023) Pascal Nasahl, Salmin Sultana, Hans Liljestrand, Karanvir Grewal, Michael LeMay, David M. Durham, David Schrammel, and Stefan Mangard. 2023\.  EC-CFI: Control-Flow Integrity via Code Encryption Counteracting Fault Attacks.  _CoRR_ abs/2301.13760 (2023).
  * Nasahl and Timmers (2019) Pascal Nasahl and Niek Timmers. 2019.  Attacking AUTOSAR using software and hardware attacks.  _escar USA_ (2019).
  * Oh et al. (2002) Nahmsuk Oh, Philip P. Shirvani, and Edward J. McCluskey. 2002\.  Control-flow checking by software signatures.  _IEEE Trans. Reliab._ 51 (2002), 111–122.
  * OpenTitan (2023) OpenTitan. 2023\.  CHIP_EARLGREY_ASIC Synthesis Results.  https://reports.opentitan.org/hw/top_earlgrey/syn/2022.07.02_00.42.20/results.html.
  * Patterson et al. (2023) David Patterson, Jeremy Bennett, Palmer Dabbelt, Cesare Garlati, G. S. Madhusudan, and Trevor Mudge. 2023\.  Embench: Open Benchmarks for Embedded Platforms.  https://www.embench.org/.
  * Reis et al. (2005) George A. Reis, Jonathan Chang, Neil Vachharajani, Ram Rangan, and David I. August. 2005.  SWIFT: Software Implemented Fault Tolerance. In _CGO_. 243–254.
  * Schilling et al. (2022) Robert Schilling, Pascal Nasahl, and Stefan Mangard. 2022\.  FIPAC: Thwarting Fault- and Software-Induced Control-Flow Attacks with ARM Pointer Authentication. In _COSADE_ _(LNCS, Vol. 13211)_. 100–124.
  * Timmers et al. (2016) Niek Timmers, Albert Spruyt, and Marc Witteman. 2016\.  Controlling PC on ARM Using Fault Injection. In _FDTC_. 25–35.
  * Vasselle et al. (2020) Aurélien Vasselle, Hugues Thiebeauld, Quentin Maouhoub, Adèle Morisset, and Sébastien Ermeneux. 2020.  Laser-Induced Fault Injection on Smartphone Bypassing the Secure Boot-Extended Version.  _IEEE Trans. Computers_ 69 (2020), 1449–1459.
  * Verbauwhede et al. (2011) Ingrid Verbauwhede, Dusko Karaklajic, and Jörn-Marc Schmidt. 2011\.  The Fault Attack Jungle - A Classification Model to Guide You. In _FDTC_. 3–8.
  * Werner et al. (2018) Mario Werner, Thomas Unterluggauer, David Schaffenrath, and Stefan Mangard. 2018.  Sponge-Based Control-Flow Protection for IoT Devices. In _EURO S &P_. 214–226.


Graph convolutional network (GCN) has been successfully applied to capture global non-consecutive and long-distance semantic information for text classification. However, while GCN-based methods have shown promising results in offline evaluations, they commonly follow a seen-token-seen-document paradigm by constructing a fixed document-token graph and cannot make inferences on new documents. It is a challenge to deploy them in online systems to infer steaming text data. In this work, we present a continual GCN model (ContGCN) to generalize inferences from observed documents to unobserved documents. Concretely, we propose a new all-token-any-document paradigm to dynamically update the document-token graph in every batch during both the training and testing phases of an online system. Moreover, we design an occurrence memory module and a self-supervised contrastive learning objective to update ContGCN in a label-free manner. A 3-month A/B test on Huawei public opinion analysis system shows ContGCN achieves 8.86% performance gain compared with state-of-the-art methods. Offline experiments on five public datasets also show ContGCN can improve inference quality. The source code will be released at <https://github.com/Jyonn/ContGCN>.

§ INTRODUCTION

As one of the fundamental tasks in natural language processing, text classification has been extensively studied for decades and used in various applications [Xu et al., 2019, Abaho et al., 2021]. In recent years, graph convolutional network (GCN) has been successfully applied in text classification [Yao, Mao, and Luo, 2019, Lin et al., 2021] to capture global non-consecutive and long-distance semantic information such as token co-occurrence in a corpus.

A line of GCN-based methods [Li et al., 2019] perform document classification by simply constructing a homogeneous graph with each document as a node and modeling inter-document relations such as citation links, which however does not exploit document-token semantic information.

Another line of GCN-based methods constructs heterogeneous document-token graphs, where each node represents a document or a token, and each edge indicates a correlation factor between two nodes.

However, they commonly follow a seen-token-seen-document (STSD) paradigm to construct a fixed document-token graph with all seen documents (labeled or unlabeled) and tokens and perform transductive inference.

Given a new document with unobserved tokens, the trained model cannot make an inference because neither the document nor the unseen tokens are included in the graph. Hence, while these methods are effective in offline evaluations, they cannot be deployed in online systems to infer streaming text data.

To address this challenge, in this paper, we propose a new all-token-any-document (ATAD) paradigm to dynamically construct a document-token graph, and based on which we present a continual GCN model (ContGCN) for text classification. Specifically, we take the vocabulary of a pretrained language model (PLM) such as BERT [Devlin et al., 2019] as the global token set, so a new document can be tokenized into seen tokens from the vocabulary.

We further form a document set which may contain any present documents (e.g., those in the current batch). The document-token graph then consists of tokens in the global token set and documents in the document set. The edge weights of the graph are dynamically calculated according to an occurrence memory module with historical token correlation information, and document embeddings are generated with pretrained semantic knowledge. In this way, ContGCN is enabled to perform inductive inference on streaming text data.

Furthermore, to address data distribution shift [Luo et al., 2022] which is prevalent in online services, we design a label-free online updating mechanism for ContGCN to save the cost and effort for periodical offline updates of the model with new text data. Specifically, we fine-tune the occurrence memory module according to the distribution shift of streaming text data and update the network parameters with a self-supervised contrastive learning objective.

ContGCN achieves favorable performance in both online and offline evaluations thanks to the proposed ATAD paradigm and label-free online updating mechanism. We have deployed ContGCN in an online text classification system – Huawei public opinion analysis system, which processes thousands of textual comments daily. A 3-month A/B test shows ContGCN achieves 8.86% performance gain compared with state-of-the-art methods. Offline evaluations on five real-world public datasets

also demonstrate the effectiveness of ContGCN.

To summarize, our contributions are listed as follows:

* We propose a novel all-token-any-document paradigm and a continual GCN model to infer unobserved streaming text data, which, to our knowledge, is the first attempt to use GCN for online text classification.

* We design a label-free updating mechanism based on an occurrence memory module and a self-supervised contrastive learning objective, which enables to update ContGCN online with unlabeled documents.

* Extensive online A/B tests on an industrial text classification system and offline evaluations on five real-world datasets demonstrate the effectiveness of ContGCN.

§ PRELIMINARIES

§.§ Graph Convolutional Network (GCN)

GCN [Welling and Kipf, 2016] is a graph encoder that aggregates information from node neighborhoods. It is composed of a stack of graph convolutional layers.
Formally, we use $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ to denote a graph, where $\mathcal{V}(\nNode=|\mathcal{V}|)$ and $\mathcal{E}$ are sets of nodes and edges, respectively.
Note that each node $v \in \mathcal{V}$ is self-connected, i.e., $(v, v) \in \mathcal{E}$.
We use $\nm \mmDim{\nNode}$ to represent initial node representations, where $\nDim$ is the embedding dimension. To aggregate information from neighborhoods, a symmetric adjacency matrix $\am \mmSqr{\nNode}$ is introduced, where $\am_{ij}$ is the correlation score of nodes $v_i$ and $v_j$ and $\am_{ii} = 1$.
The adjacency matrix is normalized as

\begin{gather}
\widetilde{\am} = \dm^{-\frac{1}{2}}\am\dm^{\frac{1}{2}},
\end{gather}

where $\dm$ is a degree matrix and $D_{ii} = \sum_{j} A_{ij}$.
At the $k$-th

convolutional layers, the node embeddings are calculated as:

\begin{gather}
\hm^{(k)} = \sigma \left( \widetilde{\am}\hm^{(k-1)}\mathbf{W}_{k} \right),
\end{gather}

where $k \in \{1, 2, \ldots, \nLayer\}$, $h$ is the total number of convolutional layers, $\sigma$ is the activation function, and $\mathbf{W}_{k} \mmSqr{\nDim}$ is a trainable matrix. Specifically, $\hm^{(0)} = \nm$.

§.§ GCN-Based Text Classification

Text classification aims to classify documents into different categories.
Formally, we use $\mathcal{D}(\nDoc=|\mathcal{D}|)$ to denote a set of given documents, which can be split into a training set $\mathcal{D}_\text{train}$ and a testing set $\mathcal{D}_\text{test}$.
Each document $\doc$ can be represented as a list of tokens, $\doc = (t_1^\ssp, t_2^\ssp, \cdots, t_{|\doc|}^\ssp)$, where $t_i^\ssp \in \mathcal{T}$ is a token in the token vocabulary set $\mathcal{T}(\nVoc = |\mathcal{T}|)$.

Existing GCN-based text classification methods [Yao, Mao, and Luo, 2019, Qiao et al., 2018, Lin et al., 2021] mainly follow a seen-token-seen-document paradigm to construct heterogeneous document-token graphs.
Specifically, they first form a seen vocabulary set $\mathcal{T}_\text{seen}$ of size $\nVoc^\prime$ ($\mathcal{T}_\text{seen} \subset \mathcal{T}$), which contains all the seen tokens in the document set $\mathcal{D}$.
Then, they construct a fixed document-token graph, whose nodes include all the seen tokens in $\mathcal{T}_\text{seen}$ and all the given documents in $\mathcal{D}$.

The adjacency matrix $\am$ of the graph is shown in Figure <ref>, which consists of a token-token symmetric matrix $\am^{(1)} \mmSqr{\nVoc^\prime}$, a document-token matrix $\am^{(2)} \mm{\nDoc}{\nVoc^\prime}$, and a document-document identity matrix $\am^{(3)} \mmSqr{\nDoc}$.
The adjacency matrix $\am$ is fixed. Finally, GCN is applied to encode and classify the documents.

Comparison of the adjacency matrices. Left: seen-token-seen-document (STSD) paradigm (e.g., BertGCN). Right: proposed all-token-any-document (ATAD) paradigm.

Framework of our ContGCN model. Green dotted lines represent operations before each phase of model training or testing. Two key components, i.e., AM Generator and Node Encoder, dynamically construct the adjacency matrix and generate node embeddings, which are then fed into a GCN encoder. Finally, our ContGCN model is trained with a classification loss and an anti-interference contrastive loss.

§ METHOD

The commonly adopted seen-token-seen-document (STSD) paradigm only allows to infer seen documents due to its transductive nature.
To address this issue, we propose a novel all-token-any-document (ATAD) paradigm that leverages the global token vocabulary and dynamically updates the document-token graph to make inferences on unobserved documents.

Based on the ATAD paradigm, we propose a continual GCN model, namely ContGCN, as
illustrated in Figure <ref>, which comprises of an adjacency matrix generator, a node encoder, and GCN encoders.

Specifically, given a batch of input documents, the adjacency matrix generator updates the adjacency matrix based on the occurrence memory module and the current batch of documents, while the node encoder produces content-based node embeddings. The GCN encoder is then employed to capture the global-aware node representations. Finally, two training objectives are applied to train the ContGCN model.

§.§ Proposed All-Token-Any-Document Paradigm

In contrast to the seen-token-seen-document paradigm, which treats seen tokens ($\mathcal{T}_\text{seen}$) and seen documents ($\mathcal{D}$) as fixed graph nodes, our proposed all-token-any-document paradigm involves constructing a document-token graph with all tokens ($\mathcal{T}$) and dynamic documents (i.e., a batch of documents $B = \{\mathbf{s}_1, \mathbf{s}_2, \cdots, \mathbf{s}_\nBat\} \subset \mathcal{D}$ where $\nBat$ is the batch size).

Our ContGCN model is designed based on the all-token-any-document paradigm. We define all tokens as the vocabulary used for PLM tokenizers, allowing unseen words to be tokenized into sub-words that are already present in the vocabulary.

When a new batch of data is fed into the model, the adjacency matrix and node embedding will be dynamically updated by the adjacency matrix generator and node encoder, respectively.

§.§ Adjacency Matrix Generator

As illustrated in Figure <ref>, the adjacency matrix consists of three matrices: a token-token matrix $\am^{(1)} \mmSqr{\nVoc}$, a document-token matrix $\am^{(2)} \mm{\nBat}{\nVoc}$, and a document-document identity matrix $\am^{(3)} \mmSqr{\nBat}$.

The token-token matrix $\am^{(1)}$ is learned from the token occurrence knowledge of the corpus and is phase-fixed. This means that it will be refined when the model enters a new training or testing phase with the emergence of new corpora and token co-occurrence knowledge.
The document-token matrix $\am^{(2)}$ is actively calculated based on the current batch of data and is used to update the adjacency matrix dynamically.
Finally, the inner-document identity matrix $\am^{(3)}$ ensures that each document is not influenced by other samples during model learning or reasoning.

The Occurrence Memory Module (OMM) is a module that incrementally records historical statistics, which includes a document counter $\nSen \in \mathbb{Z}^1$ that keeps track of the number of sentences, a token occurrence counter $\mathbf{c} \in \mathbb{Z}^{\nVoc}$ that records the number of sentences in which a token appears, and a token co-occurrence counter $\mathbf{C} \in \mathbb{Z}^{\nVoc \times \nVoc}$ that records the number of times two tokens appear in the same sentence. The OMM captures global non-consecutive semantic information and offers the following benefits: 1) it enables the dynamic calculation of the adjacency matrix for any batch of documents, and 2) it stores a large amount of previous general and domain-specific knowledge without the need for recalculation during updates. We develop a simple yet effective algorithm for updating the OMM, as described in Algorithm <ref>. As shown in Figure <ref>, the OMM is initialized with the Wikipedia corpus and updated by the training or testing data before model training or testing.
Thus, $\am^{(1)}$ will be phasely updated by PPMI (positive pointwise mutual information), which is defined as:

\begin{gather}
\am^{(1)}_{i, j} =
\begin{cases}
1, & \mbox{if } i = j, \\
\max \left( \log \left( \nSen\frac{C_{i, j}}{c(i,:)c_{j}} \right), 0 \right), & \mbox{else.}
\end{cases}
\end{gather}

To obtain the document-token correlation for each document $\mathbf{s} \in B$, we use TF-IDF (term frequency-inverse document frequency), which is calculated as:

\begin{gather}
\am^{(2)}_{\mathbf{s}, t} = \frac{\mathtt{g}(\mathbf{s}, t)}{|\mathbf{s}|} \log  \frac{\nSen}{c_t + 1},
\end{gather}

where $\mathtt{g}(\mathbf{s}, t)$ represents the number of times token $t$ appears in document $\mathbf{s}$.

The inner-document matrix $\am^{(3)}$ is an identity matrix, denoted as:

\begin{gather}
\am^{(3)}_{i, j} =
\begin{cases}
1, & \mbox{if } i = j, \\
0, & \mbox{else.}
\end{cases}
\end{gather}

Finally, the adjacency matrix $\am$ can be composed by:

\begin{gather}
\am = \begin{pmatrix} \am^{(1)} & \am^{(2)\top} \\ \am^{(2)} & \am^{(3)} \end{pmatrix}.
\end{gather}

A corpus or dataset $\mathcal{D}$, and OMM counters $s$, $\mathbf{c}$, and $\mathbf{C}$

Update document counter, i.e., $\nSen \leftarrow \nSen + \len(\mathcal{D})$;

$\textnormal{each document in the corpus or dataset } \mathcal{D}$
$\textnormal{each sentence in the document }$
Update the count of token $t_i$ if it appears in current sentence, i.e., $\mathbf{c}[t_i] \leftarrow \mathbf{c}[t_i] + 1$;

$\textnormal{each token pair } t_i \textnormal{ and } t_j \textnormal{ in the sentence}$
$t_i \neq t_j$
Update the co-occurrence count, i.e., $\mathbf{C}[t_i][t_j] \leftarrow \mathbf{C}[t_i][t_j] + 1$;

Updated OMM counters $s$, $\mathbf{c}$, and $\mathbf{C}$
Continual OMM updating algorithm

§.§ Node Encoder

The effectiveness of pretrained language models (PLMs) such as BERT [Devlin et al., 2019], RoBERTa [Liu et al., 2019], and XLNet [Yang et al., 2019] for text modeling has been widely demonstrated in various scenarios. Therefore, we utilize a PLM as a document encoder to capture semantic information for each document $\mathbf{s} \in B$:

\begin{gather}
\mathbf{E}_\ssp = \mathtt{PLM} (\mathbf{s}) \mm{\nLen}{\nDim}, \label{eq:plm}
\end{gather}

where $\nLen$ is the maximum document length for PLM, and each row of $\mathbf{E}$ is a token embedding.

We append special $\mathtt{<PAD>}$ tokens to short documents to align their length with other documents in the batch, following BERT. We average the hidden states of the first and last Transformer layers following previous works [Li et al., 2020, Su et al., 2021].

We then perform an average pooling operation on $\mathbf{E}_\ssp$ to obtain the unified document embedding $\mathbf{d}^\ssp \in \mathbb{R}^{\nDim}$.

Following BertGCN [Lin et al., 2021], we form a batch-wise node embedding $\mathbf{X}^{n} \mmDim{(\nVoc+\nBat)}$:

\begin{gather}
\mathbf{X}^n = \left(\mathbf{0}, \cdots, \mathbf{0}, \mathbf{d}^{(\mathbf{s}_1)}, \cdots, \mathbf{d}^{(\mathbf{s}_{\nBat})} \right)^\top,
\end{gather}

where $\mathbf{d}^\sj$ is the embedding of document $j$ in the current batch $B$.

However, the document embeddings will cause interference among one another when doing node message passing by GCN.

To avoid interference, for each document $j$
in the batch, we form a sample-specific node embedding $\mathbf{X}^p_\sj \mmDim{(\nVoc+\nBat)}$ by:

\begin{gather}
\mathbf{X}^p_\sj = \left(\mathbf{M}_\sj, \mathbf{0}, \cdots, \mathbf{0}, \mathbf{d}^\sj, \mathbf{0}, \cdots, \mathbf{0} \right)^\top,
\end{gather}

where $\mathbf{M}_\sj \mm{\nVoc}{\nDim}$ is a sample-specific token embedding matrix of document $j$:

\begin{gather}
\mathbf{M}_\sj(i,:) =
\begin{cases}
\mathbf{E}_\sj(k,:), & \mbox{if token } i \mbox{ of the vocabulary} \\ & \mbox{is the }k \mbox{-th token in } \mathbf{s}_j, \\
\mathbf{0}, & \mbox{otherwise.}
\end{cases} \label{eq:m}
\end{gather}

We refer to $\mathbf{X}^{n}$ as the jammed node embeddings for all documents in the current batch, and $\mathbf{X}^p_\ssp$ as the unjammed node embedding of a single document $\mathbf{s}$.

§.§ GCN Encoder

Once the adjacency matrix ($\am$) and node embeddings (unjammed $\mathbf{X}^p_\ssp$ and jammed $\mathbf{X}^n$) are generated, the GCN encoder is applied to obtain graph-enhanced node representations.

We denote the GCN-encoded unjammed and jammed node representations as $\bar{\mathbf{X}}^p_\ssp$ and $\bar{\mathbf{X}}^n$, respectively.

Next, we extract the document embeddings from $\bar{\mathbf{X}}^p_\sj (\forall \mathbf{s}_j\in B)$  to form $\mathbf{Z} \mmDim{\nBat}$ (see Figure <ref>), i.e.,

\begin{equation}
\mathbf{Z}(j,:) = \bar{\mathbf{X}}^p_\sj(j+\nVoc,:).
\end{equation}

$\mathbf{Z}$ will be used in the classification task. We then extract the document embeddings from $\bar{\mathbf{X}}^p_\sj$ to form $\mathbf{Z}^p_\sj \mmDim{\nBat}$ and those from $\bar{\mathbf{X}}^n$ to form $\mathbf{Z}^n \mmDim{\nBat}$ by:

\begin{gather}
\mathbf{Z}^p_\sj(i,:) = \bar{\mathbf{X}}^p_\sj(i+\nVoc,:)~~ \text{and}\\
\mathbf{Z}^n(i,:) = \bar{\mathbf{X}}^{n}(i+\nVoc,:),
\end{gather}

as illustrated in Figure <ref>.
$\mathbf{Z}^p_\sj $ and $\mathbf{Z}^n $ will be used in the anti-interference contrastive task.

§.§ Training Objectives

To train the model, we employ two tasks: a document classification task and an anti-interference contrastive task.

The document classification task

utilizes a multi-layer perceptron (MLP) classifier $f: \mathbb{R}^\nDim \rightarrow \mathbb{R}^\nCls$ ($\nCls$ is the number of document classes) with a softmax activation function to infer the probability distribution over all classes. The loss function can be defined as:

\begin{gather}
\mathcal{L}_{\text{cls}} = -\frac{1}{\nBat} \sum^\nBat_{j=1} \log \left(f\left( \mathbf{Z}(j,:) \right)_{l_j}\right),
\end{gather}

where $l_j$ is the class label of the $j$-th document.

For the anti-interference contrastive task, the goal is to learn a representation space where semantically similar documents are closer to each other and dissimilar documents are farther apart. Hence, in the contrastive task, we enforce the GCN encoder to learn a mapping such that the jammed embedding of document $j$ (i.e., $\mathbf{Z}^n(j,:)$) is close to its unjammed embdding (i.e., $\mathbf{Z}^p_\sj(j,:)$) while distant from the embeddings of other documents (i.e., $\mathbf{Z}^p_\sj(k,:)$, $k \neq j$) in the batch.

The loss function can be calculated by:

\begin{gather}
\mathcal{L}_{\text{aic}} = -\frac{1}{\nBat} \sum^\nBat_{j=1} \log \left( \mathbf{y}_\sj(j) \right), \text{where}\label{eq:contrastive-loss}\\
\mathbf{y}_\sj = \mathtt{softmax} \left(\mathbf{Z}^{p}_\sj \left(\mathbf{Z}^{n}(j,:)\right)^\top \right) \in \mathbb{R}^\nBat.
\end{gather}

The anti-interference contrastive task helps to learn representations robust to the interference between documents.

The overall loss function is a combination of the classification and contrastive tasks with a balancing parameter $\lambda$, denoted as:

\begin{equation}\label{eq:overall-loss}
\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda \mathcal{L}_{\text{aic}}.
\end{equation}

Label-free Updating Mechanism (LUM). The occurrence memory module and the anti-interference contrastive task enables to continually update our ContGCN model with incoming unlabeled text data during inference. Hence, we name it label-free updating mechanism (LUM).

§.§ Model Training and Update

The training and updating of the ContGCN model involves three stages.

Stage 1: Before training. Prior to training, we perform post-pretraining on the pre-trained language model by using the classification task on the PLM-enhanced document embeddings. This pre-training helps to speed up the convergence of the model during the training process.

Stage 2: During training. During the training process, we use the multi-task training objective (Eq. <ref>) to train the ContGCN model.

Stage 3: During inference. When new test data is available, we first update the occurrence memory module using Algorithm <ref>. We then finetune the ContGCN model using the auxiliary anti-interference contrastive task (Eq. <ref>) to improve model performance during inference.

§ EXPERIMENTS

§.§ Experimental Setups

§.§.§ Datasets.

As described in [Lin et al., 2021], we have performed experiments on five text classification datasets which are commonly used in real-world applications: 20-Newsgroups (20NG), Ohsumed, R52 Reuters, R8 Reuters, and Movie Review (MR) datasets. Table <ref> presents the summarized statistics of these datasets. We randomly chose 10% of the training set for validation purposes for all datasets.

§.§.§ Baselines and Variants of Our Method.
To validate the effectiveness of our proposed ContGCN model, we compare it with three types of state-of-the-art models:: 1) traditional GCN-based models that do not utilize pretrained general semantic knowledge, including TextGCN [Yao, Mao, and Luo, 2019] and TensorGCN [Liu et al., 2020];
2) transformer-based PLMs, including BERT [Devlin et al., 2019], RoBERTa [Liu et al., 2019] and XLNet [Yang et al., 2019];
3) models that combine GCN with PLM, including TG-Transformer [Zhang and Zhang, 2020], BertGCN [Lin et al., 2021] and RoBERTaGCN [Lin et al., 2021]. As our ContGCN can be plugged by different PLMs, we adopt BERT, XLNet, and RoBERTa as alternatives, denoted as , , and .

§.§.§ Implementation Details.

We adopt the Adam optimizer [Kingma and Ba, 2015] to train the network of our ContGCN model and the baseline models, which are consistent in the following parameters. The following parameters are kept consistent across all models: the number of graph convolutional layers, if applicable, is set to 3; the embedding dimension is set to 768; and the batch size is set to 64. In the post-pretraining phase, we set the learning rate for PLM to 1e-4.
During training, we set different learning rates for PLM and other randomly initialized parameters including those of the GCN network, following Lin et al., 2021. Precisely, we set 1e-5 to finetune RoBERTa and BERT, 5e-6 to finetune XLNet, and 5e-4 to other parameters.
We average the results of 10 runs as the final evaluation results.

Dataset     20NG     R8     R52     Ohsumed     MR

# Docs     18,846     7,674     9,100     7,400     10,662

# Training     11,314     5,485     6,532     3,357     7,108

# Test     7,532     2,189     2,568     4,043     3,554

# Classes     20     8     52     23     2

Avg. Length     221     66     70     136     20

Dataset statistics.

Models     20NG     R8     R52      Ohsumed     MR

TextGCN     86.3     97.1     93.6     68.4     76.7

TensorGCN     87.7     98.0     95.0     70.1     77.9

BERT     85.3     97.8     96.4     70.5     85.7

RoBERTa     83.8     97.8     96.2     70.7     89.4

XLNet     85.1     98.0     96.6     70.7     87.2

TG-Transformer     -     98.1     95.2     70.4     -

BertGCN     89.3     98.1     96.6     72.8     86.0

RoBERTaGCN     89.5     98.2     96.1     72.8     89.7

89.4     98.3     96.9     73.1     86.4

89.7     98.5     97.0     73.1     88.7

90.1     98.6     96.6     73.4     91.3

Comparison of ContGCN with state-of-the-art models in offline evaluation. The best results are in boldface, and the second best results are underlined.

§.§ Offline Evaluation

We conduct an offline evaluation of our ContGCN model and state-of-the-art baselines on five datasets.
Table <ref> presents the overall performance of all methods, and the following observations can be made. First, PLM-only methods mostly outperform GCN-only methods due to their pre-learned semantic knowledge. However, GCN-only methods can build more document-token edges for better semantic comprehension on datasets with long documents, such as 20NG, but they struggle on datasets with short documents, such as MR. Second, PLM-empowered GCN methods combine the strengths of both PLMs and GCNs and outperform both PLM-only and GCN-only methods. Third, our ContGCN model achieves state-of-the-art performance on five datasets, thanks to 1) the proposed all-token-any-document paradigm that leverages general semantic knowledge from a large Wikipedia corpus and 2) the proposed contrastive learning objective that reduces inter-document interference. Notably, our  achieves the best performance on four datasets.

Models     20NG     R8     Ohsumed

90.1     98.6     73.4

w/o Wikipedia Init     89.9     98.2     73.1

w/o OMM Updating     89.6     98.3     73.0

w/o Contrastive Loss     89.7     98.5     73.2

89.7     98.5     73.1

w/o Wikipedia Init     89.8     98.3     72.8

w/o OMM Updating     89.4     98.2     72.7

w/o Contrastive Loss     89.5     98.2     73.0

Influence of Wikipedia initialization, OMM updating, and the anti-interference contrastive task.

Variants     1/6     2/6     3/6     4/6     5/6     6/6

ContGCN$^*$     86.4     87.3     88.1     88.6     89.0     89.6

ContGCN     86.3     87.1     87.8     88.2     88.7     89.1

ContGCN$^\alpha$     86.1     86.9     87.5     87.9     88.3     88.7

ContGCN$^\beta$     86.0     86.2     86.4     86.6     86.9     87.1

Comparisons of variants of  in the online learning scenario on the 20NG dataset. ContGCN$^*$ is retrained from scratch in each session with all previously seen data. ContGCN$^\alpha$ is updated without the contrastive loss. ContGCN$^\beta$ is updated without LUM.

legend columns=3,
legend style=
column sep=2ex
legend entries=
purple,mark=*,solid,line legend
orange,mark=*,solid,line legend
olive,mark=*,solid,line legend

xticklabels=0, 0.001, 0.01, 0.02, 0.03, 0.05, 0.10, 0.20,
line width=0.35mm,

yticklabels=-0.2, -0.2, 0, 0.2, 0.4,
ylabel=Relative Accuracy%,
ylabel near ticks,
ylabel shift=-5pt,

(1, 0) (2, 0.00) (3, 0.03) (4, 0.05) (5, 0.32) (6, 0.11) (7, 0.02) (8, -0.12)

(1, 0) (2, 0.05) (3, 0.11) (4, 0.14) (5, 0.13) (6, 0.09) (7, 0.04) (8, -0.07)

(1, 0) (2, 0.01) (3, 0.05) (4, 0.11) (5, 0.26) (6, 0.28) (7, 0.24) (8, 0.09)

Influence of the parameter $\lambda$ that weights the anti-interference contrastive loss.
Relative accuracy (%) means the difference between the accuracy achieved with $\lambda = \lambda_0$ and that achieved with $\lambda = 0$.

§.§ Ablation Study

First, we study the effect of different components of ContGCN, including Wikipedia initialization, OMM updating, and anti-interference contrastive task on offline performance.
Based on the results from Table <ref>, we can conclude the following. First, on the 20NG dataset, we observed that Wikipedia initialization is less effective, which is likely because the lengthy documents already contain sufficient non-consecutive knowledge during OMM updating. Apart from this, removing any of these components leads to a decline in performance for both  and  on three datasets, confirming their effectiveness. Second, models without OMM updating show the worst performance, indicating the importance of non-consecutive semantic information in training and testing.

Next, we study the updating strategy of ContGCN in the online learning scenario. As shown in Table <ref>, we can make the following observations. First, by comparing ContGCN with ContGCN$^\alpha$ and ContGCN$^\beta$, it can be seen that both OMM updating and the contrastive loss are effective.

Second, compared with retraining from scratch, i.e., ContGCN$^*$, ContGCN requires less computational resources and time to update yet still achieves competitive performance.

§.§ Impact of Anti-interference Contrastive Learning

Here, we study the balancing parameter $\lambda$ which weights the auxiliary anti-interference contrastive loss.
We conduct experiments on the 20NG, R8 and Ohsumed datasets with  model, varying $\lambda$ in {0.001, 0.01, 0.02, 0.03, 0.05, 0.10, 0.20}.
As demonstrated in Figure <ref>, we can make the following observations. First,

the reliance on the auxiliary task varies for different datasets.
Specifically, the model achieves the best performance on the 20NG, R8, and Ohsumed datasets when $\lambda$ is set to 0.03, 0.02, and 0.05, respectively. Second, for each dataset, as $\lambda$ increases, the performance first increases and then decreases. Hence, it is essential to select a good $\lambda$.

§.§ Online Evaluation

§.§.§ Fixed Testing Data.
Figure <ref> illustrates the performance of our model and baselines in an online learning scenario where the training/updating data is incremental and the testing data is fixed. Based on the results, we can draw the following observations:
First, GCN-based methods such as TextGCN and RoBERTaGCN cannot be updated and are incapable of reasoning about unobserved data as they construct fixed graphs based on the original corpus. Therefore, their performance remains constant over time, as shown by the dotted lines in Figure <ref>.
Second, as the updating data increases, the performance of all updatable models improves.
Third, our proposed ContGCN method outperforms all baselines in all sessions.

Fourth, due to the limitations of STSD-based GCN methods, we introduce a RoBERTaGCN$_\texttt{scratch}$ model that retrains from scratch in each session with all previously seen data. However, this model still falls short compared to due to the utilization of Wikipedia Initialization and the unjammed node embedding in . Moreover, as illustrated in Figure <ref>, the updating time of RoBERTaGCN$_\texttt{scratch}$ increases almost linearly w.r.t. data size, making it unsuitable for online learning. The finetuning time ratios of ContGCN and RoBERTa are closer to or less than 1, indicating that for these models, each session takes up a comparable amount of time.

legend columns=3,
legend style=
column sep=2ex
legend entries=
purple,mark=x,dotted,line legend
orange,mark=x,solid,line legend
olive,mark=x,dotted,line legend

legend columns=2,
legend style=
column sep=2ex
legend entries=
olive,mark=x,solid,line legend
violet,mark=x,solid,line legend

xticklabels=0/6, 1/6, 2/6, 3/6, 4/6, 5/6, 6/6,
line width=0.35mm,
ytick=78, 82, 86, 90,

ylabel near ticks,
ylabel shift=-5pt,

(1, 80.5) (2, 80.5) (3, 80.5) (4, 80.5) (5, 80.5) (6, 80.5) (7, 80.5)

(1, 81.2) (2, 81.5) (3, 81.3) (4, 81.1) (5, 81.9) (6, 81.8) (7, 82.1)

(1, 84.8) (2, 84.8) (3, 84.8) (4, 84.8) (5, 84.8) (6, 84.8) (7, 84.8)

(1, 84.8) (2, 85.2) (3, 85.5) (4, 86.1) (5, 86.6) (6, 87.1) (7, 87.9)

(1, 85.7) (2, 86.3) (3, 87.1) (4, 87.8) (5, 88.2) (6, 88.7) (7, 89.1)

Reasoning Quality

xticklabels=1/6, 2/6, 3/6, 4/6, 5/6, 6/6,
line width=0.35mm,

ylabel=Finetuning Time Ratio,
ylabel near ticks,
ylabel shift=-5pt,

(1, 1.00) (2, 0.78) (3, 0.89) (4, 1.03) (5, 0.82) (6, 0.92)

(1, 1.00) (2, 1.07) (3, 1.12) (4, 1.15) (5, 1.24) (6, 1.34)

(1, 1.00) (2, 0.95) (3, 0.94) (4, 0.98) (5, 0.95) (6, 1.02)

Finetuning Efficiency

Comparison between our ContGCN model and baselines in an online learning scenario. We divide the 20NG dataset into training, testing, and updating sets by the ratio of 2:2:6. We trained each model with the training set to learn an initial version. Then, we divided the updating set into six equal parts and gradually fed each part to the model for fine-tuning. The finetuning time ratio in (b) is calculated by the finetuning time of the current session over that of the first session. For each training or updating session, we used 10% of the training set as the validation set.

§.§.§ Fixed Training Data.
We have deployed ContGCN on Huawei public opinion analysis system, under the scenario that the training data is fixed and the updating/testing data is incremental. We use an optimized variant of RoBERTa [Liu et al., 2019] – RoBERTa$_\texttt{wwm-ext}$ [Xu, 2021], tailored for Chinese text classification, which will still be referred to as RoBERTa below and in Table <ref>.
We implement RoBERTaGCN by plugging RoBERTa into BertGCN [Lin et al., 2021], which cannot infer unobserved documents due to the STSD scheme and hence the results are unavailable in the following months.
We then implement ContGCN based on RoBERTa and the proposed ATAD scheme, i.e., . After RoBERTa and  were trained offline initially (i.e., in the 0th month), we deploy them online for comparison. As illustrated in Table <ref>, our  has achieved 5.94%, 6.57%, 7.98%, and 8.86% gains in accuracy over RoBERTa in the 0th, 1st, 2nd, and 3rd month respectively. Besides, due to the distribution shift of public opinions, the accuracy drops slightly over time. However,  still maintains higher performance than RoBERTa.
Furthermore, by removing the label-free update mechanism (i.e., $^\beta$), the performance drops significantly, which demonstrates the capability of our ContGCN model in continual learning.

Models     0th     1st     2nd     3rd

RoBERTaGCN     91.7     N/A     N/A     N/A

RoBERTa     87.6     86.8     85.2     83.5

ContGCN$_\texttt{RoBERTa}^\beta$      92.8     90.3     89.9     88.2

92.8     92.5     92.0     90.9

Comparison of our ContGCN model with RoBERTa in an industrial online learning scenario. All models are first trained offline (in the 0th month) with a labeled dataset. After deployed,  performs online learning with LUM. ContGCN$_\texttt{RoBERTa}^\beta$ is a static network with parameters fixed after the initial training.

§ RELATED WORK

Graph Convolutional Network.
Graph Convolutional Networks [Welling and Kipf, 2016] (GCNs) have become increasingly popular in recent years due to their ability to capture the structural relations among data [Hamilton, Ying, and Leskovec, 2017, Li, Han, and Wu, 2018]. They can learn representations of graph data by propagating information between nodes in the graph. The popularity of GCNs can be attributed to their versatility and effectiveness in various applications [Zhang et al., 2019], including image classification [Hong et al., 2020], video understanding [Huang et al., 2020], social recommendation [Fan et al., 2019], and text classification [Yao, Mao, and Luo, 2019, Lin et al., 2021].

Text Classification. Text classification is a critical and fundamental task in the field of natural language processing. Early studies [Jacovi, Sar Shalom, and Goldberg, 2018, Sari, Rini, and Malik, 2019] utilize word embedding methods [Mikolov et al., 2013, Pennington, Socher, and Manning, 2014] or apply traditional models [Zhang, Zhao, and LeCun, 2015, Lai et al., 2015] such as convolutional neural networks [Krizhevsky, Sutskever, and Hinton, 2012] or recurrent neural networks [Rumelhart, Hinton, and Williams, 1985] to learn textual knowledge. In recent years, Transformer-based pretrained language models such as BERT [Devlin et al., 2019] have been introduced [Sun et al., 2019] for text classification due to its strong ability of semantic comprehension. However, these models do not effectively utilize global semantic information such as token co-occurrence in a corpus.

GCN-based Text Classification.
GCNs have been gaining attention in text classification, owing to their ability to model non-structured data and capture global dependencies, such as high-order neighborhood information [Yao, Mao, and Luo, 2019, Lin et al., 2021]. Unlike some GCN-based methods [Li et al., 2019, Xie et al., 2021, Li et al., 2021] that construct homogeneous document graphs, we follow Yao, Mao, and Luo, 2019 to construct a document-token graph to better capture semantic relations. In contrast to previous methods that typically construct fixed graphs and are limited in their ability to reason about unobserved documents, our proposed ContGCN model employs an all-token-any-document paradigm, enabling making inferences on new data in online systems.

§ CONCLUSION

To deploy GCN-based text classification methods to online industrial systems, we propose a ContGCN model with a novel all-token-any-document paradigm and a label-free updating mechanism, which endow the model the ability to infer unobserved documents and enable to continually update the model during inference time. To our knowledge, this is the first attempt to use GCN for online text classification. Extensive online and offline evaluations validate the effectiveness of ContGCN, which achieves favorable performance compared with various state-of-the-art methods.

§ ACKNOWLEDGMENTS

The authors would like to thank Qimai Li for valuable discussion and the anonymous reviewers for their helpful comments. Q. Liu and X. Wu were supported by GRF No.15222220 funded by the UGC and ITS/359/21FP funded by the ITC of Hong Kong.

[Abaho et al., 2021]
Abaho, M.; Bollegala, D.; Williamson, P.; and Dodd, S. 2021.
Detect and Classify – Joint Span Detection and Classification for
Health Outcomes.
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, 8709–8721. Online and Punta Cana, Dominican
Republic: Association for Computational Linguistics.

[Devlin et al., 2019]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.
BERT: Pre-training of Deep Bidirectional Transformers for Language
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis,
Minnesota: Association for Computational Linguistics.

[Fan et al., 2019]
Fan, W.; Ma, Y.; Li, Q.; He, Y.; Zhao, E.; Tang, J.; and Yin, D. 2019.
Graph Neural Networks for Social Recommendation.
In The world wide web conference, 417–426.

[Hamilton, Ying, and Leskovec, 2017]
Hamilton, W.; Ying, Z.; and Leskovec, J. 2017.
Inductive Representation Learning on Large Graphs.
Advances in neural information processing systems, 30.

[Hong et al., 2020]
Hong, D.; Gao, L.; Yao, J.; Zhang, B.; Plaza, A.; and Chanussot, J. 2020.
Graph Convolutional Networks For Hyperspectral Image Classification.
IEEE Transactions on Geoscience and Remote Sensing, 59(7):

[Huang et al., 2020]
Huang, D.; Chen, P.; Zeng, R.; Du, Q.; Tan, M.; and Gan, C. 2020.
Location-aware Graph Convolutional Networks for Video Question
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, 11021–11028.

[Jacovi, Sar Shalom, and Goldberg, 2018]
Jacovi, A.; Sar Shalom, O.; and Goldberg, Y. 2018.
Understanding Convolutional Neural Networks for Text Classification.
In Proceedings of the 2018 EMNLP Workshop BlackboxNLP:
Analyzing and Interpreting Neural Networks for NLP, 56–65. Brussels,
Belgium: Association for Computational Linguistics.

[Kingma and Ba, 2015]
Kingma, D. P.; and Ba, J. 2015.
Adam: A Method for Stochastic Optimization.
International Conference on Learning Representations.

[Krizhevsky, Sutskever, and Hinton, 2012]
Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.
Imagenet classification with deep convolutional neural networks.
Advances in neural information processing systems, 25.

[Lai et al., 2015]
Lai, S.; Xu, L.; Liu, K.; and Zhao, J. 2015.
Recurrent Convolutional Neural Networks for Text Classification.
In Proceedings of the AAAI conference on artificial
intelligence, volume 29.

[Li et al., 2020]
Li, B.; Zhou, H.; He, J.; Wang, M.; Yang, Y.; and Li, L. 2020.
On the Sentence Embeddings from Pre-trained Language Models.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), 9119–9130. Online: Association for
Computational Linguistics.

[Li, Han, and Wu, 2018]
Li, Q.; Han, Z.; and Wu, X.-M. 2018.
Deeper Insights into Graph Convolutional Networks for Semi-supervised
In Proceedings of the AAAI conference on artificial
intelligence, volume 32.

[Li et al., 2019]
Li, Q.; Wu, X.-M.; Liu, H.; Zhang, X.; and Guan, Z. 2019.
Label Efficient Semi-supervised Learning via Graph Filtering.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, 9582–9591.

[Li et al., 2021]
Li, Q.; Zhang, X.; Liu, H.; Dai, Q.; and Wu, X.-M. 2021.
Dimensionwise Separable 2-D Graph Convolution for Unsupervised and
Semi-supervised Learning on Graphs.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge
Discovery & Data Mining, 953–963.

[Lin et al., 2021]
Lin, Y.; Meng, Y.; Sun, X.; Han, Q.; Kuang, K.; Li, J.; and Wu, F. 2021.
BertGCN: Transductive Text Classification by Combining GNN and BERT.
In Findings of the Association for Computational Linguistics:
ACL-IJCNLP 2021, 1456–1462.

[Liu et al., 2020]
Liu, X.; You, X.; Zhang, X.; Wu, J.; and Lv, P. 2020.
Tensor Graph Convolutional Networks for Text Classification.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, 8409–8416.

[Liu et al., 2019]
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.;
Zettlemoyer, L.; and Stoyanov, V. 2019.
RoBERTa: A Robustly Optimized BERT Pretraining Approach.
CoRR, abs/1907.11692.

[Luo et al., 2022]
Luo, R.; Sinha, R.; Hindy, A.; Zhao, S.; Savarese, S.; Schmerling, E.; and
Pavone, M. 2022.
Online Distribution Shift Detection via Recency Prediction.
arXiv preprint arXiv:2211.09916.

[Mikolov et al., 2013]
Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013.
Distributed Representations of Words and Phrases and Their
In Advances in neural information processing systems,

[Pennington, Socher, and Manning, 2014]
Pennington, J.; Socher, R.; and Manning, C. D. 2014.
Glove: Global Vectors for Word Representation.
In Proceedings of the 2014 conference on empirical methods in
natural language processing (EMNLP), 1532–1543.

[Qiao et al., 2018]
Qiao, C.; Huang, B.; Niu, G.; Li, D.; Dong, D.; He, W.; Yu, D.; and Wu, H.
A New Method of Region Embedding for Text Classification.
In International Conference on Learning Representations.

[Rumelhart, Hinton, and Williams, 1985]
Rumelhart, D. E.; Hinton, G. E.; and Williams, R. J. 1985.
Learning Internal Representations by Error Propagation.
Technical report, California Univ San Diego La Jolla Inst for
Cognitive Science.

[Sari, Rini, and Malik, 2019]
Sari, W.; Rini, D.; and Malik, R. 2019.
Text Classification Using Long Short-Term Memory With GloVe Features.
Jurnal Ilmiah Teknik Elektro Komputer dan Informatika, 5(2):

[Su et al., 2021]
Su, J.; Cao, J.; Liu, W.; and Ou, Y. 2021.
Whitening Sentence Representations for Better Semantics and Faster
arXiv preprint arXiv:2103.15316.

[Sun et al., 2019]
Sun, C.; Qiu, X.; Xu, Y.; and Huang, X. 2019.
How to Fine-tune BERT for Text Classification?
In Chinese Computational Linguistics: 18th China National
Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18,
194–206. Springer.

[Welling and Kipf, 2016]
Welling, M.; and Kipf, T. N. 2016.
Semi-supervised Classification with Graph Convolutional Networks.
In J. International Conference on Learning Representations
(ICLR 2017).

[Xie et al., 2021]
Xie, Q.; Huang, J.; Du, P.; Peng, M.; and Nie, J.-Y. 2021.
Inductive Topic Variational Graph Auto-Encoder for Text
In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, 4218–4227. Online: Association for Computational Linguistics.

[Xu et al., 2019]
Xu, G.; Meng, Y.; Qiu, X.; Yu, Z.; and Wu, X. 2019.
Sentiment Analysis of Comment Texts Based on BiLSTM.
Ieee Access, 7: 51522–51532.

[Xu, 2021]
Xu, Z. 2021.
RoBERTa-wwm-ext Fine-Tuning for Chinese Text Classification.
arXiv preprint arXiv:2103.00492.

[Yang et al., 2019]
Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V.
XLNet: Generalized Autoregressive Pretraining for Language
In Advances in Neural Information Processing Systems,
volume 32.

[Yang et al., 2019]
Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V.
Xlnet: Generalized Autoregressive Pretraining for Language
Advances in neural information processing systems, 32.

[Yao, Mao, and Luo, 2019]
Yao, L.; Mao, C.; and Luo, Y. 2019.
Graph Convolutional Networks for Text Classification.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 33, 7370–7377.

[Zhang and Zhang, 2020]
Zhang, H.; and Zhang, J. 2020.
Text Graph Transformer for Document Classification.
In Conference on Empirical Methods in Natural Language
Processing (EMNLP).

[Zhang et al., 2019]
Zhang, S.; Tong, H.; Xu, J.; and Maciejewski, R. 2019.
Graph Convolutional Networks: A Comprehensive Review.
Computational Social Networks, 6(1): 1–23.

[Zhang, Zhao, and LeCun, 2015]
Zhang, X.; Zhao, J.; and LeCun, Y. 2015.
Character-level Convolutional Networks for Text Classification.
In Advances in neural information processing systems,