# Multi-Class Unsourced Random Access
via Coded Demixing

†Vamsi K. Amalladinne, §Allen Hao, §Stefano Rini, †Jean-Francois Chamberland
†Department of Electrical and Computer Engineering, Texas A&M University
§Department of Electrical and Computer Engineering, National Yang Ming Chiao
Tung University (NYCU)  This material is based upon work supported, in part,
by the National Science Foundation (NSF) under Grant CCF-1619085, and by
Qualcomm Technologies, Inc., through their University Relations Program.

###### Abstract

Unsourced random access (URA) is a recently proposed communication paradigm
attuned to machine-driven data transfers. In the original URA formulation, all
the active devices share the same number of bits per packet. The scenario
where several classes of devices transmit concurrently has so far received
little attention. An initial solution to this problem takes the form of group
successive interference cancellation, where codewords from a class of devices
with more resources are recovered first, followed by the decoding of the
remaining messages. This article introduces a joint iterative decoding
approach rooted in approximate message passing. This framework has a
concatenated coding structure borrowed from the single-class coded compressed
sensing and admits a solution that offers performance improvement at little
added computational complexity. Our findings point to new connections between
multi-class URA and compressive demixing. The performance of the envisioned
algorithm is validated through numerical simulations.

###### Index Terms:

Unsourced random access, approximate message passing, coded compressed
sensing, compressive demixing.

##  I Introduction and Background

Machine-type communication (MTC) is poised to increase sharply over the coming
decades [1, 2]. This poses a great challenge to the foundation on which
wireless infrastructures are relying, as MTC often produces short packets and
fleeting connections. This results in a constantly changing collection of
devices wishing to transmit bursty data. In this context, it becomes
impractical to adopt the enrollment-estimation-scheduling paradigm, as the
cost of acquiring side information on the channel state becomes prohibitive.
This reality is reflected in the current interest in grant-free schemes and
uncoordinated wireless access.

Another aspect of coordinated system that is hard to maintain in MTC is the
implicit agreement between an access point and an active device regarding the
MAC layer configuration. This consideration serves as a motivation behind the
unsourced random access (URA) model introduced by Polyanskiy [3]. A
distinguishing feature of this approach is that active devices must share a
same encoding function. The signal sent by a device therefore depends solely
on the message payload, and not on its identity. If a device wishes to be
recognized, then it must embed an identifier within its message. Altogether,
the URA model captures several fundamental aspects of MTC: this has led to a
rapid growth of literature on the topic [4, 5, 6, 7, 8, 9, 10, 11, 12].

An aspect of URA that has received little consideration is the heterogeneous
scenario where various types of device are present. In such situations,
devices from distinct groups may wish to communicate messages of different
lengths and they may have varying power budgets. This reality can be
accommodated by allowing group-based encoding and separating groups in signal
space [13]. Interestingly, standard URA admits a compressed sensing (CS)
interpretation where signals sent by active devices are collectively viewed as
sparse indices in the space of possible messages. Under this viewpoint, the
heterogeneous URA framework becomes similar to compressive demixing in very
large dimensional spaces [14]. In this article, we combine recent advances in
coded compressed sensing [7, 8, 9] and established concepts in demixing [15,
16, 17, 18, 19] to develop a novel communication scheme for the two-class URA
setting.

##  II System Model and Encoding Process

In this section, we begin by describing the two-class URA architecture.
Consider the scenario in which $K_{1}$ active devices form group 1 and $K_{2}$
devices form group 2, each user wishing to send a message to the access point.
These transmissions take place over an uncoordinated multiple access channel.
Active devices are cognizant of frame boundaries and they collectively signal
over $n$ channel uses (real degrees of freedom). The signal received at the
destination is of the form

$\textstyle\mathbf{y}=\sum_{i=1}^{K_{1}}\mathbf{x}_{i}^{(1)}+\sum_{i=1}^{K_{2}}\mathbf{x}_{i}^{(2)}+\mathbf{z}$
(1)

where signal $\mathbf{x}_{\iota}^{(g)}\in\mathcal{C}_{g}\subset\mathbb{R}^{n}$
is the codeword transmitted by active device $\iota\in[1,\ldots,K_{g}]$ in
group $g\in\\{1,2\\}$, and $\mathcal{C}_{g}$ is the codebook associated with
the group $g$. The noise component $\mathbf{z}$ is composed of independent
Gaussian elements, each with distribution $\mathcal{N}(0,1)$. Adhering to the
URA framework, we emphasize that all the active devices within a group share a
same codebook; consequently, $\mathbf{x}_{\iota}^{(g)}$ is a function of the
payload of device $\iota$, and not of its identity.

Our proposed algorithm loosely parallels coded compressed sensing (CCS). The
selection of a codeword mimics the encoding process obtained by combining the
outer code of Amalladinne et al. [20, 7] and the SPARC-inspired inner encoding
of Fengler et al. [8, 21], albeit we must account for the two groups. More
specifically, information bits are encoded using a concatenated structure as
in [9, 22]. The outer code is an LDPC code over a large alphabet. Every output
symbol is turned into an index vector, which contains zeros everywhere except
for a single location where it features a one. These index vectors are
concatenated into a SPARC-like vector [23, 24, 25], which is subsequently
multiplied by a sensing matrix. Such a construction is somewhat intricate, and
it can hardly be explained in detail within a short article. We point the
reader to [7] for an extensive description of the scheme. Below, we shall only
provide a synopsis of the approach and, then, we emphasize the modifications
that must be performed to accommodate two groups of devices.

Consider a sequence of information bits $\mathbf{w}\in\\{0,1\\}^{w}$. This
sequence $\mathbf{w}$ is encoded using an LDPC code, leading to a codeword
$\mathbf{v}$ which is then partitioned into $L$ blocks $\mathbf{v}(\ell)$ of
length $v=w+p$, so that
$\mathbf{v}={\mathbf{v}(1)}{\mathbf{v}(2)}\cdots{\mathbf{v}(L)}$. This LDPC
code is designed to permit efficient decoding of several codewords on a same
graph using belief propagation (BP) through the application of fast Fourier
transform (FFT) or similar techniques [26]. As part of the next encoding step,
each sub-block $\mathbf{v}(\ell)$ is transformed into a message index of
length $m=2^{v/L}$. Mathematically, we have

$\begin{split}\mathbf{m}(\ell)&=f(\mathbf{v}(\ell)),\end{split}$ (2)

where the function $f(\cdot)$ is a bijection between $\\{0,1\\}^{v/L}$ and
standard basis elements of length $m$. The SPARC-like message $\mathbf{m}$ is
subsequently obtained as the concatenation of these index vectors, that is
$\mathbf{m}=\mathbf{m}(1)\mathbf{m}(2)\cdots\mathbf{m}(L)$. In the original
URA formulation, the binary vector $\mathbf{m}$ is leveraged to generate
transmit signal as $\boldsymbol{\Phi}\mathbf{m}$. This is illustrated in Fig.
1. The signal aggregate produced by all the active devices then assumes the
form $\boldsymbol{\Phi}\mathbf{s}$, where $\mathbf{s}=\sum_{i}\mathbf{m}_{i}$
is the sum of the SPARC-like messages from all the active devices.

Message
$\mathbf{w}$$\mathbf{v}(1)$$\mathbf{v}(2)$$\vdots$$\vdots$$\mathbf{v}(L)$$\vdots$Bipartite
Graph$\mathbf{v}$$\mathbf{m}$Indexing$\mathbf{m}(1)$$\mathbf{m}(2)$$\vdots$$\mathbf{m}(L)$$f$$f$$f$
Figure 1:  A depiction of the CCS encoding process in Sec. II. The function
$f(\cdot)$ in (2) maps strings of bits to index vectors.

To accommodate two groups of devices, we must expand this construction. This
warrants the creation of two distinct codes $\mathcal{C}_{1}$ and
$\mathcal{C}_{2}$, both of which follow an encoding architecture akin to Fig.
1. Thus, the received signal becomes

$\begin{split}\mathbf{y}&=\boldsymbol{\Phi}_{1}\mathbf{s}_{1}+\boldsymbol{\Phi}_{2}\mathbf{s}_{2}+\mathbf{z}.\end{split}$
(3)

Signal $\mathbf{s}_{1}$ is the sum of the SPARC-like vectors
$\mathbf{m}^{(1)}_{i}$ encoded using $\mathcal{C}_{1}$; and $\mathbf{s}_{2}$
is the sum of the vectors $\mathbf{m}^{(2)}_{i}$ encoded using
$\mathcal{C}_{2}$. We note that both $\mathbf{s}_{1}$ and $\mathbf{s}_{2}$
possess sparse structures. This hints at a connection between (3) and
compressive demixing.

######  Remark 1.

There are at least two ways to arrive at (3). The simplest avenue is to encode
the two groups separately, with devices in group 1 leading to $\mathbf{s}_{1}$
and devices in group 2, $\mathbf{s}_{2}$. The second option is to create
layers as in [13]. Suppose that messages within group 1 are smaller than those
in group 2, i.e., $w_{1}<w_{2}$. Then, all the devices in group 1 employ
$\mathcal{C}_{1}$. On the other hand, the devices in group 2 encode their
first $w_{1}$ bits with $\mathcal{C}_{1}$ and, after adding linking parities,
they encode the remaining $w_{2}-w_{1}$ bits using $\mathcal{C}_{2}$. These
distinct approaches lead to essentially equivalent mathematical formulations,
up to parameterization. Therefore, we discuss the simpler form with the
understanding that findings extend to the two alternatives.

##  III Demixing and Decoding Algorithm

Having established the encoding for our multi-class URA problem, we now turn
our attention to the decoding process. Conceptually, the receiver seeks to
recover $\mathbf{s}_{1}$ and $\mathbf{s}_{2}$; and it must disentangle the
information messages embedded in each aggregate. The first task can be viewed
as support recovery through compressive demixing, and the second objective
corresponds to message disambiguation. As in [9], we propose a decoding
framework that performs these two undertakings concurrently. The building
blocks for our algorithm include BP on factor graphs [27] and approximate
message passing (AMP) applied to non-separable functions [28].

###  III-A Belief Propagation (BP)

The application of BP to bipartite LDPC graphs is well understood [27] and we
shall only briefly review it here. It can be summarized as follows:
$\boldsymbol{\lambda}_{\ell}$ denotes the trivial factor associated with a
local observation; $\boldsymbol{\mu}_{s\to a}$ represents the message from
variable $s$ to factor $a$; and, $\boldsymbol{\mu}_{a\to s}$ is the message
from factor $a$ to node $s$. Under suitable conditions, BP efficiently
computes marginal probabilities at every variable node $s$.

$s_{1}$$s_{2}$$\vdots$$s_{L}$$\vdots$$a_{1}$$\vdots$$a_{P}$Local
ObservationsFFT-based Factors$\boldsymbol{\mu}_{s_{1}\to
a_{1}}$$\boldsymbol{\mu}_{a_{P}\to s_{L}}$$\boldsymbol{\lambda}_{2}$ Figure 2:
Our scheme employs factor graphs over large alphabets, one per group, to
partially recover several codewords at once.

Still, there is an important distinction between standard decoding on factor
graph and its use in URA settings. The decoder seeks to recover multiple
messages on a same graph simultaneously. This distinguishing aspect and its
implications are carefully described in [22]. Note that it forces the LDPC
code to be over a very large alphabet. Also, it changes the way
$\\{\boldsymbol{\lambda}_{\ell}\\}$ are initialized, a consideration we
delayed until the next section. Nevertheless, we can interpret vector
$\boldsymbol{\lambda}_{\ell}$ for one group as a collection of local
estimates. Entry $\boldsymbol{\lambda}_{\ell}(k)$ is an estimate for the event
that a designated device has produced basis element $\mathbf{e}_{k}$ within
section $\ell$, i.e., a proxy for
$\mathrm{Pr}(\mathbf{m}_{i}(\ell)=\mathbf{e}_{k})$.

Once trivial factor nodes are initialized, the message passing rules are
essentially standard. A message going from check node $a_{p}$ to variable node
$s$ is given by

$\boldsymbol{\mu}_{a_{p}\to
s}(k)=\sum_{\mathbf{k}_{a_{p}}:k_{p}=k}\mathcal{G}_{a_{p}}\left(\mathbf{k}_{a_{p}}\right)\prod_{s_{j}\in
N(a_{p})\setminus s}\boldsymbol{\mu}_{s_{j}\to a_{p}}(k_{j}),$ (4)

where $N(a_{p})$ is the graph neighborhood of $a_{p}$ and indicator function
$\mathcal{G}_{a_{p}}(\cdot)$ assesses the local consistency of the index sub-
vector $\mathbf{k}_{a_{p}}=(k_{s}:s\in N(a_{p}))$. Similarly, a message passed
from variable node $s_{\ell}$ to check node $a$ can be expressed as

$\boldsymbol{\mu}_{s_{\ell}\rightarrow
a}(k)\propto\boldsymbol{\lambda}_{\ell}(k)\prod_{a_{p}\in N(s_{\ell})\setminus
a}\boldsymbol{\mu}_{a_{p}\to s_{\ell}}(k),$ (5)

where the ‘$\propto$’ symbol indicates that the measure should be normalized.

At any point in the iterative process, the belief vector on section $\ell$
based on extrinsic information is proportional to the product of the messages
from adjoining check factors, i.e.

$\boldsymbol{\mu}_{s_{\ell}}(k)=\prod_{a\in N(s_{\ell})}\boldsymbol{\mu}_{a\to
s_{\ell}}(k).$ (6)

The estimated marginal distribution of a specific device having transmitted
index $k$ at variable node $s_{\ell}$ is proportional to the product of the
current messages from all adjoining factors, including its intrinsic
information, that is

$\begin{split}p_{s_{\ell}}(k)&\propto\boldsymbol{\lambda}_{\ell}(k)\prod_{a\in
N(s_{\ell})}\boldsymbol{\mu}_{a\to
s_{\ell}}(k)=\boldsymbol{\lambda}_{\ell}(k)\boldsymbol{\mu}_{s_{\ell}}(k).\end{split}$
(7)

A normalized version of
$\boldsymbol{\lambda}_{\ell}(k)\boldsymbol{\mu}_{s_{\ell}}(k)$ can be viewed
as an estimate for probability
$\mathrm{Pr}\left(\mathbf{m}_{i}(\ell)=\mathbf{e}_{k}\right)$ (within a
group), where $i$ is fixed.

Again, in the context of multi-class URA and (3), there are actually two
factor graphs: one associated with $\mathcal{C}_{1}$ and the other, with
$\mathcal{C}_{2}$. BP is similar in these two cases, which justifies our
unified treatment. Below, we draw a distinction between the two codes wherever
appropriate. It is also worth noting that, during joint decoding, we utilize
BP in an unusual way with frequent reinitialization and only a few steps of BP
per round. The motivation behind this strategy is explained below.

###  III-B Approximate Message Passing (AMP)

In this section, we explain the component of the decoding algorithm associated
with the inner code. The mathematical structure in (3) hints at several
demixing algorithms found in the literature to address similar tasks [16, 17,
18, 19]. Yet, as mentioned in the introduction, a distinguishing feature of
our setting lies in the very large dimensions of the raw index vectors. Much
like in the case of CCS, this precludes the direct application of existing
demixing methods. As such, we strive to create schemes that harness lessons
from the past while also admitting a computationally tractable implementation.
Specifically, we wish to leverage insights from demixing and transpose them
into an AMP framework, which has recently been adapted to the standard URA
formulation [8, 9]. The appeal of AMP-based solutions stems, partly, from
their low-complexity and mathematical tractability [29, 30].

To conform to the structure of typical AMP formulations, we rewrite the
received signal in (3) as

$\begin{split}\mathbf{y}=\mathbf{A}\begin{bmatrix}d_{1}\mathbf{s}_{1}\\\
d_{2}\mathbf{s}_{2}\end{bmatrix}+\mathbf{z}=\begin{bmatrix}\mathbf{A}_{1}&\mathbf{A}_{2}\end{bmatrix}\begin{bmatrix}d_{1}\mathbf{s}_{1}\\\
d_{2}\mathbf{s}_{2}\end{bmatrix}+\mathbf{z}.\end{split}$ (8)

where $d_{g}\mathbf{A}_{g}=\boldsymbol{\Phi}_{g}$. In this abstraction, matrix
$\mathbf{A}$ has independent, zero-mean Gaussian entries. The variance of
these entries is selected as to normalize the 2-norm of every column,
$\mathbb{E}\left\|\mathbf{A}_{:,\iota}\right\|^{2}=1$. Constant $d_{g}$ is a
non-negative amplitude; it accounts for the transmit power employed by devices
in each group. We emphasize that (8), on top of having a standard AMP format,
matches the canonical compressive demixing equation found in [18], albeit with
an additional noise term. In its current form, this representation invites a
composite iterative AMP decoder that alternates between (9) and (10):

$\displaystyle\mathbf{z}^{(t)}=\mathbf{y}-\mathbf{A}\begin{bmatrix}d_{1}\mathbf{s}_{1}^{(t)}\\\
d_{2}\mathbf{s}_{2}^{(t)}\end{bmatrix}+\frac{\mathbf{z}^{(t-1)}}{n}\operatorname{div}\boldsymbol{\eta}^{(t-1)}\left(\mathbf{r}^{(t-1)}\right)$
(9) $\displaystyle\begin{bmatrix}\mathbf{s}_{1}^{(t+1)}\\\
\mathbf{s}_{2}^{(t+1)}\end{bmatrix}=\boldsymbol{\eta}^{(t)}\left(\mathbf{r}^{(t)}\right)=\begin{bmatrix}\boldsymbol{\eta}_{1}^{(t)}\left(\mathbf{r}_{1}^{(t)}\right)\\\
\boldsymbol{\eta}_{2}^{(t)}\left(\mathbf{r}_{2}^{(t)}\right)\end{bmatrix},$
(10)

where the _effective observation_ $\mathbf{r}^{(t)}$ is given by

$\mathbf{r}^{(t)}=\begin{bmatrix}\mathbf{r}_{1}^{(t)}\\\
\mathbf{r}_{2}^{(t)}\end{bmatrix}=\mathbf{A}^{\mathrm{T}}\mathbf{z}^{(t)}+\begin{bmatrix}d_{1}\mathbf{s}_{1}^{(t)}\\\
d_{2}\mathbf{s}_{2}^{(t)}\end{bmatrix}.$ (11)

The iterative process is initiated with $\mathbf{s}^{(0)}=\mathbf{0}$ and
$\mathbf{z}^{(0)}=\mathbf{y}$. Equation (9) is the computation of the
_residual_ augmented by the characteristic AMP Onsager term [30]. The second
equation is a state update, which is performed through denoising using
function $\boldsymbol{\eta}^{(t)}(\cdot)$. The remainder of this section is
devoted to defining and justifying the denoising functions.

An astounding property of AMP is that, under certain conditions, the effective
observation is asymptotically distributed as the true state plus Gaussian
noise. For the problem at hand, in the limit where the system gets larger,
$\mathbf{r}^{(t)}$ becomes approximately distributed as

$\begin{bmatrix}d_{1}\mathbf{s}_{1}\\\
d_{2}\mathbf{s}_{2}\end{bmatrix}+\tau_{t}\boldsymbol{\zeta}_{t},$ (12)

where the entries in $\boldsymbol{\zeta}_{t}$ are independent
$\mathcal{N}(0,1)$ random variables and $\tau_{t}$ is a deterministic
quantity. This phenomenon underlies the rationale behind our denoising
functions.

The first ingredient of our denoiser is the posterior mean estimate (PME) of
Fengler et al. [21], a low-complexity estimator that encourages sparsity in
$\mathbf{s}^{(t)}$. Their work advocates an element-wise estimate of the form

$\hat{s}_{g}\left(q,r,\tau\right)=\frac{q\exp\left(-\frac{\left(r-d_{g}\right)^{2}}{2\tau^{2}}\right)}{q\exp\left(-\frac{\left(r-d_{g}\right)^{2}}{2\tau^{2}}\right)+(1-q)\exp\left(-\frac{r^{2}}{2\tau^{2}}\right)}$
(13)

where $q$ is the prior probability of entry $s$ being equal to one. As its
name suggests, $\hat{s}_{g}\left(q,r,\tau\right)$ is equal to the mean of $s$
under the aforementioned Gaussian approximation $r\sim d_{g}s+\tau\zeta$. This
approach performs well for the single-class URA problem, especially in view of
its low complexity.

The PME denoiser found in [21] employs uninformative priors based on the known
sparsity level of $\mathbf{s}$. The approach was subsequently improved after
realizing that priors can be enhanced by performing one round of BP on the
factor graph of the outer LDPC code [22]. Specifically, in this latter case,
$\boldsymbol{\lambda}_{\ell}$ is initialized based on the PME of the
corresponding entries in $\mathbf{s}$. This vector estimate is then refined
using (4) and (5). Finally, the distribution afforded by (6) is then utilized
as priors for a final round of PME computation, leading to the output of the
denoiser. Details are available in [22] and, as such, we do not reproduce them
in this article. Conceptually, the enhanced version provides a means to embed
the local structure of the outer code in the AMP iterate. Accordingly, it can
improve performance significantly while adding little to the computational
complexity of the CCS approach.

The dynamic PME denoiser can be adapted to the current context, with its
multiple classes. We propose the following candidate implementation. As in
(10), we use two instances of the denoiser, one for each group,

$\boldsymbol{\eta}_{g}^{(t)}(\mathbf{r}_{g})\quad g\in\\{1,2\\}.$ (14)

These functions are the dynamic PME denoiser defined by

$\boldsymbol{\eta}_{g}^{(t)}(\mathbf{r}_{g})=\hat{\mathbf{s}}_{g,1}(\mathbf{r}_{g},\tau_{t})\cdots\hat{\mathbf{s}}_{g,L_{g}}(\mathbf{r}_{g},\tau_{t}).$
(15)

The number of sections in (15) matches the number of symbols produced by the
LDPC code in $\mathcal{C}_{g}$ (see Fig. 1). Furthermore, individual entries
are given by

$\hat{\mathbf{s}}_{g,\ell}(\mathbf{r}_{g},\tau_{t})=\left(\hat{s}_{g}\left(\mathbf{q}(\ell,k),\mathbf{r}(\ell,k),\tau_{t}\right):k\in
0,\ldots,m_{g}-1\right),$

where $\\{\mathbf{q}(\ell,k)\\}$ are the BP extrinsic estimate of (6) employed
as prior probabilities in the PME.

An important subtlety in extending previous results to the current, more
elaborate scenario is that $\tau_{t}$ is shared by the two parallel denoisers
$\boldsymbol{\eta}_{1}^{(t)}$ and $\boldsymbol{\eta}_{2}^{(t)}$. It must
therefore be computed (or evaluated) through the joint state evolution,
leading to

$\begin{split}\tau_{t+1}^{2}=\sigma^{2}+\lim_{n\rightarrow\infty}\frac{1}{n}\mathbb{E}\left\|\mathbf{D}\boldsymbol{\eta}^{(t)}\left(\begin{bmatrix}d_{1}\mathbf{s}_{1}\\\
d_{2}\mathbf{s}_{2}\end{bmatrix}+\tau_{t}\boldsymbol{\zeta}_{t}\right)-\begin{bmatrix}d_{1}\mathbf{s}_{1}\\\
d_{2}\mathbf{s}_{2}\end{bmatrix}\right\|^{2},\end{split}$

where $\mathbf{D}=\operatorname{diag}(d_{1}\mathbf{I},d_{2}\mathbf{I})$
accounts for the signal amplitudes and $\sigma^{2}$ is the noise variance in
(1). In practice, $\tau_{t}$ is often approximated by
$\tau_{t}^{2}\approx\|\mathbf{z}^{(t)}\|^{2}/n$. A second nuance in AMP for
multi-class URA is the computation of the Onsager correction. To get the
proper form, we note that

$\operatorname{div}\mathbf{D}\boldsymbol{\eta}^{(t)}(\mathbf{r})=d_{1}\operatorname{div}\boldsymbol{\eta}_{1}^{(t)}(\mathbf{r}_{1})+d_{2}\operatorname{div}\boldsymbol{\eta}_{2}^{(t)}(\mathbf{r}_{2}).$
(16)

Adapting results from [7, Prop. 8], we then obtain

$\operatorname{div}\mathbf{D}\boldsymbol{\eta}^{(t)}(\mathbf{r})=\frac{1}{\tau_{t}^{2}}\left(\big{\|}\mathbf{D}^{2}\boldsymbol{\eta}^{(t)}(\mathbf{r})\big{\|}_{1}-\big{\|}\mathbf{D}\boldsymbol{\eta}^{(t)}(\mathbf{r})\big{\|}^{2}\right).$
(17)

This compact form takes advantage of the identities

$\displaystyle\big{\|}\mathbf{D}\boldsymbol{\eta}^{(t)}(\mathbf{r})\big{\|}^{2}$
$\displaystyle=\big{\|}d_{1}\boldsymbol{\eta}_{1}^{(t)}(\mathbf{r}_{1})\big{\|}^{2}+\big{\|}d_{2}\boldsymbol{\eta}_{2}^{(t)}(\mathbf{r}_{2})\big{\|}^{2}$
$\displaystyle\big{\|}\mathbf{D}^{2}\boldsymbol{\eta}^{(t)}(\mathbf{r})\big{\|}_{1}$
$\displaystyle=\big{\|}d_{1}^{2}\boldsymbol{\eta}_{1}^{(t)}(\mathbf{r}_{1})\big{\|}_{1}+\big{\|}d_{2}^{2}\boldsymbol{\eta}_{2}^{(t)}(\mathbf{r}_{2})\big{\|}_{1}.$

These results crucially assume that the number of BP iterations on each LDPC
factor graph is strictly less than its shortest cycle. The computation of the
Onsager correction would become more involved otherwise. In practice, this is
not an issue because every AMP composite step entails the initiation of the
local observations followed by a single round of BP on each graph. Expression
(17) can be substituted as the Onsager term in (9), thus completing the
description of the AMP algorithm for the two-class setting.

###  III-C Disambiguation

Once the AMP algorithm has converged, the messages embedded within
$\hat{\mathbf{s}}_{1}$ and $\hat{\mathbf{s}}_{2}$ still need to be
disentangled. The presence of the LDPC-based message passing within the
denoising functions encourages local consistency, but may fall short of
decoding. To address this issue, it suffices to run a version of the outer
decoding algorithm first introduced in [7]. Starting from root sections, the
algorithm keeps track of all possible paths. A message is added to the output
list when it fulfills all the factors of the outer code. This step is somewhat
standard, as it is done separately for $\mathcal{C}_{1}$ and
$\mathcal{C}_{2}$. Additional details can be found in [7, 9].

##  IV Practical Considerations and Performance

As mentioned above, AMP is a powerful conceptual framework for the design and
analysis of efficient algorithms. Still, in practice, it can be beneficial to
deviate slightly from theoretical underpinnings, especially for exceedingly
large spaces. Certain decisions can greatly reduce computational complexity.

###  IV-A Implementation Details

One modification commonly found in the CCS literature is to create
$\mathbf{A}$ by sampling rows of a large Hadamard matrix (except the row
composed of all ones), rather than generate it randomly. Empirical evidence
suggests that this does not significantly affect performance, while
circumventing the need to store a large matrix. Furthermore, the matrix
multiplication steps in (9) and (11) can then be performed using the fast
Walsh-Hadamard transform.

For the problem at hand, we could create $\mathbf{A}$ in a similar manner.
Yet, inspecting (11) and (14), we gather that $\mathbf{r}_{1}^{(t)}$ and
$\mathbf{r}_{2}^{(t)}$ are employed separately within the denoiser. It
therefore suffices to produce these sub-vectors independently. Thus,
$\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ can be generated in a disjoint fashion,
possibly each through the sampling of a matrix amenable to fast transform
techniques. This approach retains the structural properties of the problem,
and lowers the computational load slightly. Nevertheless, it is important that
$\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ maintain good incoherence properties.
Natural candidates for this step include standard Hadamard matrices, but also
extend to matrices constructed using second-order Reed-Muller functions [31].
These latter matrices feature good correlation properties and they have
appeared in the context of single-class CCS in the past [6], although their
use therein is quite different.

###  IV-B Numerical Simulations

We compare the performance of the proposed multi-class approach against two
baselines for multi-class CCS. The first implementation executes (group)
_treating interference as noise_ (TIN). That is, a single-class recovery
algorithm for $\mathcal{C}_{1}$ is applied to $\mathbf{y}$; similar steps are
followed for $\mathcal{C}_{2}$, again using $\mathbf{y}$ as the input. This
scheme would arise naturally if the two device groups were assigned to
distinct access points that are not cooperating, but nevertheless share a same
spectral band. The second multi-class approach is group _successive
interference cancellation_ (SIC). Therein, a single-class recovery algorithm
for $\mathcal{C}_{1}$ is applied to $\mathbf{y}$, which yields estimate
$\hat{\mathbf{s}}_{1}$. Interference from group 1 is then subtracted from
observation vector $\mathbf{y}$, leading to updated observation
$\mathbf{y}_{2}=\mathbf{y}-\mathbf{A}_{1}\hat{s}_{1}$. A single-class
algorithm for $\mathcal{C}_{2}$ is subsequently used on the refined
observation $\mathbf{y}_{2}$. The third option is the novel multi-class CCS-
AMP algorithm introduced in this article. It can be categorized as joint
decoding (JD).

The system parameters for these three candidate schemes are identical. Group
sizes are taken to be $K_{1}=K_{2}=25$ active devices. The payloads are
$w_{1}=128$ and $w_{2}=96$ bits. The rate of the outer codes are $1/2$ and
$3/8$ for the first and second class. The message from every device is encoded
into a block of $n=38400$ channel uses. The _energy-per-bit_ for both groups
is identical; and it is denoted by ${E_{\mathrm{b}}}/{N_{0}}$: this implies
that devices in group 1 are using more energy, as they are transmitting more
information bits. The factor graphs for the outer code employed in our
simulations are analogous to the designs in [22]. Information messages are
partitioned into fragments of 16 bits and, hence, the alphabet size of the
outer code in both cases is $2^{16}$. The factor graphs have parameters
$L_{1}=L_{2}=16$. Both $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ are formed by
sampling rows of Hadamard matrices (excluding rows of all ones).

The algorithm performs one round of BP, after reinitialization, per AMP
composite step. Under the parameters above, the AMP algorithm converges
rapidly. The disambiguation is subsequently performed on the factor graph of
the outer code. The $K_{g}$ most promising codewords are returned as output of
the last step. Results do not use phased decoding where top candidates are
identified and peeled before repeating the overall CCS-AMP decoding process.
This approach provides a slight improvement in performance, but comes at a
substantial cost in terms of computational complexity. The three
implementations described in this section feature essentially equivalent
computational complexity.

$1.8$$2$$2.2$$2.4$$2.6$$2.8$$10^{-1}$$10^{0}$$E_{\mathbf{b}}/N_{0}$ (dB)Per-
User Error Rate $P_{\mathrm{e}}$Group 1: TINGroup 1: SICGroup 1: JDGroup 2:
TINGroup 2: SICGroup 2: JD

Figure 3: This plot shows a comparison of per-user error rate for the two
groups under the three candidate implementations. The proposed novel multi-
class URA algorithm performs best at essentially no additional computation
load.

Illustrative numerical results are summarized in Fig. 3, where per-user
probabilities of error are plotted for both groups and the three candidate
implementations as a function of energy-per-bit ${E_{\mathrm{b}}}/{N_{0}}$.
Every point on this graph is averaged over $100$ trials for statistical
accuracy. As expected, group SIC outperforms group TIN; yet we emphasize that
the group $1$, which is decoded first in SIC, experiences the exact same
performance under these two schemes. Moreover, the multi-class CCS-AMP
algorithm performs significantly better than the previous two schemes. This is
hardly surprising because joint decoding typically outperforms sequential
decoding schemes or TIN-based approaches. However, we emphasize two important
aspects of our findings. First, the form of the Onsager in the AMP iterate
retains a simple mathematical form despite the presence of the two classes and
the coupled state evolution equation. Second and more importantly, the greater
performance of joint decoding comes at essentially no additional complexity.
The same operations must be performed in sequence for all three
implementations. This particular aspect is somewhat surprising because joint
decoding schemes are typically much more demanding from a computational point
of view. In closing, it is worth noting that Fig. 3 contains no benchmarks,
other than those produced within this work. This is a byproduct of the fact
that standard demixing algorithms are not meant to work at dimensions on the
order of $2^{96}$ or $2^{128}$.

##  V Conclusion

In this paper, an extension of coded compressed sensing to the multi-class
unsourced random access scenario is studied. A joint decoding approach is
proposed to concurrently decode messages emanating from multiple classes of
users, leveraging insights from compressed demixing. Within the framework of
approximate message passing (AMP), the computational complexity of the
proposed scheme is the same as that of alternate implementations such as
treating interference as noise and successive interference cancellation.
However, these other schemes perform poorly compared to concurrent multi-class
decoding. The ability to decode classes of device jointly with essentially no
added complexity is surprising and should be embraced. Our findings provide a
blueprint for generic demixing in very high dimensional spaces, which forms a
promising avenue for future research.

## References

  * [1] Liang Liu, Erik G. Larsson, Wei Yu, Petar Popovski, Cedomir Stefanovic, and Elisabeth De Carvalho,  “Sparse signal processing for grant-free massive connectivity: A future paradigm for random access protocols in the Internet of Things,”  IEEE Signal Processing Mag., vol. 35, no. 5, pp. 88–99, 2018.
  * [2] Andrea Munari, Federico Clazzer, Osvaldo Simeone, and Zoran Utkovski,  “Grant-free access for IoT in beyond-5G systems: The potential of receiver diversity,”  in 6G Wireless Summit. IEEE, 2020, pp. 1–5.
  * [3] Yury Polyanskiy,  “A perspective on massive random-access,”  in Proc. Int. Symp. on Information Theory, 2017, pp. 2523–2527.
  * [4] Or Ordentlich and Yury Polyanskiy,  “Low complexity schemes for the random access Gaussian channel,”  in Proc. Int. Symp. on Information Theory, 2017, pp. 2528–2532.
  * [5] Evgeny Marshakov, Gleb Balitskiy, Kirill Andreev, and Alexey Frolov,  “A polar code based unsourced random access for the Gaussian MAC,”  in Proc. Vehicular Tech. Conf. IEEE, 2019, pp. 1–5.
  * [6] Robert Calderbank and Andrew Thompson,  “Chirrup: a practical algorithm for unsourced multiple access,”  Information and Inference: A Journal of the IMA, vol. 9, no. 4, pp. 875–897, 2020.
  * [7] Vamsi K. Amalladinne, Jean-Francois Chamberland, and Krishna R. Narayanan,  “A coded compressed sensing scheme for unsourced multiple access,”  IEEE Trans. Inform. Theory, vol. 66, no. 10, pp. 6509–6533, October 2020.
  * [8] Alexander Fengler, Peter Jung, and Giuseppe Caire,  “SPARCs and AMP for unsourced random access,”  in Proc. Int. Symp. on Information Theory. IEEE, 2019, pp. 2843–2847.
  * [9] V. K. Amalladinne, A. K. Pradhan, C. Rush, J.-F. Chamberland, and K. R. Narayanan,  “On approximate message passing for unsourced access with coded compressed sensing,”  in Proc. Int. Symp. on Information Theory. 2020, pp. 2995–3000, IEEE.
  * [10] Gustavo Kasper Facenda and Danilo Silva,  “Efficient scheduling for the massive random access Gaussian channel,”  IEEE Trans. on Wireless Comm., vol. 19, no. 11, pp. 7598–7609, 2020\.
  * [11] Alexis Decurninge, Ingmar Land, and Maxime Guillaud,  “Tensor-based modulation for unsourced massive random access,”  arXiv preprint arXiv:2006.06797, 2020.
  * [12] Volodymyr Shyianov, Faouzi Bellili, Amine Mezghani, and Ekram Hossain,  “Massive unsourced random access based on uncoupled compressive sensing: Another blessing of massive MIMO,”  arXiv preprint arXiv:2002.03044, 2020.
  * [13] Allen Hao, Stefano Rini, Vamsi Amalladinne, Asit Kumar Pradhan, and Jean-Francois Chamberland,  “An exploration of the heterogeneous unsourced MAC,” 2020.
  * [14] Michael B McCoy and Joel A Tropp,  “Sharp recovery bounds for convex demixing, with applications,”  Foundations of Computational Mathematics, vol. 14, no. 3, pp. 503–567, 2014.
  * [15] Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders,  “Atomic decomposition by basis pursuit,”  SIAM Review, vol. 43, no. 1, pp. 129–159, 2001.
  * [16] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein,  “Distributed optimization and statistical learning via the alternating direction method of multipliers,”  Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.
  * [17] Chinmay Hegde and Richard G. Baraniuk,  “Signal recovery on incoherent manifolds,”  IEEE Trans. Inform. Theory, vol. 58, no. 12, pp. 7204–7214, 2012\.
  * [18] Dennis Amelunxen, Martin Lotz, Michael B. McCoy, and Joel A. Tropp,  “Living on the edge: Phase transitions in convex programs with random data,”  Information and Inference, vol. 3, no. 3, pp. 224–294, 2014.
  * [19] Michael B. McCoy, Volkan Cevher, Quoc Tran Dinh, Afsaneh Asaei, and Luca Baldassarre,  “Convexity in source separation: Models, geometry, and algorithms,”  spmag, vol. 31, no. 3, pp. 87–95, 2014.
  * [20] Vamsi K. Amalladinne, Avinash Vem, Dileep Kumar Soma, Krishna R. Narayanan, and Jean-Francois Chamberland,  “A coupled compressive sensing scheme for unsourced multiple access,”  in Proc. Intl. Conf. Acoustics Speech and Sig. Proc. IEEE, 2018, pp. 6628–6632.
  * [21] Alexander Fengler, Peter Jung, and Giuseppe Caire,  “SPARCs for unsourced random access,”  arXiv preprint arXiv:1901.06234, 2019.
  * [22] Vamsi K. Amalladinne, Asit Kumar Pradhan, Cynthia Rush, Jean-Francois Chamberland, and Krishna R. Narayanan,  “Unsourced random access with coded compressed sensing: Integrating AMP and belief propagation,”  arXiv preprint arXiv:2010.04364, 2020.
  * [23] Antony Joseph and Andrew R. Barron,  “Fast sparse superposition codes have near exponential error probability for $R<C$,”  IEEE Trans. Inform. Theory, vol. 60, no. 2, pp. 919–942, 2013.
  * [24] Cynthia Rush, Adam Greig, and Ramji Venkataramanan,  “Capacity-achieving sparse superposition codes via approximate message passing decoding,”  IEEE Trans. Inform. Theory, vol. 63, no. 3, pp. 1476–1500, 2017\.
  * [25] Ramji Venkataramanan, Sekhar Tatikonda, and Andrew Barron,  “Sparse regression codes,”  Foundations and Trends in Communications and Information Theory, vol. 15, no. 1-2, pp. 1–195, 2019.
  * [26] Alban Goupil, Maxime Colas, Guillaume Gelle, and David Declercq,  “FFT-based BP decoding of general LDPC codes over Abelian groups,”  IEEE Trans. on Commun., vol. 55, no. 4, pp. 644–649, 2007.
  * [27] Frank R. Kschischang, Brendan J Frey, and Hans-Andrea Loeliger,  “Factor graphs and the sum-product algorithm,”  IEEE Trans. Inform. Theory, vol. 47, no. 2, pp. 498–519, 2001.
  * [28] Raphael Berthier, Andrea Montanari, and Phan-Minh Nguyen,  “State evolution for approximate message passing with non-separable functions,”  Information and Inference: A Journal of the IMA, vol. 9, no. 1, pp. 33–79, 2020.
  * [29] David L. Donoho, Arian Maleki, and Andrea Montanari,  “Message-passing algorithms for compressed sensing,”  Proceedings of the National Academy of Sciences, vol. 106, no. 45, pp. 18914–18919, 2009.
  * [30] Mohsen Bayati and Andrea Montanari,  “The dynamics of message passing on dense graphs, with applications to compressed sensing,”  IEEE Trans. Inform. Theory, vol. 57, no. 2, pp. 764–785, 2011.
  * [31] Stephen D. Howard, A. Robert Calderbank, and Stephen J. Searle,  “A fast reconstruction algorithm for deterministic compressive sensing using second order Reed-Muller codes,”  in Conference on Information Sciences and Systems. IEEE, 2008, pp. 11–15.


# Leveraging Language Models to Detect Greenwashing

Avalon Vinella1∗ Margaret Capetz1∗ Rebecca Pattichis1∗
Christina Chance1∗ Reshmi Ghosh2
1 University of California, Los Angeles
2 Microsoft Research
∗ Equal Author Contribution
{avinella, mcapetz17, pattichi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In recent years, climate change repercussions have increasingly captured
public interest. Consequently, corporations are emphasizing their
environmental efforts in sustainability reports to bolster their public image.
Yet, the absence of stringent regulations in review of such reports allows
potential greenwashing. In this study, we introduce a novel methodology to
train a language model on generated labels for greenwashing risk. Our primary
contributions encompass: developing a mathematical formulation to quantify
greenwashing risk, a fine-tuned ClimateBERT model for this problem, and a
comparative analysis of results. On a test set comprising of sustainability
reports, our best model achieved an average accuracy score of 86.34% and F1
score of 0.67, demonstrating that our methods show a promising direction of
exploration for this task.

##  1 Introduction

As climate change has moved further to the forefront of social and political
discussions, corporations have an increased stake in presenting themselves as
environmentally conscious to investors, government agencies, and consumers.
Greenwashing is broadly defined as when an entity creates an overly positive
corporate image through selective disclosure Bowen and Aragon-Correa (2014).
In other words, authors of corporate impact reports may present misleading
information about their sustainability efforts in order to maintain public
favor and profit, rather than being transparent about their potentially
negative environmental impact. Natural Language Processing (NLP) can aid with
greenwashing detection in the dense language of corporate sustainability
reports, and this potential has been recently explored in previous studies.
However, a major obstacle is the relative subjectivity of a definition for
greenwashing, which is difficult to quantify. In this work, we present
methodology for generating groundtruth labels for greenwashing risk based on
climate specific attributes. We validate our results with gathered metrics of
reported corporate carbon emissions from our collaborators.

##  2 Related Work

We build our work off of previous studies and models that have addressed
climate related issues in an NLP context and defined greenwashing.

NLP for Climate Work. Many works utilize NLP methods to assess environmental
risk and impact. For example, Kang and Kim (2022) and Polignano et al. (2022)
employ sentence similarity, sentiment classification, and text extraction to
assess corporate sustainability reports. Recently, Kim et al. (2023) created a
search service for greenwashing pattern detection and evaluation system.

ClimateBERT. ClimateBERT is a BERT architecture-based model, more specifically
RoBERTa-based, pre-trained on a corpus of climate-related texts, which has
improved performance on various climate-related downstream tasks (2022).
ClimateBERT has been fine-tuned for climate-related tasks such as text
detection, sentiment classification, etc Bingler et al. (2023). We build off
of this model in our work.

Definition of Greenwashing. (2011) defined Greenwashing as the intersection of
a firm’s poor environmental performance, and positive communication about
environmental performance. Additionally, (2011-03) defines corporate
greenwashing as "the selective disclosure of positive information about a
company’s environmental or social performance while withholding negative
information on these dimensions." Milgrom (1992). This suggests that companies
tend to view sustainability information as optional to disclose based on
whether or not it will positively impact them.

##  3 Methodology

###  3.1 Data

ClimateBERT Data. The ClimateBERT model was pre-trained on 2 million samples
of news articles, research abstracts, and corporate climate reports (years
2015 to 2020) Webersinke et al. (2022). The public ClimateBERT dataset
includes paragraphs sourced from firm’s financial disclosures Bingler et al.
(2023). We use this data for fine-tuning ClimateBERT for greenwashing risk
classification by extracting text from the corpus that was identified as
climate-related text from all dataset splits. Then, we re-split into training
and validation sets of 80/20 percent (1,056/264 examples), respectively.

Open for Good Data. Open for Good is a project at the Center for Impact at the
University of California, Los Angeles Anderson School of Management tackling
transparent reporting of corporate sustainability Delmas et al. (2023). We use
their carbon emissions environmental impact data, which is classified into
Scope 1, Scope 2, and Scope 3 by the US Department of Energy. Scope 1
emissions are company’s direct production of a good, Scope 2 emissions include
indirect emissions for purchased electricity, and Scope 3 emissions include
emissions from the value chain of the product’s life-cycle. We use these
sustainability reports as our test set.

###  3.2 Generating Ground Truth for Greenwashing Risk

While greenwashing has consistent implications, defining its qualities can be
subjective and context-dependent. In addition, given the legal significance of
sustainability reports, there are often instances of evasive language, such as
“The Group is not aware of any noise pollution that could negatively impact
the environment.” These cases are not necessarily greenwashing, as they are
not trying to positively re-frame some adverse action, but more so legal
hedging Vass (2017).

We have identified the following characteristics typical of greenwashing
language in corporate sustainability reports: 1. Absence of explicit climate-
related commitments and actions, 2. Use of non-specific language, 3. overly
optimistic sentiment, and 4. lack of evasive or hedging terms. Lueg and Lueg
(2020) propose a method to model the relationship between a company’s market
risk and its sustainability performance, including implementing a linear
combination to assess various types of risk based on company’s attributes. We
therefore adopt a similar approach of using a linear combination of specific
metrics to determine greenwashing risk in sustainability-related text.

We present a calculation for greenwashing risk as a linear combination of the
attributes outlined above. Climate commitment and specificity are collected
from fine-tuned ClimateBERT models. While ClimateBERT has been fine-tuned for
climate sentiment, due to their ternary labeling scheme, we instead use the
pre-trained SiBERT model Hartmann et al. (2023) for a binary sentiment label.
This is more aligned with the standard "positive" and "negative" sentiments.
Lastly, we use a comprehensive list of phrases that represent legal hedging
vocabulary (see Appendix A) to identify instances in the data wherein there
was usage of legal hedging language.

To establish a mathematical formulation for greenwashing, we sourced 10
representative exemplars from the ClimateBERT dataset, which were annotated
and categorized by subject-matter experts. Employing these instances in a
least squares linear regression analysis, we leveraged the four previously
delineated metrics (i.e., sentiment, climate specificity, commitment, and
hedging) against the labels derived from expert annotation.

Additionally, utilizing insights around definitions of greenwashing the
metrics we are using, we iteratively developed another linear combination for
comparison. We additionally provided a threshold to fit the binary nature of
the task.

##  4 Results

Groundtruth Labeling. The linear combination where coefficients were found
through least squares is presented below. Because our final label is binary,
we pass the raw output of this equation through a sigmoid, and set a threshold
of 0.67 (comes from data-specific observations). The equation exhibits a
residual of 0.71. This relatively high residual suggests a sub-optimal fit,
which could be attributed to the limited sample size used for the regression.
Additionally, potential discrepancies between input metrics and output values
may play a role, an aspect we will explore further in the subsequent
limitations section.

$\displaystyle y=0.71*sentiment+0.14*commitment-0.86*specificity-0.71*hedging$

The second equation, developed from our understanding of greenwashing and
metric definitions is below. We received an agreement of 6 out of 10 of the
expert human-annotated examples in which non-impact report samples accounted
for some incorrect and unclear labels (see Discussion section).

$\displaystyle x=-sentiment+commitment+specificity+hedging\text{,
}y=\mathbbm{1}[x\leq 0]$

Fine-Tuning ClimateBERT for Greenwashing Risk. We fine-tuned the ClimateBERT
model for greenwashing risk detection on data labeled using the first
equation, and experimented with freezing all but the classification head vs.
fine-tuning all layers. We used the Adam Optimizer with a linear learning rate
of 5e-5, 0 warm up steps, and 70 epochs. Each experiment ran a total of 10
times (different seeds) to report mean and standard deviation (see Appendix B
for plots of these results).

We found that not freezing the RoBERTa layers resulted in higher mean
validation accuracy and F1 scores as well as a smaller deviance, suggesting
that the non-frozen model may be slightly more robust than the frozen
configuration. This also indicates that our task is specific enough to benefit
from training all layers in order to improve the accuracy of ClimateBERT, and
that this approach of fine-tuning would be appropriate for determining
greenwashing risk.

Model |  | RoBERTa
---
Layer
| Validation
---
Accuracy
| F1
---
Score
1 | Frozen |  66.52 $\pm$ 2.39 |  0.68 $\pm$ 0.01
2 | Not frozen |  70.78 $\pm$ 2.03 |  0.70 $\pm$ 0.02

Table 1:  Validation results for the frozen vs. unfrozen ClimateBERT fine-
tuning experiments. This table reports the mean accuracy and F1 scores with
their respective standard deviation.

OOD Evaluation. We select the run that falls in the mode of accuracy
(validation accuracy shows up at least twice) and has the highest F1 score
within that subset (see Appendix B). Our OOD test set (from the Open for Good
Data) come from APA Corporation, Devon Energy, and Diamondback Energy in the
oil and gas sector and Autodesk, NVIDIA, and ServiceNow in the tech sector.
The sections relevant to environmental impact were manually identified and
extracted. To accommodate the maximum input length of the models, these
sections were split into paragraph-aligned chunks determined by a character
count upper-limit before being tokenized. Finally, the report’s overall label
comes from a majority voting of its intermediate climate-related paragraphs.

Both models correctly predicted a negative greenwashing risk for each report
with significantly higher accuracy than the validation set; however, the F1
variation is much higher (Table 2). It is likely that this is due to an
imbalance of positive versus negative samples in each report and how they were
chunked (see Discussion).

Company | Model Config 1 | Model Config 2
---|---|---
Acc. | F1 | Acc. | F1
DiamondBack | 80.70 | 0.62 | 82.46 | 0.67
Devon | 86.67 | 0.65 | 83.33 | 0.63
APA | 86.49 | 0.76 | 85.14 | 0.74
Autodesk | 96.49 | 0.83 | 85.96 | 0.56
ServiceNow | 89.47 | 0.50 | 89.47 | 0.67
NVIDIA | 77.78 | 0.43 | 91.67 | 0.72

Table 2:  Accuracy and F1 scores reported for each company’s climate impact
statement.

##  5 Discussion

There are potential improvements in our ground truth generation. For example,
the ClimateBERT models are preliminary, and our ground truth for our data is
dependent on the accuracy of these models. During human annotation of our sub
sample, for example, we noticed the following examples had matching attributes
(from ClimateBERT) but were given opposite labels by our domain-experts.

Text | Label
---|---
| The Group faces many other r isks which, although important and subject to
regular review,
---
have been assessed as less significant and are not listed here. These include,
for example,
natural catastrophe and business interruption risks and certain financial
risks….
1
| The cement industry is associated with high CO2 intensity and LafargeHolcim
is exposed
---
to a variety of regulatory frameworks to reduce emissions, some of which may
be
under revision. These frameworks can affect the business activities of
LafargeHolcim.
In addition, a perception of the sector as a high emitter could impact our
reputation,
thus reducing our attractiveness to investors, employees and potential
employees.
0

Table 3: Two examples that had matching attributes but opposite greenwashing
risk labels. Attribute values were 1 for sentiment and 0 for the rest.

During annotation, experts noted that while the first example was greenwashing
due to obfuscation, the second was not because they are admitting to bad
performance. It is these type of nuances that the intermediate attributes may
fail to capture. Ultimately, we would prefer to have the ground truth
extensively expert-validated.

Another indicator of greenwashing to consider in our test set is the reported
carbon emissions (Scope 1 and 2) for each company relative to its industry’s
average. If a company has a high risk of greenwashing, their emissions may be
higher than average in their domain relative to company size, which we
approximate using annual revenue (as in Appendix C). Given this data, there
might be reason to intuit that APA, whose emissions are incredibly high
relative to similar companies, has a risk of using greenwashing language, but
neither our ground truth equation nor model reflect this. While it is not
definitive evidence, this discrepancy could suggest some error in our
formulations.

To avoid using test data that ClimateBERT had seen in it’s training, we only
used sustainability reports from 2021 and onwards. Because of this, we were
limited by the amount of data in the Open For Good data set that met this
requirement, skewing our distribution of labels. Moreover, the ClimateBERT
training corpus contains data that is not necessarily relevant for the task of
greenwashing detection, such as third-party audits or reviews, and may add
noise to our training and validation sets.

Lastly, we are limited to analyzing the sustainability reports paragraph-by-
paragraph given the large length of the reports. This may impact the final
score in both the ground truth generation and model results due to the
possible loss of context, and these labels may also be impacted depending on
how the text is parsed.

##  6 Conclusion

We present a novel method of quantitatively defining greenwashing language and
utilize it to fine-tune ClimateBERT, a BERT model trained on climate-related
textual data, to identify risks of greenwashing in corporate sustainability
reports. Our best model achieved an average accuracy score of 86.34% and F1
score of 0.67 on our test set of sustainability reports, which indicates
further room for research and improvement in both the ground truth label
formulation and training of the language model. We hope that these
contributions and results will further aid the development of NLP tools for
greenwashing detection.

## Ethics Statement

Due to the limitations of our work as well as the inconsistent definition of
greenwashing, we utilize the language of potential risk of greenwashing rather
than categorizing something as greenwashing itself. With this understanding,
our work has no clear ethical concerns as we aren’t producing data that can be
mistreated and have no potential negative applications to the field.

## Acknowledgements

Thank you to our collaborators at the UCLA Anderson School of Management for
sharing their greenwashing insights with us and providing us with the Open for
Good data, specifically Kelly Clark and Magali Delmas. Thank you to the
ClimateBERT team for providing a pre-trained model and data for our project.

## References

  * Bingler et al. [2023] Julia Bingler, Mathias Kraus, Markus Leippold, and Nicolas Webersinke.  How cheap talk in climate disclosures relates to climate initiatives, corporate emissions, and reputation risk.  Working paper, Available at SSRN 4000708, 2023.
  * Bowen and Aragon-Correa [2014] Frances Bowen and Juan Antonio Aragon-Correa.  Greenwashing in corporate environmentalism research and practice: The importance of what we say and do.  _Organization & Environment_, 27(2):107–112, 2014.  doi: 10.1177/1086026614537078.
  * Delmas et al. [2023] M. Delmas, K. Clark, and T. Timmer.  Open for good initiative: Corporate sustainability database, 2023.  URL https://www.anderson.ucla.edu/about/centers/impactanderson/open-for-good-transparency-index.
  * Delmas and Burbano [2011] Magali A. Delmas and Vanessa Cuerel Burbano.  The drivers of greenwashing.  _California Management Review_ , 54(1):64–87, 2011.  doi: 10.1525/cmr.2011.54.1.64.  URL https://doi.org/10.1525/cmr.2011.54.1.64.
  * Hartmann et al. [2023] Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp.  More than a feeling: Accuracy and application of sentiment analysis.  _International Journal of Research in Marketing_ , 40(1):75–87, 2023.  doi: https://doi.org/10.1016/j.ijresmar.2022.05.005.  URL https://www.sciencedirect.com/science/article/pii/S0167811622000477.
  * Kang and Kim [2022] Hyewon Kang and Jinho Kim.  Analyzing and visualizing text information in corporate sustainability reports using natural language processing methods.  _Applied Sciences_ , 12(11), 2022.  ISSN 2076-3417.  doi: 10.3390/app12115614.  URL https://www.mdpi.com/2076-3417/12/11/5614.
  * Kim et al. [2023] Ji Sue Kim, Jun Bo Sim, Yoon Ju Kim, Min Kyu Park, Se Jong Oh, and Ill Chul Doo.  Establishment of nlp-based greenwashing pattern detection service.  In Ji Su Park, Laurence T. Yang, Yi Pan, and Jong Hyuk Park, editors, _Advances in Computer Science and Ubiquitous Computing_ , pages 253–259, Singapore, 2023. Springer Nature Singapore.  ISBN 978-981-99-1252-0.
  * Lueg and Lueg [2020] Klarissa Lueg and Rainer Lueg.  Detecting green-washing or substantial organizational communication: A model for testing two-way interaction between risk and sustainability reporting.  _Sustainability_ , 12(6), 2020.  doi: https://doi.org/10.3390/su12062520.  URL https://www.mdpi.com/2071-1050/12/6/2520.
  * Lyon and Maxwell [2011-03] Thomas P Lyon and John W Maxwell.  Greenwash: Corporate environmental disclosure under threat of audit.  _Journal of economics & management strategy_, 20(1), 2011-03.  ISSN 1058-6407.
  * Milgrom [1992] Paul R. Milgrom.  _Economics, organization, and management /_.  Prentice-Hall,, Englewood Cliffs, N.J. :, 1992.  ISBN 0132246503.
  * Polignano et al. [2022] Marco Polignano, Nicola Bellantuono, Francesco Paolo Lagrasta, Sergio Caputo, Pierpaolo Pontrandolfo, and Giovanni Semeraro.  An NLP approach for the analysis of global reporting initiative indexes from corporate sustainability reports.  In _Proceedings of the First Computing Social Responsibility Workshop within the 13th Language Resources and Evaluation Conference_ , pages 1–8, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.csrnlp-1.1.
  * Vass [2017] Holly Vass.  Lexical verb hedging in legal discourse: The case of law journal articles and supreme court majority and dissenting opinions.  _English for Specific Purposes_ , 48:17–31, 2017.  doi: https://doi.org/10.1016/j.esp.2017.07.001.  URL https://www.sciencedirect.com/science/article/pii/S0889490617301989.
  * Webersinke et al. [2022] Nicolas Webersinke, Mathias Kraus, Julia Bingler, and Markus Leippold.  ClimateBERT: A Pretrained Language Model for Climate-Related Text.  In _Proceedings of AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges_ , 2022.  doi: https://doi.org/10.48550/arXiv.2212.13631.
  * WSJ [2023] WSJ.  Wall street journal markets, 2023.  URL https://www.wsj.com/market-data/quotes/company-list.

##  Appendix A ChatGPT Prompting Details

To generate the list of phrases and/or words that we use for tagging a piece
of text, we prompted the May 24 version of ChatGPT. The three turn dialogue
prompt used is in Table 5 as well as the generated hedging phrases. The
expanded list of hedging phrases provides more options to capture the nuances
of hedging language in a legal setting.This list is used in Python code for
various purposes, such as analyzing legal texts or generating automated
responses.

Note that this list is a bit broad, as identified by our collaborators. In
other words, there are some words that, if present in a text and only that
word/phrase is present, it shouldn’t necessarily be counted as legal hedging.
We discuss improving our greenwashing risk score in our limitations through
further hyper-parameter tuning, and this is one area that could benefit from
future development.

##  Appendix B Model Experiments

Figure 1: Accuracy and F1 scores of the frozen and non-frozen ClimateBERT
models during fine-tuning.

Figure 1 reports the accuracy and F1 scores for each of ten experiments done
for frozen and non-frozen training on ClimateBERT. Frozen model 6 and non-
frozen model 4 were used to predict the test set.

##  Appendix C Corporate GHG Emissions and Revenue

Table 4 reports each company’s Scope 1 and Scope 2 Carbon Dioxide Equivalents
(CO2e) emissions relative to the average emissions of their domain (oil or
tech) for that year, as well as their revenue in that same year Delmas et al.
[2023], WSJ [2023].

Company | Scope 1 CO2e | Scope 2 CO2e (relative to average) | Annual Revenue
---|---|---|---
Market | Location | Uncategorized
DiamondBack | -20 | - | +0 | - | 2.8
Devon | +6.9 | - | - | -0.36 | 4.8
APA | +31 | - | - | +1.3 | 4.0
Autodesk | -0.34 | -0.30 | -0.47 | - | 3.8
ServiceNow | -0.33 | -0.11 | -0.30 | - | 4.5
NVIDIA | -0.32 | +0.36 | +0.18 | - | 17
Table 4:  Each company’s reported Scope 1 and Scope 2 emissions measured in
hundred thousand metric tonnes, relative to the hundred thousand average
emissions of their domain (oil or tech) for that year, as well as their
revenue in billions of USD. User Generate a list of phrases or words that
represent deflection languages. Examples include: "not aware", "unsure"
---
ChatGPT 1. I’m not certain. 2. I can’t say for sure. 3. I’m not entirely
convinced. 4. It’s hard to say. 5. I’m not familiar with that. 6. I don’t have
all the information. 7. … User Generate a list of phrases or words that
represent deflection languages seen in legal language. Examples include: "not
aware", "unsure":
ChatGPT 1. Alleged 2. Purported 3. Supposed 4. Ostensibly 5. Arguably 6.
Presumably 7. Putatively 8. … User Can you generate the above list as a Python
list of string?
ChatGPT Certainly! Here’s the list of deflection phrases in legal language as
a Python list of strings: deflectionPhrases = [ "Alleged", "Purported",
"Supposed", "Ostensibly", "Arguably", "Presumably", "Putatively", "Seemingly",
"Reportedly", "Apparent", "So-called", "Allegedly", "Potentially", "Possibly",
"Putative", "Suggested", "Circumstantially", "Put forth", "By all accounts",
"Inferred"]
User Can you append more words to the previous list related to deflection
languages for the legal setting?
ChatGPT Certainly! Here’s an extended list of deflection phrases in legal
language:
deflectionPhrases = [ "Alleged", "Purported", "Supposed", "Ostensibly",
"Arguably", "Presumably", "Putatively", "Seemingly", "Reportedly", "Apparent",
"So-called", "Allegedly", "Potentially", "Possibly", "Putative", "Suggested",
"Circumstantially", "Put forth", "By all accounts", "Inferred", "Conceivably",
"Inconclusively", "Tentatively", "Speculatively", "Hypothetically", "In all
likelihood", "Plausibly", "Implied", "Indirectly", "Presumptively",
"Theoretically", "Presumed", "Conditionally", "Preliminarily",
"Provisionally", "Notoriously", "Put forward", "Understood", "It is said",
"Subject to interpretation", "Perceived", "In a sense", "To a degree",
"Assumed", "Putatively", "Vaguely", "Broadly", "Loosely", "Dubiously",
"Evidently", "Inferred", "Outwardly", "Reputedly", "Tacitly", "Unofficially",
"Putatively", "Ostensible", "Presumptuous", "Unconfirmed" ]
Table 5:  Three turn dialogue to generate the list of phrases and/or words
that we use for tagging a piece of text, we prompted the May 24 version of
ChatGPT.