# On the Open Prompt Challenge in Conditional Audio Generation

###### Abstract

Text-to-audio generation (TTA) produces audio from a text description,
learning from pairs of audio samples and hand-annotated text. However,
commercializing audio generation is challenging as user-input prompts are
often under-specified when compared to text descriptions used to train TTA
models. In this work, we treat TTA models as a “blackbox” and address the user
prompt challenge with two key insights: (1) User prompts are generally under-
specified, leading to a large alignment gap between user prompts and training
prompts. (2) There is a distribution of audio descriptions for which TTA
models are better at generating higher quality audio, which we refer to as
_“audionese”_. To this end, we rewrite prompts with instruction-tuned models
and propose utilizing text-audio alignment as feedback signals via margin
ranking learning for audio improvements. On both objective and subjective
human evaluations, we observed marked improvements in both text-audio
alignment and music audio quality.

Index Terms—  text-to-audio generation, prompt engineering, distributional
drift

##  1 Introduction

Text-to-audio (TTA) generation has witnessed significant advancements in
recent years, enabling the conversion of textual descriptions into high-
fidelity audio representations [1, 2]. TTA models have been trained using
paired data consisting of hand-annotated texts and corresponding audio
samples, leveraging neural approaches to learn the mapping between text and
audio.

Despite these advancements, scarcity in paired text-audio data has created an
inherent difficulty in synthesizing high-quality and coherent audio from text.
Creating text descriptions of general audio is considerably harder than
describing images [3]. MusicLM [4] outlines two challenges in creating music
prompts: (1) Expressing the essential features of acoustic scenes and music is
a complex task that cannot be easily accomplished using only a few words. (2)
The temporal dimension in audio introduces a structural complexity that
renders sequence-wide captions less effective as annotations compared to image
captions.

Fig. 1:  Alignment of open prompts: Our approach involves an LLM-based prompt
rewriter to align user (open) prompts to “audionese”, which is a distribution
of text capable of producing higher quality audio given the blackboxed text
encoder and audio decoder.

The problem is exacerbated when conditional audio models are put into actual
product usage – it results in misalignment111This train-test deviation has
long been studied in past alignment research [5, 6, 7, 8].  between user
prompts and annotated prompts, making it difficult for audio decoders to
synthesize audio samples that accurately represent the user intents. Such
prompt alignment is worse in musical domains – in line with previous studies
[4], we observed that majority of users lack the expertise to construct
musical prompts that are as descriptive as those in the training set (which
are typically annotated by domain experts). Therefore, (open) user prompts are
often too abstract and underspecified (e.g., _“pop song”_) compared to the
annotated, elaborate prompts that a TTA model is typically trained on (e.g.,
_“Moody keyboard and drum centric pop song featuring neo-R &B chordal
information and layered barbershop harmonies”_).

In this work, we provide a preliminary study of the open prompt challenge in
text-to-audio generation where we expose TTA models to out-of-distribution
user text prompts. We focus our efforts on musical audio (as opposed to speech
or other general acoustic stimuli) because of the aforementioned gap between
general user and expert prompts for music. We leverage instruction-tuned large
language models (LLMs) to perform prompt rewriting and improve upon the LLMs
with audio feedback and margin rank learning to increase their ability to
output prompts capable of producing higher quality audio. Overall, we observe
improvements over audio quality, text-audio alignments, and human preference.

##  2 Open Prompts Collection

Text-to-audio (TTA) models generate audio from a text description $x$ such as
“The pop rock music features a male voice singing.” TTA systems are designed
to generate a wide range of high dimensional audio signals $y$ by modeling the
learned compact latent space $Z$. Text encoders of TTA systems condition on
$x$ to sample latent code $z$, which the audio decoder uses as the prior to
sample $y$. This process hinges on the alignment between the learned parameter
$\theta$ of both text encoder and audio decoder. Thus, the problem of prompt
rewriting targets at converting text inputs $x$ into $x^{\prime}$ such that
they are closer to the _audionese_ and can thus better leverage $\theta$ to
produce high-fidelity audio samples (refer Figure 1). Here we define audionese
as the text distribution that produces the best audio quality metrics for a
given TTA model parameterized by parameters $\theta$, which only exists
theoretically as a result of the model training and has its root in input
perturbation and model complexity [9].

#### Out-of-distribution prompts.

For our study, we set up collections runs with $30$ non-expert users with
varying knowledge about music. We collected $300$ user input texts by asking
users to enter free-form text prompts for generating music. This resulted in a
wide range of musical prompt topics, many of which did not belong to musical
domains (e.g., we found prompts relating to cuisines, sports, and politics.)

#### Expert vs. user prompts.

We compared the linguistic complexity of expert and open prompts, by examining
their distributional difference along information density (or entropy)
$log(\frac{1}{P(w_{j}|t_{i})})$ [10]. Here, $w_{j}$ is $j^{th}$ token and
$t_{i}$ is the set of tokens in prompts.

Fig. 2:  Comparison of user and expert prompts: Plot of normalized CLAP and
information density scores where audio descriptive prompts range from expert-
annotated musical prompts to user-input prompts, which subsequently result in
various text-audio alignment (CLAP). The disparity is due to difference in
ease of annotation where expert-annotation are costly to obtain as opposed to
arbitrary open prompts. The dotted line represents processed versions of the
user prompts, rewritten by a finetuned LLM with the intent to push them closer
to the _audionese_.

User Input |  Instruction |  Enhanced Prompt
---|---|---
_Cupcake_ |  _Write a detailed description describing the music with instruments, tone, pitch, rhythm, atmosphere, style, and all the audio details._ |  _The music for the cupcake video is upbeat and cheerful, featuring a mix of bouncy electronic elements and live instruments such as guitar and drums. The tone is light and playful, with a catchy melody and a steady beat that evokes feelings of happiness and joy._
_Write a detailed description describing the music with instruments, tone, pitch, rhythm, atmosphere, style, and all the audio details._ |  _Rewrite the following prompts to be diverse and different_ |  _1\. Describe the instruments used in the music, including their tone, pitch, and rhythm. 2. Provide details about the overall sound and atmosphere of the music, including the style and genre. 3. Include any special effects or layers that enhance the music, such as layers of guitars or synths._

Table 1: The provided example represents a pair of instruction-input and
output generated by instruction-tuned LLMs. Here, the LM must infer a latent
document-level concept to generate coherent next tokens during pretraining. At
test time, in-context learning occurs when the LM also infers a shared latent
concept between examples in a prompt.

Further, we use the CLAP metric [11] as a proxy for alignment between text and
audio pairs. CLAP consists of language and audio encoders trained with a
contrastive learning objective to project audio and text descriptions into the
same latent space and can thus serve as a useful estimate of their alignment.
To verify this, we also conducted a quick study to correlate the CLAP metric
to human evaluations of text-audio alignment, finding significant correlations
at (r=0.35, p¡.05).

We then conducted analyses comparing CLAP scores to informational density of
the text prompt, summarized in Figure 2: (A) Relationship between information
density and CLAP is observed to be monotonically increasing, which suggests a
correlation between how much information is packed within each prompt and how
well it aligns with the generated audio. (B) There is a large disparity
between expert and open prompts for both CLAP and information density. Further
analysis shows that open prompts not only use a smaller vocabulary but the
average prompt length is also lower by $5$ tokens.

##  3 The Approach

We employ instruction-tuned large language models, i.e. FLAN-T5 variants [12]
and LaMini-LM [13] as base models $f(\cdot)$ (_Instruct_), and improve upon
them with audio feedback in the form of text-audio ranking. Each prompt $x$ is
passed into base model $f(x)$ to obtain a converted prompt $x^{\prime}$. Since
the base models are instruction-tuned, they possess generic instruction
capabilities that enabled them to rewrite user prompts with some level of
audio details. We employ this approach as the zero-shot approach
(_Instruct+0-shot_) of rewriting prompts. However, the ability for these
models to converse in _audionese_ without any finetuning is still largely
limited. In what follows we describe how supervised update is performed to
achieve this.

###  3.1 Open Prompt Rewriting with Feedback Alignment

#### Step 1: Audio feedback.

As earlier, we use the CLAP score as a pointwise metric to evaluate text-audio
alignment for each pair by computing a score in the range $[0,1]$. We first
generate a diverse set of enhanced prompts $x^{\prime}$ by employing the zero-
shot LLM approach (described above) on user prompts. This thus gives us pairs
of the original user prompt $x$ and the rewritten prompt $x^{\prime}$ and CLAP
scores of each which can then be used for ranking the items in each pair.

#### Step 2: Rank-based alignment.

For _Instruct+Feedback_ , we introduce the margin ranking loss, which
evaluates two types of prompts, denoted as $x$ and $x^{\prime}$. The variable
$y$ represents the ranking expectation, where a value of 1 for $y$ indicates
the desired ranking order where $x$ is ranked higher than $x^{\prime}$.
Conversely, a value of -1 indicates the desired ranking order $x^{\prime}$
higher than $x$. The loss is computed using the formula:
$\max(0,-y\cdot(x-x^{\prime})+m)$, where $m$ represents the margin, a constant
determining the minimum difference required between the ranked values to avoid
incurring a penalty. If the difference between $x$ and $x^{\prime}$ falls
below the margin, indicating a satisfactory ranking, then the loss is assumed
to be $0$ to signify its desired property. See sample generation at Table 2.

User Input | Enhanced Prompt (Instruct+Feedback)
---|---
bumpy jazz | The music is bumpy and jazzy
sad harp solo | The harp solo is sad and haunting
cozy warm hip hop beat | a hip hop beat with a warm atmosphere
fashion upbeat | The music is upbeat and energetic
gnarly beats that are fast and crazy | a sonic journey

Table 2: The provided example represents a pair of instruction-input and
output that is utilized to train instruction-tuned LLMs.

| LaMini-LM (125M) | LaMini-LM (1.5B) | FLAN-T5-small (80M) | FLAN-T5-base (250M) | FLAN-T5-large (780M)
---|---|---|---|---|---
Open Prompts | 0.0556 (100.00)
Instruct | 0.0123 (1.58) | 0.0232 (3.27) | 0.0744 (56.61) | 0.0680 (61.20) | 0.0701 (30.50)
Instruct+0-shot | 0.0175 (1.88) | 0.0299 (4.31) | 0.0747 (56.68) | 0.0693 (62.74) | 0.0763 (31.71)
Instruct+Feedback | 0.126 (1.32) | 0.115 (2.59) | 0.0739 (55.01) | 0.0769 (27.96) | 0.0809 (29.46)

Table 3:  Benchmarks showing the CLAP scores for various approaches including
the collected _open prompts_ , _Instruct_ with _0-shot_ and
_Instruct+Feedback_. We indicate SacreBLEU (%) with original prompt as
reference to show the extent of textual deviation. We also compute the CLAP
scores with the audio samples generated from test user prompts to show the
theoretical lower bound.

##  4 Experiments and Results

#### Configs.

In this work, we employ AudioLDM [14] to generate realistic speech and piano
music audio samples. AudioLDM uses the CLAP model [11] as the text encoder to
obtain the text embeddings and applies a diffusion model to predict the
quantized mel spectrogram features of the target audio. We avoid using LLMs
larger than 1.5B due to resource contraints as per realtime inference. Thus,
we use LaMini-LM-[125M, 1.5B] [13], which are decoder-only language models;
and FLAN-T5 are encoder-decoder based models [12] with sizes 250M, 780M, and
up to 3B due to empirical memory and latency constraints. Our codebase is
released and built with the Huggingface library [15]. Audio samples are
evaluated with CLAP [11] for automatic text-audio alignment and
subjective/objective human evaluation [14, 4] for audio quality assessment
based on human preference. We sampled $250$ expert-annotated prompts from
MusicCaps [4] to test for performance degradation and use the collected $300$
open prompts with $250$ as test samples, and the remaining $50$ for training.

#### Main results.

We first present the main results on Table 3 where we compare _Original_ ,
_0-shot-refinement_ , _Instruct_ , and _Instruct+Feedback_. To demonstrate the
efficacy of our proposed approach, _Instruct+Feedback_ , we compare it against
two alternative methods: _Instruct+0-shot_ and the base instruction-tuned LLM
(_Instruct_), where we observe marked and consistent CLAP-based improvement
over open prompts. We also observe that the proposed technique results in a
plot resembling the hypothesized audionese, and we attribute this finding to
the ability of LLMs to embellish open prompts.

#### Ablation studies.

By leveraging the CLAP-based feedback, _Instruct+Feedback_ improves unseen
prompts as well, suggesting its ability to project the prompt into a text
distribution more attuned to the TTA models. In contrast, _Instruct+0-shot_
relies solely on pre-trained models and does not benefit from user feedback
when exposed to unseen prompts. Overall, our experimental results consistently
show that _Instruct+Feedback_ achieves higher CLAP scores as compared to
_Instruct+0-shot_ and the original prompts.

Model | MusicCaps (Close) | User (Open)
---|---|---
SBJ(%) | OBJ | SBJ(%) | OBJ
Original | 24.4 | 3.47 | 10.4 | 1.53
Instruct | 23.6 | 3.58 | 28.4 | 3.63
Instruct+0-shot | 26.8 | 3.39 | 29.6 | 3.65
Instruct+Feedback | 25.2 | 3.47 | 31.6 | 3.71

Table 4: Human evaluation of generated audio samples (with FLAN-T5-large) for
_close_ and _open_ prompts. Five annotators were asked to evaluate the
_Subjective_ (side-by-side audio preference) and _Objective_ (relevance to
prompts on scale 1-5).

#### Effectiveness in low resource scenarios.

Further, while the training set consists of $50$ prompts, CLAP-based
improvement was observed with as little as $5$ prompts, and gradually
plateaued at $10$ samples (see Figure 3) – this opens up the possibility of
online learning of prompt rewriter models, where shifts in the distribution of
user open prompts can be readily acquired. Thus, we think that the approach is
suitable for online learning setups as well, which we save for future works.

In practice, we found the margin rank learning process to be rather brittle.
We set the learning rate to be 3e-4 and with training samples up to $50$
samples, and observed that the attained CLAP scores to go significantly higher
as more samples are added, but at the huge cost of the text similarity with
the original prompts. To avoid drastic deviation from the original user intent
(and hence the objective human evaluation), we pick lower training sample
sizes between $5$-$10$, depending on the SacreBLEU threshold, and stop
training when SacreBLEU goes below $20$ points. We summarize the relationship
between SacreBLEU and CLAP below in Figure 3.

#### Choice of model architectures.

Moreover, we observe more visible improvements with encoder-decoder based
architecture as shown in the FLAN-T5 series, while decoder-only LaMini-LM
seems to display better CLAP scores, but results in extremely low SacreBLEU
scores, which translates to text-level degradation upon further examination.
We attribute this to the encoder’s ability to more robustly encode full
sequence all-at-once, thereby either mitigating the noise or accounting for
the full context before decoding.

Fig. 3:  Learning curve: The figure illustrates the learning curve at small
sample size of $50$, with an iterval size of $5$. We report the CLAP and
ScareBLEU scores.

#### Human preference study.

Overall, we found that _Instruct+Feedback_ generates higher-quality and more
contextually appropriate prompts, but the base model _Instruct_ offers the
greater quality improvement. The alignment technique we use is limited by
CLAP’s ability to effectively identify decent audio qualities, as shown the
MusicCaps test set – leading to our hypothesis that CLAP provides the greatest
improvements for open, abstract prompts, rather than elaborate prompts.

#### Correlation of CLAP with human preference.

Interestingly, the gap between the preferential ratio between Open _Original
Prompts_ and _Instruct_ is greater than that as measured by CLAP (Table 4). In
terms of objective evaluation, _Instruct+Feedback_ is the clear winner and
yields a slightly larger gap than the case of subjective evaluation. However,
we measure the strength of CLAP correlation with human preference, and found
that the correlation coefficient to be a mere $0.242$ – which is a limiting
factor of our experiments, so we hope to explore with better metrics to
account for audio samples’ temporal information with human preference
incorporated.

##  5 Conclusions

This work addresses the open prompt challenge for commercialized TTA
generation by proposing the concept of ”audionese” and enhancing user prompts.
Our observation highlights vague user prompts causing alignment issues with
training data. Certain audio descriptions yield improved TTA model results,
named ” _audionese_ ”, exposing intricacies and emphasizing the need for
better alignment and _audionese_ comprehension. We analyze the distribution of
audio text prompts and propose prompt enhancement techniques using
instruction-tuned large language models. Through extensive experiments, the
proposed approach demonstrates significant improvements in audio metrics
compared to the original user prompts, as validated by objective and
subjective human evaluations.

## References

  * [1] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu,  “Diffsound: Discrete diffusion model for text-to-sound generation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [2] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi,  “Audiogen: Textually guided audio generation,”  arXiv preprint arXiv:2209.15352, 2022.
  * [3] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever,  “Zero-shot text-to-image generation,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8821–8831.
  * [4] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al.,  “Musiclm: Generating music from text,”  arXiv preprint arXiv:2301.11325, 2023.
  * [5] Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel,  “Peer: A collaborative language model,”  ArXiv, vol. abs/2208.11663, 2022.
  * [6] Michihiro Yasunaga and Percy Liang,  “Graph-based, self-supervised program repair from diagnostic feedback,”  37th Int. Conf. Mach. Learn. ICML 2020, vol. PartF168147-14, pp. 10730–10739, 2020.
  * [7] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi,  “Generating sequences by learning to self-correct,”  arXiv preprint arXiv:2211.00053, 2022.
  * [8] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.,  “Constitutional ai: Harmlessness from ai feedback,”  arXiv preprint arXiv:2212.08073, 2022.
  * [9] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein,  “Sensitivity and generalization in neural networks: an empirical study,”  in International Conference on Learning Representations.
  * [10] Vera Demberg and Frank Keller,  “Data from eye-tracking corpora as evidence for theories of syntactic processing complexity,”  Cognition, vol. 109, no. 2, pp. 193–210, 2008.
  * [11] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang,  “Clap: Learning audio concepts from natural language supervision,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [12] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.,  “Scaling instruction-finetuned language models,”  arXiv preprint arXiv:2210.11416, 2022.
  * [13] Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji,  “Lamini-lm: A diverse herd of distilled models from large-scale instructions,”  arXiv preprint arXiv:2304.14402, 2023.
  * [14] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley,  “Audioldm: Text-to-audio generation with latent diffusion models,”  arXiv preprint arXiv:2301.12503, 2023.
  * [15] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.,  “Transformers: State-of-the-art natural language processing,”  in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.


# Ascent and descent of WCT operators on Orlicz spaces

S. Nojoumi and S. Shamsigamchi<EMAIL_ADDRESS>Department of Mathematics,
Payame Noor University (PNU), P. O. Box: 19395-3697, Tehran- Iran

###### Abstract.

In this paper we are concerned with weighted conditional type(WCT) operators
on Orlicz spaces. We prove that all WCT operators have finite ascent. Also, we
provide some sufficient conditions for WCT operators to have finite descent.
As a consequence we find some decompositions for Orlicz space $L^{\Phi}(\mu)$.
In the sequel we discuss power bounded WCT operators and some results on their
Cesaro boundedness.

###### Key words and phrases:

weighted conditional expectation, ascent, descent, Orlicz space.

###### 2010 Mathematics Subject Classification:

46E30, 47A05

##  1\.  Introduction and Preliminaries

Let $X$ be a linear space and $T:X\longrightarrow X$ be a linear operator with
domain $\mathcal{D}(T)$ and range $\mathcal{R}(T)$ in $X$. The null space of
the iterates of $T$, $T^{n}$, is denoted by $\mathcal{N}(T^{n})$, and we know
that the null spaces of $T^{n}$’s form an increasing chain of subspaces
$\\{0\\}=\mathcal{N}(T^{0})\subset\mathcal{N}(T)\subset\mathcal{N}(T^{2})\subset\ldots$.
Also the ranges of iterates of $T$ form a nested chain of subspaces
$X=\mathcal{R}(T^{0})\supset\mathcal{R}(T)\supset\mathcal{R}(T^{2})\supset\ldots$.
Note that if $\mathcal{N}(T^{k})$ coincides with $\mathcal{N}(T^{k+1})$ for
some $k$, it coincides with all $\mathcal{N}(T^{n})$ for $n>k$. The smallest
non-negative integer $k$ such that $\mathcal{N}(T^{k})=\mathcal{N}(T^{k+1})$
is called the ascent of $T$ and denotes by $\alpha(T)$. If there is no such
$k$, then we set $\alpha(T)=\infty$. Also if
$\mathcal{R}(T^{k})=\mathcal{R}(T^{k+1})$, for some non-negative integer $k$,
then $\mathcal{R}(T^{n})=\mathcal{R}(T^{k})$ for all $n>k$. The smallest non-
negative integer $k$ such that $\mathcal{R}(T^{k})=\mathcal{R}(T^{k+1})$ is
called descent of $T$ and denotes by $\delta(T)$. We set $\delta(T)=\infty$
when there is no such $k$. When ascent and descent of an operator are finite,
then they are equal and the linear space $X$ can be decomposed into the direct
sum of the null and range spaces of a suitable iterates of $T$. The ascent and
descent of an operator can be used to characterize when an operator can be
broken into a nilpotent piece and an invertible one; see, for example, [1,
18]. For some results on ascent and descent of bounded operators in general
setting see, for example, [19, 20].
The operator $T$ is called power bounded if the norms of $T^{k}$, $k\geq 0$,
are uniformly bounded $(\sup_{k}\|T^{k}\|<\infty)$, and Cesaro bounded if the
Cesaro means $A_{n}(T)=n^{-1}\sum^{n-1}_{i=0}T^{i}$ are uniformly bounded.
Here we recall the concepts on Orlicz spaces. A function
$\Phi:\mathbb{R}\rightarrow[0,\infty]$ is called a Young function if $\Phi$ is
convex, even, and $\Phi(0)=0$; we will also assume that $\Phi$ is neither
identically zero nor identically infinite on $(0,\infty)$. The fact that
$\Phi(0)=0$, along with the convexity of $\Phi$, implies that
$\lim_{x\rightarrow 0^{+}}\Phi(x)=0$; while $\Phi\neq 0$, again along with the
convexity of $\Phi$, implies that $\lim_{x\rightarrow\infty}\Phi(x)=\infty$.
We set $a_{\Phi}:=\sup\\{x\geq 0:\Phi(x)=0\\}$ and
$b_{\Phi}:=\sup\\{x>0:\Phi(x)<\infty\\}.$ Then it can be checked that $\Phi$
is continuous and nondecreasing on $[0,b_{\Phi})$ and strictly increasing on
$[a_{\Phi},b_{\Phi})$. We also assume the left-continuity of the function
$\Phi$ at $b_{\Phi}$, i.e. $\lim_{x\rightarrow
b_{\Phi}^{-}}\Phi(x)=\Phi(b_{\Phi})$.

To each Young function $\Phi$ is associated another convex function
$\Psi:\mathbb{R}\rightarrow[0,\infty)$ with similar properties, defined by

$\Psi(y)=\sup\\{x|y|-\Phi(x):x\geq 0\\}\quad(y\in\mathbb{R}).$

The function $\Psi$ is called the function complementary to $\Phi$ in the
sense of Young. Also, for any measurable function $f$ on the measure space
$(X,\Sigma,\mu)$ we set $I_{\Phi}(f)=\int_{X}\Phi(f)d\mu$. Any pair of
complementary functions $(\Phi,\Psi)$ satisfies Young’s inequality
$xy\leq\Phi(x)+\Psi(y)\,\,(x,y\geq 0)$.

The generalized inverse of the Young function $\Phi$ is defined by

$\Phi^{-1}(y)=\inf\\{x\geq 0:\Phi(x)>y\\}\quad(y\in[0,\infty)).$

Notice that if $x\geq 0$, then $\Phi\big{(}\Phi^{-1}(x)\big{)}\leq x$, and if
$\Phi(x)<\infty$, we also have $x\leq\Phi^{-1}\big{(}\Phi(x)\big{)}$. There
are equalities in either case when $\Phi$ is a Young function vanishing only
at zero and taking only finite values. Also, if $(\Phi,\Psi)$ is a pair of
complementary Young functions, then

(1.1) $x<\Phi^{-1}(x)\Psi^{-1}(x)\leq 2x$

for all $x\geq 0$ (Proposition 2.1.1(ii) [17]).

By an $N$-function we mean a Young function vanishing only at zero, taking
only finite values, and such that $\lim_{x\rightarrow\infty}\Phi(x)/x=\infty$
and $\lim_{x\rightarrow 0^{+}}\Phi(x)/x=0$. Note that then $a_{\Phi}=0,$
$b_{\Phi}=\infty$, and, as we said above, $\Phi$ is continuous and strictly
increasing on $[0,\infty)$. Moreover, a function complementary to an
$N$-function is again an $N$-function.

A Young function $\Phi$ is said to satisfy the $\Delta_{2}$-condition at
$\infty$ if $\Phi(2x)\leq K\Phi(x)\;(x\geq x_{0})$ for some constants $K>0$
and $x_{0}>0$. A Young function $\Phi$ satisfies the $\Delta_{2}$-condition
globally if $\Phi(2x)\leq K\Phi(x)\;(x\geq 0)$ for some $K>0$.

A Young function $\Phi$ is said to satisfy the $\Delta^{\prime}$-condition
(respectively, the $\nabla^{\prime}$-condition) at $\infty$, if there exist
$c>0$ (respectively, $b>0$) and $x_{0}>0$ such that

$\Phi(xy)\leq c\,\Phi(x)\Phi(y)\quad(x,y\geq x_{0})$ $(\mbox{respectively,
}\Phi(bxy)\geq\Phi(x)\Phi(y)\quad(x,y\geq x_{0})).$

If $x_{0}=0$, these conditions are said to hold globally. Notice that if
$\Phi\in\Delta^{\prime}$, then $\Phi\in\Delta_{2}$ (both at $\infty$ and
globally).

Let $\Phi,\Psi$ be Young functions. Then $\Phi$ is called stronger than $\Psi$
at $\infty$, which is denoted by
$\Phi\mathrel{\overset{\makebox[0.0pt]{\mbox{\scriptsize$\ell$
}}}{\succ}}\Psi$ [or
$\Psi\mathrel{\overset{\makebox[0.0pt]{\mbox{\scriptsize$\ell$}}}{\prec}}\Phi$],
if

$\Psi(x)\leq\Phi(ax)\quad(x\geq x_{0})$

for some $a\geq 0$ and $x_{0}>0$; if $x_{0}=0$, this condition is said to hold
globally and is then denoted by
$\Phi\mathrel{\overset{\makebox[0.0pt]{\mbox{\scriptsize$a$}}}{\succ}}\Psi$
[or
$\Psi\mathrel{\overset{\makebox[0.0pt]{\mbox{\scriptsize$a$}}}{\prec}}\Phi$].

For a given complete $\sigma$-finite measure space $(X,\Sigma,\mu)$, let
$L^{0}(\Sigma)$ be the linear space of equivalence classes of
$\Sigma$-measurable real-valued functions on $X$, that is, we identify
functions equal $\mu$-almost everywhere on $X$. The support $S(f)$ of a
measurable function $f$ is defined by $S(f):=\\{x\in X:f(x)\neq 0\\}$. For a
given Young function $\Phi$, the space

$L^{\Phi}(\mu)=\left\\{f\in L^{0}(\Sigma):\exists
k>0,\int_{X}\Phi(kf)d\mu<\infty\right\\}$

is called an Orlicz space. Define the functional

$N_{\Phi}(f)=\inf\\{k>0:\int_{X}\Phi(\frac{f}{k})d\mu\leq 1\\}.$

$(L^{\Phi}(\mu),N_{\Phi}(.))$ is a normed linear space. If a.e. equal
functions are identified, then $(L^{\Phi}(\mu),N_{\Phi}(.))$ is a Banach
space, the basic measure space $(X,\Sigma,\mu)$ is unrestricted.

$\|f\|_{\Phi}=\inf\left\\{k>0:\int_{X}\Phi(f/k)d\mu\leq 1\right\\}.$

The couple $(L^{\Phi}(\Sigma),\|\cdot\|_{\Phi})$ is called the Orlicz space
generated by a Young function $\Phi$. Let $\Phi(x)=|x|^{p}/p$ with
$1<p<\infty$; $\Phi$ is then a Young function and
$\Psi(x)=|x|^{p^{\prime}}/p^{\prime}$, with $1/p+1/p^{\prime}=1$, is the Young
function complementary to $\Phi$. Thus, with this function $\Phi$ we retrieve
the classical Lebesgue space $L^{p}(\Sigma)$, i.e.
$L^{\Phi}(\Sigma)=L^{p}(\Sigma)$.

Recall that an atom of the measure space $(X,\Sigma,\mu)$ is a set
$A\in\Sigma$ with $\mu(A)>0$ such that if $F\in\Sigma$ and $F\subset A$, then
either $\mu(F)=0$ or $\mu(F)=\mu(A)$. A measure space $(X,\Sigma,\mu)$ with no
atoms is called a non-atomic measure space. It is well-known that if
$(X,\Sigma,\mu)$ is a $\sigma$-finite measure space, then for every measurable
real-valued function $f$ on $X$ and every atom $A$, there is a unique scalar,
denoted by $f(A)$, such that $f=f(A)\,\,\mu$-a.e. on $A$. Also, if
$(X,\Sigma,\mu)$ is a $\sigma$-finite measure space that fails to be non-
atomic, there is a non-empty countable set of pairwise disjoint atoms
$\\{A_{n}\\}_{n\in\mathbb{N}}$ with the property that
${B}:=X\setminus\bigcup_{n\in\mathbb{N}}A_{n}$ contains no atoms [21].
For a sub-$\sigma$-finite algebra $\mathcal{A}\subseteq\Sigma$, the
conditional expectation operator associated with $\mathcal{A}$ is the mapping
$f\rightarrow E^{\mathcal{A}}f$, defined for all non-negative, measurable
function $f$ as well as for all $f\in L^{1}(\Sigma)$ and $f\in
L^{\infty}(\Sigma)$, where $E^{\mathcal{A}}f$, by the Radon-Nikodym theorem,
is the unique $\mathcal{A}$-measurable function satisfying

$\int_{A}fd\mu=\int_{A}E^{\mathcal{A}}fd\mu,\ \ \ \forall A\in\mathcal{A}.$

As an operator on $L^{1}({\Sigma})$ and $L^{\infty}(\Sigma)$,
$E^{\mathcal{A}}$ is idempotent and
$E^{\mathcal{A}}(L^{\infty}(\Sigma))=L^{\infty}(\mathcal{A})$ and
$E^{\mathcal{A}}(L^{1}(\Sigma))=L^{1}(\mathcal{A})$. Thus it can be defined on
all interpolation spaces of $L^{1}$ and $L^{\infty}$ such as, Orlicz spaces
[2]. If there is no possibility of confusion, we write $E(f)$ in place of
$E^{\mathcal{A}}(f)$. This operator will play a major role in our work and we
list here some of its useful properties:

$\bullet$ If $g$ is $\mathcal{A}$-measurable, then $E(fg)=E(f)g$.

$\bullet$ $\varphi(E(f))\leq E(\varphi(f))$, where $\varphi$ is a convex
function.

$\bullet$ If $f\geq 0$, then $E(f)\geq 0$; if $f>0$, then $E(f)>0$.

$\bullet$ For each $f\geq 0$, $S(f)\subseteq S(E(f))$, where $S(f)=\\{x\in
X;f(x)\neq 0\\}$.
A detailed discussion and verification of most of these properties may be
found in [16].

Let $f\in L^{\Phi}(\Sigma)$. It is not difficult to see that $\Phi(E(f))\leq
E(\Phi(f))$ and so by some elementary computations we get that
$\|E(f)\|\leq\|f\|$ i.e, $E$ is a contraction on the Orlicz spaces. As we
defined in [6], we say that the pair $(E,\Phi)$ satisfies the generalized
conditional-type Hölder-inequality (or briefly GCH-inequality) if there exists
some positive constant $C$ such that for all $f\in L^{\Phi}(\mu)$ and $g\in
L^{\Psi}(\mu)$ we have

$E(|fg|)\leq C\Phi^{-1}(E(\Phi(|f|)))\Psi^{-1}(E(\Psi(|g|))),$

where $\Psi$ is the complementary Young function of $\Phi$. There are many
examples of pairs $(E,\Phi)$ that satisfy GCH-inequality in [6].

Finally in the following we give another key lemma that is important in our
investigation. The proof is an easy exercise.

######  Lemma 1.1.

If $\Phi$ is a Young function and $f$ is a $\Sigma$-measurable function such
that $E(f)$ and $E(\Phi(f))$ are defined, then $S(E(f))=S(E(\Phi(f)))$.

We keep the above notations throughout the paper.
Weighted conditional type operators have been studied by many mathematicians
in recent years [5, 6, 7, 8, 10]. For the importance of WCT operators we refer
the interested readers to [4, 3, 15, 12]. In this paper we are going to
investigate ascent and descent of weighted conditional type (WCT) operators on
Orlicz spaces. To this end, first we show that all WCT operators have finite
ascent. Also, we provide some sufficient conditions for them to have finite
descent. As a consequence, we find some decompositions for Orlicz space
$L^{\Phi}(\mu)$. Finally we find some results on power bounded WCT operators
and Cesaro bounded WCT operators.

##  2\.  Main Results

In this section we determine which hypothesis let us get that WCT operator
$T=M_{w}EM_{u}$ is a bounded operator on the Orlicz space $L^{\Phi}(\mu)$.
After that we will discuss the conditions under which the bounded operator $T$
has finite ascent and descent. Also, we will find some sufficient condition
for $T$ to have closed range. Some other results will be obtained. Here we
define weighted conditional type operators on Orlicz spaces.

######  Definition 2.1.

Let $\Phi$ be a Young function and $u,w:X\rightarrow\mathbb{C}$ be a
measurable function on the measure space $(X,\Sigma,\mu)$. The weighted
conditional type operator (WCT operator) from $L^{\Phi}(\Sigma)$ into
$L^{\Phi}(\Sigma)$ is defined by $wEM_{u}(f)=wE(uf)$ for every $f\in
L^{\Phi}(\Sigma)$ such that $wE(uf)\in L^{\Phi}(\mathcal{A})$.

In the next theorem we give a condition under which the WCT operator
$T=M_{w}EM_{u}$ is bounded on the Orlicz space $L^{\Phi}(\mu)$.

######  Theorem 2.2.

Let $(\Phi,\Psi)$ be a pair of complementary Young’s functions such that
satisfies GCH-inequality and $T=M_{w}EM_{u}$ be WCT operator. If
$w\Psi^{-1}(E(\Psi(u)))\in L^{\infty}(\Sigma)$, then $T$ is a bounded operator
on $L^{\Phi}(\mu)$.

###### Proof.

Since $(\Phi,\Psi)$ satisfies GCH-inequality, then there exists $C>0$ such
that for all $f\in L^{\Phi}(\mu)$ and $g\in L^{\Psi}(\mu)$ we have

$E(|fg|)\leq C\Phi^{-1}(E(\Phi(|f|)))\Psi^{-1}(E(\Psi(|g|))).$

Let $M=\|w\Psi^{-1}(E(\Psi(u)))\|_{\infty}$. For each $f\in L^{\Phi}(\mu)$ we
have

$\displaystyle\int_{X}\Phi(\frac{wE(uf)}{CMN_{\Phi}(f)}d\mu$
$\displaystyle\leq\int_{X}\Phi(\frac{wC\Phi^{-1}(E(\Phi(\frac{|f|}{N_{\Phi}(f)})))\Psi^{-1}(E(\Psi(|u|)))}{CM}d\mu$
$\displaystyle\leq\int_{X}\Phi(\Phi^{-1}(E(\Phi(\frac{|f|}{N_{\Phi}(f)})))d\mu$
$\displaystyle=\int_{X}\Phi(\frac{|f|}{N_{\Phi}(f)})d\mu$ $\displaystyle\leq
1.$

By these observations we get that $N_{\Phi}(wE(uf))\leq CMN_{\Phi}(f)$, so $T$
is bounded. ∎

Now in the next Lemma we see that for every $n\in\mathbb{N}$, $T^{n}$ is again
a WCT operator.

######  Lemma 2.3.

Let $n\in\mathbb{N}$ and $T=M_{w}EM_{u}$ be a bounded operator on
$L^{\Phi}(\Sigma)$. Then we have

$T^{n}=M_{(E(uw))^{n-1}}M_{w}EM_{u}.$

###### Proof.

By induction and straight forward calculations one can get the proof. ∎

In the sequel we find that every bounded WCT operator has finite ascent.

######  Proposition 2.4.

Let the WCT operator $T=M_{w}EM_{u}$ be bounded on the Orlicz space
$L^{\Phi}(\Sigma)$. The for every $n\in\mathbb{N}$,
$\mathcal{N}(T^{2})=\mathcal{N}(T^{n+2})$. As a result $\alpha(T)\leq 2$.

###### Proof.

Let $n\in\mathbb{N}\setminus\\{1\\}$, then by Lemma 2.3 we have

$T^{n}(f)=(E(uw))^{n-1}wE(uf)$

and specially for $n=2$, $T^{2}(f)=(E(uw))wE(uf)$, for $f\in L^{\Phi}(\mu)$.
It is clear that $T^{2}(f)=0$ if and only if $T^{n}(f)=0$, for all $n\geq 2$.
So $\mathcal{N}(T^{2})=\mathcal{N}(T^{n})$.

∎

For our main results we need to find some sufficient conditions for closedness
of range of WCT operator $T=M_{w}EM_{u}$. So in the next proposition we
provide some conditions under which $T$ has closed range.

######  Proposition 2.5.

Let $(\Phi,\Psi)$ a pair of complementary Young’s functions that satisfies
GCH-inequality, WCT operator $T$ be bounded on $L^{\Phi}(\Sigma)$, $\mu(B)=0$
and the set

$H=\\{n\in\mathbb{N}:w(A_{n})\Psi^{-1}(E(\Psi(u)))(A_{n})\neq 0\\}$

be finite. Then $T$ has closed range.

###### Proof.

Since $(\Phi,\Psi)$ satisfies GCH-inequality, then there exists $C>0$ such
that for every $f\in L^{\Phi}(\mu)$,

$wE(uf)\leq wC\Phi^{-1}(E(\Phi(|f|)))\Psi^{-1}(E(\Psi(|u|))),$

so

$S(wE(uf))\subset S(w)\cap
S(\Psi^{-1}(E(\Psi(|u|))))=S(w\Psi^{-1}(E(\Psi(|u|)))).$

Moreover, by our assumptions we have

$S_{0}:=S(w\Psi^{-1}(E(\Psi(|u|))))=\cup\\{A_{n}:n\in H\\}.$

Hence we get that $\mathcal{R}(T)\subset
L^{\Phi}(S_{0},\Sigma_{S_{0}},\mu_{S_{0}})$. Since $H$ is finite and
$\Sigma$-atoms are disjoint, then we get that
$L^{\Phi}(S_{0},\Sigma_{S_{0}},\mu_{S_{0}})$ is finite dimensional and
therefore $\mathcal{R}(T)$ is finite dimensional. Consequently,
$\mathcal{R}(T)$ is closed. ∎

Here we obtain a condition under which $T^{n}$ has closed range for all
$n\in\mathbb{N}$.

######  Theorem 2.6.

Let $(\Phi,\Psi)$ a pair of complementary Young’s functions that satisfies
GCH-inequality, WCT operator $T$ be bounded on $L^{\Phi}(\Sigma)$, $\mu(B)=0$
and the set

$H_{k}=\\{n\in\mathbb{N}:E(wu)^{k}(A_{n}).w(A_{n})\Psi^{-1}(E(\Psi(u)))(A_{n})\neq
0\\}$

be finite, for some $k\in\mathbb{N}$. Then $T^{n}$ has closed range, for all
$n\in\mathbb{N}\setminus\\{1\\}$. Consequently, if

$H=H_{0}=\\{n\in\mathbb{N}:E(wu)^{k}(A_{n}).w(A_{n})\Psi^{-1}(E(\Psi(u)))(A_{n})\neq
0\\}$

is finite, then $T^{n}$ has closed range, for all $n\in\mathbb{N}$.

###### Proof.

As we had in the Proposition 2.4 for each $n\in\mathbb{N}$,
$T^{n}=M_{w_{n}}EM_{u}$, in which $w_{n}=wE(uw)^{n-1}$, so $T^{n}$ is also a
WCT operator. Moreover, by our assumptions, for every $m,n\in\mathbb{N}$, with
$m\neq n$ we have $H_{n}=H_{m}$. Hence if $H_{k}$ is finite for some
$k\in\mathbb{N}$, then by Proposition 2.5 we get that $T^{n}$ has closed range
for all $n\in\mathbb{N}$. In addition if $H$ is finite then $T^{n}$ has closed
range for all $n\in\mathbb{N}$, because $H_{k}\subseteq H$, for every
$k\in\mathbb{N}$. ∎

In the following we have some consequences on closedness of range of $T$ and
some decompositions for $L^{\Phi}(\mu)$.

######  Remark 2.7.

Let WCT operator $T=M_{w}EM_{u}$ be bounded on $L^{\Phi}(\Sigma)$. If
$\mathcal{R}(M_{E(wu)^{n-1}}T)$ is closed for some $n>2$ or
$\mathcal{R}(M_{E(wu)^{j}}T)+\mathcal{N}(M_{E(wu)^{k}}T)$ is closed for some
positive integers with $j+k=n$, then for
$\mathcal{R}(T^{n})=\mathcal{R}(M_{E(wu)^{n-1}}T)$ is closed for all $n\geq 2$
and $\mathcal{R}(M_{E(wu)^{j}}T)+\mathcal{N}(M_{E(wu)^{k}}T)$ is closed for
all $j+k\geq 2$. Moreover,
$L^{\Phi}(\Sigma)=\mathcal{R}(M_{E(wu)}T)+\mathcal{N}(M_{E(wu)}T)$.

###### Proof.

As we proved in Proposition 2.4, $\alpha(T)\leq 2$. Therefore by Theorem 2.1
of [9] we get the proof. ∎

Now by mixing Theorem 2.6 and Remark 2.7 we get the next corollary.

######  Corollary 2.8.

Let $(\Phi,\Psi)$ a pair of complementary Young’s functions that satisfies
GCH-inequality, WCT operator $T$ be bounded on $L^{\Phi}(\Sigma)$, $\mu(B)=0$
and the set

$H_{k}=\\{n\in\mathbb{N}:E(wu)^{k}(A_{n}).w(A_{n})\Psi^{-1}(E(\Psi(u)))(A_{n})\neq
0\\}$

be finite, for some $k\in\mathbb{N}$. Then
$\mathcal{R}(M_{E(wu)^{j}}T)+\mathcal{N}(M_{E(wu)^{k}}T)$ is closed for all
$j+k\geq 2$. Moreover,
$L^{\Phi}(\Sigma)=\mathcal{R}(M_{E(wu)}T)+\mathcal{N}(M_{E(wu)}T)$.

######  Proposition 2.9.

If $w\Psi^{-1}(E(\Psi(u)))\in L^{\infty}(\Sigma)$, then the following hold:

a) The sequence $\\{\|E(wu)\|_{\infty}\\}_{n}$ is uniformly bounded if and
only if $\|E(wu)\|_{\infty}\leq 1$.

b) The WCT operator $T$ is power bounded on the Orlicz space
$L^{\Phi}(\Sigma)$ if and only if $|E(wu)|<1$ on $S(\Phi^{-1}(E(\Phi(w))))\cap
S(\Psi^{-1}(E(\Psi(u))))$.

###### Proof.

a) For the proof of this part one can see Theorem 2.5, part (a), [5].
b) Let $T$ be power bounded. Then there exists $D>0$ such that

$\|T^{n}\|=\|M_{E(wu)^{n-1}}T\|\leq D,\ \ \ \ \forall n\in\mathbb{N}.$

As we discussed before, since $(\Phi,\Psi)$ satisfies GCH-inequality, then for
every $f\in L^{\Phi}(\mu)$,

$T(f)=wE(uf)\leq wC\Phi^{-1}(E(\Phi(|f|)))\Psi^{-1}(E(\Psi(|u|))),$

for some $C>0$. Hence

$\displaystyle S(wE(uf))$ $\displaystyle\subset S(w)\cap
S(\Psi^{-1}(E(\Psi(|u|))))$ $\displaystyle=S(w\Psi^{-1}(E(\Psi(|u|))))$
$\displaystyle\subseteq S(\Psi^{-1}(E(\Psi(|w|))))S(\Psi^{-1}(E(\Psi(|u|)))).$

Also, by GCH-inequality we have

$S(E(wu)\subseteq S(\Psi^{-1}(E(\Psi(|w|))))S(\Psi^{-1}(E(\Psi(|u|)))).$

Moreover, $S(T^{n}(f))=S(E(wu)^{n-1})\cap S(Tf)$. By these observations we get
that if $\|E(wu)\|>1$ on a set of positive measure, then the non-zero WCT
operator $T$ can’t be power bounded.
Conversely, let

$|E(wu)|<1\ \ \ \text{on}\ \ S(\Phi^{-1}(E(\Phi(w))))\cap
S(\Psi^{-1}(E(\Psi(u)))).$

Then $\|E(wu)^{n}\|\leq C$ or some $C>0$ and so for every $n\in\mathbb{N}$,
$\|T^{n}\|\leq C\|T\|$. Thus $T$ is power bounded. ∎

In the sequel we provide some conditions under which WCT operators have finite
descent.

######  Theorem 2.10.

If $w\Psi^{-1}(E(\Psi(u)))\in L^{\infty}(\Sigma)$, and $E(uw)$ is bounded away
from zero, then for each $n\in\mathbb{N}$,
$\mathcal{R}(T^{n+2})=\mathcal{R}(T^{2})$ and so $T$ has finite descent.

###### Proof.

It is clear that for any $n\in\mathbb{N}$,
$\mathcal{R}(T^{n+2})\subseteq\mathcal{R}(T^{2})$. For the converse, let
$g\in\mathcal{R}(T^{2})$. Then there exists $f\in L^{\Phi}(\mu)$ such that
$T^{2}(f)=E(wu)T(f)$. Also, since $E(uw)$ is bounded away from zero, then
$\frac{1}{E(wu)}\chi_{S(E(wu))}\in L^{\infty}(\mu)$. Hence
$\frac{1}{E(wu)^{n}}\chi_{S(E(wu))}f\in L^{\Phi}(\mu)$.

$\displaystyle g$ $\displaystyle=E(wu)T(f)$
$\displaystyle=\frac{1}{E(wu)^{n}}\chi_{S(E(wu))}E(wu)^{n+1}T(f)$
$\displaystyle=E(wu)^{n+1}T(\frac{1}{E(wu)^{n}}\chi_{S(E(wu))}f)$
$\displaystyle=T^{n+2}(\frac{1}{E(wu)^{n}}\chi_{S(E(wu))}f).$

This implies that $g\in\mathcal{R}(T^{n+2})$. Consequently we have
$\mathcal{R}(T^{n+2})=\mathcal{R}(T^{2})$. ∎

######  Corollary 2.11.

Let $w\Psi^{-1}(E(\Psi(u)))\in L^{\infty}(\Sigma)$, $E(wu)$ be bounded away
from zero and $T=M_{w}EM_{u}$ be WCT operator on the Orlicz spaces
$L^{\Phi}(\Sigma)$. Then $\delta(T)\leq 2$.

Till now we have obtained that bounded WCT operators $T=M_{w}EM_{u}$ have
finite ascent with $\alpha(T)\leq 2$ and also under a weak conditions have
finite descent with $\delta(T)\leq 2$. In the next Proposition some more
results affected by finite ascent and descent.

######  Proposition 2.12.

Let $T=M_{w}EM_{u}$ be bounded WCT operator on the Orlicz spaces
$L^{\Phi}(\Sigma)$. Then $\mathcal{R}(T^{2})\cap\mathcal{N}(T^{m})=\\{0\\}$,
for every $m\geq 1$. Also, if $E(wu)$ be bounded away from zero, then
$L^{\Phi}(\mu)=\mathcal{R}(T^{n})+\mathcal{N}(T^{2})$, for some (equivalently,
all) $n\geq 1$.

###### Proof.

As it is known

$T^{2}(\mathcal{N}(T^{2+n}))=\mathcal{R}(T^{2})\cap\mathcal{N}(T^{n}).$

This implies that $\mathcal{R}(T^{2})\cap\mathcal{N}(T^{n})=\\{0\\}$, because
$\mathcal{N}(T^{2})=\mathcal{N}(T^{n+2})$.
Moreover, we know that
$T^{-2}(\mathcal{R}(T^{2+n})=\mathcal{R}(T^{n})+\mathcal{N}(T^{2})$ and this
equation gives us $L^{\Phi}(\mu)=\mathcal{R}(T^{n})+\mathcal{N}(T^{2})$, for
some (equivalently, all) $n\geq 1$. This completes the proof. ∎

For a bounded linear operator $T$ on an arbitrary Banach space the Cesaro
means is defined as

$A_{n}(T)=\frac{I+T+T^{2}+...+T^{n-1}}{n}\ \ \ \ \ \ n\in\mathbb{N}.$

Ergodic theory is concerned with the existence of the limit of the sequence
$\\{A_{n}(T)\\}_{n\in\mathbb{N}}$ in various operator topologies. For
investigation the convergence of this sequence one can use the following
simple formulas([11], [13] and [14])

(2.1) $\frac{T^{n}}{n}=\frac{n+1}{n}A_{n+1}(T)-A_{n}(T),$ (2.2)
$(I-T)A_{n}(T)=\frac{I-T^{n}}{n}$ (2.3)
$I-A_{n}(T)=(I-T)\frac{T^{n-2}+2T^{n-3}+...+(n-2)T+(n-1)I}{n}.$

If we apply it for the bounded WCT operators $T=M_{w}EM_{u}$ on the Banach
space $L^{\Phi}(\mu)$, then we have

$A_{n}(T)=n^{-1}(I+M_{v_{n}}T),\ \ \ \ \ \ \forall
n\in\mathbb{N}\setminus\\{1\\},$

and $A_{1}(T)=I$, in which $v_{n}=\sum^{n-2}_{i=0}E(uw)^{i}$.

######  Proposition 2.13.

Let $|E(wu)|<1$ on $S(\Phi^{-1}(E(\Phi(w))))\cap S(\Psi^{-1}(E(\Psi(u))))$,
$T=M_{w}EM_{u}$ be bounded WCT operator on the Orlicz spaces
$L^{\Phi}(\Sigma)$. Then $\alpha(I-T)\leq 1$ and $\alpha(I-T^{*})\leq 1$.

###### Proof.

Under our assumptions and by the Proposition 2.9 we get that $T$ is power
bounded. So by the above observations we get that the sequence
$\\{\frac{T^{n}}{n}(f)\\}$ tends to zero for every
$f\in\mathcal{N}((I-T)^{2})$. Hence by the Lemma 1.3 and Theorem 3.2 of [9] we
get the result. ∎

Here, in the next theorem, we provide a dense subset of $L^{\Phi}(\mu)$ by
means of the finite ascent and descent of bounded WCT operators.

######  Theorem 2.14.

Let $T=M_{w}EM_{u}$ be bounded WCT operator on the Orlicz spaces
$L^{\Phi}(\Sigma)$ such that $\mathcal{R}(T^{2})$ is closed. Then
$\mathcal{R}(T^{2})\cap\mathcal{N}(T^{2})=\\{0\\}$ and
$\mathcal{R}(T^{2})+\mathcal{N}(T^{2})$ is dense in $L^{\Phi}(\mu)$.

###### Proof.

For bounded WCT operator $T=M_{w}EM_{u}$(equivalently for $T^{*}$) we have
obtained that its ascent is finite and specially $\alpha(T)=\alpha(T^{*})\leq
2$. Hence by the Proposition 2.12 we get that
$\mathcal{R}(T^{2})\cap\mathcal{N}(T^{2})=\\{0\\}$ and
$\mathcal{R}(T^{*^{2}})\cap\mathcal{N}(T^{*^{2}})=\\{0\\}$. Since
$\mathcal{R}(T^{2})^{\perp}=\mathcal{N}(T^{*^{2}})$, then we get that

$(\mathcal{R}(T^{2})+\mathcal{N}(T^{2}))^{\perp}=\mathcal{R}(T^{2})^{\perp}\cap\mathcal{N}(T^{2})^{\perp}=\mathcal{R}(T^{*^{2}})\cap\mathcal{N}(T^{*^{2}})=\\{0\\}.$

This implies that $\mathcal{R}(T^{2})+\mathcal{N}(T^{2})$ is dense in
$L^{\Phi}(\mu)$. ∎

In the sequel we obtain that $L^{\Phi}(\mu)$ can be written as direct sum of
its two closed sub-spaces.

######  Proposition 2.15.

Let $T=M_{w}EM_{u}$ be bounded WCT operator on the Orlicz spaces
$L^{\Phi}(\Sigma)$ such that $|E(wu)|<1$ on $S(\Phi^{-1}(E(\Phi(w))))\cap
S(\Psi^{-1}(E(\Psi(u))))$ and $\mathcal{R}(I-T)$ is closed. Then
$L^{\Phi}(\mu)=\mathcal{R}(I-T)\oplus\mathcal{N}(I-T)$.

###### Proof.

Since $|E(wu)|<1$ on $S(\Phi^{-1}(E(\Phi(w))))\cap S(\Psi^{-1}(E(\Psi(u))))$,
then by Proposition 2.9 we get that $T$ is power bounded. So
$\\{\frac{T^{n}}{n}\\}$ is convergent in weak operator topology on
$L^{\Phi}(\mu)$. So by Theorem 4.4 of [9] we have the result. ∎

Finally we have the following result.

######  Theorem 2.16.

Let $|E(wu)|<1$ on $S(\Phi^{-1}(E(\Phi(w))))\cap S(\Psi^{-1}(E(\Psi(u))))$ and
$T=M_{w}EM_{u}$ . Then $\overline{\mathcal{R}(I-T)}=L^{p}(\mathcal{F})$ and
equivalently we have the followings:
(i) $\mathcal{R}(I-T)=L^{\Phi}(\mu)$;
(ii) $I-T$ is invertible;
(iii)
$\\{\|(I-T)^{-1}(\frac{(n-1)I-M_{v_{n}}T}{n})\|\\}_{n\in\mathbb{N}}=\\{\|n^{-1}(M_{w_{n}}T+(n-1)I)\|\\}_{n\in\mathbb{N}}$
is bounded;
(iv)
$\\{(I-T)^{-1}(\frac{(n-1)I-M_{v_{n}}T}{n})(f)\\}_{n\in\mathbb{N}}=\\{n^{-1}(M_{w_{n}}Tf+(n-1)f)\\}_{n\in\mathbb{N}}$
converges for all $f\in L^{\Phi}(\mu)$. In which
$w_{n}=\sum^{n-2}_{i=1}(n-i-1)E(uw)^{i-1}$.
In this case, $n^{-1}(M_{w_{n}}Tf+(n-1)f)\rightarrow(I-T)^{-1}(f)$ for all
$f\in L^{\Phi}(\mu)$.
(v) $A_{n}(T)(f)$ converges to a $T$-invariant limit for all $f\in
L^{\Phi}(\mu)$.

###### Proof.

By our assumptions we get that $T$ is power bounded and so for every $f\in
L^{\Phi}(\mu)$ we have $A_{n}(T)(f)\rightarrow 0$. As is defined in [9]

$B_{n}(T)=n^{-1}(T^{n-2}+2T^{n-3}+....+(n-2)T+(n-1)I).$

Hence for $T=M_{w}EM_{u}$ we have:

$B_{n}(T)=n^{-1}(M_{w_{n}}T+(n-1)I),$

in which $w_{n}=\sum^{n-2}_{i=1}(n-i-1)E(uw)^{i-1}$. Therefore by Proposition
4.5 of [9] we have the proof of (i)-(iv). Since
$\\{\|E(uw)^{n}\|_{\infty}\\}_{n\in\mathbb{N}}$ is uniformly bounded, then
$T=M_{w}EM_{u}$ is power bounded. Also the closure of a norm bounded subset of
$L^{\Phi}(\mu)$ is weakly compact. Hence it’s weakly sequentially compact.
Then we get (v). ∎

## References

  * [1] Y. A. Abramovich, C. D. Aliprantis, An invitation to operator theory, Amer. Math. Soc., Providence, Rhode Island, 2002.
  * [2] C. Bennett and R. Sharpley, Interpolation of operators, Academic Press, INC, 1988.
  * [3] R. G. Douglas, Contractive projections on an L1 space, Pacific J. Math. 15 (1965), 443–462.
  * [4] P. G. Dodds, C. B. Huijsmans, B. De Pagter, Characterizations of conditional expectation-type operators. Pac. J. Math. 141(1), 55-77 (1990)
  * [5] Y. Estaremi, On a class of bounded operators and their ascent and descent, Re. Cir. Mate. Pal. 69 (2020), 1393-1399.
  * [6] Y. Estaremi. On the Algebra of WCE Operators. Rocky Mountain J. Math. 48 (2018), 501-517.
  * [7] Y. Estaremi, Some classes of weighted conditional type operators and their spectra, Positivity. 19 (2015), 83-93.
  * [8] Y. Estaremi, M. R. Jabbarzadeh, Weighted lambert type operators on L p-spaces. Oper. Matrices 1(2013) 101–116.
  * [9] S. Grabiner, J. Zemanek, Ascent, descent and ergodic properties of linear operators, J. Operator theory. 48 (2002), 69-81.
  * [10] J. J. Grobler and B. De Pagter, Operators representable as multiplication-conditional expecta- tion operators, J. Operator Theory 48 (2002), 15–40.
  * [11] U. Krengel, Ergodic theorems, de Gruyter, Berlin 1985.
  * [12] A. Lamber, Lp multipliers and nested sigma-algebras, Oper. Theory Adv. Appl. 104 (1998), 147– 153\.
  * [13] Yu. Lyubich, J. Zemanek, Precompactness in the uniorm ergodic theory, Studia Math, 112 (1994), 89- 97.
  * [14] M. Mbekhta, J. Zemanek, Sur le theoreme ergodique uniforme et le spectre, C. R. Acad. Sci. Paris Ser. I Math. 317 (1993), 1155- 1158.
  * [15] S. T. Moy, Characterizations of conditional expectation as a transformation on function spaces, Pacific J. Math. 4 (1954), 47–63.
  * [16] M. M. Rao, Conditional measure and applications, Marcel Dekker, New York, 1993.
  * [17] M.M. Rao, Z.D. Ren, Theory of Orlicz spaces, Marcel Dekker, New York, 1991.
  * [18] A.E. Taylor, D. C. Lay, Introduction to functional analysis, 2nd ed., Wiley, New York.1980.
  * [19] A. E. Taylor, Theorems on ascent, descent, nullity and defect of linear operators, Math. Ann., 163 (1966) 18-49.
  * [20] B. Yood, Ascent, descent and roots of Fredholm operators, Studia Math., 158 (2003) 219-226.
  * [21] A. C. Zaanen, Integration, 2nd ed., North-Holland, Amsterdam, 1967\.


# A Single Graph Convolution is All You Need: Efficient Grayscale Image
Classification

###### Abstract

Image classifiers often rely on convolutional neural networks (CNN) for their
tasks, which are inherently more heavyweight than multilayer perceptrons
(MLPs), which can be problematic in real-time applications. Additionally, many
image classification models work on both RGB and grayscale datasets.
Classifiers that operate solely on grayscale images are much less common.
Grayscale image classification has diverse applications, including but not
limited to medical image classification and synthetic aperture radar (SAR)
automatic target recognition (ATR). Thus, we present a novel grayscale (single
channel) image classification approach using a vectorized view of images. We
exploit the lightweightness of MLPs by viewing images as a vector and reducing
our problem setting to the grayscale image classification setting. We find
that using a single graph convolutional layer batch-wise increases accuracy
and reduces variance in the performance of our model. Moreover, we develop a
customized accelerator on FPGA for the proposed model with several
optimizations to improve its performance. Our experimental results on
benchmark grayscale image datasets demonstrate the effectiveness of the
proposed model, achieving vastly lower latency (up to 16$\times$ less) and
competitive or leading performance compared to other state-of-the-art image
classification models on various domain-specific grayscale image
classification datasets.

Index Terms—  GCN, grayscale, image classification, MLP, low-latency

##  1 Introduction

As the demand and popularity of real-time systems increase, low-latency
machine learning has become increasingly important. With more and more
consumers interacting with machine learning models through the cloud, the
speed at which those models can deliver results is critical. Consumers expect
fast and accurate results; any latency can lead to a poor user experience.
Moreover, low-latency machine learning is essential in real-time applications,
such as autonomous vehicles or stock market trading, where decisions must be
made quickly and accurately. In these scenarios, delays caused by high latency
can result in severe consequences and even cause inaccurate downstream
calculations [1].

A particular instance where low-latency machine learning is needed is
grayscale image classification. For example, a targeting system on a satellite
is costly, and decisions must be made using SAR efficiently and accurately.
Examples like this are where low-latency grayscale image classification comes
into play. It is often the case that image classifiers work on RGB datasets
and grayscale image datasets, but seldom do modern image classifiers focus
solely on the grayscale setting. RGB models are overkill for the grayscale
setting, as the grayscale problem allows us to focus on a single channel.
Models focusing on grayscale image classification are naturally more
efficient, as they can concentrate on a single channel rather than three.
Thus, many image classifiers that generalize to the grayscale image
classification are not truly optimized for the grayscale case. For these
reasons, we present a lightweight grayscale image classifier capable of
achieving up to $16\times$ lower latency than other state-of-the-art machine
learning models.

From a trustworthy visual data processing perspective, the demand for
grayscale image classification requires data to be collected from various
domains with high resolution and correctness so that we can train a robust
machine learning model. Additionally, recent advancements in machine learning
rely on convolutional neural networks, which often suffer from high
computation costs, large memory requirements, and many network parameters,
resulting in poor inference latency, poor scalability, and weak
trustworthiness.

The inherent novelties of our model are the following: our proposed method is
the first that vectorizes an image in a fully connected manner and inputs the
resultant into a single-layer graph convolutional network (GCN). We also find
that a single GCN layer is enough to stabilize the performance of our shallow
model. Additionally, our proposed method benefits from a batch-wise attention
term, allowing our shallow model to capture interdependencies between images
and form connections for classification. Finally, by focusing on grayscale
imagery, we can focus on a streamlined method for grayscale image
classification rather than concentrating on the RGB setting. A result of these
novelties is extremely low latency and high throughput for image
classification.

With the recent technological advances, modern FPGAs contain many hardware
resources [2], including DSPs, LUTs, BRAMs, and URAMs. The programmable nature
of FPGAs allows users to develop a customized data path and on-chip memory
organization, leading to high-performance implementations. Consequently, FPGAs
have emerged as an appealing option for executing time-sensitive machine
learning tasks with reduced latency and power. For instance, FPGAs have been
used for accelerating machine learning [3] and graph analytic tasks [4]. Given
that our model is a lightweight image classifier, we find it suitable to be
deployed on an FPGA. We can perform inference on the FPGA without returning
the intermediate results to external memory. This ensures low-latency
inference by capitalizing the fine-grained data parallelism inherent in FPGAs.
In contrast, CPUs and GPUs exploit coarse-grained thread-level parallelism and
have complex cache hierarchies, which are unsuitable for low-latency
inference. Thus, this paper makes the following contributions:

  * •

We present a lightweight, graph-based neural network for grayscale image
classification. Specifically, we (1) apply image vectorization, (2) construct
a graph for each batch of images and apply a single graph convolution, and (3)
propose a weighted-sum mechanism to capture batch-wise dependencies.

  * •

We implement our proposed method on FPGA, including the following design
methodology: (1) a portable and parameterized hardware template using high-
level synthesis, (2) layer-by-layer design to maximize runtime hardware
resource utilization, and (3) a one-time data load strategy to reduce external
memory accesses.

  * •

Experiments show that our model achieves competitive or leading accuracy with
respect to other popular state-of-the-art models while vastly reducing latency
and model complexity.

  * •

We implement our model on a state-of-the-art FPGA board, Xilinx Alveo U200.
Compared with state-of-the-art GPU implementation, our FPGA implementation
achieves $2.78\times$ speedup in latency and $2.1\times$ speedup in
throughput.

##  2 Problem Definition

The problem is to design a lightweight system capable of handling high volumes
of data with low latency. The solution should be optimized for performance and
scalability while minimizing resource utilization, a necessary component of
many real-time machine learning applications. The system should be able to
process and respond to requests quickly, with minimal delays. High throughput
and low latency are critical requirements for this system, which must handle
many concurrent requests without compromising performance. We define latency
and throughput in the following ways:

Throughput $\displaystyle=\frac{\text{Total number of images
processed}}{\text{Total inference time}}$ Latency $\displaystyle=\text{Total
time for a single inference}$

Latency refers to the total time (from start to finish) it takes to gather
predictions for a model in one batch (a standard approach). A lightweight
machine learning model aims to maximize throughput and accuracy while
minimizing latency.

##  3 Related Work

###  3.1 MLP Approaches

Our model combines various components of simple models and is inherently
different from current works in low-latency image classification. Some recent
architectures involve simple MLP-based models. Touvron et al. introduced
ResMLP [5], an image classifier based solely on MLPs. ResMLP is trained in a
self-supervised manner with patches interacting together. Touvron et al.
highlight their model’s high throughput properties and accuracy. ResMLP uses
patches from the image and alternates linear layers where patches interact and
a two-layer feed-forward network where channels interact independently per
patch. Additionally, MLP-Mixer [6] uses a similar patching method, which also
attains competitive accuracy on RGB image datasets compared to other CNNs and
transformer models. Our proposed method uses the results from a single-layer
MLP to feed into a graph neural network, during which we skip the information
from the three-channel RGB setting and only consider the single-channel
grayscale problem. This is inherently different than the methods mentioned
earlier, as they use patching approaches while we focus on the vectorization
of pixels.

###  3.2 Graph Image Construction Methods

The dense graph mapping that utilizes each pixel as a node in a graph is used
and mentioned by [7, 8]. For this paper, we employ the same terminology.
Additionally, Zhang et al. presented a novel graph neural network architecture
and examined its low-latency properties on the MSTAR dataset using the dense
graph [9]. Our proposed method differs from dense graph methods, as we
vectorize an image rather than using the entire grid as a graph.

Han et al. [10] form a graph from the image by splitting the image into
patches, much like a transformer. A deep graph neural network learns on the
patches similarly to a transformer but in a graph structure. Our structure
does not form a graph where each patch is a node in a graph; instead, we
create a graph from the resultant of a vectorized image passed through a
single-layer MLP.

Mondal et al. proposed applying graph neural networks on a minibatch of images
[11]. Mondal et al. claim that this method improves performance for
adversarial attacks. We use the proposed method to stabilize the performance
of a highly shallow model. The graph neural network, in this case, allows
learning to be conducted in a graph form, connecting images containing similar
qualities.

Besides the model proposed by Zhang et al., all the methods mentioned focus on
the RGB setting. This is overkill for grayscale image classification. Focusing
on a single channel allows us to develop a more streamlined solution rather
than forcing a model to operate on RGB datasets and having the grayscale
setting come as an afterthought. Doing so allows us to reduce computational
costs.

##  4 Overview and Architecture

This section describes our model architecture (GECCO: Grayscale Efficient
Channelwise Classification Operative). The overall process is summarized in
Figure 1.

Fig. 1: GECCO Architecture

Overall Architecture.  Many existing methods do not focus on the latency of
their design and its implications. Additionally, the vast majority of image
classification models focus on the performance of their work in the RGB
setting, rarely citing the performance of datasets in various domains. We
address these problems by presenting a novel architecture, focused on low
latency and the grayscale image setting.

Our model vectorizes a batch of images, which allows us to use a fully
connected layer pixel-wise for low computation time rather than the
traditional reliance on convolutional neural networks. The proposed method
uses the resultant from MLPs and constructs a graph batch-wise, where each
node corresponds to the flattened image outputted from an MLP. We then apply a
batch-wise attention term, inputted into an MLP for classification. Image
Vectorization.  For each image in a batch, we view the image as a vector. For
a tensor $\mathbf{X}\in\mathbb{R}^{B\times H\times W}$ where $B$ is the batch
size, $H$ and $W$ are the height and width of an image, we flatten the tensor
to $\mathbf{X}_{1}\in\mathbb{R}^{B\times(H\cdot W)}$. Viewing an image as a
vector allows our model to skip the traditional convolutional neural network,
which views the image as a grid and cuts computation time.

Simple MLP Layer.  We input $\mathbf{X}_{1}$ into a linear MLP with output
dimensionality $D_{out}$. Formally,
$\text{\scriptsize{MLP}}(\mathbf{X}_{1})\rightarrow\mathbf{X}_{2}\in\mathbb{R}^{B\times
D_{out}}$. Formally, MLP = $\sigma(\sum_{j}w_{j}x_{j}+b)$ where $\sigma$ is a
general activation term, each $w$ is a learned weight vector, each $x$ is a
data vector, and $b$ is a bias term. We then apply a standard layer norm and
dropout functions. We use a standard activation function to $\mathbf{X}_{2}$,
followed by the standard dropout and MAXPOOL1D functions.

Graph Construction.  Our graph construction consists of using each resultant
vector from MLP as a node in a graph, such that the number of nodes in the
graph corresponds with the number of images in a batch. This means that for
each batch, a vectorized image is each node in the graph with feature size
$\mathbb{R}^{D_{out}/2}$, and each image is connected to every other image in
a batch. Formally, we calculate the adjacency matrix $\mathbf{A}$ as
$\mathbf{A}_{ij}=1$, which connects all nodes. We then perform message passing
amongst vectors, meaning the message that is aggregated from $i$’s graph
neighborhood, given its neighbors $j$.

Graph Convolution.  Our single GCN layer learns from similar features of
images within its minibatch. Generally, a graph convolutional layer updates
the representations of nodes by aggregating each node’s neighbor’s
representation. It takes an input $\bm{h_{i}}$ and maps an output
$\bm{h_{i}}\mapsto\bm{h^{\prime}_{i}}$ given its neighbors
$\mathcal{N}_{i}=\\{j\in\mathcal{V}\mid(j,i)\in\mathcal{E}\\}$ where
$\bm{h^{\prime}}_{i}$ is defined as

$\bm{h^{\prime}}_{i}=f_{\theta}\left(\bm{h}_{i},\text{\scriptsize{AGGREGATE}}\left(\\{\bm{h_{j}}\mid
j\in\mathcal{N}_{i}\\}\right)\right).$

In our case, the input for each node $\bm{h_{i}}$ is the output from the MLP
layer of a vectorized image. We claim that this component increases and
stabilizes accuracy in the case of our simple MLP-based model. Section 5.2.1
presents a study of accuracy on MNIST with the graph convolutional layer
varying. We find that the Residual Gated Graph Convolutional Network [12]
outperforms in all categories, and thus, we use this as a backbone for our
model.

From the result of inputting $\mathbf{X}_{2}$ into the previous dropout and
MAXPOOL1D functions, we fix the resultant matrix $\mathbf{X}_{3}$. From these
functions, $\mathbf{X}_{3}$ inputs into a single-layer GCN with dimension
$\mathbb{R}^{B\times D_{out}}$ and maps to the same output dimension
$\mathbb{R}^{B\times D_{out}}$ where the output is denoted $\mathbf{X}_{4}$.

Thus, we can denote a general GCN term output $\mathbf{X}_{4}$. We then
consider the output term layer as $\mathbf{X}_{4}$ as
$\mathbf{X}_{4}=\sigma\left(\mathbf{A}\mathbf{X}_{3}\mathbf{W}\right)$ for the
general GCN case where $\mathbf{W}$ is a learned layer.

From the resultant matrix $\mathbf{X}_{4}$, we apply a standard batch
normalization term, activation function, and MAXPOOL1D functions.

The intuition behind this module is that it is helpful for such a shallow
model to draw connections between images with similar properties batch-wise to
stabilize accuracy. Adding graph convolutional layers past a single layer does
not increase accuracy and deters performance for our model.

Batch-wise Attention, Residual Connections, & Output. Then, we propose the
addition of a residual connection with batch-wise attention, defined as

$\displaystyle\mathbf{X}_{5}$
$\displaystyle=\left(\frac{\sigma\left(\mathbf{X}_{4}\mathbf{X}^{\top}_{4}\right)}{\sum_{i=1}^{B}\sum_{j=1}^{B}\sigma\left(\mathbf{X}_{4}\mathbf{X}_{4}^{\top}\right)_{ij}}\right)\mathbf{X}_{4}$

where $\sigma$ is the standard sigmoid function. The batch-wise attention term
allows the model to further capture similar features from each image to
another batch-wise. Relating similar properties from images to each other
boosts accuracy in our case; refer to our ablation study in Table 7.

The residual term is defined $\mathbf{X}_{6}=\mathbf{X}_{5}+\mathbf{X}_{4}$.
The residual term allows our model to capture similar features while
minimizing the risk of gradient explosion and retains information from the
previous $\mathbf{X}_{4}$ step. By multiplying a softmax-like term with the
output of a previous graph convolution, our intuition is that we weigh the
correspondence of each pixel compared to other similar pixels within similar
images batch-wise. This method differs from the standard query-key-value
attention mechanism. It is more suitable to our case, as we allow batch-wise
feature assimilation, allowing the model to capture similar features between
images in a batch. In contrast, attention mechanisms focus on individual
features, which may not be as effective in capturing batch-wise similarities.
We then feed the residual term into an MLP for classification results.111We
make our code publicly available at https://github.com/GECCOProject/GECCO

##  5 Experiments

###  5.1 Datasets

Datasets from several domains are examined to gauge the effectiveness of GECCO
in diverse settings. We use the popular MNIST dataset [13], Fashion-MNIST
[14], MSTAR, and CXR [15] summarized in the following manner:

  * •

MNIST is a grayscale handwritten dataset with $(28,28)$ pixel image sizes and
$10$ different object categories. The training size for this dataset is
$60000$, and the testing size is $10000$.

  * •

Fashion-MNIST contains $(28,28)$ sized grayscale images from $10$ categories
with a training size of $60000$, and a testing size of $10000$.

  * •

MSTAR is a SAR ATR dataset with a training size of $2747$ and testing size of
$2425$ SAR images of $10$ different vehicle categories. We resize this dataset
to $(128,128)$ pixels.

  * •

CXR is a chest X-ray dataset containing 5863 X-ray images and 2 categories
(Pneumonia/Normal). The images are $(224,224)$ pixels. The training size is
$5216$, and the testing size is $624$.

###  5.2 Results

####  5.2.1 Backbone

For Table 1, we choose ResConv as the backbone of our model because it has the
most desirable characteristics for applying a graph convolutional layer.

Table 1: Performance of GCN Layers Varying on MNIST

Convolutional Layer |  Top-1 Accuracy |  Throughput ($\text{imgs}/\text{ms}$) |  Latency (ms)
---|---|---|---
GCN [16] |  $97.09\%$ |  $54.58\pm 8.47$ |  $4.68\pm 0.72$
TAGConv [17] |  $97.39\%$ |  $58.39\pm 9.01$ |  $4.37\pm 0.67$
SAGEConv [18] |  $97.77\%$ |  $58.84\pm 8.98$ |  $4.32\pm 0.66$
ChebConv [19] |  $97.50\%$ |  $51.98\pm 8.65$ |  $4.91\pm 0.82$
ResConv [12] |  $\bf{98.04\%}$ |  $\bf{62.44\pm 9.87}$ |  $\bf{4.10\pm 0.65}$

####  5.2.2 Experimental Performance

Experimental performance includes the top-1 accuracy, inference throughput,
and inference latency. We perform our inference batch-wise as a means to
reduce latency. These metrics vary across each dataset.

In Tables 2, 3, 4, and 5, we present a summary of our findings. We report the
best-performing accuracy, average throughputs, and latencies with their
standard deviations. Our model outperforms every other model in terms of
throughput and latency across all datasets, leads accuracy on the MSTAR
dataset, and performs competitively in terms of accuracy on all datasets.

We perform the remaining experiments on a state-of-the-art NVIDIA RTX A5000
GPU. Additionally, we compare our model to the top-performing variants of VGG
[20], the variant of the popular ViT [21], the ViT for small-sized datasets
(SS-ViT) [22], FastViT [23], Swin Transformer [24], and ResNet [25] models. We
use the open-source packages PyTorch and HuggingFace for model building and
the PyTorch Op-Counter for operation counting. Performing the remaining
experiments on the same hardware system is vital in fostering a fair
comparison for each model.

Table 2: MNIST Performance

Model |  Top-1 Accuracy |  Throughput ($\text{imgs}/\text{ms}$) |  Latency (ms)
---|---|---|---
Swin-T |  $98.30\%$ |  $4.69\pm 0.21$ |  $54.53\pm 2.44$
SS-ViT |  $98.09\%$ |  $8.38\pm 1.42$ |  $30.46\pm 5.17$
VGG16 |  $\bf{99.54\%}$ |  $39.02\pm 4.98$ |  $6.55\pm 0.84$
FastViT |  $98.44\%$ |  $7.51\pm 0.51$ |  $34.05\pm 2.31$
ResNet34 |  $99.06\%$ |  $12.88\pm 1.02$ |  $19.86\pm 1.57$
GECCO |  $98.04\%$ |  $\bf{62.44\pm 9.87}$ |  $\bf{4.10\pm 0.65}$

Table 3: Fashion-MNIST Performance

Model |  Top-1 Accuracy |  Throughput ($\text{imgs}/\text{ms}$) |  Latency (ms)
---|---|---|---
Swin-T |  $88.73\%$ |  $4.58\pm 0.31$ |  $55.81\pm 3.78$
SS-ViT |  $87.88\%$ |  $8.27\pm 1.56$ |  $30.97\pm 5.83$
VGG16 |  $\bf{92.48\%}$ |  $24.98\pm 5.77$ |  $10.24\pm 2.36$
FastViT |  $87.94\%$ |  $7.28\pm 0.75$ |  $35.07\pm 3.63$
ResNet34 |  $88.50\%$ |  $11.98\pm 1.05$ |  $21.35\pm 1.87$
GECCO |  $88.09\%$ |  $\bf{59.08\pm 9.04}$ |  $\bf{4.32\pm 0.66}$

Table 4: MSTAR Performance

Model |  Top-1 Accuracy |  Throughput ($\text{imgs}/\text{ms}$) |  Latency (ms)
---|---|---|---
Swin-T |  $86.04\%$ |  $5.44\pm 0.37$ |  $46.98\pm 3.20$
SS-ViT |  $95.61\%$ |  $9.14\pm 1.72$ |  $27.97\pm 5.26$
VGG16 |  $93.13\%$ |  $6.75\pm 1.34$ |  $37.89\pm 7.52$
FastViT |  $91.78\%$ |  $4.16\pm 0.52$ |  $61.44\pm 7.69$
ResNet34 |  $98.64\%$ |  $12.44\pm 0.84$ |  $20.48\pm 1.39$
GECCO |  $\bf{99.47\%}$ |  $\bf{67.17\pm 15.24}$ |  $\bf{3.80\pm 0.86}$

Table 5: CXR Performance

Model |  Top-1 Accuracy |  Throughput ($\text{imgs}/\text{ms}$) |  Latency (ms)
---|---|---|---
Swin-T |  $73.66\%$ |  $1.08\pm 0.21$ |  $236.71\pm 46.09$
SS-ViT |  $71.09\%$ |  $4.09\pm 0.84$ |  $62.35\pm 12.85$
VGG16 |  $\bf{82.01\%}$ |  $3.03\pm 1.02$ |  $84.10\pm 28.43$
FastViT |  $75.46\%$ |  $4.24\pm 1.00$ |  $60.30\pm 14.24$
ResNet34 |  $78.31\%$ |  $2.41\pm 0.44$ |  $105.84\pm 19.39$
GECCO |  $79.87\%$ |  $\bf{14.04\pm 3.19}$ |  $\bf{18.05\pm 4.14}$

####  5.2.3 Model Complexity Metrics

Model complexity metrics for this paper include the number of multiply-
accumulate operations, the number of model parameters, model size, and the
number of layers. In other words, suppose accumulator $a$ counts an operation
of arbitrary $b,c\in\mathbb{R}$. We count the number of multiply-accumulate
operations as $a\leftarrow a+(b\times c)$. Additionally, the layer count
metric is an essential factor of latency. Increasing the number of layers will
also improve the latency of a model’s inference. The goal of an effective
machine learning model is to maximize throughput while minimizing the number
of MACs and the number of layers, in our case.

We measure the model complexity of our model against other popular machine
learning models that we have chosen in Table 6. Our model outperforms in all
categories regarding our chosen model complexity metrics, highlighting its
lightweightness.

Table 6: Model Complexity Metrics

Model |  # MACs |  # Parameters |  Model Size (Mb) |  # Layers
---|---|---|---|---
Swin-T |  $2.12\times 10^{10}$ |  $2.75\times 10^{7}$ |  $109.9$ |  $167$
SS-ViT |  $1.55\times 10^{10}$ |  $4.85\times 10^{6}$ |  $19.62$ |  $79$
VGG16 |  $9.51\times 10^{9}$ |  $4.69\times 10^{6}$ |  $18.75$ |  $20$
FastViT |  $7.16\times 10^{8}$ |  $4.02\times 10^{6}$ |  $16.1$ |  $226$
ResNet34 |  $4.47\times 10^{9}$ |  $2.13\times 10^{7}$ |  $85.1$ |  $92$
GECCO |  $\bf{1.22\times 10^{6}}$ |  $\bf{1.90\times 10^{4}}$ |  $\bf{0.075}$ |  $\bf{10}$

####  5.2.4 Ablation Study

We perform an ablation study to verify that the components of our proposed
model contribute positively to the overall accuracy.

Table 7: Ablation Study

Mini-batch GNN |  Weighted Sum Residual Term |  Accuracy on MNIST
---|---|---
✓ |  ✓ |  $98.02\%$
✗ |  ✓ |  $93.75\%$
✓ |  ✗ |  $92.17\%$
✗ |  ✗ |  $86.55\%$

Additionally, we find that only a single graph convolutional layer is enough
to reduce the variance and increase the accuracy of our model. Refer to Figure
2.

Fig. 2: Accuracy on MNIST vs. Number of Graph Convolutional Layers

###  5.3 Discussion

Across multiple datasets, GECCO achieves leading or competitive accuracy
compared to other state-of-the-art image classifiers. With respect to model
complexity, it is clear that GECCO outperforms other machine learning models,
highlighting our model’s low latency and lightweight properties.

It is difficult for our model to generalize to the RGB setting. We attribute
this challenge to the vectorization process that our model uses. Learning on
three channels poses a complexity challenge, as GECCO is very shallow and
simple, thus making it challenging to learn on three separate channels.

Our proposed method does not make use of positional embeddings or class
tokens. GECCO can learn essential features by the weighted residual term.
Additionally, we tested the addition of positional embeddings and class tokens
and found no improvement in accuracy across various datasets. We note that the
$\mathbf{X}_{5}$ residual term adds positional awareness to the model.

###  5.4 FPGA Implementation

We develop an accelerator for the proposed model on a state-of-the-art FPGA,
Xilinx Alveo U200 board, to highlight the model’s efficiency and compatibility
with hardware further. It has 3 Super Logic Regions (SLRs), 4 DDR memory
banks, 1182k Look-up tables, 6840 DSPs, 75.9 Mb of BRAM, and 270 Mb of URAM.
The FPGA kernels are developed using the Xilinx High-level Synthesis (HLS)
tool to expedite the design process.

We perform the following optimizations for our FPGA design: (1) _Portable
design_ : We design a parameterized hardware template using HLS. It is
portable to different FPGA platforms, including embedded and data-center
FPGAs. (2) _Resource sharing_ : The model is executed layer-by-layer. Each
layer in the model is decomposed into basic kernel functions. The basic kernel
functions, including matrix multiplication, elementwise activation, column-
wise and row-wise summations, max pooling, and various other elementwise
operations, are implemented separately and subsequently invoked within their
corresponding layers. Due to the reuse of these fundamental kernel functions
across multiple layers, FPGA resources are shared among the different layers,
maximizing runtime hardware resource utilization. (3) _One-load strategy_ : We
employ a one-time data load strategy that enables us to load the required data
from DDR only once. All other data required for the computations are stored in
on-chip memory, reducing inference latency. Figure 3 illustrates the overall
hardware architecture of our design.

We use the Vitis Analyzer tool to provide insights into resource utilization,
latency, and throughput. Table 8 reports the results obtained for the MNIST
dataset. Given the model’s compact design and resource efficiency, it can be
accommodated within a single SLR. Hence, we deploy multiple accelerator
instances across multiple SLRs, each with one instance. This increases the
inference throughput. In table 8 shows the latency obtained for a single
inference and the throughput achieved by running the design on 3 SLRs
concurrently.

We compare our FPGA implementation with the baseline GPU implementation. The
GPU baseline is executed on an NVIDIA RTX A5000 GPU, which operates at 1170
MHz and has a memory bandwidth of 768 GB/s. On the other hand, the FPGA
operates at 300 MHz and has an external memory bandwidth of 77 GB/s. The GPU
baseline is comparable with the FPGA in terms of the platform. Although the
GPU has higher peak performance and memory bandwidth, our FPGA implementation
achieves a latency reduction of $2.78\times$ and a throughput improvement of
$2.1\times$. This speedup is attributed to the optimizations mentioned above
adopted in our implementation.

Fig. 3: Overview of Hardware Architecture Table 8: Resource Utilization (per
SLR), Latency, and Throughput for MNIST

Latency |  $1.47$ ms
---|---
Throughput |  $130.61$ imgs/ms
BRAMs |  $956~{}(22\%)$
DSPs |  $1226~{}(17\%)$
LUTs |  $459K~{}(38\%)$
FFs |  $597K~{}(25\%)$

##  6 Conclusion

This work presented a novel architecture based on MLPs and graph convolutional
layers. We benchmarked our model on popular grayscale image datasets and
highlighted the strength of our model in terms of its experimental performance
and model complexity metrics. We emphasized the importance of a lightweight
image classifier and examined the vast domain of grayscale image datasets.
Overall, a lightweight and low-latency image classifier can improve the
efficiency and effectiveness of various applications that rely on image
processing.

Our model performs well in various domains, including general image
classification, SAR ATR, and medical image classification. Additionally, we
implement our model on FPGA to show that our model is hardware-friendly, which
is an added benefit of any image classification model.

We presented the novelties of our model, which include viewing an image as a
vector in a fully connected manner and inputting the resultant into a single-
layer GCN. We also found that graph neural networks can increase the accuracy
and reduce the variance in performance of shallow models like ours.
Additionally, a simple batch-wise attention term allows our shallow model to
capture interdependencies between images and form connections for
classification. These novelties result in a highly low latency and high
throughput model.

## References

  * [1] Kaoru Ota, Minh Son Dao, Vasileios Mezaris, and Francesco G. B. De Natale,  “Deep learning for mobile multimedia: A survey,”  ACM Trans. Multimedia Comput. Commun. Appl., vol. 13, no. 3s, jun 2017.
  * [2] Tan Nguyen, Samuel Williams, Marco Siracusa, Colin MacLean, Douglas Doerfler, and Nicholas J. Wright,  “The performance and energy efficiency potential of fpgas in scientific computing,”  in 2020 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), 2020, pp. 8–19.
  * [3] Fahad Siddiqui, Sam Amiri, Umar Ibrahim Minhas, Tiantai Deng, Roger Woods, Karen Rafferty, and Daniel Crookes,  “Fpga-based processor acceleration for image processing applications,”  Journal of Imaging, vol. 5, no. 1, pp. 16, 2019.
  * [4] Bingyi Zhang, Hanqing Zeng, and Viktor Prasanna,  “Graphagile: An fpga-based overlay accelerator for low-latency gnn inference,”  IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 9, pp. 2580–2597, 2023.
  * [5] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou,  “Resmlp: Feedforward networks for image classification with data-efficient training,” 2021.
  * [6] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy,  “Mlp-mixer: An all-mlp architecture for vision,” 2021.
  * [7] Benjamin Sanchez-Lengeling, Emily Reif, Adam Pearce, and Alexander B. Wiltschko,  “A gentle introduction to graph neural networks,”  Distill, 2021,  https://distill.pub/2021/gnn-intro.
  * [8] Naman Goyal and David Steiner,  “Graph neural networks for image classification and reinforcement learning using graph representations,” 2022.
  * [9] Bingyi Zhang, Rajgopal Kannan, Viktor Prasanna, and Carl Busart,  “Accurate, low-latency, efficient sar automatic target recognition on fpga,”  in 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL). Aug. 2022, IEEE.
  * [10] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu,  “Vision gnn: An image is worth graph of nodes,” 2022.
  * [11] Arnab Kumar Mondal, Vineet Jain, and Kaleem Siddiqi,  “Mini-batch graphs for robust image classification,” 2021.
  * [12] Xavier Bresson and Thomas Laurent,  “Residual gated graph convnets,” 2018.
  * [13] Yann LeCun, Corinna Cortes, and CJ Burges,  “Mnist handwritten digit database,”  ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, vol. 2, 2010.
  * [14] Han Xiao, Kashif Rasul, and Roland Vollgraf,  “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” 2017.
  * [15] Daniel Kermany,  “Labeled optical coherence tomography (oct) and chest x-ray images for classification,” 2018.
  * [16] Thomas N. Kipf and Max Welling,  “Semi-supervised classification with graph convolutional networks,” 2017\.
  * [17] Jian Du, Shanghang Zhang, Guanhang Wu, Jose M. F. Moura, and Soummya Kar,  “Topology adaptive graph convolutional networks,” 2018.
  * [18] William L. Hamilton, Rex Ying, and Jure Leskovec,  “Inductive representation learning on large graphs,” 2018.
  * [19] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst,  “Convolutional neural networks on graphs with fast localized spectral filtering,” 2017.
  * [20] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,” 2015.
  * [21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021.
  * [22] Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song,  “Vision transformer for small-size datasets,”  CoRR, vol. abs/2112.13492, 2021.
  * [23] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan,  “Fastvit: A fast hybrid vision transformer using structural reparameterization,” 2023.
  * [24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,” 2015.