# FarsInstruct: Empowering Large Language Models
for Persian Instruction Understanding

Hojjat<EMAIL_ADDRESS>
Department of Computer Engineering
Isfahan University of Technology  Ziba<EMAIL_ADDRESS>
Department of Computer Engineering
Shahid Bahonar University of Kerman  Abbas<EMAIL_ADDRESS>
Department of Computer Engineering
University of Tehran  Hossein<EMAIL_ADDRESS>
Department of Computer Engineering
Isfahan University of Technology

###### Abstract

Instruction-tuned large language models, such as T0, have demonstrated
remarkable capabilities in following instructions across various domains.
However, their proficiency remains notably deficient in many low-resource
languages. To address this challenge, we introduce FarsInstruct: a
comprehensive instruction dataset designed to enhance the instruction-
following ability of large language models specifically for the Persian
language—a significant yet underrepresented language globally. FarsInstruct
encompasses a wide range of task types and datasets, each containing a mix of
straightforward to complex manual written instructions, as well as
translations from Public Pool of Prompts, ensuring a rich linguistic and
cultural representation. Furthermore, we introduce Co-CoLA, a framework
designed to enhance the multi-task adaptability of LoRA-tuned models. Through
extensive experimental analyses, our study showcases the effectiveness of
FarsInstruct dataset coupled with training by Co-CoLA
framework111https://github.com/Hojjat-Mokhtarabadi/FarsInstruct, in improving
the performance of large language models within the Persian context. As of the
current writing, FarsInstruct comprises more than 200 templates across 21
distinct datasets, and we intend to update it consistently, thus augmenting
its applicability.

Keywords: Instruction-tuned LLMs, Low-resource languages, Parameter efficient
fine-tuning

FarsInstruct: Empowering Large Language Models
for Persian Instruction Understanding

Hojjat<EMAIL_ADDRESS>Department of Computer
Engineering Isfahan University of Technology and Ziba
<EMAIL_ADDRESS>Department of Computer Engineering Shahid
Bahonar University of Kerman and Abbas<EMAIL_ADDRESS>Department of Computer Engineering University of Tehran and Hossein
<EMAIL_ADDRESS>Department of Computer Engineering Isfahan
University of Technology

##  1 Introduction

The modern era of artificial intelligence is marked by numerous breakthroughs,
among which is the rise of large language models (LLMs). These models, such as
PaLM (chowdhery2022palm), GPT4 (openai2024gpt4), and Llama2 (touvron2023llama)
with continuous scaling of their parameters and training data, are known to
exhibit emergent properties. wei2022emergent considers an ability to be
emergent if it is not present in smaller models but is present in larger
models. This is an unpredictable phenomenon that can not be predicted simply
by extrapolating the performance of smaller models. One such ability is
instruction-following, which enables models to execute unseen natural language
processing (NLP) tasks from reading instructions provided within the input
text. Previously, the capability for instruction-following was primarily
attributed to the scale of these models. However, recent studies have
demonstrated that instruction-following does not exclusively rely on the large
size of language models (sanh2022multitask). By instruction-tuning on a
collection of instructional NLP tasks, smaller language models can learn to
follow prompts. This approach has proven to be remarkably efficient, allowing
these smaller models to perform competitively and, in some specific tasks,
even outperform their larger counterparts (sanh2022multitask;
wei2021finetuned; wang-etal-2022-super). Instruction-tuning emerges as a vital
technique in the evolution of language models, involving training a model on a
wide range of tasks described through natural language instructions. This
method diverges from traditional task-specific fine-tuning, offering a more
generalized and versatile approach to model training, thus contributing
significantly to the advancement of LLMs.

Despite the steady progress of instruction-tuned language models, they still
struggle to accurately grasp the subtleties of low-resourced languages due to
the scarcity of prompted data and challenges inherent in translating English
datasets into other languages (naous2024having; ramesh-etal-2023-fairness;
vanmassenhove-etal-2021-machine). While efforts have been made to compile
extensive multilingual instruction-following datasets, gaps remain in creating
diverse and complex prompts for languages like Persian. For example, the
SuperNaturalInstructions benchmark (wang-etal-2022-super), encompassing
various task types across 55 languages, contains merely 2.1% of Persian
content. Similarly, the Aya Dataset (singh2024aya), a human-curated effort to
enhance AI’s instruction-following abilities across 65 languages, includes 1%
of Persian content. This underscores the disparity in the diversity and
quantity of the Persian Language tasks compared to other languages.

In this study, we propose FarsInstruct, a comprehensive prompted dataset
tailored to the Persian language. It comprises a mixture of manually written
instructions ranging from basic to proficient language levels, as well as
translations from Public Pool of Prompts (P3) (sanh2022multitask) which is a
collection of prompted English datasets. In particular, we created more than
200 prompt templates (roughly 10 templates for each of the 21 unique public
datasets) that we selected from a variety of sources. These datasets
collectively cover ten different task categories: Text Summarization, Textual
Entailment, Text Classification, Sentiment Analysis, Word Sense
Disambiguation, Query Paraphrasing, Question Answering, Reading Comprehension,
Named Entity Recognition (NER), and Translation. Figure 1 depicts an instance
of a prompt within our dataset, and a detailed overview of FarsInsturct
dataset is provided in Section 3.

Figure 1: An example of the prompts utilized in the training process. The
Persian version of the prompt is employed for training purposes, while the
translated English version is provided to enhance comprehension. The
instruction component is highlighted in blue, the data field is marked in
orange, and the target answer is indicated in gray. In Appendix Prompts, this
example is shown in PromptSource environment.

Additionally, in order to facilitate the multi-task adaptation of our model
and mitigate the problem of catastrophic forgetting, we introduce Co-CoLA, an
integration of CoLA (xia2024chain) with rehearsal training
(kirkpatrick2017overcoming). More specifically, we adopt an iterative
optimization framework that merges learned low-rank matrices into the model
parameters and reinitializes optimization for new LoRA modules. At each
iteration, we involve retraining a subset of data from previously learned
tasks and mixing it with the current task’s data during training. With this
periodic revisiting of earlier tasks, the model maintains performance on both
old and new tasks while preserving computational efficiency. Section 4
presents an in-depth explanation of Co-CoLA method.

FarsInstruct is publicly available and open-source and we are committed to
enhancing it by continually expanding our dataset with a broader range of
tasks, instruction entries, and modalities. We hope this dataset fills the
critical gap and serves as a valuable resource to the NLP community.

##  2 Related work

Instruction-tuning: In the landscape of AI, the capabilities of LLMs have
expanded far beyond mere text processing. These sophisticated models are now
being fine-tuned in a practice known as instruction-tuning, where models are
trained with specific input-output pairs drawn from a wide array of data
sources. This technique enables a pre-trained LLM to produce tailored outputs
based on given inputs, enhancing its versatility and effectiveness. FLAN
(wei2021finetuned) and T0 (sanh2022multitask) pioneered the exploration of
instruction-tuned language models, each contributing significantly to the
field. FLAN (wei2021finetuned) adapted a 137-billion parameter pre-trained
model, refining it with over 60 NLP datasets using natural language
instructions. On the other hand, T0 (sanh2022multitask) applied instruction
tuning to various T5 models across 2073 prompts from 177 datasets.
SuperNaturalInstruction (wang-etal-2022-super) further advanced the field by
assembling a comprehensive benchmark featuring 1,616 expert-written NLP tasks,
covering 76 unique task types, and extending support to multiple languages.
xP3 (muennighoff2022crosslingual) expanded on P3’s groundwork
(sanh2022multitask) by including content from 46 languages, adding new tasks
like Translation and Program Synthesis that P3 had not tackled. In a similar
expansive effort, Aya (singh2024aya) emerged as a significant multilingual
project, featuring an impressive collection of 513 million instances across
114 languages, achieved through a collaborative research effort that involved
fluent speakers from around the world to compile and complete instructional
content. Our dataset distinguishes itself from these collections in its depth
and adaptability, especially with the inclusion of more challenging Persian
tasks, offering a high level of detail not found in many multilingual efforts.
While most such projects primarily use machine translations and cover a narrow
range of tasks, our dataset presents a wide array of culturally and
linguistically rich tasks.

Parameter effecient fine-tuning: Conventional full-parameter fine-tuning
becomes computationally impractical as the model size and the number of
downstream tasks increase. To address this challenge, recent advancements in
parameter-efficient fine-tuning methods suggest training only a small portion
of parameters while keeping the majority of pre-trained model parameters
unchanged. One of the most widely used paradigms in parameter-efficient fine
turning is Low-Rank Adaptation (LoRA) (hu2021lora). LoRA only modifies a
small, low-rank portion of the model’s weights. This is achieved by adding
low-rank matrices to the model’s weights during training. Despite the
significant computational advantage of LoRA, it falls short in multi-task
adaptation, and also kalajdzievski2024scaling showed that PEFT strategies,
such as LoRA, are still susceptible to catastrophic forgetting. MultiLoRA
(wang2023multilora) addresses the limitations of LoRA by reducing the
dominance of top singular vectors, horizontally scaling LoRA modules, and
altering the initialization of adaptation matrices, which leads to improved
performance across multiple tasks with minimal additional parameters. MixLoRA
(li2024mixlora) introduces multiple LoRA-based experts within a frozen pre-
trained model using a top-k routing strategy to efficiently distribute tasks,
independently configure attention layer adapters, and apply auxiliary load
balance loss, significantly enhancing performance while reducing GPU memory
consumption and training latency. Additionally, CoLA(xia2024chain) introduces
an iterative optimization framework designed to improve the fine-tuning of
LLMs by employing multiple iterations of LoRA. In this paper, we design Co-
CoLA to address the issue of catastrophic forgetting, while ensuring an
effective multi-task adaption.

##  3 FarsInstruct Dataset

With about 130 million222https://en.wikipedia.org/wiki/Persian_language
speakers, Persian — also referred to as Farsi in Iran — is an important
language in the Middle East and Central Asia. FarsInstruct represents a
project to provide a comprehensive public prompted dataset for the Persian
community. As of this writing, FarsInstruct has more than 200 carefully
designed and created prompt templates for 21 already-published public datasets
and some translations from existing prompted datasets. Unlike multilingual
collections focusing on common tasks such as Text Summarization and Question
Answering, FarsInstruct introduces more innovative and challenging tasks,
including Named Entity Recognition and Word Sense Disambiguation. The creation
procedure, statistics, task augmentation, and quality of the dataset are
covered in detail in the following subsections. Additional illustrations and
tables are provided in the Appendix Dataset Illustrations, Prompts.

###  3.1 Dataset Construction

The development of FarsInstruct entailed transforming Persian NLP datasets
into their prompted format, described in plain language. This process involved
a combination of manual ideation, during which our team meticulously
brainstormed and refined prompt templates, along with invaluable insights from
Persian language instructors. For datasets with multiple data fields, prompts
were crafted to interrelate these fields, as elaborated in Section 3.2.
Additionally, synonyms were employed to diversify the instructions within the
prompts and reduce repetition. Each prompt template falls into one of two
classes: categorization or generation. Categorization prompts guide the model
in classifying text into predefined categories from dataset labels or
identified through dataset analysis. In contrast, generation prompts require
the model to produce full-length text, such as summarizing longer texts or
answering questions based on the provided information. These instructions also
include scenarios where the model needs to generate missing content from
partial text inputs.

Figure 2: Detailed depiction of 11 task types utilized in our dataset. Each
box within the figure lists the specific datasets associated with the
respective task type. Datasets designated for training are highlighted in
blue, and those reserved for testing are marked in orange. Additionally,
manual datasets, which have been specifically curated and prompted by our
team, are enclosed with solid borders. In contrast, datasets that have been
translated from English to Persian are enclosed with dashed borders.

To efficiently create a large collection of prompts, we primarily utilized
PromptSource (bach2022promptsource), an open-source tool designed for
creating, sharing, and managing prompts for NLP tasks. A key design choice in
bach2022promptsource is the use of Jinja2 as a templating language, providing
the flexibility crucial for crafting clear and effective prompts.
Specifically, a template is a function that maps dataset examples into input-
output natural language pairs, while a prompt is the combination of an input
template and a target template along with a collection of specific meta-data.
Instructions are specific directives within input-templates that guide the
model’s behavior. However, as the original version of PromptSource did not
support Persian, we modified its source code to accommodate Persian datasets.
Since this system is originally integrated with Huggingface Datasets (lhoest-
etal-2021-datasets) library, we gathered datasets from various sources and
consolidated into a unified public repository on
HuggingFace333https://huggingface.co/PNLPhub. Appendix Prompts provides some
exmaples of prompt template.

In addition to manual templating, we have decided to translate a subset of
three question-answering tasks from the P3 dataset sanh2022multitask. This
decision was made to enhance the comprehensiveness and utility of our work by
providing a broader scope of data. To ensure a high-quality translation, we
utilized the No Language Left Behind (NLLB) (costa2022no) machine translation
model, capable of single-sentence translations between 200 languages and
dialects in various scripts. We employed the largest NLLB model with 3.3B
parameters to achieve the best performance. A complete list of manually
templated and translated datasets is given in Figure 2.

Finally, since the datasets were sourced from multiple repositories, we
applied a series of pre-processing steps such as deduplication and stripping
out non-alphanumeric characters like emojis to ensure normalized text across
all data.

###  3.2 Task Augmentation

It is widely recognized that instruction-tuned models benefit significantly
from extensive and varied tasks. Given this context, we focus on developing
diverse prompts, spanning from basic to proficient language levels.
Furthermore, drawing from the methodologies outlined in FLAN Collection
(longpre2023flan), T0 (sanh2022multitask), and MetaICL (min-
etal-2022-metaicl), we enhance task diversity by mixing and swapping different
data fields within a given dataset. For instance, whereas a dataset might
initially be structured to evaluate a model’s ability to answer question x
given input y, we train the model to generate question x when provided with
answer y. This approach effectively broadens the spectrum of prompts within a
limited data pool.

###  3.3 Data statistics

The statistics of final dataset after applying templates across all datasets
is presented in Figure 3. Table 1 also presents the total number of
categorization and generation prompts for each task type.

Figure 3: Distribution of NLP tasks across the FarsInstruct dataset, highlighting the expanded data volumes post-prompt application and the number of prompts designed per task type. Task Type | Cat | Gen
---|---|---
Question Answering | 1 | 9
Translation | 2 | 10
NER (Named Entity Recognition) | 4 | 19
Multiple Choice QA | 9 | 1
Word Sense Disambiguation | 10 | 0
Classification | 15 | 12
Summarization | 4 | 15
Reading Comprehension | 2 | 18
Query Paraphrasing | 10 | 7
Sentiment Analysis | 24 | 13
Textual Entailment | 16 | 5
Table 1:  List of task types, along with the number of categorization and
generation prompts dedicated to each task type. The expanded version of this
table can be found in the Appendix Dataset Illustrations.

###  3.4 Quality Control

We selectively chose publicly available Persian datasets predominantly used
for single-task fine-tuning, as their extensive use ensures high quality.
Furthermore, to ensure the accuracy and quality of the instructions, we
conduct human evaluations through consultations with the general public and
experts in the field of literature. This review process allowed us to assess
the instructions from multiple perspectives and incorporate cultural and
linguistic nuances, critical for ensuring the prompts’ clarity, accuracy, and
relevance.

##  4 Methodology and Experimental Setup

To maintain our model’s robustness and generalization capabilities, we
integrate the CoLA framework (xia2024chain) with continual learning
(kirkpatrick2017overcoming). This section offers a thorough overview of the
training procedure and evaluation setup.

###  4.1 Training Procedure

Given the significant computational demands of full fine-tuning, we aim to
employ LoRA for the training procedure, specifically using the FarsInstruct
dataset. However, as highlighted in studies by (wang2023multilora;
li2024mixlora), LoRA tends to underperform in multi-task training scenarios
due to its limitations in capturing complex interactions between tasks,
leading to suboptimal performance. To mitigate this challenge, Chain of LoRA
(CoLA) (xia2024chain), presents an iterative optimization framework based on
the principles of the Frank-Wolfe algorithm (also known as the Conditional
Gradient Method). This method involves an iterative process of fine-tuning on
a single task, merging it with the base model, and reinitializing with a new
LoRA module. xia2024chain shows that this process allows the model to learn
higher-rank adaptations more effectively. Another persistent challenge
affecting the performance of LoRA-tuned models is catastrophic forgetting.
kalajdzievski2024scaling observed a strong inverse linear relationship between
the fine-tuning performance and the amount of forgetting when fine-tuning LLMs
with LoRA.

In this study we propose Continual-Chain of LoRA (Co-CoLA), an extension of
CoLA framework which incorporates rehearsal with replay during training. More
specifically, rehearsal training is an approach within the continual learning
framework that involves revisiting a portion of previously learned tasks
during training new tasks. Despite the limited success of continual learning
frameworks, the study by scialom2022fine demonstrated that continual training
of language models, such as T0 sanh2022multitask with rehearsal, can
effectively help them in comprehending new instruction via instruction
composition, resulting in better generalization and improved performance on
new tasks.

The core mathematical operation in LoRA involves updating the low-rank
matrices $A$ and $B$, which are applied to modify the transformer layers of
the model. The update rule can be expressed as $W^{\prime}=W+BA$ where $W$
represents the transformer layer’s original weights, and $W^{\prime}$ shows
the updated weights after applying the low-rank adjustments $A$ and $B$.
Essentially, Co-CoLA structures this training procedure into an iterative
three phases:

Tuning: In this phase, following the standard LoRA, the base model weights
remain frozen, while only the model’s LoRA parameters (represented by matrices
$A$ and $B$) are fine-tuned. Additionally, a subset of previously trained data
is replayed along with the new data. Formally, given the sequence
$T=(T_{1},\ldots,T_{n})$ where $T_{i}$ represents the training data after
applying an individual template, the training data augmented with rehearsal is
defined as:

$T_{i}^{r}=T_{i}\cup\left(\sum_{j=1}^{i-1}rT_{j}\right)$ (1)

where r is the rehearsal hyper-parameter that controls the percentage of
examples sampled from previous templates $T_{1},\ldots,T_{n}$.

Merging: After the tuning phase, the newly updated LoRA parameters are merged
with the existing model weights. These merged weights are fixed and do not
receive any gradient update in subsequent steps.

Expanding: The final phase involves preparing the model for subsequent
training rounds by reinitializing the LoRA modules with new parameters
($A^{\prime}$ and $B^{\prime}$). Following hu2021lora $A^{\prime}$ adopts
Gaussian initialization and $B^{\prime}$ is initialized to zero.

An illustration of this iterative three-staged approach is provided in Figure
4.

Figure 4: The Continual-Chain of LoRA Training Procedure

###  4.2 Evaluation Setup

Evaluation Tasks: Our model’s performance was evaluated through two categories
of task types: those that were part of the training dataset ("Held in") and
those introduced to the model for the first time during evaluation ("Held
out"). The evaluation dataset encompasses three distinct types of tasks:
Sentiment analysis and Query paraphrasing, classified as “Held in” tasks and
Textual Entailment which is categorized as a “Held out” task. As illustrated
in Figure 2, the evaluation includes one dataset each for sentiment analysis
and paraphrase identification, alongside two datasets dedicated to entailment
tasks.

Evaluation Metric: To assess the performance of our model relative to several
baseline models, we utilized the ROUGE-L metric, which measures the overlap of
n-grams between the generated text and reference texts. Specifically, we
concentrated on the F1-scores of ROUGE-L, a metric that integrates precision
and recall to provide a balanced evaluation. As demonstrated by wang-
etal-2022-super, the rankings produced by this metric exhibit a strong
correlation with accuracy for categorization templates.

##  5 Results

To investigate the applicability of FarsInstruct, we choose the Ava model and
instruction-tune it using the Co-CoLA framework on a diverse set of templates.
Our results were compared against a series of monolingual and multilingual
instruction-tuned models and to effectively assess the performance of our
model we conduct both quantitative and linguistic evaluations. For a
comprehensive overview of the training configuration, please refer to the
Appendix Training Configuration.

Task | Type | Model | ROUGE-L
---|---|---|---
parsinlu query paraphrasing |  Held In | Aya-13B | 45.58
PersianMind-7B | 17.07
Mistral-7B | 6.89
Dorna-8B | 3.85
Ava-8B | 6.67
Ava-LoRA-8B | 8.73
Co-CoLA-8B | 45.86
Digikala Sentiment Analysis |  Held In | Aya-13B | 28.41
PersianMind-7B | 18.19
Mistral-7B | 2.46
Dorna-8B | 2.42
Ava-8B | 8.69
Ava-LoRA-8B | 5.72
Co-CoLA-8B | 40.87
FarsTail |  Held Out | Aya-13B | 37.61
PersianMind-7B | 17.05
Mistral-7B | 5.74
Dorna-8B | 4.81
Ava-8B | 12.48
Ava-LoRA-8B | 9.07
Co-CoLA-8B | 36.35
Parsinlu Entailment |  Held Out | Aya-13B | 42.64
PersianMind-7B | 4.45
Mistral-7B | 4.93
Dorna-8B | 3.32
Ava-8B | 15.04
Ava-LoRA-8B | 7.18
Co-CoLA-8B | 55.32
Table 2: ROUGE-L F1 Scores for Different Models across Tasks

###  5.1 Quantitative Evaluation

We evaluate our model against several existing models fine-tuned on
instruction-specific data. Specifically, PersianMind (huggingface_persianmind)
is a Llama-2 7B based model, trained in 3 phases on different Persian
datasets. Though their training data is unavailable, Dorna
(huggingface_dorna_llama) and Ava (huggingface_ava_llama) are newly introduced
models, fine-tuned on the Llama-3 8B model for Persian tasks. Aya
(huggingface_aya_101) is a 13B encoder-decoder model trained on a subset of 25
million samples from the Aya dataset and Mistral-7B
(huggingface_mistral_7b_instruct) is a decoder-only model trained on publicly
available prompted datasets

Table 2 summarizes the comparative performance of various models, including
our proposed method, Co-CoLA, across several NLP Datasets: ParsiNLU Query
Paraphrasing, Digikala Sentiment Analysis, FarsTail, and ParsiNLU Entailment.
These models are evaluated using ROUGE-L F1 scores. As illustrated in Table 2,
Co-CoLA performs comparably well to the Aya model, despite having fewer
parameters and being trained on less instruction data and significantly
outperforms all other models, indicating the effectiveness of Co-CoLA. The
factors contributing to this performance gap are further discussed in Section
6. Moreover, the scores of Ava-LoRA, reflecting the performance of raw LoRA
fine-tuning of Ava on FarsInstruct, are inferior to those achieved with Co-
CoLA training, highlighting the effectiveness of our method.

Figure 5: Comparative performance of different models on Persian language
tasks using the ROUGE-L metric. The bar chart depicts the superior performance
of Co-CoLA across multiple tasks, particularly excelling in the ParsiNLU
Entailment task.

###  5.2 Linguistic Evaluation

Our comprehensive linguistic evaluation aimed to further substantiate the
effectiveness of Co-CoLA in handling the nuances of the Persian language,
compared to the baseline model Ava. The evaluation specifically focused on
analyzing the models’ capabilities in terms of coherence, relevance, and
linguistic quality, which are critical for assessing the applicability of
language models in real-world scenarios.

####  5.2.1 Evaluation Setup

The evaluation involved detailed analysis by language experts who assessed the
output from both models based on predefined criteria. This approach ensures an
unbiased evaluation of the models’ performance in generating contextually
appropriate and linguistically accurate content.

####  5.2.2 Evaluation Criteria

The linguistic outputs were evaluated based on three main criteria:

Coherence: This assesses the logical flow and connectivity of the text
produced by the models.

Relevance: This measures how well the model’s output adheres to the context
provided in the input.

Linguistic Quality: This evaluates the grammatical accuracy, punctuation, and
stylistic appropriateness of the text.

####  5.2.3 Evaluation Results

The evaluation results are summarized in Table 3, which provides a clear
comparative analysis of the performance of the two models across all assessed
criteria. The scores indicate that while Ava scored slightly higher in
coherence, Co-CoLA outperformed Ava in relevance and linguistic quality,
suggesting its superior ability to produce contextually accurate and
linguistically refined outputs.

Criteria | Co-CoLA | Ava
---|---|---
Coherence | 4.2 | 4.3
Relevance | 3.7 | 3.2
Linguistic Quality | 4.6 | 4.0
Table 3: Average Scores from Linguistic Evaluation

The higher scores of Co-CoLA in relevance and linguistic quality demonstrate
its effectiveness in producing not only grammatically correct but also
contextually relevant outputs, which is essential for real-world applications.
These results underscore the potential of Co-CoLA in enhancing the linguistic
handling of Persian language tasks, setting a benchmark for future
developments in language model applications.

##  6 Discussion

Figure 5 provides a detailed breakdown of the overall performance reported in
Table 2. Each dot in the plot represents the ROUGE-L F1 score of the given
model on the selected template. As clearly illustrated, other Persian
instruction-tuned models fail to achieve a high ROUGE-L F1 score. One
significant factor contributing to this disparity is the low precision score.
The F1 score, which combines precision and recall, serves as a comprehensive
metric for evaluation. Precision measures the proportion of the longest common
subsequence (LCS) in the candidate text that matches the reference text, while
recall measures the proportion of the LCS in the reference text that is
present in the candidate text. Although these models achieve acceptable recall
scores, they fall short in precision, a critical metric for categorization
templates. In contrast, Aya demonstrates proficiency in handling both
generation and categorization templates within the Persian context. Compared
to Aya, Co-CoLA enhances the model’s ability to manage both categorization and
generation tasks effectively while being less computationally expensive.

##  7 Conclusion

This study introduces significant advancements with FarsInstruct and Co-CoLA,
addressing critical gaps in the processing and instruction-following
capabilities for Persian, a low-resource language. FarsInstruct, with its
diverse tasks ranging from text summarization to named entity recognition, has
proven to enhance language model performance as shown through rigorous ROUGE
evaluations and human assessments. This dataset not only enriches multilingual
model training but also establishes a new standard for language model
instruction tuning.

Further, Co-CoLA leverages the strengths of CoLA with rehearsal training to
mitigate catastrophic forgetting and improve multi-task adaptation, through
its iterative optimization framework. This allows for sustained model
performance over diverse tasks while optimizing computational resources.
Looking ahead, the focus will be on expanding the scope of these datasets to
cover more tasks and modalities, thereby driving further innovations in cross-
lingual language understanding and promoting AI inclusivity.

##  8 Limitations

This section delineates the principal limitations of our study, which, while
providing substantial contributions to Persian NLP, presents challenges that
could be addressed in future developments to enhance its utility and
applicability in broader linguistic contexts:

Data Diversity and Representation: Although FarsInstruct broadens the corpus
of Persian language resources, it does not fully capture the rich tapestry of
dialects and sociolects that characterize the Persian-speaking world. Also,
the collected templates are generally biased towards short responses, which
might affect the overall performance of the model.

Complexity of Instructions: The dataset prompts vary in complexity but still
may not sufficiently challenge or train models to handle the types of complex
instructions encountered in everyday human interactions. Real-world
applications often demand a high level of interpretative depth and context
awareness—qualities that current models may struggle with when trained on
existing datasets. Future versions of FarsInstruct could benefit from
integrating prompts that require higher-order cognitive processing, such as
irony, metaphor understanding, and techniques that involve prompting the model
to break down complex tasks into intermediate steps, mimicking human reasoning
processes (wei2022chain).

Dependency on External Datasets: The effectiveness of the FarsInstruct dataset
is contingent upon the quality and variety of the external datasets. This
dependency creates vulnerability, as biases or errors in source datasets may
be passed to FarsInstruct.A rigorous process for source data, coupled with
efforts to develop original, high-quality training materials, could diminish
reliance on external datasets and enhance the overall integrity of the
dataset.

Evaluation Metrics: The metrics currently used to evaluate models trained on
FarsInstruct may not fully capture the nuanced and multifaceted aspects of
language comprehension and generation. Furthermore, for certain tasks such as
rewriting, ROUGE-L may not serve as an adequate measure of quality.

Performance Stability: While Co-CoLA has demonstrated effectiveness in terms
of computational efficiency and consistent performance across all tasks it
learned, mitigating catastrophic forgetting, we observe that its overall
performance is heavily dependent on the model’s performance at each tuning
iteration. We leave potential solutions to this problem to future work.

## Appendix

## Training Configuration

All implementations were carried out using PyTorch, Transformers (wolf-
etal-2020-transformers) and Accelerate (accelerate) library. For efficient
training, we randomly selected 25 prompt templates and applied them to their
corresponding datasets. Consequently, for example, a dataset with two selected
templates would be upsampled to twice its original size. We then sampled a
minimum of 10,000 examples from each dataset, based on the specific template
and dataset length, to create the current training data. The rehearsal hyper-
parameter of Co-CoLA was set to 0.01. We used Paged-AdamW as the base
optimizer and trained for a total of four epochs in each tuning phase. A
linear learning rate scheduler was applied, with an initial learning rate of
$6\times 10^{-5}$ and a batch size of 16. For implementing LoRA, we utilized
the PEFT (peft) library for convenience.

## Dataset Illustrations

Dataset | Categorization | Generation
---|---|---
DigiMag | 9 | 1
Digikala_sentiment_analysis | 9 | 1
ExaPPC | 3 | 4
FarsTail | 8 | 2
ParsABSA | 5 | 1
ParsiNLU_entailment | 8 | 3
ParsiNLU_multiple_choice | 9 | 1
ParsiNLU_query_paraphrasing | 7 | 3
ParsiNLU_reading_comprehension | 1 | 9
ParsiNLU_sentiment | 3 | 7
ParsiNLU_translation_En_FA | 1 | 5
ParsiNLU_translation_FA_En | 1 | 5
PEYMA | 1 | 9
Persian_NER | 3 | 10
Persian_news | 3 | 3
Persian_QA | 1 | 9
Pn_summary | 3 | 8
Snappfood_sentiment_analysis | 7 | 4
Syntran_FA | 1 | 9
Wiki_summary | 1 | 7
XL_WiC | 10 | 0
Table 4: Detailed Overview of Datasets Utilized for Categorization and
Generation Tasks. Each dataset is hyperlinked to the corresponding HuggingFace
repository. As shown in this table Categorization and Generation tasks are not
equally distributed across all datasets. Some datasets, such as Digimag, are
originally designed for categorization tasks. We have enhanced these datasets
by incorporating generation prompts. Conversely, translation tasks, which are
inherently generative, have been augmented with categorization prompts. This
dual-purpose approach enriches the datasets, facilitating both categorization
and generation tasks and providing a more versatile training and testing
framework. This table provides insight into the distribution and
specialization of prompts across different datasets, highlighting the balance
and focus within the training and testing framework.

Figure 6: A treemap visualization that organizes datasets by task type, post-
instruction application size, and data category (training vs. testing). Each
primary rectangle represents a distinct task type within the natural language
processing field, encompassing areas such as Question Answering,
Classification, Translation, and more. Within these primary rectangles,
smaller sub-rectangles represent individual datasets. The area of each sub-
rectangle is scaled to the logarithm of the size of the dataset to accommodate
the broad variance in dataset sizes, ensuring a more balanced visual
representation that allows for the inclusion of both large and small datasets
on the same scale.

## Prompts

Figure 7: A prompt example shown in Promptsource environment. PromptSource is
an advanced toolkit designed for creating, sharing, and utilizing natural
language prompts. Prompts function as mappings that convert examples from
datasets into natural language inputs and corresponding target outputs. In
PromptSource, we develop input templates, target templates, and choice
templates. Inputs typically consist of questions or instructions, while the
output code specifies the expected answer or result. For Categorization tasks,
the choice template includes predefined options for answering questions, while
Generation tasks do not require this template. In this picture, The "Metrics"
box is set to measure Accuracy for Categorization tasks, and the "Prompt
Language" used is Farsi (Persian). "Answer choices" are provided within the
template, which comprises an instruction followed by data fields. The premise
and hypothesis are selected from the "Data Schema" on the left side of the
interface. The $|||$ symbol separates instructions from outputs, and the
output employs Jinja code for conditional logic: if the label is c, it outputs
(no); if the label is e, it outputs (yes); and if the label is n, it outputs
(cannot determine).

See pages 2- of figures/FarsInstruct.pdf