The advancement of function-calling agent models requires diverse, reliable, and high-quality datasets.
This paper presents APIGen, an automated data generation pipeline designed to synthesize verifiable high-quality datasets for function-calling applications. We leverage APIGen and collect 3,673 executable APIs across 21 different categories to generate diverse function-calling datasets in a scalable and structured manner. Each data in our dataset is verified through three hierarchical stages: format checking, actual function executions, and semantic verification, ensuring its reliability and correctness.
We demonstrate that models trained with our curated datasets, even with only 7B parameters, can achieve state-of-the-art performance on the Berkeley Function-Calling Benchmark, outperforming multiple GPT-4 models. Moreover, our 1B model achieves exceptional performance, surpassing GPT-3.5-Turbo and Claude-3 Haiku. We release a dataset containing 60,000 high-quality entries, aiming to advance the field of function-calling agent domains. The dataset is available on Huggingface [<https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k>] and the project homepage [<https://apigen-pipeline.github.io/>].

§ INTRODUCTION

Function-calling agents represent a significant advancement in artificial intelligence, specifically within the realm of Large Language Models (LLMs). These models, such as GPT4 [1], Gemini [2], and Mistral [3], have evolved to not only understand and generate human-like text but also to execute functional API calls based on natural language instructions. For instance, consider a user requesting the weather in Palo Alto, as illustrated in Fig. <ref>. The function-calling agent interprets this query, accesses the relevant API—such as —and retrieves the weather information, all in real-time. This capability extends the utility of LLMs beyond simple conversation tasks to include dynamic interactions with a variety of digital services and applications, ranging from social media platforms to financial services [4, 5, 6, 7, 8].

Despite their growing popularity and potential, the deployment of function-calling agents is often hampered by the quality of the datasets used for training. Current datasets are largely static and lack comprehensive verification, leading to potential inaccuracies and inefficiencies of model fine-tuning in real-world applications [9, 10, 11, 12]. This limitation is particularly evident when models trained on these datasets encounter new, unseen APIs.
For example, a model trained primarily on restaurant booking APIs may struggle when suddenly tasked with retrieving stock market data, as it lacks the specific training data or the adaptability to handle new domains.

To address these challenges, we introduce APIGen, an Automated PIpeline for Generating verifiable and diverse function-calling datasets. Our framework is designed to facilitate the fine-tuning of function-calling LLMs by providing high-quality, diverse datasets that better reflect the variability and complexity of real-world API use.
Crucially, each generated data point undergoes rigorous multi-stage verification processes—format, execution, and semantic—to ensure accuracy and applicability.
We fine-tune function-calling models using the dataset generated by APIGen. The results show the strong performance of our models, surpassing many existing powerful LLMs with much fewer parameters, highlighting the effectiveness of APIGen and the high quality of the dataset it produces.

With APIGen, we release a comprehensive dataset containing 60,000 entries with 3,673 APIs across 21 categories. They include various query styles, such as parallel function calling data (asking the agent to produce multiple concurrent function calls in a single response) [9], which is rarely found in public datasets, to the best of our knowledge. This large-scale synthetic dataset is intended to catalyze further research and development in the field of function-calling agents, offering researchers and developers a foundation for training and testing their models. The data is available on Huggingface and our project homepage.

The contributions of this work are summarized as follows:

* We introduce APIGen, a function-calling data generation pipeline that features verifiability, scalability, and diversity of the data. APIGen is compatible with a range of models and APIs to construct high-quality synthetic function-calling datasets.

* We train two function-calling models of different sizes, 1.3B and 6.7B, using APIGen-constructed training data. Extensive experiments demonstrate that the 6.7B model achieves a rank of 6th on the Berkeley Function-Calling Leaderboard [9], surpassing GPT-4o and Gemini-1.5-Pro, while the 1.3B model outperforms GPT-3.5-Turbo.

* We also release a synthetic function-calling dataset containing 60,000 high-quality data generated by APIGen using several strong open-source LLMs, which can potentially benefit the research community in developing advanced function-calling models.

Workflow of an LLM-based function-calling agent.

§ RELATED WORK

Tool-use Agent.
Recent works have developed frameworks and models that
enable LLMs to interact with APIs and tools [13, 14, 15, 16, 17, 18, 19, 20]. RestGPT [21] connects LLMs to RESTful APIs using a Planner, API selector, and API executor to handle complex instructions. Toolformer [22] is an early work that enables agents to use tools like Question Answering, Calculator, and Wikipedia Search through a supervised-finetuned model.
[23] designs the xLAM model, showing strong tool usage capability across several benchmarks. Octopus-v4 [24] presents a methodology to incorporate multiple specialized language models to solve corresponding tasks. While
NexusRaven [5] and Gorilla OpenFunctions-v2 [25] are strong open-sourced models that focus on function calling, neither provides access to their training datasets.

Agent Datasets.
Several datasets have been created to support the development of agent models. AgentInstruct [18] consists of 6 datasets for different agent tasks, including AlfWorld [26], WebShop [27], Mind2Web [28], Knowledge Graph, Operating System, and Database [29]. APIBank [12] is a benchmark designed for tool-augmented LLMs, providing a training set containing tool-use dialogues from various APIs. Toolalpaca [30] constructs a varied and well-structured tool-use dataset by randomly selecting APIs and generating documentation using ChatGPT. ToolBench [10] creates an instruction-tuning dataset for tool use by collecting diverse REST APIs and generating their descriptions using ChatGPT. AgentOhana [23] and Lumos [31] design a unified data and training pipeline for efficient agent learning, covering multiple different datasets and environments. However, most of these datasets were not rigorously verified, and usually contain noisy data.

Recent studies have established several benchmarks to assess agent abilities on various tasks such as web interactions, reasoning, decision making, function calling, and tool usage [27, 8, 29, 7, 32, 10, 11, 33, 34, 35]. Specifically, AgentBoard [34] includes 9 tasks, with ToolOperation and ToolQuery designed to evaluate agent ability on multi-turn interaction with external tools. ToolEval [10] assesses functional calling capabilities via RapidAPI, containing around 1,000 test cases and asking GPT-3.5 to assess the Win Rate. Furthermore, the Berkeley Function-Calling Leaderboard (BFCL) [9] provides a robust and comprehensive framework to evaluate models' abilities to call functions, with 2,000 test cases covering a wide range of scenarios. We use BFCL as our testing ground as it provides the most thorough comparison among popular LLMs.

§ APIGEN FRAMEWORK

This section introduces the detailed design of APIGen, an Automated PIpeline for Generating verifiable and diverse function-calling datasets. Our framework is designed with three key factors in mind: data quality, data diversity, and collection scalability. We achieve these through the key modules shown in Fig. <ref>: the multi-stage data verification process ensures data quality, the seed QA (query-answer) data sampler, API sampler, and various prompt templates ensure diversity, and our structured modular design using a unified format enables the system to scale to diverse API sources, including but not limited to Python functions and representational state transfer (REST) APIs.

Illustration of the post-process filters.

§.§ Data Generation Overview

Figure <ref> outlines the data generation process using the APIGen framework, which begins by sampling one or more APIs and example query-answer (QA) pairs (seed data) from the library, then formatting them into a standardized JSON format (see Fig. <ref> for examples). A prompt template is selected based on the desired data generation objectives, which steers the LLM in generating relevant query-answer pairs. Each answer in the generated pairs is a function call formatted in JSON.

The adoption of a standardized JSON format for APIs, function calls, and generator outputs (as shown in Figure <ref>) provides several advantages. Firstly, it establishes a structural way to verify whether the generator's output contains all necessary fields. Outputs that fail to comply with these format requirements are discarded. Secondly, the JSON structure enables efficient checking of function calls for correct parsing and validity of arguments. Calls that include arguments not present in the API library or hallucinate non-existent functions are excluded, enhancing the overall quality of the dataset.
Another key benefit is the scalability it enables. With this uniform format, APIGen can easily incorporate data from diverse sources (Python functions, REST APIs, etc) by developing format converters that adapt them into these basic JSON elements, without modifying other core components, such as the prompting library, making the framework highly adaptable and extensible.

The generated function calls are subjected to a multi-stage verification process to ensure their correctness and relevance.
First, a format checker verifies correct JSON formatting and parseability. Next, the API execution engine processes the calls and sends the results and queries to a semantic checker, another LLM, which assesses alignment between the function calls, execution results, and query objectives. Data points passing all stages are added back to the seed dataset as high-quality examples to enhance future generation diversity.
We detail each checker in the next section.

JSON data format examples.

§.§ Multi-Stage Data Verification

Prioritizing quality is crucial, as previous research has shown that small amounts of high-quality fine-tuning data can substantially enhance model performance on domain-specific tasks [36]. This motivates our multi-stage dataset verification process to align large language models effectively.

The key insight driving our framework design is that, unlike synthetic chat data which can be difficult to evaluate, function-calling answers can be directly executed via their corresponding APIs. This enables checking if the output API and parameters' formats are correct, if the generated API calls are executable, and if execution results match the query's intent, etc. Based on this observation, we propose a three-stage verification process:

Stage 1: Format Checker: This stage performs sanity checks to filter out poorly formatted or incomplete data. The LLM output must strictly follow a JSON format with the "query" and "answer" fields, as shown in Fig. <ref>. We usually also include an additional "thought" field, which is known as the chain-of-thought (CoT) prompting technique [37], to increase the pass rate of the generated data. The data is discarded if these fields cannot be properly extracted for function calls. Additionally, the function calls are checked for correct JSON parsing and valid arguments. Generated calls whose arguments or functions are not present in the given APIs are eliminated to reduce hallucination and improve data quality.

Stage 2: Execution Checker: Well-formatted function calls from Stage 1 are executed against the appropriate backend (e.g. Python functions are directly imported and executed in a separate subprocess, while REST APIs are called to obtain results and status codes). Unsuccessful executions are filtered out, and fine-grained error messages are provided for failures, including argument type errors, invalid parameters, runtime errors, timeout, syntax errors, missing arguments, etc.

Stage 3: Semantic Checker: Successful Stage 2 execution results, available functions, and the generated query are formatted and passed to another LLM to assess if the results semantically align with the query's objective. Query-answer pairs that execute successfully but produce meaningless results due to infeasible queries or incorrect arguments are filtered out. The main decision factors for this stage are: 1) whether the function call aligns with the query's objective and has proper arguments; 2) whether the function call and arguments are appropriately chosen from the available functions; 3) whether the number of function calls matches the user's intent; 4) whether the execution results contain errors or indicate unsuccessful function execution; 5)  whether the execution results are relevant and match the query's purpose. APIGen's design offers the flexibility to select one or multiple LLMs as checkers, and the filtering rules can be readily adjusted—either tightened or relaxed—depending on specific use cases.

Data points that pass all three verification stages are regarded as high-quality and added back to improve future diverse data generation. This multi-stage verification process is the key to ensuring the APIGen framework produces a dataset that is not only diverse but also of verifiable high quality, enabling more effective fine-tuning of LLMs to domain-specific API-related tasks.

§.§ Methods to Improve Dataset Diversity

Encouraging diversity in training datasets is crucial for developing robust function-calling agents that can handle a wide range of real-world scenarios. In APIGen, we promote data diversity through multiple perspectives, including query style diversity, sampling diversity, and API diversity.

Query Style Diversity.
APIGen's dataset is structured into four main categories: simple, multiple, parallel, and parallel multiple, each designed to challenge and enhance the model's capabilities in different usage scenarios. These categories are inspired by the Berkeley function-calling benchmark [9] and are controlled by corresponding prompts and seed data. We show examples of them in the supplementary material. The categories are as follows:

* Simple: This query style includes straightforward scenarios where a single function call is made based on the user's input with a single provided JSON format API description.

* Multiple: In this style, user queries could be answered by one of several function calls. The challenge lies in selecting the most appropriate function from multiple provided APIs. It represents one of the most common real-world use cases.

* Parallel: This query style requires executing multiple function calls simultaneously in response to a single user query, which may consist of one or more sentences but with only one API provided. For instance, if the user wants to know the weather in both  and , the model should call the  function twice with corresponding city names in a single response.

* Parallel Multiple: This query style combines the parallel and multiple categories, where multiple function and API documents are provided, and each function call might be invoked multiple times based on the query's requirements.

While there exist publicly available training data for simple and multiple categories [38, 10], however,
to the best of our knowledge, we offer the first large-scale and high-quality datasets that include the parallel-related function-calling scenario.

Sampling Diversity.
APIGen utilizes a sampling system designed to maximize the diversity and relevance of the generated datasets, which include three main components, as shown in Fig. <ref>:

* API Sampler: This module extracts one or more function descriptions from executable API libraries, standardizing them into a uniform JSON format. The diverse sources of APIs ensure a wide range of function calls are available for inclusion in the training dataset.

* Example Sampler: It samples a specified number of seed examples corresponding to the different categories. These examples are transformed into structured queries, function descriptions, and answers, serving as an important few-shot reference for data generation.

* Prompt Sampler: This sampler draws from a diverse prompt library to generate a variety of query-answer pairs.
The prompts for each query style contain different contexts, ranging from simple, concise query-answer pairs to more realistic scenarios, such as ambiguous or misspelled user requests, enhancing the model's ability to handle real-world interactions.

We provide some prompt templates and seed data in the supplementary material.
In APIGen, the number of examples and APIs sampled for each dataset iteration is randomly chosen from a predefined range. This randomization enhances dataset variability by preventing repetitive patterns and ensuring a broad coverage of scenarios. We next introduce our API diversity.

§ DATASET PREPARATION AND COLLECTION

We begin by discussing our dataset preparation process, which includes selecting and cleaning API libraries. Then we present our dataset collection setup and an overview of the resulting dataset.

§.§ Dataset API Sources

To ensure a high-quality and diverse dataset, we focused on collecting real-world APIs that could be readily executed and came with thorough documentation. We primarily sourced APIs from ToolBench [10], a comprehensive tool-use dataset that includes 16,464 REST APIs across 49 coarse-grained categories from RapidAPI Hub. This hub is a leading marketplace featuring a vast array of developer-contributed APIs.
To further enhance the usability and quality of the APIs, we perform the following filtering and cleaning procedures on the ToolBench dataset:

* Data Quality Filtering: We remove APIs with incorrectly parsed documentation and those lacking required or optional parameters. APIs requiring no parameters were excluded to maintain the challenge level appropriate for our dataset needs.

* API Accessibility Testing: We tested API accessibility by making requests to each endpoint using example parameters provided in the dataset and through the Stable Toolbench server [38]. APIs that could not be executed or returned errors, such as timeouts or invalid endpoints, were discarded.

* Docstring Regeneration: To improve the quality of API documentation, we regenerated docstrings for the APIs that have noisy and unusable descriptions.

The category distribution of the 3,673 executable APIs.

After cleaning, we obtain 3,539 executable REST APIs with good documentation.
Additionally, we incorporated Python functions as another API type, inspired by the executable evaluation categories of the Berkeley function-calling benchmark [9]. We collected 134 well-documented Python functions covering diverse fields such as mathematics, finance, and data management. Sample API examples are provided in the supplementary material.

The original ToolBench dataset contained semantically overlapping categories such as  and . We consolidated these into 21 distinct categories to ensure clarity and balance across the dataset. Figure <ref> illustrates the distribution of the 3,673 executable APIs across these redefined categories, spanning sectors like technology, social sciences, education, and sports.
This diverse collection of APIs provides a strong foundation for synthetic data generation and is a valuable asset for ensuring data quality and reliability.

§.§ Collection Setup and Dataset Details

To validate the effectiveness of the APIGen framework, we generated datasets targeting various query styles as outlined in Section <ref>. We utilized several base LLMs for data generation, including DeepSeek-V2-Chat (236B) [39], DeepSeek-Coder-33B-Inst [40], Mixtral-8x22B-Inst, and Mixtral-8x7B-Inst [3]. For each model, our target was to generate 40,000 data points by sampling different combinations of APIs, seed data, and prompt templates. To foster diversity in the generated responses, we set the generation temperature to 0.7 across all models. Examples of the prompt templates and APIs used are provided in the supplementary materials for reference.

Table <ref> presents statistics for the data generation process with different models, including the total verified data point count and the number of filtered data points at each verification stage. The filtering process successfully removes many low-quality data points due to formatting issues, execution errors, or failure to pass the semantic check. The first two stages, format checker and execution checker, typically filter out the majority of low-quality data. These data points often have infeasible argument ranges, incorrect types, missing required parameters, or more severe issues such as hallucination of function calls or parameters. Our systematic verification process provides a rigorous way to reduce the occurrence of these situations.

Filtering statistics for the generated datasets using different base LLMs.

Model     Verified Data     Fail Format     Fail Execution     Fail Semantic     Pass Rate
DeepSeek-Coder-33B-Inst     13,769     4,311     15,496     6,424     34.42%

Mixtral-8x7B-Inst     15,385     3,311     12,341     7,963     38.46%

Mixtral-8x22B-Inst     26,384     1,680     5,073     6,863     65.96%

DeepSeek-V2-Chat (236B)     33,659     817     3,359     2,165     84.15%

The semantic checker also plays a crucial role in filtering generated data that does not align with the query's objectives. For instance, if a user's query contains multiple requests, but the returned results only address one, or if the generated function-call data and execution results do not match the user's query, the data point will be filtered out. Including these data points in the training set for model training could potentially harm the performance, as demonstrated in the experiments.

We observe that stronger models like DeepSeek-V2-Chat and Mixtral-8x22B-Inst have better format-following capabilities and higher pass rates, while the two relatively smaller models have a much higher likelihood of producing data that cannot be executed. This suggests that when using weaker models to generate data, a strict verification process is recommended to filter out low-quality data.

We are releasing approximately 60,000 high-quality function-calling datasets generated from the two strongest models: Mixtral-8x22B-Inst and DeepSeek-V2-Chat (236B). These datasets include all the query styles mentioned in Sec. <ref> and cover a wide range of practical situations, with 3,673 diverse APIs across 21 categories. Each data point has been verified using real-world APIs to ensure its validity and usefulness. By making this dataset publicly available, we aim to benefit the research community and facilitate future work in this area.

§ EXPERIMENTS

§.§ Experiment Setup

To evaluate the utility and effectiveness of the collected dataset, we conducted experiments by training function-calling models with the generated data. Our aim is to answer two key questions:
1) To what extent can the generated data boost the model's function-calling capability, and how does it compare to existing models?
2) How effective is the APIGen framework in filtering out low-quality data?

To address these questions, we train two versions of base models: DeepSeek-Coder-1.3B-instruct and DeepSeek-Coder-7B-instruct-v1.5 [40] using the xLAM (large action model) training pipeline proposed in [23]. We refer to these models as xLAM-1B (FC) and xLAM-7B (FC), where FC stands for the Function-Calling mode, similar to this mode in other existing models that output JSON-format function calls [1, 25, 41, 4]. We compare the performance of these small-sized models against state-of-the-art models, including different versions of GPT-4 series [1], Claude-3 series [42], Gemini series [2], Llama3 [43], Mixtral [3], OpenFunctions-v2 [25], Command R+ [41], etc.

Benchmark. We evaluate the trained models' performance on the Berkeley Function-Calling Benchmark (BFCL) [9], which provides a comprehensive evaluation framework for assessing the function-calling capabilities of LLMs across various programming languages and application domains. Designed to reflect real-world use cases, the BFCL includes 2,000 testing cases, covering complex scenarios such as parallel and multiple-function calls. The benchmark contains diverse API sources like Java, JavaScript, and Python, offering a detailed analysis of each model's ability to correctly interpret and execute commands under different conditions.
BFCL serves as a highly detailed and scalable benchmark for evaluating LLMs' function-calling capabilities and provides a leaderboard to track the most recent and powerful LLMs, both commercialized and open-source.

Evaluation Metrics.
The Berkeley Function-Calling Leaderboard (BFCL) evaluates LLMs using two main categories: Abstract Syntax Tree (AST) Evaluation and Executable Function Evaluation. The AST evaluation focuses on the syntactic accuracy of the generated function calls, ensuring that the model's output matches a predefined function documentation in structure and parameters. This includes checks for correct function names, required parameters, and appropriate data types. The Executable Function Evaluation goes a step further by running the generated function calls to verify their operational correctness. This executable test ensures that the functions not only compile but also execute correctly, providing the expected results, which is crucial for practical applications where real-time performance is essential.

§.§ Experiment Results Analysis

Can the generated data improve the model's function-calling capability and how does it compare to other most powerful models?
The performance of our models, xLAM-7B and xLAM-1B, as presented in Table <ref>, highlights the effectiveness of our APIGen framework and the quality of the datasets produced.
Notably, our xLAM-7B model ranks 6th among the most powerful LLMs listed on the BFCL leaderboard [<https://gorilla.cs.berkeley.edu/leaderboard.html#leaderboard>], surpassing several versions of GPT-4 (GPT-4o, GPT4-Turbo-FC), Llama3-70B, multiple Claude-3 models, and a series of strong models which are known for their exceptional capabilities in various tasks, including function-calling.
This achievement demonstrates the significant impact of our high-quality dataset on the model's function-calling performance.

Our smaller xLAM-1B model also shows remarkable results, securing the 24th position and outperforming many larger models, such as Claude-3 Haiku [42], Command-R-Plus [41], DBRX-Instruct [44], Mistral-large [3], and GPT-3.5-Turbo-0125.
The results highlight the effectiveness of the APIGen pipeline in enhancing a model's function-calling capabilities, even with a much smaller size.
Both xLAM-7B and xLAM-1B demonstrate substantial improvements in handling complex query types, particularly in the parallel and multiple function-calling scenarios, which are typically underrepresented in existing publicly available dataset. This validates the value of our pipeline and datasets in addressing practical scenarios involving complex API interactions and multiple concurrent API calls, especially considering that the base model, DeepSeek-Coder-v1.5, only ranks 45th on the leaderboard and performs poorly in these categories.

Performance comparison of using different stage's datasets from APIGen. “+Fail Semantic Data" and “+Fail Execution Data" meaning adding the filtered dataset from stage 3 and stage 2 to the training set.

Next, we answer the question: how effective is the APIGen framework in filtering out low-quality data? We conducted an ablation study by adding the datasets that were filtered out by stage 3 (semantic checker) and stage 2 (execution checker) back to the training set, simulating situations where generated data is used without the rigorous verification process. The performance comparison on the BFCL benchmark, shown in Fig. <ref>, reveals that using these filtered datasets for training harms the final performance, with a more significant impact on the smaller model. This indicates that directly using generated data might not yield the best results and demonstrates the effectiveness of our APIGen framework in filtering out low-quality data.

These results provide compelling evidence for the effectiveness of the APIGen framework in generating high-quality, diverse datasets for function-calling tasks. The impressive performance achieved by our small-sized models highlights the efficiency of our approach, demonstrating that by focusing on data quality and diversity, we can effectively boost the performance of smaller models, making them competitive with much larger ones in this function-calling agent domain.

§ CONCLUSION

In this paper, we introduced APIGen, a novel framework that generates reliable and diverse function-calling datasets by leveraging a multi-stage verification process. Our experiments demonstrate the effectiveness of APIGen in producing high-quality datasets from a wide range of executable APIs. This has significant implications for the development of more efficient and accessible language models, as it shows that high-quality data can be as important as model size in achieving strong performance.
By enabling smaller models to achieve competitive results and significantly enhancing their function-calling capabilities, our approach and released dataset open up new possibilities for the development of efficient and powerful language models in the agent tool-use domains.

However, the current version of APIGen and the generated dataset have some limitations. Presently, the framework and dataset only consider REST APIs and Python functions. Additionally, although APIGen is a general framework, it currently only implements the generation procedure for single-turn function calling.
Future work will focus on extending APIGen to support more scenarios, programming languages, and APIs. We also plan to extend the framework to handle multi-turn and more complex interactions between agents and tools. Despite these limitations, we believe that APIGen and the generated dataset represent a significant step forward in the development of efficient and effective function-calling agents.

[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.

[2]
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530, 2024.

[3]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
arXiv preprint arXiv:2401.04088, 2024.

[4]
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.
Gorilla: Large language model connected with massive apis.
arXiv preprint arXiv:2305.15334, 2023.

[5]
Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang.
Nexusraven: a commercially-permissive language model for function calling.
In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.

[6]
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al.
Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents.
arXiv preprint arXiv:2308.05960, 2023.

[7]
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al.
Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents.
ICLR, 2023.

[8]
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al.
Webarena: A realistic web environment for building autonomous agents.
arXiv preprint arXiv:2307.13854, 2023.

[9]
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.
Berkeley function calling leaderboard.
<https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html>, 2024.

[10]
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
ICLR, 2024.

[11]
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji.
Mint: Evaluating llms in multi-turn interaction with tools and language feedback.
arXiv preprint arXiv:2309.10691, 2023.

[12]
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.
Api-bank: A comprehensive benchmark for tool-augmented llms.
In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.

[13]
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023.

[14]
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al.
Metagpt: Meta programming for multi-agent collaborative framework.
arXiv preprint arXiv:2308.00352, 2023.

[15]
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.
Fireact: Toward language agent fine-tuning.
arXiv preprint arXiv:2310.05915, 2023.

[16]
Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, and Ding Zhao.
Creative robot tool use with large language models.
arXiv preprint arXiv:2310.13065, 2023.

[17]
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al.
Lemur: Harmonizing natural language and code for language agents.
arXiv preprint arXiv:2310.06830, 2023.

[18]
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.
Agenttuning: Enabling generalized agent abilities for llms.
arXiv preprint arXiv:2310.12823, 2023.

[19]
Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, and Rasool Fakoor.
Tail: Task-specific adapters for imitation learning with large pretrained models.
arXiv preprint arXiv:2310.05905, 2023.

[20]
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu.
Webvoyager: Building an end-to-end web agent with large multimodal models.
arXiv preprint arXiv:2401.13919, 2024.

[21]
Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.
Restgpt: Connecting large language models with real-world restful apis, 2023.

[22]
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
Advances in Neural Information Processing Systems, 36, 2024.

[23]
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al.
Agentohana: Design unified data and training pipeline for effective agent learning.
arXiv preprint arXiv:2402.15506, 2024.

[24]
Wei Chen and Zhiyuan Li.
Octopus v4: Graph of language models.
arXiv preprint arXiv:2404.19296, 2024.

[25]
Charlie Cheng-Jie Ji, Huanzhi Mao, Fanjia Yan, Shishir Patil, Tianjun Zhang, Ion Stoica, and Joseph Gonzalez.
Gorilla openfunctions v2.
In <https://gorilla.cs.berkeley.edu//blogs/7_open_functions_v2.html>, 2024.

[26]
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
Alfworld: Aligning text and embodied environments for interactive learning.
arXiv preprint arXiv:2010.03768, 2020.

[27]
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents.
Advances in Neural Information Processing Systems, 35:20744–20757, 2022.

[28]
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
Advances in Neural Information Processing Systems, 36, 2024.

[29]
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al.
Agentbench: Evaluating llms as agents.
arXiv preprint arXiv:2308.03688, 2023.

[30]
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.
Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.
arXiv preprint arXiv:2306.05301, 2023.

[31]
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin.
Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs.
arXiv preprint arXiv:2311.05657, 2023.

[32]
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, et al.
Metatool benchmark for large language models: Deciding whether to use tools and which to use.
arXiv preprint arXiv:2310.03128, 2023.

[33]
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al.
Datasets and benchmarks for offline safe reinforcement learning.
arXiv preprint arXiv:2306.09303, 2023.

[34]
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He.
Agentboard: An analytical evaluation board of multi-turn llm agents.
arXiv preprint arXiv:2401.13178, 2024.

[35]
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al.
Agentlite: A lightweight library for building and advancing task-oriented llm agent system.
arXiv preprint arXiv:2402.15538, 2024.

[36]
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.
Lima: Less is more for alignment.
Advances in Neural Information Processing Systems, 36, 2024.

[37]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
Advances in neural information processing systems, 35:24824–24837, 2022.

[38]
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu.
Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.

[39]
Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.

[40]
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al.
Deepseek-coder: When the large language model meets programming–the rise of code intelligence.
arXiv preprint arXiv:2401.14196, 2024.

[41]
Command r plus: Enhanced retrieval-augmented generation with microsoft azure.
<https://cohere.com/blog/command-r-plus-microsoft-azure>, 2024.
Accessed: 2024-04-04.

[42]
AI Anthropic.
The claude 3 model family: Opus, sonnet, haiku.
Claude-3 Model Card, 2024.

[43]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.

[44]
Introducing dbrx: A new state-of-the-art open llm.
<https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm>, 2024.
Accessed: 2024-03-27.

§ DATASET DOCUMENTATION AND ACCESSIBILITY

§.§ Dataset Documentation and Intended Uses

The dataset generated using the APIGen framework is intended for training and evaluating function-calling agents. The dataset consists of diverse query-answer pairs, where the answers are verified function calls that could address the requested query with provided APIs. The APIs and function calls are in a standardized JSON format, as demonstrated in the main paper Fig. 3. More details of the format and examples are available in Appendix <ref>. The dataset covers a wide range of API categories and includes various query styles, such as simple, multiple, parallel, and parallel multiple function calls, as introduced in [9].

Hosting, Licensing, and Maintenance Plan.
The dataset currently can be viewed and downloaded from our project homepage [<https://apigen-pipeline.github.io/>] or via Huggingface [<https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k>].
All datasets are licensed under the Creative Commons Attribution 4.0 License (CC BY).
We also plan to open-source the trained models on Huggingface once after the company's legal approval.
As for maintenance, we have established a long-term plan to keep the datasets up-to-date, correct any potential issues, and provide support to users. We also aim to expand these datasets further based on new advances in the field, thus continually promoting progress in the field of function-calling agent training.

Author Responsibility Statement.
As the authors, we hereby affirm that we bear full responsibility for the datasets provided in this submission. We confirm that to the best of our knowledge, no rights are violated in the collection, distribution, and use of these datasets.

§.§ JSON Data Format and Examples

This JSON data format is used to represent a query along with the available tools and the corresponding answers. Here's a description of the format:

§.§.§ Dataset Structure

The JSON data structure comprises three main keys: , a string representing the problem statement; , an array of tools each defined by properties such as , , and  that further describe each tool's required and optional parameters with their types and descriptions; and , an array detailing responses with the tool used () and the arguments provided () for each answer, thereby aligning tools with their respective query intentions.
The detailed description of each data point's entries is as follows.

*  (string): The query or problem statement.

*  (array): An array of available tools that can be used to solve the query.

* Each tool is represented as an object with the following properties:

*  (string): The name of the tool.

*  (string): A brief description of what the tool does.

*  (object): An object representing the parameters required by the tool.

* Each parameter is represented as a key-value pair, where the key is the parameter name and the value is an object with the following properties:

*  (string): The data type of the parameter (e.g., "integer", "float", "array").

*  (string): A brief description of the parameter.

*  (boolean): Indicates whether the parameter is required or optional.

*  (array): An array of answers corresponding to the query.

* Each answer is represented as an object with the following properties:

*  (string): The name of the tool used to generate the answer.

*  (object): An object representing the arguments passed to the tool to generate the answer.

* Each argument is represented as a key-value pair, where the key is the parameter name and the value is the corresponding value.

§.§.§ Example Data

Here's an example JSON data for the simplest scenario.

"query": "What is the weather in Palo Alto?",
"tools": [

"name": "weather_api.get_current_weather",
"description": "Retrieves the current weather conditions for a specified location.",
"type": "string",
"description": "The name of the city or geographic location.",
"required": true
"type": "string",
"description": "The units for temperature measurement (e.g., 'Celsius', 'Fahrenheit').",
"required": false

"answers": [

"name": "weather_api.get_current_weather",
"location": "Palo Alto",
"units": "Celsius"

In this example, the query asks about the current weather in Palo Alto.
The tools array contains a single entry for , describing the tool used to retrieve weather data, including parameters for location and units.
The answers array lists the specific API call made with the location set as  and units as .

Here's an example JSON data for the parallel function-calling category, i.e., the user's query contains multiple intentions and the answers contain multiple parallel tool calls:

"query": "Find the sum of all the multiples of 3 and 5 between 1 and 1000. Also find the product of the first five prime numbers.",
"tools": [

"name": "math_toolkit.sum_of_multiples",
"description": "Find the sum of all multiples of specified numbers within a specified range.",
"type": "integer",
"description": "The start of the range (inclusive).",
"required": true
"type": "integer",
"description": "The end of the range (inclusive).",
"required": true
"type": "array",
"description": "The numbers to find multiples of.",
"required": true

"name": "math_toolkit.product_of_primes",
"description": "Find the product of the first n prime numbers.",
"type": "integer",
"description": "The number of prime numbers to multiply together.",
"required": true

"answers": [

"name": "math_toolkit.sum_of_multiples",
"lower_limit": 1,
"upper_limit": 1000,
"multiples": [3, 5]

"name": "math_toolkit.product_of_primes",
"count": 5

In this example, the query asks to find the sum of multiples of 3 and 5 between 1 and 1000, and also find the product of the first five prime numbers. The available tools are  and , along with their parameter descriptions. The  array provides the specific tool and arguments used to generate each answer.

§.§ Human Evaluation of Dataset Quality

To ensure that the three-stage verification process employed by APIGen produces a high-quality dataset, we conduct a human evaluation on a sample of the generated data. We engage three human evaluators to manually inspect a total of 600 samples from our released dataset. The evaluators assess the quality of each sample based on factors such as the accuracy of parameter values and the appropriateness of the number of API calls.

The results of the human evaluation reveal that only 28 out of the 600 inspected samples have minor issues, such as inaccurate parameter values or more API calls than expected. This means that the majority of the data, approximately 95.3%, are of very high quality. The high quality of the dataset can be attributed to the format and execution checkers implemented in the APIGen pipeline.

The format checker ensures that the generated data adheres to the specified JSON format and contains all the necessary fields. This step helps to filter out poorly formatted or incomplete data points. The execution checker, on the other hand, executes the generated function calls against the appropriate backend and verifies their successful execution. By providing real execution results, the execution checker plays a crucial role in filtering out cases that might be difficult to identify by an LLM-based semantic checker alone.

The combination of these two checkers, along with the final semantic checker, creates a robust verification process that effectively filters out low-quality data points. The human evaluation results confirm the effectiveness of this approach, demonstrating that APIGen is capable of generating high-quality datasets for training function-calling agents.

§ DATASET GENERATION AND EXPERIMENT DETAILS

§.§ Generator LLM Prompt

[title=Example Prompt for the Generator to Generate Parallel Function-Calling Data]
You are a data labeler. The responsibility for you is to generate a set of diverse queries and corresponding answers for the given functions in JSON format.

Construct queries and answers that exemplifies how to use these functions in a practical scenario. Include in each query specific, plausible values for each parameter. For instance, if the function requires a date, use a typical and reasonable date.

Ensure the query:
- Is clear and concise
- Contain multiple parallel queries in natural language for the given functions, they could use either the same function with different arguments or different functions
- Demonstrates typical use cases
- Includes all necessary parameters in a meaningful way. For numerical parameters, it could be either numerals or words
- Across a variety level of difficulties, ranging from beginner and advanced use cases
- The corresponding result's parameter types and ranges match with the functions descriptions.

Ensure the answer:
- Is a list of function calls in JSON format.
- The length of the answer list should be equal to the number of requests in the query
- Can solve all the requests in the query effectively

Here are examples of queries and corresponding answers for similar functions:

Note that the query could be interpreted as a combination of several independent requests.

Based on these examples and the above instructions, generate number diverse query and answer pairs for the functions `func_name`.
The detailed functions description is as follows:

Now please generate number diverse query and answer pairs following the above format.

The template provided outlines the prompt for an LLM to generate datasets as data labelers, emphasizing the diversity of query types and complexity to ensure thorough coverage of potential real-world applications. It specifies the importance of generating clear, concise queries and precisely formatted JSON responses. Sampled data, used to populate the  field, and API information, filling the  and  fields, enable a structured approach to dataset generation.
The  specifies the enforced JSON output format, as shown below.

[title=Example Format Instruction to Generate Parallel Function-Calling Data]
The output MUST strictly adhere to the following JSON format, and NO other text MUST be included:

"query": "The generated query.",
"answers": [

"name": "api_name",
"arg_name": "value",
... (more arguments as required)

... (more API calls as required)

The enforced JSON output format facilitates efficient data extraction and cost-effective generation. By requesting multiple query-answer pairs in a single inference with the  field—referred to here as a "batching" technique—token usage and costs are significantly reduced.

§.§ Semantic Checker LLM Prompt

We prompted another LLM as the semantic checker to evaluate whether the execution results and the tool calls align with the user query. We could use multiple LLMs with different prompts as checkers here to increase the credibility of this verification stage. We provide one example prompt as follows.

[title=Example Prompt for the Semantic Checker to Verify the Data]
As a data quality evaluator, you must assess the alignment between a user query, corresponding function calls, and their execution results.
These function calls and results are generated by other models, and your task is to ensure these results accurately reflect the user's intentions.

Do not pass if:

1. The function call does not align with the query's objective, or the input arguments appear incorrect.
2. The function call and arguments are not properly chosen from the available functions.
3. The number of function calls does not correspond to the user's intentions.
4. The execution results are irrelevant and do not match the function's purpose.
5. The execution results contain errors or reflect that the function calls were not executed successfully.

Given Information:
- All Available Functions:

- User Query: query

- Generated Function Calls: func_call

- Execution Results: execution_result

Note: The query may have multiple intentions. Functions may be placeholders, and execution results may be truncated due to length, which is acceptable and should not cause a failure.
The main decision factor is wheather the function calls accurately reflect the query's intentions and the function descriptions.

Provide your reasoning in the thought section and decide if the data passes (answer yes or no).
If not passing, concisely explain your reasons in the thought section; otherwise, leave this section blank.

Your response MUST strictly adhere to the following JSON format, and NO other text MUST be included.

"thought": "Concisely describe your reasoning here",
"pass": "yes" or "no"

Here, the  field is the same as the generator, while the  and  are the key fields to determine whether the generated data successfully address the 's intention.
We also enforce the model to output a JSON-formatted string, and then extract whether we should give a pass to this data point.

§.§ Model Training

We train two function-calling models of different sizes, xLAM-1B (FC) and xLAM-7B (FC), using the dataset generated by APIGen. The training pipeline mainly follows the AgentOhana paper [23]. We use 8 NVIDIA A100 40GB GPUs for training both models.

Since the Berkeley Function-Calling Benchmark [9] contains a relevance detection category, which evaluates a model's ability to distinguish non-relevant queries and tools, we extend APIGen to generate relevance detection data points from the generated datasets. These data points cover two types of scenarios:

* The provided tools cannot solve the query (e.g., query: "I want to know the weather in Palo Alto on Dec 25, 2023," provided tool: ).

* The provided tools are missing key arguments to solve the query (e.g., query: "I want to know the weather in Palo Alto on Dec 25, 2023," provided tool: ).

In both cases, the correct output is an empty tool call or a concise explanation indicating that the model should refuse to answer due to insufficient or irrelevant information.

We create 8,000 such data points from the collected dataset by 1) randomly discarding some tools that will be called in the answer or 2) randomly dropping some required parameters that were used in the generated tool calls. Then we relabel the answer to be an empty tool call or with a concise explanation.
By incorporating relevance detection data points into our training datasets, we can enhance the model's performance in determining when the provided tools are not suitable for addressing a given query. This enables the training of agents that can effectively assess the relevance of the available tools and respond appropriately, either by utilizing the relevant tools or by refraining from answering when the necessary information is lacking.

When training the model, we fill in the sampled query and available tools to the training prompt template, and then ask the model to predict the corresponding tool calls in specified JSON format. The training prompt template is as follows:
[title=Model Training Prompt]
[BEGIN OF TASK INSTRUCTION]
You are an expert in composing functions. You are given a question and a set of possible functions.
Based on the question, you will need to make one or more function/tool calls to achieve the purpose.
If none of the function can be used, point it out and refuse to answer.
If the given question lacks the parameters required by the function, also point it out.
[END OF TASK INSTRUCTION]

[BEGIN OF AVAILABLE TOOLS]
[END OF AVAILABLE TOOLS]

[BEGIN OF FORMAT INSTRUCTION]
The output MUST strictly adhere to the following JSON format, and NO other text MUST be included.
The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make tool_calls an empty list '[]'

"tool_calls": [
"name": "func_name1", "arguments": "argument1": "value1", "argument2": "value2",
... (more tool calls as required)

[END OF FORMAT INSTRUCTION]

[BEGIN OF QUERY]
User Query: query
[END OF QUERY]

The training hyperparameters for our models include a learning rate of $5 \times 10^{-6}$, four epochs, and use of the AdamW optimizer. Other settings include a cutoff length of 2048, a per-device batch size of six, two gradient accumulation steps, a cosine learning rate scheduler with 50 warmup steps, and the bfloat16 (BF16) data type.


# p for Power: Need for Agency before Political Participation

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

# A for Agency: Building Towards _p_ olitical Participation

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

# A Before P: Agency Before Participation

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

# P for Power: Building Agency towards _p_ olitical Participation

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

# p for _p_ olitical: Building Agency For and Beyond Participation

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

# p for _p_ olitical: Participation Without Agency Is Not Enough

Aakash Gautam  Virginia Tech, USA<EMAIL_ADDRESS>and  Deborah Tatar  Virginia
Tech, USA<EMAIL_ADDRESS>
(2020)

###### Abstract.

Participatory Design’s vision of democratic participation assumes
participants’ feelings of agency in envisioning a collective future. But this
assumption may be leaky when dealing with vulnerable populations. We reflect
on the results of a series of activities aimed at supporting agentic-future-
envisionment with a group of sex-trafficking survivors in Nepal. We observed a
growing sense among the survivors that they could play a role in bringing
about change in their families. They also became aware of how they could
interact with available institutional resources. Reflecting on the
observations, we argue that building participant agency on the small and
personal interactions is necessary before demanding larger Political
participation. In particular, a value of PD, especially for vulnerable
populations, can lie in the process itself if it helps participants position
themselves as actors in the larger world.

sensitive setting, vulnerable population, empowerment, agency

††journalyear: 2020††copyright: acmlicensed††conference: Proceedings of the
16th Participatory Design Conference 2020- Participation(s) Otherwise - Vol.
2; June 15–20, 2020; Manizales, Colombia††booktitle: Proceedings of the 16th
Participatory Design Conference 2020- Participation(s) Otherwise - Vol. 2 (PDC
’20: Vol. 2), June 15–20, 2020, Manizales, Colombia††price: 15.00††doi:
10.1145/3384772.3385142††isbn: 978-1-4503-7606-8/20/06††ccs: Human-centered
computing Field studies††ccs: Human-centered computing PD

##  1\. Introduction

Over the past three years, we have been working with an anti-trafficking non-
governmental organization (NGO) in Nepal and the sex-trafficking survivors
supported by the NGO. We call this NGO “Survivor Organization” (SO) as it was
established by a group of sex-trafficking survivors and many staff members at
different levels of the organization are trafficking survivors. A part of our
work involves exploring socio-technical interventions to support the survivors
achieve what SO calls “dignified reintegration”. Survivors face a myriad of
challenges in reintegration including social stigma held against trafficked
persons in Nepali society, hurdles in bureaucratic processes such as in
obtaining a citizenship certificate, and lack of employment opportunities
(Simkhada, 2008; Poudel, 2009; Joshi, 2001; Sharma, 2015; Crawford and
Kaufman, 2008; Richardson et al., 2009).

While over the past three years we have built a relationship with SO, the
space is contested and requires careful goal-balancing. The first author is a
native Nepali familiar with the cultural norms but as a privileged male
researcher from the West, he is an outsider and is seen as an outsider. Our
knowledge of the complexities of the setting is limited, the goals set by SO
for the survivors do not necessarily align with the survivors’ visions, and
the survivors’ dependency on SO for support results in them having limited
power to negotiate (Gautam et al., 2018). Part of our work has been in seeking
ways to support the survivor agency with the hope that they feel emboldened to
face challenges arising in their reintegration journey.

The work reported here began with an exercise to hear how the survivors —
henceforth called _sister-survivors_ to better reflect their nomenclature —
envisioned their future with respect to themselves, their family, and others
in society. While we heard varied expressions regarding their future, they all
envisioned a better society, one that did not hate trafficking survivors and
other marginalized groups. However, they did not see themselves playing a role
in bringing about the envisioned changes. We then conducted a second session
around child marriage and human trafficking, the two problems that the sister-
survivors said were common in their villages. We discussed factors that cause
the problem, listed actors involved in the issue, and identified ways in which
they could interact and act along with those actors to mitigate the problem.

We reflect on the discussions that ensued during the sessions. The sister-
survivors othered the larger institutions and broader society, and saw
themselves being distant from the processes involved in bringing change. Upon
narrowing the lens to look at the societal problems close to their home, they
could imagine themselves interacting with known actors to play an active role
in bringing about change. Further, imagining their role in attending to
personal, day-to-day interactions, that is, small “p” _p_ olitical engagement,
helped in forming more concrete visions for engagement with larger elements of
Nepali society, that is, large “P” Political engagement. We argue that a value
of participatory design (PD) can be in enabling the participants to realize
their agency in day-to-day interactions. This, in turn, builds towards
interactions with the larger world. We add to Beck’s argument for political
movement in PD (Beck, 2002) with a call to focus on the personal _p_ olitics
rather than institutional Politics, especially when working with vulnerable
groups.

##  2\. Background and Literature Review

Traditionally, PD has focused on future possibilities and alternatives (e.g.
(Björgvinsson et al., 2010; Kensing and Blomberg, 1998; Kyng, 2015)). It grew
during an era where the concern was on the impact of technology in the
workplace and the recognition for workers (and unions) to regulate new
technology (Bødker and Kyng, 2018). The discourse centered around democratic
control and Politics, with a focus on the workers’ influence on technology and
its adaptation, expansion of choices through alternatives, and engagement with
the wider network such as worker unions to strengthen democratic ideals (e.g.
(Nygaard and Terje Bergo, 1975; Bødker et al., 1987; Kyng and Mathiassen,
1979)). The assumption was that the participants had agency to act in
relationship with one another and the larger structures such as trade unions
to promote their interests in both the design of technology and the broader
structures beyond the project.

Modern PD adapted to technology being commonplace (Brereton and Buur, 2008;
Bødker and Kyng, 2018). With it, the focus moved from the Politics to the
project-level relationship between designers and stakeholders, and the various
tools, techniques, and facilitation involved in realizing it (Bødker and Kyng,
2018; Kyng, 2010; Light and Akama, 2012). But the assumption of participants’
agency such as in promoting self-interests through the use of those
participatory tools remained (e.g. (Halskov and Dalsgård, 2006; Beck et al.,
2008; Bratteteig and Wagner, 2012; Brandt, 2006)). For example, Beck (Beck,
2002), in arguing for a move to a Political PD, calls for people to become
“stroppy” users — users who demand change in the larger system by
problematizing their day-to-day interactions. However, such problematizing
requires agency; not all participants may feel that they have such agency.

We see here that the sister-survivors feel that they may not have agency to
act in relationship to larger institutions or even their own family. This
problematizes PD’s assumption of participants having agency to envision future
possibilities and alternatives, and raises concern on the effectiveness of
participation alone in realizing democratic ideals both within the project and
in the broader structure. Following Beck’s call for PD to “encompass work
motivated in political conscience” (Beck, 2002), we present a possible step
towards building agency by encouraging participants to attend to the power
relations and interactions in day-to-day life i.e. _p_ olitics.

##  3\. Methodology

The work reported here builds on an earlier work where we introduced a web
application contextualized around crafting to a group of sister-survivors
(Gautam et al., 2020). SO trained all the sister-survivors to create local
handicrafts. The web application was presented as a way of widening future
possibilities by selling the handicrafts online.

During the study, the sister-survivors mentioned that they had limited control
in their interactions within and outside of SO. We developed a document to
support them to envision their future with respect to various aspects of their
lives, and with it, chart possible pathways to gain some control. The first
workshop was conducted with a group of nine sister-survivors in January 2019.
We iterated on the document and the activity based on our reflection from the
workshop. In this paper, we present findings from the second iteration which
we conducted in August 2019.

###  3.1. The Sister-Survivors

Ten sister-survivors who were living at SO’s shelter home participated in the
study. They were between the ages of 11 and 20. All of them were enrolled in
“morning schools”; they had classes from 6:30 am to 9:30 am. Three of them had
just started going to school. Three others were in the seventh grade, the
highest among them, but the school moved students up a grade every six months
until the sixth grade so this was their fourth year of formal schooling.

The sister-survivors are vulnerable in a number of ways: they are young, have
experienced traumatic ordeals, many have been shunned by their families, and
all of them are dependent on SO for support. The opportunities made available
to them are influenced by the resources available to SO. For example, the
sister-survivors started going to school because another NGO sponsored the
entire cost for SO to send them to school.

###  3.2. Future Envisioning Activity

We began the activity by eliciting values around six aspects of the
participants’ lives: “me”, “my family”, “my society”, “my crafts and skills”,
“my source of income”, and “me and my technology”. We discussed their future
vision in relation to those aspects at three different stages of their life:
(1) when they are about to leave the shelter home, (2) when they feel they are
successful, and (3) when they become old. In an earlier iteration, the three
stages were 1 year, 3 years, and 5 years. However, those numbers held little
meaning to the sister-survivors since they were of different ages, had been at
SO for different duration, and had different outlook for reintegration.

The document (see Figure 1) created to facilitate the activity was designed
considering that many survivors are illiterate (Thapa Kshetri et al., 2018) .
We had seen SO’s walls covered with posters created by survivors using
newspaper cutouts, drawings, and texts. The document was created such that the
sister-survivors could express themselves similarly or simply by speaking
about those aspects of their lives.

Figure 1. Left: S3’s vision of her society and her skills when she feels she
is successful. Right: Collective response listing the causes, actors, and
possible ways for them to be involved against human trafficking.

On the first day, the sister-survivors shared and discussed their views about
the future at the time when they are about to leave the shelter home. We
followed up the next day with a discussion on two societal problems that they
had mentioned on the first day: child marriage and human trafficking. During
this activity, the discussion centered around (1) different factors that lead
to the problem, (2) actors who perpetuate and/or can help in mitigating the
problem, and (3) ways in which they could act together with the actors to
mitigate the problem. Eight of the ten sister-survivors participated in this
activity; two of them joined us only at the end. On the third day, we
discussed their vision of the future at the two other stages of their lives.
Each of the three sessions was around two hours long.

###  3.3. Data Collection and Analysis

At the beginning of each session, we asked for verbal consent from the sister-
survivors to record audio and started recording only after obtaining consent
from all. The first author translated and transcribed the recordings. The
first round of coding involved closely following the text and summarizing the
interactions. Following that, the first author merged frequent occurrences
into lower-level codes (e.g. “Seeing self in raising awareness”). Following
Saldaña, these codes were discussed with the second author, and in multiple
rounds, the codes were merged into higher-level codes (Saldaña, 2015). While
the data cover multiple aspects of the sister-survivors’ lives, here we focus
on their expressions with regards to themselves, their families, and members
and institutions in their society.

##  4\. Findings

We observed elements of the sister-survivors’ initial, limited agency with
respect to their family and institutions. As the sessions progressed, they saw
possibilities of interacting with other actors and playing an active role in
bringing about the changes they imagined.

###  4.1. Limited Agency When Envisioning Future

####  4.1.1. Lack of Agency Within Family

Most of the sister-survivors felt that they were placed at the fringe of their
family’s decision-making process. They discussed the pressure imposed upon
them by family members trying to model others in the community. For example,
S10 mentioned, “They [family] will say things like ‘so and so’s son and
daughter are now married, you too should marry’, [that] creates stress.”
Similarly, the influence of societal norm of discriminating against daughters
came up frequently as heard in S8’s statement, “they [family] don’t educate
daughters. They educate only sons.” To this, S9 added that daughters are seen
by families as “types that leave”, suggesting their limited influence in their
family’s decisions. Similarly, S6, recounting observations from her village,
noted the significance of family pressure on young girls, “Everyone in our
village runs away and does it [marries]” and added that they elope at a young
age “because the parents try to force them. So, they choose their own and run
away instead.”

The sister-survivors expressed further marginalization in decision-making
resulting from the pressure imposed upon their family by members of the
society. S8, for example, highlighted the way others pressurize their family,
“They say, ‘Your daughter has grown up, now get her married’ and that puts
pressure [on the family]” which led S7 to comment, “More than the parents, it
is those outsiders that put pressure.” Similarly, when discussing factors that
cause human trafficking, S5 mentioned that a major factor of trafficking is
the societal pressure on families to send daughters to work. These intricate
relationships between the influence members of society have over families, and
the power family members hold over a girl child’s life repeatedly came up
leading S5 to comment, “Everything seems to lead to that same thing: family
and neighbors.”

####  4.1.2. Distant View of Institutions

The sister-survivors expressed views of being distant from existing
institutions. They mentioned that the police could play a role in tackling
both child marriage and human trafficking. Some, like S1, thought that members
of the society and the police could get together to find a resolution to
social problems whereas S6 and S9 saw the police’s role limited to raise
awareness in villages. But none expressed views in which they or others in the
society could leverage the police for help.

Similarly, the sister-survivors wanted the government to help but felt distant
from the government and did not see ways to engage with the government or to
seek support from it. S8 mentioned that child marriage is prevalent despite
being illegal because “the government overlooks these issues in most places.”
S10 too felt that the “government does not pay attention” to mitigate child
marriage. Similarly, S2 felt that the government did not take enough
responsibility in mitigating human trafficking and expressed, “The government
should raise public awareness through programs like street plays.” This is
particularly noteworthy because S2 had conducted street plays for SO and yet
did not see herself playing a role in working with institutions to raise
awareness.

####  4.1.3. Limited Possibilities for Action

All of them wished for a better society by the time they leave SO and were
concerned about being accepted as heard in S8’s expression, “I wish that when
I join society, it doesn’t look down upon me. May my society not hate me.” S4
expressed her wishes more positively, “I want my society to look at me with
kindness and in a good light. May it love me and may it do good to other
survivors like me.” None except S9 mentioned a role they could play in helping
move the society towards their vision. S9 wrote, “I want to be able to say
that my society is very nice. I wish nothing bad goes on in my society” and
added, “I also want to help in making my society better”, but when asked, she
did not know what she could do to make her society better.

The lack of agency in influencing members of the society could be heard
throughout the earlier discussions. For example, S10 mentioned the
helplessness survivors may feel when they encounter hatred from others in
society, “It is hard to mix with society. One may say something today, someone
else may say something later on, and someone else may say something, and that
will irritate them [survivors] and they can’t stay there anymore. They may not
have any option left.” When we asked about possible actions to reduce hate and
discrimination, S10 suggested “raising awareness such as through street
plays”, an approach that modeled SO’s operation but did not see herself doing
it. Like S10, others too expressed views where they did not see themselves
being involved in taking action.

###  4.2. Envisioning _p_ olitical Action

As the sessions progressed, the sister-survivors saw possibilities to make
their families stronger, leverage institutional resources, and also act on
their own. They discussed ways in which they could interact with known actors
such as their family members and neighbors, suggesting an increasing move
towards _p_ olitical action.

####  4.2.1. Making Family Stronger

All the sister-survivors saw a need for families to be strong as heard in
S10’s expression, “Our own family has to be strong, that’s the main thing.” To
this, S8 added, “First, mom and dad have to live harmoniously together. That’s
needed. Then others can’t look down upon us. All [family members] have to love
each other. That probably will help.”

They discussed ways in which they could help strengthen their families. Some
mentioned the need for themselves to have confidence and strength before
supporting their family as heard in S7’s plan of action, “First, we have to
have self-confidence and be strong. That’s needed.” S2 mentioned the need to
become a model example and raise awareness and, like S8, also suggested a need
for family cooperation, “First of all, we have to be good. Also, dad and mom
should listen to one another. And we should raise a bit of awareness among
family members and outside [the family].”

Like S2, others too saw themselves interacting with family and community
members to raise awareness. They had earlier mentioned that people were
unaware of factors that lead to societal problems. To address this, S8 wanted
to raise public awareness by “telling them [family and neighbors] that they
shouldn’t do so [child marriage].” Such assertive action plans were
increasingly formulated as the sister-survivors began charting out various
factors and actors involved in the societal problems, and ways in which they
would interact with those actors in their day-to-day lives.

####  4.2.2. Interacting with Existing Institutions

The sister-survivors acknowledged that they alone may not be able to raise
awareness and bring about change. As the second session progressed, they
expressed the need to engage with others and leverage external resources to
tackle societal problems. Particularly, they expressed a desire to leverage
existing institutional power held by different local actors as heard in S10’s
plan to “bring in the police or NGOs or other people like teachers, who can
help, people who can advise families” while discussing ways to mitigate child
marriage.

Once the sister-survivors identified the police as potential actors who could
help, they discussed plans to engage with the police. Acknowledging the power
of the police to deter community members from perpetuating societal problems,
they saw possibilities of getting the police to “advise and warn families and
neighbors” (S6). Further, they saw possibilities of engaging the police to
arrest people, including their own family members, if the people did not heed
to advice and warning. The sister-survivors also saw the potential of the
police as a resource for support. It could be heard in S8’s fallback plan if
her family did not support against trafficking, “… the family should support
[against trafficker] as needed. If the family does not support, then [I will]
go to the police.”

However, the sister-survivors were also concerned about the excessive use of
police force. S10, for example, mentioned, “It’s not possible to arrest
everyone.” This led to a discussion on engaging with other institutions,
particularly NGOs and the government. S10 had earlier expressed that child
marriage is prevalent because “ the government does not pay attention”. Later,
she saw the possibility to engage and draw the government’s attention, “if we
raise voice in unity, the government may pay attention.” She later added,
“there is nothing we can’t do, we can do it but the government has to help a
little bit”, suggesting an interdependence of individual and institutions.

####  4.2.3. Identifying Possible Actions

We could hear more assertive statements as the sessions progressed. S10 noted
that while staff members discussed issues related to human trafficking with
the sister-survivors, those issues were “never talked about in this detail”,
suggesting a limited understanding of what they could do to mitigate the
problem. At the end of the second session, S10 added that she now knew what
she could do, “I have learned a bit about what needs to be done like if I go
there [home] and see that child marriage is happening, I feel like I can
probably do something. I feel I can at least counsel and advice.”

Raising awareness in villages and raising their voice to make the government
pay attention to overlooked problems in their society were the two major
action plans discussed during the sessions. All the sister-survivors wanted to
create videos and share them online and through television to raise awareness,
particularly against human trafficking. Even though some of them feared being
identified as a trafficked person once they showed their face, all of them
expressed willingness to show their faces if needed to convey the message and
raise awareness.

##  5\. Concluding Discussion

In the sister-survivors’ accounts of the future, they wanted a supportive,
happy family, and an inclusive and caring society but did not see themselves
involved in realizing these visions. In fact, their expressions highlighted a
lack of agency in their lives, in their family’s decision-making process, and
in leveraging institutional resources. In the second session, we narrowed the
institutional Politics to _p_ olitics, focusing on problems they had seen in
their villages. Doing this allowed them to discuss how to change their
families and other members of society and, importantly, to imagine themselves
playing an active role in bringing about the changes they imagined. We
eventually saw enough agency for them to imagine engaging actively with the
broader Nepali society through, for example, creating videos to raise
awareness against human trafficking.

We believe that two inter-dependent aspects of the activity were useful.
First, focusing on personal issues that were prevalent close to their home
helped them identify factors and the actors involved in those issues. Second,
focusing on their relationship with particular known actors — rather than
abstract institutions — helped illuminate possibilities for interaction and
invited plans for action. These approaches broke through the “othering” that
we saw when the sister-survivors tried to think about more abstract
institutions. While Politics was perceived to be beyond reach, _p_ olitics
embodied in personal responses through known relationships was not.

Imagining engaging with others is not the same as actual engagement. Moreover,
the known – and many unknown – social pressures that the sister-survivors face
in society during their reintegration may restrict their actual engagement
with other actors. In this sense, the work we report here is limited. A more
prolonged and deeper engagement is needed. But envisionment may have value of
its own in their lives and the possibility of partaking in _p_ olitics can be
of value in preparing them to engage with larger elements of Nepali society,
that is, in large “P” Political engagement.

Envisioning future possibilities and alternatives, putting forth self (or
group) interest, and negotiating positions and control are fundamental tenets
of PD. In fact, Bødker and Kyng contend that PD – the PD that matters – needs
to move from a focus on co-design sessions towards engagement beyond the
project to deal with Political issues around democratic control (Bødker and
Kyng, 2018). They posit that “democratic control is one of the most
challenging” tasks for PD. They are concerned that modern PD lacks Political
outcomes (Bødker and Kyng, 2018). We agree that the outcome of PD has to be
Political for PD to support the realization of democratic ideals. However, we
have to acknowledge that such political stances require participants’ agency
to engage in the ensuing interactions and actions. In some cases, such as in
ours, participants may not feel that they have and indeed may not have
sufficient agency. Thus, PD must involve enhancing the agency of those who may
be on the margins. More broadly, “What are the values that matter?” and “Who
gets to establish those values?” are questions that are inherent in PD and
should be examined through negotiations between designers, participants, _and_
the broader community. For PD to matter in issues beyond the sessions, we have
to begin by being aware of participants’ agency or the lack of it, support in
building it if needed, helping them to understand more about power and agency,
and emphasize _p_ olitical interaction before striving for democratic control.

###### Acknowledgements.

We wish to thank the sister-survivors and SO staff members for their time and
support.

## References

  * (1)
  * Beck et al. (2008) Elke Beck, Marianna Obrist, Regina Bernhaupt, and Manfred Tscheligi. 2008.  Instant card technique: how and why to apply in user-centered design. In _Proceedings of the Tenth Anniversary Conference on Participatory Design 2008_. Indiana University, 162–165.
  * Beck (2002) Eevi E Beck. 2002\.  P for political: Participation is not enough.  _Scandinavian Journal of Information Systems_ 14, 1 (2002), 1\.
  * Björgvinsson et al. (2010) Erling Björgvinsson, Pelle Ehn, and Per-Anders Hillgren. 2010\.  Participatory design and democratizing innovation. In _Proceedings of the 11th Biennial participatory design conference_. ACM, 41–50.
  * Bødker and Kyng (2018) Susanne Bødker and Morten Kyng. 2018.  Participatory design that matters—Facing the big issues.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 25, 1 (2018), 4.
  * Bødker et al. (1987) Susanne Bødker, Morten Kyng, Pelle Ehn, John Kammersgaard, and Yngve Sundblad. 1987.  A utopian experience: On design of powerful computer-based tools for skilled graphic workers. In _Computers and democracy_. Gower Publishing, 251–278.
  * Brandt (2006) Eva Brandt. 2006\.  Designing exploratory design games: a framework for participation in participatory design?. In _Proceedings of the ninth conference on Participatory design: Expanding boundaries in design-Volume 1_. ACM, 57–66.
  * Bratteteig and Wagner (2012) Tone Bratteteig and Ina Wagner. 2012.  Spaces for participatory creativity.  _CoDesign_ 8, 2-3 (2012), 105–126.
  * Brereton and Buur (2008) Margot Brereton and Jacob Buur. 2008.  New challenges for design participation in the era of ubiquitous computing.  _Co-Design_ 4, 2 (2008), 101–113.
  * Crawford and Kaufman (2008) Mary Crawford and Michelle R Kaufman. 2008.  Sex trafficking in Nepal: Survivor characteristics and long-term outcomes.  _Violence against women_ 14, 8 (2008), 905–916.
  * Gautam et al. (2018) Aakash Gautam, Chandani Shrestha, Andrew Kulak, Steve Harrison, and Deborah Tatar. 2018.  Participatory Tensions in Working with a Vulnerable Population. In _Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial - Volume 2_ _(PDC ’18)_. ACM, New York, NY, USA.
  * Gautam et al. (2020) Aakash Gautam, Deborah Tatar, and Steve Harrison. 2020\.  Crafting, Communality, and Computing: Building on Existing Strengths To Support a Vulnerable Population. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Halskov and Dalsgård (2006) Kim Halskov and Peter Dalsgård. 2006.  Inspiration card workshops. In _Proceedings of the 6th conference on Designing Interactive systems_. ACM, 2–11.
  * Joshi (2001) Sushma Joshi. 2001\.  ‘Cheli-Beti’: Discourses of trafficking and constructions of gender, citizenship and nation in modern Nepal.  _South Asia: Journal of South Asian Studies_ 24, s1 (2001), 157–175.
  * Kensing and Blomberg (1998) Finn Kensing and Jeanette Blomberg. 1998.  Participatory design: Issues and concerns.  _Computer Supported Cooperative Work (CSCW)_ 7, 3-4 (1998), 167–185.
  * Kyng (2010) Morten Kyng. 2010\.  Bridging the Gap Between Politics and Techniques: On the next practices of participatory design.  _Scandinavian J. Inf. Systems_ 22, 1 (2010), 5\.
  * Kyng (2015) Morten Kyng. 2015\.  On creating and sustaining alternatives: the case of Danish telehealth. In _Proceedings of The Fifth Decennial Aarhus Conference on Critical Alternatives_. Aarhus University Press, 5–16.
  * Kyng and Mathiassen (1979) Morten Kyng and Lars Mathiassen. 1979.  _Systems development and trade union activities_.  Computer Science Department, Aarhus University.
  * Light and Akama (2012) Ann Light and Yoko Akama. 2012.  The human touch: participatory practice and the role of facilitation in designing with communities. In _Proceedings of the 12th Participatory Design Conference: Research Papers-Volume 1_. 61–70.
  * Nygaard and Terje Bergo (1975) Kristen Nygaard and Olav Terje Bergo. 1975.  The trade unions-New users of research.  _Personnel review_ 4, 2 (1975), 5–10.
  * Poudel (2009) Meena Poudel. 2009\.  _Dealing with hidden issues: Social rejection experienced by trafficked women in Nepal_.  Ph.D. Dissertation. Newcastle University.
  * Richardson et al. (2009) Diane Richardson, Meena Poudel, and Nina Laurie. 2009\.  Sexual trafficking in Nepal: constructing citizenship and livelihoods.  _Gender, Place & Culture_ 16, 3 (2009), 259–278.
  * Saldaña (2015) Johnny Saldaña. 2015\.  _The coding manual for qualitative researchers_.  Sage.
  * Sharma (2015) Jyoti Sharma. 2015\.  _Sex trafficking in Nepal: women’s experiences of reintegration_.  Ph.D. Dissertation. The University of Queensland.
  * Simkhada (2008) Padam Simkhada. 2008\.  Life histories and survival strategies amongst sexually trafficked girls in Nepal.  _Children & society_ 22, 3 (2008), 235–248.
  * Thapa Kshetri et al. (2018) Kamal Thapa Kshetri, Yesoda Banjade, and Govind Subedi. 2018\.  _Trafficking in Persons National Report_.  Technical Report. National Human Rights Commission Nepal.  http://www.nhrcnepal.org/nhrc_new/doc/newsletter/NHRC_National_Report_TIP_in_Nepal_September_2018.pdf