#  BotSIM: An End-to-End Bot Simulation Toolkit
for Commercial Task-Oriented Dialog Systems

Guangsen Wang Shafiq Joty Junnan Li Steven C.H. Hoi
Salesforce Research
{guangsen.wang, sjoty, junnan.li<EMAIL_ADDRESS>
###### Abstract

We introduce BotSIM, a modular, open-source Bot SIMulation environment with
dialog generation, user simulation and conversation analytics capabilities.
BotSIM aims to serve as a one-stop solution for large-scale data-efficient
end-to-end evaluation, diagnosis and remediation of commercial task-oriented
dialog (TOD) systems to significantly accelerate commercial bot development
and evaluation, reduce cost and time-to-market. BotSIM adopts a layered design
comprising the infrastructure layer, the adaptor layer and the application
layer. The infrastructure layer hosts key models and components to support
BotSIM’s major functionalities via a streamlined “generation-simulation-
remediation” pipeline. The adaptor layer is used to extend BotSIM to
accommodate new bot platforms. The application layer provides a suite of
command line tools and a Web App to significantly lower the entry barrier for
BotSIM users such as bot admins or practitioners. In this report, we focus on
the technical designs of various system components. A detailed case study
using Einstein BotBuilder is also presented to show how to apply BotSIM
pipeline for bot evaluation and remediation. The detailed system descriptions
can be found in our system demo paper (Wang et al., 2022). The toolkit is
available at: _https://github.com/salesforce/BotSIM_.

##  1 Introduction

Figure 1: BotSIM System Design.

The typical dialog system development cycle consists of dialog design, pre-
deployment testing, deployment, performance monitoring, model improvement and
iteration. As in any production software system, effective and comprehensive
testing at all stages is of paramount importance. Unfortunately, _evaluating
and troubleshooting_ production TOD systems is still a largely manual process
requiring large amount of real human conversations with the bots. This process
is time-consuming, expensive, and inevitably fails to capture the breadth of
language variation present in the real world (Tan et al., 2021). The time- and
labor-intensive nature of such an approach is further exacerbated when the
developer significantly changes the dialog flows, since new sets of test
dialogs will need to be collected (Benvie et al., 2020). Performing
comprehensive end-to-end evaluation of both NLU and dialog-level performance
(_e.g._ , task success rate) is highly challenging due to the need for further
annotated efforts. Finally, there is a lack of analytical tools for
interpreting test results and troubleshooting underlying bot issues. To
address these limitations, we present _BotSIM_ , an open source Bot SIMulation
environment with dialog generation, user simulation (Schatzmann et al., 2007)
and conversation analytics capabilities. BotSIM aims to serve as a one-stop
solution for bot practitioners to perform large-scale data-efficient end-to-
end evaluation, diagnosis and remediation of commercial TOD systems. The
modular design also allows bot developers to extend the framework to more bot
platforms.

The toolkit design is depicted in Figure 1. It consists of three layers: the
infrastructure layer, the adaptor layer and the application layer. The
infrastructure layer supports the upper layers through the key modules (NLU
and NLG) and components (the generator, simulator and remediator). They are
the “engines” to power BotSIM’s “generation-simulation-remediation” pipeline
for end-to-end diagnosis, evaluation and remediation of commercial TOD bots.
More details regarding the functionalities of these modules and components are
given in (Wang et al., 2022). BotSIM currently supports two bot platforms,
including Salesforce Einstein BotBuilder and Google DialogFlow CX. The adaptor
layer is designed for bot developers to extend BotSIM to new bot platforms. In
addition to defining a set of unified parser and client interfaces, the
adaptor layer also provides some reference implementations for developers to
start their own implementations of the platform-dependent parsers and chat API
clients. The application layer offers a suite of command line tools and an
easy-to-use Web App. The Web app is designed for bot admins to directly apply
BotSIM to their bots for evaluation and remediation without diving into the
technical details of various system components. On the other hand, the command
line tools are provided to bot practitioners for a better understanding of the
toolkit to customize their own evaluation and remediation pipeline. They can
learn more about how different BotSIM components work in terms of the required
inputs, expected outputs and their functionalities. We have provide detailed
tutorials of how to use these tools in our code documentation.

##  2 Related Work

| Methods | Stages | Automation | Metrics
---|---|---|---|---
Regression | End-to-end | Pre-deployment | Monitoring |  | Test case
---
curation
| User
---
Simulation
NLU |  | Task
---
Completion
CX | ✓ |  |  | ✓ |  |  |  |
Watson | ✓ |  |  | ✓ |  |  | ✓ | ✓
Botium | ✓ |  |  |  |  |  | ✓ |
BotSIM | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

Table 1: Comparison of bot evaluation capabilities of the reviewed commercial
bot platforms

We present the related work of BotSIM from both toolkit design and application
perspectives.

###  2.1 Toolkit Design

To the best of our knowledge, there are no open source toolkits designed for
simulation, evaluation and remediation of commercial chat bots. Therefore, we
review some of the research toolkits/libraries designed for TOD systems and
are equipped with user simulators. They include ConvLab-2 (Zhu et al., 2020),
TC-Bot Li et al. (2016, 2017) and user-simulator Shi et al. (2019).

  * •

ConvLab-2 111https://github.com/thu-coai/ConvLab-2 is a research toolkit for
building TOD systems with state-of-the-art dialog models. It was used in DSTC9
Track 2 222https://convlab.github.io/index.html for the multi-domain TOD
challenge. In addition, it also can perform end-to-end evaluation, and
diagnose the weakness of the research systems. The essential differences to
BotSIM include 1) the toolkit is limited to research TOD systems to support
research models and datasets; 2) the evaluation is based on the annotated
dialogs.

  * •

TC-Bot 333https://github.com/MiuLab/TC-Bot is another research-oriented
framework for building end-to-end neural-based TOD systems.TC-Bot utilises a
dialog-act level agenda-based user simulator to interact with the neural-based
end-to-end dialog agent for policy training. Some of our simulator rules and
designs are inspired and adapted from a simpler version of the TC-Bot called
GO-Bot-DRL 444https://github.com/maxbren/GO-Bot-DRL.

  * •

The “user-simulator” 555https://github.com/wyshi/user-simulator provides
implementations and comparisons of different user simulators, such as the
agenda-based user simulator and neural-based user simulator based on Sequicity
(Lei et al., 2018). The work aims to provide an evaluation framework for user
simulator study in terms of their impacts on the trained dialog policies.

The focus of these research toolkits is to build an end-to-end (neural-based)
dialog agent prototype constrained to certain domains (_e.g._ , movie booking,
restaurant domain of the MultiWOZ (Budzianowski et al., 2018) dataset). On the
contrary, BotSIM is designed as a holistic bot simulation environment for
task-agnostic end-to-end evaluation, diagnosis and remediation of commercial
bots. The agenda-based user simulator is only a small part of the toolkit as
shown in Figure 1. It is designed for bot practitioners to significantly
accelerate commercial bot development and evaluation, reducing human efforts,
cost and time-to-market.

###  2.2 Commercial Bot Evaluation

The overall comparison of different platforms in terms of testing capabilities
is given in Table 1. The detailed system reviews can be found in the system
demo paper (Wang et al., 2022). Most current commercial platforms only focus
on the regression testing rather than NLU or task-completion metrics. While
regression testing is important to ensure correct and consistent system
behaviours, it is also vital to perform pre-deployment evaluation to avoid
poor user adoption and retention rate. Although some of the platforms offer
turn-level NLU performance metrics, they require human efforts in curating or
annotating a large number of test cases. In addition, the NLU metrics do not
directly translate to goal completion performance. On the other hand, BotSIM
can help circumvent these limitations via large scale automatic dialog
generation and simulation.

##  3 BotSIM Design

In this section, we present more details of the BotSIM design in Figure 1. The
key design principles of BotSIM include modularity, extensibility and
usability. These principles allow BotSIM to be adopted both by developers as a
framework and bot end-users as an easy-to-use application. To achieve these,
BotSIM adopts a layered design comprising the infrastructure layer, the
adaptor layer and the application layer.

###  3.1 Infrastructure Layer

As the name suggests, the infrastructure layer is designed to offer
fundamental model support for the framework. BotSIM’s “generation-simulation-
remediation” pipeline is powered by the models and components that reside in
this layer. The models include the natural language understanding (NLU),
natural language generation (NLG) models and the key components include the
generator, the simulator and the remediator.

botsim.models package hosts BotSIM’s NLU and NLG models. From a dialogue
system perspective, BotSIM can be viewed as a counterpart to a chatbot as
shown in Figure 2: it needs to “understand” chatbot messages (NLU) and
“respond” in natural languages (NLG) to carry on the conversation. Currently,
fuzzy matching-based NLU and template-based NLG models are provided for
efficiency reasons. More advanced NLU and NLG models can also be incorporated
by the developers by following the recipes given in the code  documentation.

botsim.modules consists of the three key components to power BotSIM’s
“generation-simulation-remediation” pipeline.

  * •

botsim.modules.generator provides two major functionalities: 1) the parser
parses the input bot designs in the form of either metadata (Einstein
BotBuilder) or API (DialogFlow CX) to infer the dialog-act maps (BotSIM’s
NLU); 2) the large pre-trained language model based paraphraser generates
paraphrases from the input intent utterances. These paraphrases are used as
intent queries in the simulation goals to probe bots’ intent models, which
allows BotSIM to perform large-scale data-efficient bot evaluation even before
bots are deployed.

  * •

botsim.modules.simulator implements the dialog-act level agenda-based user
simulation in abus. It also defines a simulation API client interface
simulation_client_base.

  * •

botsim.modules.remediator analyzes the simulated dialogs and produces the
performance metrics and conversational analytics to support the dashboard
visualisation. These metrics include both the end-to-end performance such as
the task completion rates and the NLU performance. It also offers a suite of
analytical tools to provide actionable insights to troubleshoot and improve
the current systems. Such tools include confusion matrix analysis and
visualisation (analytics, visualization_cm), dialog design graph exploration
(dialog_graphs).

###  3.2 Adaptor Layer

The adaptor layer is designed for bot developers to extend BotSIM to new bot
platforms. To cover new bot platforms, the following two most important
platform-specific modules of the layer must be implemented. We also provided
more details in the code documentation. To accommodate a new bot platform,
create a new package under botsim.platforms and implement the following
classes.

parser acts as an “adaptor” to unify bot definitions (_e.g._ conversation
flows, intents/tasks) from different platforms to a common representation of
dialog act maps. The dialog act maps are used as BotSIM NLU to map bot
messages to dialog acts. The rational is that most commercial TOD bots follow
a “rule-action-message” design scheme and there exist clear mappings from
system messages to rules/actions. For example, according to bot definitions,
“May I get your email?” (bot message) is used to “Collect” (action) the
“Email” (slot) with entity type “Email” from the user. One of the most
important parser function is to parse input bot definitions (_e.g._ ,
MetaData, API) to capture such mappings, infer the “request_Email” dialog act
and add the bot message to the mapping candidates of the dialog act. An
example of the dialog act map is given in Figure 3. The major parser functions
are listed in the code snippet below:

⬇

1class Parser:

2 """ Parser interface"""

3 def __init__(self):

4 # The aggregated dialog act maps to be used as

5 # BotSIM NLU

6 self.dialog_act_maps = {}

7 # mapping from dialogs to their entities

8 self.dialog_ontology = {}

9 # conversation graph data to support path explorer

10 self.conv_graph_visualisation_data = {}

11 def extract_local_dialog_act_map(self):

12 def conversation_graph_modelling(self, local_dialog_act_maps):

13 """Model local dialog act maps and their transitions as a graph.

14 :param local_dialog_act_maps: local dialog act maps (one per dialog)
obtained from the parser (extract_local_dialog_act_map)

15 :return:

16 dialog_act_maps: aggregated dialog act maps obtained via graph traversal.
These maps need to be reviewed/revised by users before used as BotSIM NLU

17 conv_graph_visualisation_data: conversation graph data to support dialog
path visualisation

18 """

19 def extract_ontology(self):

Note the implementations of the parsers are highly platform-dependent. They
require the developers to have access to bot platform, design and API
documents to understand how bots are designed and how user inputs are elicited
by the bots. There is also need to constantly revisit the implementations to
incorporate new features that might be missed in the current implementations.
We have provided our current implementations of BotBuilder and DialogFlow CX
parsers for references under botsim.platforms.botbuilder and
botsim.platforms.dialogflow_cx.

simulation_client is the other platform-dependent component for BotSIM to
exchange conversations with bots via API calls.

⬇

1from botsim.modules.simulator.user_simulator import UserSimulator

2class UserSimulatorClientInterface:

3 def __int__(self, config):

4 self.config = config

5 self._prepare_simulation()

6 def _prepare_simulation(self):

7 """prepare simulation according to the simulation configuration"""

8 def perform_batch_simulation(self, simulation_goals, dialog_name,
start_episode, config, batch_size=25):

9 """perform a batch of dialog user simulations using simulation_goals

10 :param simulation_goals: list of all simulation goals

11 :param dialog_name: name of the dialog under simulation

12 :param start_episode: starting goal index of simulation_goals

13 :param config: simulation settings including information such as API tokens

14 :param batch_size: number of simulation sessions per batch"""

15 user_simulator = UserSimulator(simulation_goals, config)

16 ...

Figure 2 depicts how a dialogue turn between the bot and BotSIM is conducted
via bot APIs. Using dialog act maps as NLU, a rule-based dialog state manager
(policy implemented in botsim.modules.simulator) takes in the bot dialog acts
and produces the corresponding user dialog acts. The user dialog acts are
converted to natural language responses by the template-based NLG. The natural
language responses are sent back via API. The conversation ends when the task
has been successfully finished or an error has been captured. Similar to the
parser, the implementation of the client is also highly platform-dependent.
Developers can refer to our implementations for BotBuilder and DialogFlow CX
when extending BotSIM to new bot platforms.

Figure 2: An exchange of dialog turns between bot and BotSIM during dialog
simulation

###  3.3 Application Layer

The application layer is designed to significantly flatten the learning curves
of BotSIM for both bot developers/practitioners and end-users.

Command Line Tools botsim.cli contains a set of command line tools for
practitioners to learn more about the major BotSIM components. The
“generation-simulation-evaluation” pipeline has been split into multiple
stages to expose the required inputs and expected outputs for each stage. They
serve as basic building blocks for bot practitioners to build their customized
pipelines or apply only certain tasks rather than the whole BotSIM pipeline.

⬇

1python botsim/cli/run_generator_parser.py \

2 \--platform $platform \

3 \--test_name $name

4python botsim/cli/run_generator_paraphraser.py \

5 \--platform $platform \

6 \--test_name $name

7python botsim/cli/run_generator_goal_generation.py \

8 \--platform $platform \

9 \--test_name $name

10python botsim/cli/run_simulator.py \

11 \--platform $platform \

12 \--test_name $name

13python botsim/cli/run_remediator.py \

14 \--platform $platform \

15 \--test_name $name

Streamlit Web App botsim.streamlit_app is a multi-page easy-to-use Web app for
end users such as bot admins without diving into technical details. The app
can be built as a docker image and deployed to various cloud platforms (_e.g._
GCP) for access to more computation resources. We use Streamlit
666https://streamlit.io/ to build the front-end pages. Flask is used to
implement the backend APIs for Streamlit to invoke BotSIM functionalities. The
app is also equipped with a SQL database to keep track of simulation stages
and simulation performance. BotSIM supports two types of SQL databases
including Sqlite3 and Postgres.

##  4 Einstein BotBuilder Case Study

In this section, we show how to perform end-to-end evaluation and remediation
of the pre-built “Template Bot” from the Salesforce Einstein BotBuilder
platform. We follow BotSIM’s “generation-simulation-remediation” pipeline as
detailed in (Wang et al., 2022). The “Template Bot” has six dialog intents.
Each intent has a set of hand-crafted training utterances. For controllable
experiments, we sample 150 utterances per dialog as the training set (train-
original) and use the remaining for evaluation (eval-original). The six
intents are: “Transfer to agent (TA)”, “End chat (EC)”, “Connect with sales
(CS)”, “Check issue status (CI)”, “Check order status (CO)” and “Report an
issue (RI)”. The dataset information is given in Table 2.

Dataset | Intent enquiries | TA | EC | CS | CI | CO | RI
---|---|---|---|---|---|---|---
Train | train-original | 150 | 150 | 150 | 150 | 150 | 150
train-augmented | 255 | 184 | 212 | 268 | 215 | 294
Dev | train-paraphrases | 1465 | 1467 | 1754 | 1989 | 1895 | 1786
Eval | eval-original | 182 | 145 | 183 | 222 | 205 | 178
eval-paraphrases | 1190 | 933 | 1648 | 2172 | 1936 | 1795
Table 2: Dataset information for the Einstein Template Bot case study.

###  4.1 Parse bot metadata

The required inputs for BotSIM include 1) bot design metadata containing the
bot designs (_e.g._ , intents/dialogs, entities) 2) intent utterance metadata
3) LiveAgent API information. They can be retrieved from users’ Salesforce
org. BotSIM starts by parsing the input metadata and generates the NLU (Dialog
Act Maps) and NLG (Response) templates needed for dialog simulation.

###  4.2 Revise dialog act maps and ontology

Figure 3: Revised dialog act map for dialog intent “Check the status of an
existing issue” of Einstein BotBuilder

The generated NLU dialog act maps convert bot messages to dialog acts via
fuzzy matching during dialog simulation. An example of dialog act maps is
shown in Figure 3. The aggregated dialog act map is inferred automatically by
the parser by modelling the bot design as graphs: 1) individual dialogs are
firstly parsed to get their “local” dialog act maps; 2) individual dialogs
(including sub-dialogs such as “Case Lookup”) are modelled as vertices
associated with their “local” dialog act maps; 3) dialog transitions of the
entire bot are modelled as graph edges. The final aggregated dialog act map of
a dialog is created by collecting all the “local” dialog act maps along the
paths starting from the dialog node to the successful dialogs (_e.g.,_
“End_Chat”). Meanwhile, the parser also extracts the bot entity ontology. It
lists all entities and their randomly initialized values for each dialog.

As the only “human-in-the-loop” step in the BotSIM pipeline, review or
revision of the automatically inferred dialog act maps and ontology are
required by BotSIM users. For dialog act maps, the two special dialog acts,
“dialog_success_message” and “intent_success_message”, are the golden labels
indicating a successful task completion and a correct intent classification,
respectively. They are inferred heuristically by regarding the first dialog
message as the “intent_success_message” and the last message as the
“dialog_success_mesage”. Users are required to verify these two dialog acts
for each evaluation dialog to ensure their correctness. Note the review of
dialog act maps is a one-time effort unless there are significant changes made
to the bot design. Since the entity values of the ontology are mostly related
to users’ products and services, the randomly initialised values in the
ontology file may be replaced with real ones in order to evaluate the named
entity recognition (NER) model. The revised dialog act maps and the ontology
are subsequently used to create the simulation goals as in Figure 4.3 for
dialog simulation.

###  4.3 Paraphrase and goal generation

Agenda-based user simulation requires pre-defined goals to ensure that the
simulated user behaves in a consistent, goal-directed manner Schatzmann et al.
(2007). The goals consist of a set of constraints ( dialog acts and entity
slot-value pairs) needed to complete the task. An exemplar simulation goal for
“check the status of an existing issue” is given below:

⬇

1{

2 "Goal": {

3 "Check_the_status_of_an_existing_issue_0": {

4 "inform_slots": {

5 "Has_case_number@_Boolean": "no",

6 "Case_Number@Case_Number_Entity": "P3V4S6",

7 "Email_for_Look_Up@Email_Address_Entity": <EMAIL_ADDRESS>
8 "intent": "I would like to know about my issue.",

9 "Needs_transfer_to_agent@_Boolean": "no",

10 "Needs_something_else@_Boolean": "no",

11 "Goodbye@Goodbye_Entity": "goodbye",

12 "Needs_to_add_case_comment@_Boolean": "no",

13 "Case_Comments@_Text": "P94RMU"

14 },

15 "request_slots": {

16 "Check_the_status_of_an_existing_issue": "UNK"

17 },

18 "name": "Check_the_status_of_an_existing_issue"

19 }

20 }

21}

Note how the entity slots are taken from the dialog act maps in Figure 3. In
particular, the “intent” slot contains the intent query to test the intent
models. To enable pre-deployment testing without any human-written test cases,
we apply the paraphrasing models to the “train-original” utterances to get the
“train-paraphrases” dataset. Compared to the number of original utterances,
the “train-paraphrases” dataset is roughly ten times larger as shown in Table
2. Simulation goals are created by taking the “train-paraphrases” as the
intent queries to simulate the variations in real user intent queries. The
“train-paraphrases” goals are subsequently used as the development set to
perform end-to-end evaluation of the dialog system via dialog simulation.

###  4.4 Dialog simulation results

In this study, we focus on the intent model for two reasons. Firstly, the
intent model can be retrained. Secondly, wince we do not have any customer
data, the entity values in the goals are randomly generated and may not
reflect the real-world values. After dialog simulation with “train-
paraphrases” goals, the original intent training sets are augmented with the
wrongly classified intent queries (“train-augmented”) to re-train the intent
model according to the remediation suggestions. We then compare the
performance before and after retraining on the goals created from the human-
written “eval-original” utterances in Table 3 in terms of intent F1 scores.

Model | Eval. | TA | EC | CS | CI | CO | RI
---|---|---|---|---|---|---|---
Baseline | original | 0.92$\pm$0.03 | 0.95$\pm$0.02 | 0.89$\pm$0.03 | 0.93$\pm$0.03 | 0.94$\pm$0.02 | 0.82$\pm$0.04
paraphr. | 0.88$\pm$0.01 | 0.93$\pm$0.01 | 0.85$\pm$0.01 | 0.91$\pm$0.01 | 0.93$\pm$0.01 | 0.77 $\pm$0.02
Retrained | original | 0.92$\pm$0.03 | 0.97$\pm$0.02 | 0.93$\pm$0.03 | 0.95$\pm$0.02 | 0.96$\pm$0.02 | 0.87$\pm$0.04
paraphr. | 0.89$\pm$0.01 | 0.94$\pm$0.01 | 0.90$\pm$0.01 | 0.94$\pm$0.01 | 0.94$\pm$0.01 | 0.80$\pm$0.02
Table 3: Results for the Einstein Bots case study, before and after retraining
the intent model with the augmented training set (F1 with 95% confidence
interval computed with 10K bootstrapped samples).

We observe consistent improvements for all intents after model retraining. The
more challenging the intents (lower F1s) are, _e.g.,_ “Report an issue” and
“Connect with sales”, the larger performance gains are observed compared to
the easier intents such as “End Chat” (higher F1s). This demonstrates the
efficacy of BotSIM in intent model improvement. We conjecture the improved
performance of the more challenging intents is due to more paraphrases being
selected for retraining to better cover the language variation.

###  4.5 Diagnosis and Remediation Dashboard

The remediator generates health diagnosis reports, performs analyses, and
provides actionable recommendations to troubleshoot and improve dialog
systems. The major dashboard components are presented in Figure 4, 5, 6, 7.

#### Bot health reports.

The bot “health” dashboard consists of a set of multi-scale performance
reports. At the highest level, users can have a historical view of most recent
simulation/test sessions (_e.g.,_ after each major bot update). The historical
performance comparison can help users evaluate the impacts of bot changes
quantitatively, from which they can make decisions like whether or not keep
certain changes. In the test session performance summary report, users can
have the details of a selected test session including the data distribution,
overall dialog performance metrics across all dialog intents of the test
session. The dialog-specific performance report provides detailed detailed
intent and NER performance of the selected dialig intent. Through the dialog-
specific performance report, one can quickly identify the most confusing
intents and entities. This saves significant efforts and helps better
allocation of resources for troubleshooting and bot improvement.

#### Remediation recommendations.

In addition to the diagnosis reports, the remediator also provides actionable
insights/suggestions for users to remedy some of the identified issues. The
root causes of the failed conversations are identified via backtracking of the
simulation agenda. The recommendation dashboards (Figure 5 and Figure 6) allow
detailed investigation of all intent or NER errors along with their
corresponding simulated chat logs. For intent models, the paraphrase intent
queries that lead to intent errors are grouped by the original intent
utterances. These original intent utterances are sorted by the number of
intent errors of their paraphrases in descending order (drop-down list of the
Remediation suggestions in Figure 5). Depending on the wrongly classified
intents, the remediator would suggest some follow-up actions. For example, 1)
augmenting the intent training set with the queries deemed to be out-of-domain
by the current intent model, 2) moving the intent utterance to another intent
if most of paraphrases of the former intent utterance are classified to the
latter intent. Note the suggestions are meant to be used as guidelines rather
than strictly followed. More importantly, they can always be extended by users
to include domain expertise in troubleshooting bots related to their
products/services.

#### Conversation analytics.

Another useful component of the Remediator is the suite of conversation
analytical tools as shown in Figure 7. They further help bot practitioners
gain more insights for troubleshooting and improving their dialog systems. The
confusion matrix analysis breaks down the intent model performance into
(sortable) recall, precision and F1 accuracies to help identify the worse
performing intents. It also detects potential intent overlaps using the
clustering algorithms in (Thoma, 2017) based on the performance metrics.
Another useful analytical tool is the tSNE (van der Maaten and Hinton, 2008)
clustering of the intent utterances using sentence transformer (Reimers and
Gurevych, 2019) embeddings. The tSNE visualisation enables users to gauge the
training data quality. It is also an effective tool in identifying overlapping
intents and can potential benefit new intent discovery as well.

#### Dialog path explorer

Lastly, powered by parsers’ conversation graph modelling capability, the
dialog path explorer can be used to visualise different dialog paths of the
current bot design. For example, users can select the “source” and “target”
dialogs and explore the generated dialog paths. Not only is the tool valuable
for comprehensive testing coverage of conversation paths, it also offers a
controllable approach to troubleshooting dialog design related errors or even
helping improve the current bot design.

##  5 Conclusion

We presented the design of BotSIM, an open source, modular end-to-end bot
simulation toolkit for evaluation, diagnosis and remediation of commercial TOD
systems. Through the streamlined “generation-simulation-remediation“ pipeline,
BotSIM can be adopted to accelerate commercial bot development and evaluation,
reduce cost and time-to-market. Thanks to the layered design, not only can it
be used as a framework for bot developers to extend BotSIM to new bot
platforms, it also offers a suite of easy-to-use command line tools and Web
App for bot practitioners to directly apply BotSIM to their bots. We have
open-sourced the toolkit at _https://github.com/salesforce/BotSIM_ including
the Streamlit Web App. We also provides detailed documentations to accompany
the code. We welcome contributions from the community to help improve and
extend BotSIM further.

##  6 Limitations

For efficiency reasons, BotSIM adopts a template-based NLG model for
converting user dialog acts to natural languages. Although the template-NLG is
more controllable and flexible compared to the model-based NLG, they may lack
naturalness. One possible future improvement includes a combination of
template-based NLG and the model-based NLG. For example, we can train a model-
based NLG to generate templates (Wiseman et al., 2018) for BotSIM’s response
templates. In this way, both efficiency and naturalness can be achieved.
Currently, BotSIM is trained and evaluated utilizing English text. We leave
multi-lingual bot simulation capability as one of our future works.

##  7 Broader Impact

The pretrained language-model based paraphrasers (T5-base and Pegasus) used in
this study are pretrained and finetuned with large scale of text corpora
scraped from the web, which may contain biases. These biases may even be
propagated to the generated paraphrases, causing harm to the subject of these
stereotypes. Although the paraphrasing models are only applied to generate the
testing intent queries, BotSIM users are advised to take into consideration
these ethical issues and may wish to manually inspect or otherwise filter the
generated paraphrases. It is also noteworthy that to prevent any data privacy
leakage, the information produced in the simulation (the entity values in the
BotSIM ontology) is randomly generated, and therefore fake. This includes the
email addresses, names.

Figure 4: Remediator dashboard: bot health reports.  Figure 5: Remediator
dashboard: intent model remediation suggestions.  Figure 6: Remediator
dashboard: NER remediation suggestions.  Figure 7: Remediator dashboard:
Conversational analytical tools.

## References

  * Benvie et al. (2020) Adam Benvie, Eric Wayne, and Matthew Arnold. 2020.  Watson assistant continuous improvement best practices.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz – a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.
  * Lei et al. (2018) Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018\.  Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1437–1447, Melbourne, Australia. Association for Computational Linguistics.
  * Li et al. (2016) Xiujun Li, Zachary C. Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016.  A user simulator for task-completion dialogues.  _CoRR_ , abs/1612.05688.
  * Li et al. (2017) Xuijun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017.  End-to-end task-completion neural dialogue systems.  In _Proceedings of The 8th International Joint Conference on Natural Language Processing_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * Schatzmann et al. (2007) Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007\.  Agenda-based user simulation for bootstrapping a POMDP dialogue system.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers_ , pages 149–152, Rochester, New York. Association for Computational Linguistics.
  * Shi et al. (2019) Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019.  How to build user simulators to train rl-based dialog systems.  _arXiv preprint arXiv:1909.01388_.
  * Tan et al. (2021) Samson Tan, Shafiq Joty, Kathy Baxter, Araz Taeihagh, Gregory A. Bennett, and Min-Yen Kan. 2021.  Reliability testing for natural language processing systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4153–4169, Online. Association for Computational Linguistics.
  * Thoma (2017) Martin Thoma. 2017.  Analysis and optimization of convolutional neural network architectures.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605.
  * Wang et al. (2022) Guangsen Wang, Samson Tan, Shafqi Joty, Guang Wu, Jimmy Au, and Steven Hoi. 2022\.  Botsim: An end-to-end bot simulation framework for commercial task-oriented dialog systems.
  * Wiseman et al. (2018) Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2018.  Learning neural templates for text generation.  _CoRR_ , abs/1808.10122.
  * Zhu et al. (2020) Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. 2020.  Convlab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.


Figure 1: Monte Carlo simulations of runs estimator ($C^{(0)}$) for stop-loss
index with $S=0.9$. Data are simulated from AR(1) with $\rho=0.5$ and
$r_{n}=8$ (left panel), $\rho=0.9$ and $r_{n}=9$ (right panel), $\theta=0.078$
$\alpha=4$ and the number of order statistics $k=40$. Dotted lines indicated
the true value of the cluster index.


# Wikipedia Edit Number Prediction based on Temporal Dynamics Only

Dell Zhang  DCSIS
Birkbeck, University of London
Malet Street
London WC1E 7HX, UK
Email<EMAIL_ADDRESS>
###### Abstract

In this paper, we describe our approach to the Wikipedia Participation
Challenge which aims to predict the number of edits a Wikipedia editor will
make in the next 5 months. The best submission from our team, “zeditor”,
achieved 41.7% improvement over WMF’s baseline predictive model and the final
rank of 3rd place among 96 teams. An interesting characteristic of our
approach is that only temporal dynamics features (i.e., how the number of
edits changes in recent periods, etc.) are used in a self-supervised learning
framework, which makes it easy to be generalised to other application domains.

###### Index Terms:

social media; user modelling, data mining; machine learning.

##  I Introduction

Wikipedia is “a free, web-based, collaborative, multilingual encyclopaedia
project” supported by the non-profit Wikimedia Foundation (WMF). Started in
2001, Wikipedia has become the largest and most popular general reference
knowledge source on the Internet. Almost all of its 19.7 million articles can
be edited by anyone with access to the site, and it has about 90,000 regularly
active volunteer editors around the world. However, it has recently been
observed that Wikipedia growth has slowed down significantly [1]. In
particular, WMF has reported
that111http://strategy.wikimedia.org/wiki/March_2011_Update:

> Between 2005 and 2007, newbies started having real trouble successfully
> joining the Wikimedia community. Before 2005 in the English Wikipedia,
> nearly 40% of new editors would still be active a year after their first
> edit. After 2007, only about 12-15% of new editors were still active a year
> after their first edit. Post-2007, lots of people were still trying to
> become Wikipedia editors. What had changed, though, is that they were
> increasingly failing to integrate into the Wikipedia community, and failing
> increasingly quickly. The Wikimedia community had become too hard to
> penetrate.

It is therefore of utter importance to understand quantitatively what factors
determine editors’ future editing behaviour (why they continue editing, change
the pace of editing, or stop editing), in order to ensure that the Wikipedia
community can continue to grow in terms of size and diversity.

The Wikipedia Participation Challenge222http://www.kaggle.com/c/wikichallenge,
sponsored by WMF and hosted by Kaggle, request contestants to build a
predictive model that could accurately predict the number of edits a Wikipedia
editor would make in the next 5 months based on his edit history so far. Such
a predictive model may be able to help WMF in figuring out how people can be
encouraged to become, and remain, active contributors to Wikipedia.

The ‘training’ dataset consists of randomly sampled active editors with their
full history of editing activities on the English Wikipedia (the first 6
namespaces only) in the period from 2001-01-01 to 2010-09-01. An editor is
considered “active” if he or she made at least one edit in the last one year
period, i.e., from 2009-09-01 to 2010-09-01. For each edit, the available
information includes its user_id, article_id, revision_id, namespace,
timestamp, etc.

The predictive model to be constructed should predict, for each editor from
the ‘training’ dataset, how many edits would be made in the 5 months after the
end date of the ‘training’ dataset, i.e., from 2010-09-01 to 2011-02-01. The
predictive model’s accuracy is going to be measured by the Root Mean Squared
Logarithmic Error (RMSLE):

$\epsilon=\sqrt{\frac{1}{n}\sum_{i=1}^{n}{(\log(1+p_{i})-\log(1+a_{i}))^{2}}}\
,$ (1)

where $n$ is total number of editors in the dataset, $\log(\cdot)$ is the
natural logarithm function, $p_{i}$ and $a_{i}$ are the predicted and actual
edit numbers respectively for editor $i$ in the next 5 month period.

The best submission from our team, “zeditor”, achieved 41.7% improvement over
WMF’s baseline predictive model and the final rank of 3rd place among 96
teams. An interesting characteristic of our approach is that only temporal
dynamics features (i.e., how the number of edits changes in recent periods,
etc.) are used in a self-supervised learning framework, which makes it easy to
be generalised to other application domains.

The rest of this paper is organised as follows. In Section II, we present our
approach in details. In Section III, we show the experimental results. In
Section IV, we review the related work. In Section V, we make conclusions.

##  II Approach

Our basic idea is to build a predictive model $f$ (that estimates an active
editor’s future number of edits based on his recent edit history) through
self-supervised learning, as illustrated schematically in Figure 1. The
approach is called “self-supervised” to emphasise the fact that it does not
require any manual labelling of data (as in standard _supervised learning_
[2]) but extracts the needed labels from data automatically.

Figure 1: Our self-supervised learning framework.

To facilitate the description of our approach, we shall from now on talk about
any time-length in the unit of months and refer to any time-point as the real
number of months passed since the beginning date of the dataset. So for the
official dataset ‘training’, the timestamp “2001-06-16 00:00:00” would be 5.5
because it is five and a half months since 2001-01-01.

Let $t_{\text{test}}$ denote the time-point when we would like to predict each
active editor’s number of edits in the next 5 months. To train the predictive
model, we would move 5 months backwards and assume that we were at the time-
point $t_{\text{train}}=t_{\text{test}}-5$. Thus we could know the actual
number of edits made by each active editor in those 5 months after
$t_{\text{train}}$, i.e., the label for our machine learning (regression)
methods. Specifically, the target value for regression would be set as
$y_{i}=\log(1+a_{i})$ where $a_{i}$ is the actual number of edits in the next
5 months. In this way, the _squared error_ loss function
$L(f(\mathbf{x}),y)=(f(\mathbf{x})-y)^{2}$ used by most machine learning
methods (including those in our experiments and final submission) would
connect the _empirical risk_ [2] directly to the evaluation metric RMSLE:

$R_{emp}(f)=\frac{1}{n}\sum_{i=1}^{n}L(f(\mathbf{x}_{i}),y_{i})=\epsilon^{2}\
.$ (2)

Given a time-point (either $t_{\text{train}}$ or $t_{\text{test}}$), each
active editor $i$ would be represented as a vector $\mathbf{x}_{i}$ that
consists of the following temporal dynamics features:

  * •

the number of edits in recent periods of time;

  * •

the number of edited articles in recent periods of time;

  * •

the length of time between the first edit and the last edit, scaled
logarithmically.

The periods used in our final submission for the above temporal dynamics
features are

$\frac{1}{16},\frac{1}{8},\frac{1}{4},\frac{1}{2},1,2,4,12,36,108$

where the length of period first doubles at each step from $\frac{1}{16}$ to 4
and then triples at each step from 4 to 108. The usage of such temporal
dynamics features was inspired by the decent performance of the “most-
recent-5-months-benchmark” — if using the exact number of edits in just one
period (the last 5 months) for prediction could work reasonably well, we
should be able to achieve a better performance by using many more recent
periods. The periods were chosen to be at exponentially increasing temporal
scales, because we conjecture that the influence of an editing activity to the
editor’s future editing behaviour would be exponentially decaying along with
the time distance away from now. The process of _exponential decay_
333http://en.wikipedia.org/wiki/Exponential_decay occurs in numerous natural
phenomena, and it has been widely used in temporal applications where it is
desirable to gradually discount the history of past events [3]. One reason for
changing from doubling to tripling midway through is to include the special
period of 12-months (i.e., one year) that has been used to define the “active”
editors. The periods will be capped by the time scope of the given dataset
(e.g., 106 for the additional dataset ‘moredata’) in case they are out of
range.

We have also introduced a constant _drift_ term (i.e., how much the average
number of edits would change after 5 months) into the formula of making final
predictions, which is a crude way to cover the global shift of target values
along with time. Again, its value is estimated from the situation 5 months
ago.

The concise pseudo-code of our algorithms for learning and predicting is shown
in Figure 2. The complete source code will be made available at the author’s
homepage444http://www.dcs.bbk.ac.uk/~dell/.

Learning

  * •

$t=t_{\text{train}}$ (i.e., $t_{\text{test}}-5$)

  * •

for each active editor $i$ who made at least one edit in $[t-12,t)$:

    * –

represent the editor as a vector $\mathbf{x}_{i}$ consisting of temporal
dynamics features (please refer to the above description)

    * –

label the editor by $y_{i}=\log(1+a_{i})$ where $a_{i}$ is the actual number
of edits in $[t,t+5)$

  * •

learn a predictive model/function $f:x\rightarrow y$ from
$(\mathbf{x}_{i},y_{i})$ pairs using a regression technique such as GBT

  * •

estimate the drift $d$ by comparing the average number of edits in $[t-5,t)$
and that in $[t,t+5)$

Predicting

  * •

$t=t_{\text{test}}$ (e.g., $116$ for the dataset ‘training’)

  * •

for each active editor $i$ who made at least one edit in $[t-12,t)$:

    * –

represent the editor as a vector $\mathbf{x}_{i}$ consisting of temporal
dynamics features (please refer to the above description)

    * –

compute $\hat{y}_{i}$ = $f(\mathbf{x}_{i})$ using the learnt $f$

    * –

output $p_{i}=\exp(\max(\hat{y}_{i}+d,0))-1$ as the predicted number of edits
in $[t,t+5)$

Figure 2: Our algorithms for learning and predicting.

##  III Experiments

###  III-A Datasets

There are three datasets available to all contestants:

  * •

‘training’ is the official dataset for training and testing;

  * •

‘validation’ is the official dataset for validation;

  * •

‘moredata’ is the additional dataset generously provided by Twan van
Laarhoven555http://www.kaggle.com/c/wikichallenge/forums/t/719/more-training-
data.

The characteristics of each dataset are shown in Table I.

TABLE I: The characteristics of each dataset. dataset | $t_{\text{train}}$ | $t_{\text{test}}$ | #editors | #edits
---|---|---|---|---
‘validation’ | 79 | 84 | 4856 | 274820
‘moredata’ | 106 | 111 | 23584 | 5717049
‘training’ | 111 | 116 | 44514 | 22326031

Since we did not have local access to the true labels (target values) of the
dataset ‘training’, we only used it to make the final submission, but
conducted our experiments (for parameter tuning etc.) on the other two
datasets ‘validation’ and ‘moredata’. It is noteworthy that these two datasets
‘validation’ and ‘moredata’ had been filtered to contain only active editors
(who made at least one edit in the last one year period) in order to make them
exhibit the same _survivorship bias_
666http://en.wikipedia.org/wiki/Survivorship_bias as the dataset ‘training’.
This might (partially) ensure that the experimental findings on the former two
datasets could be transferred to the latter one.

###  III-B Tools

We have only used Python777http://www.python.org/ (equipped with
Numpy888http://numpy.scipy.org/) to write small programs for analysing data
and making predictions. The machine learning methods that we have tried for
our regression task all come from two _open-source_ Python modules: one is
scikit-learn999http://scikit-learn.sourceforge.net/, and the other is
OpenCV101010http://opencv.willowgarage.com/wiki/.

###  III-C Results

First, we compare different machine learning methods (with their default
parameter values) in terms of their prediction performances (RMSLE). The
methods being compared include:

  * •

Ordinary Least Squares (OLS)111111http://scikit-
learn.sourceforge.net/modules/linear_model.html#ordinary-least-squares-ols,

  * •

Support Vector Machine (SVM)121212http://scikit-
learn.sourceforge.net/modules/svm.html,

  * •

K Nearest Neighbours
(KNN)131313http://opencv.itseez.com/modules/ml/doc/k_nearest_neighbors.html,

  * •

Artificial Neural Network
(ANN)141414http://opencv.itseez.com/modules/ml/doc/neural_networks.html,

  * •

Gradient Boosted Trees
(GBT)151515http://opencv.itseez.com/modules/ml/doc/gradient_boosted_trees.html.

The experimental results are shown in Table II and Figure 3. Gradient Boosted
Trees (GBT)161616http://en.wikipedia.org/wiki/Gradient_boosting [4, 5] clearly
outperformed all the other machine learning methods on both datasets. GBT (aka
GBM, MART and TreeNet) represents a general and powerful machine learning
method that builds an ensemble of _weak_ tree learners in a greedy fashion. It
evolved from the application of boosting to regression trees [2]. The general
idea is to compute a sequence of very simple trees, where each successive tree
is built for the prediction residuals of all preceding trees on a randomly
selected subsample of the full training dataset. Eventually a “weighted
additive expansion” of those trees can produce an excellent fit of the
predicted values to the observed values. It allows optimisation of any
differentiable loss function. Here we just use the _squared error_ for the
reasons given in Section II. The success of GBT in our task is probably
attributable to (i) its ability to capture the complex nonlinear relationship
between the target variable and the features, (ii) its insensitivity to
different feature value ranges as well as outliers, and (iii) its resistance
to overfitting via regularisation mechanisms such as shrinkage and subsampling
[4, 5].

TABLE II: The prediction performances of different machine learning methods (with their default parameter values). learning method | ‘validation’ | ‘moredata’
---|---|---
OLS | 0.832351 | 0.869779
SVM | 0.901698 | 0.732814
KNN | 0.833288 | 0.690832
ANN | 0.987345 | 1.040396
GBT | 0.820805 | 0.635807

(a) validation

(b) moredata

Figure 3: The prediction performances of different machine learning methods
(with their default parameter values).

Second, we investigate how GBT’s most important parameter weak_count — the
number of weak tree learners — affects its prediction performance for our
task. Tuning weak_count is our major means of controlling the model complexity
to avoid underfitting or overfitting. The experimental results are shown in
Table III and Figure 4. It seems that on big datasets like ‘moredata’, a
higher value of weak_count (i.e., more weak tree learners) would be
beneficial, but on small datasets like ‘validation’, it might increase the
risk of overfitting.

TABLE III: The prediction performances of GBT with different number of weak tree learners (weak_count). GBT weak_count | ‘validation’ | ‘moredata’
---|---|---
200 | 0.820805 | 0.635807
400 | 0.817789 | 0.616876
600 | 0.817483 | 0.614507
800 | 0.817614 | 0.613757
1000 | 0.818726 | 0.613530
1200 | 0.819804 | 0.613465
1400 | 0.819998 | 0.613671

(a) validation

(b) moredata

Figure 4: The prediction performances of GBT with different number of weak
tree learners (weak_count).

Third, we demonstrate how the prediction performance changes when we use more
and more periods to generate temporal dynamics features: we start from just
the shortest period ($\frac{1}{16}$) and then each time we add the next longer
period to the series (see Section II). The experimental results are shown in
Table IV and Figure 5. It seems that making use of more periods for temporal
dynamics features usually helps, but the pay-off gradually diminishes.

TABLE IV: The prediction performances of GBT using different number of periods for temporal dynamics features. GBT #periods | ‘validation’ | ‘moredata’
---|---|---
1 | 0.861111 | 0.788450
2 | 0.857575 | 0.760365
3 | 0.849440 | 0.728888
4 | 0.841127 | 0.696196
5 | 0.836116 | 0.669754
6 | 0.830619 | 0.647883
7 | 0.829393 | 0.629062
8 | 0.816459 | 0.614429
9 | 0.818515 | 0.613749
10 | 0.818726 | 0.613530

(a) validation

(b) moredata

Figure 5: The prediction performances of GBT using different number of periods
for temporal dynamics features.

###  III-D Submissions

Since ‘moredata’ is more similar than ‘validation’ to the official dataset
‘training’ in terms of the time scope and the number of editors, we applied
the best working algorithm, GBT, with the optimal parameter setting on
‘moredata’ (weak_count = 1000), to make the final submission based on
‘training’. It got an RMSLE score of 0.862582 on the private leaderboard,
which is roughly 41.7% better than WMF’s baseline predictive model. The final
rank of our team, “zeditor”, is the 3rd place among 96 teams.

##  IV Related Work

The global slowdown of Wikipedia’s growth rate (both in the number of editors
and the number of edits per month) has been studied [1]. It is found that
medium-frequency editors now cover a lower percentage of the total population
while high frequency editors continue to increase the number of their edits.
Moreover, there are increased patterns of conflict and dominance (e.g.,
greater resistance to new edits in particular those from occasional editors),
which may be the consequence of the increasingly limited opportunities in
making novel contributions. These findings could guide us to generate other
kinds of useful features to tackle the problem of edit number prediction.
Furthermore, researchers have also investigated other activities of
Wikipedia’s editors, such as voting on the promotion of Wikipedia admins [6].

In addition to Wikipedia, the temporal dynamics of online users’ behaviour has
been explored and exploited in web search [7, 8, 9, 10], social tagging [11,
12], blogging [13], twittering [14], and collaborative filtering [15]. The
_power law_ [16] and the _exponential decay_ [3] seem to be recurrent themes
across application domains.

##  V Conclusions

Our most important insight is that a Wikipedia editor’s future behaviour can
be largely determined by the temporal dynamics of his recent behaviour. We are
a bit surprised that just temporal dynamics features can go such a long way
when we choose proper temporal scales and employ a powerful machine learning
method. Human beings seem to be working and living in a more mechanical way
than one might have thought. Since such temporal dynamics features are
actually independent of any semantics or knowledge about this specific
problem, our approach could be easily generalised to other application
domains, such as predicting the future supermarket spendings of shoppers
(e.g., the dunnhumby’s Shopper
Challenge171717http://www.kaggle.com/c/dunnhumbychallenge), predicting the
future hospital admissions of patients (e.g., the Heritage Health Prize
Competition181818http://www.heritagehealthprize.com/c/hhp), and so on, based
on historical behavioural data.

Have we answered the question that we asked at the beginning of this paper?
Yes and No. On one hand, we have built a predictive model which can be used to
identify those editors who are likely to become inactive, or in other words,
who need special care and attention to be kept — if an editor is going to
leave the Wikipedia community, there would probably be early signals in the
temporal dynamics of his recent behaviour. On the other hand, that predictive
model is pretty much a black box — it does not reveal the underlying reasons
why editors become inactive, and therefore it cannot tell us how to encourage
editors to remain active. For the ultimate purpose of Wikipedia’s sustainable
growth, we will need to investigate which attributes of an editor (his
articles’ category distribution, his relationship with other editors, etc.)
and also which recent events happened to him (his articles being deleted, his
revisions being reverted, unfair comments about his edits being received,
etc.) could affect his behaviour. Due to the time constraints and the dataset
limitations (for example, the lack of information about articles and comments
in the datasets ‘validation’ and ‘moredata’), we have to leave it to future
work.

Long live Wikipedia!

## Acknowledgment

We would like to thank WMF and Kaggle for their wonderful job in organising
this interesting contest. We are grateful to Twan van Laarhoven for creating
and sharing the additional dataset ‘moredata’. We also appreciate the
reviewers’ helpful comments.

## References

  * [1] B. Suh, G. Convertino, E. H. Chi, and P. Pirolli, “The singularity is not near: Slowing growth of Wikipedia,” in _Proceedings of the 2009 International Symposium on Wikis (WikiSym)_ , Orlando, FL, USA, 2009.
  * [2] T. Hastie, R. Tibshirani, and J. Friedman, _The Elements of Statistical Learning: Data Mining, Inference, and Prediction_ , 2nd ed. Springer, 2009.
  * [3] C. C. Aggarwal, J. Han, J. Wang, and P. S. Yu, “A framework for projected clustering of high dimensional data streams,” in _Proceedings of the 13th International Conference on Very Large Data Bases (VLDB)_ , Toronto, Canada, 2004, pp. 852–863.
  * [4] J. Friedman, “Greedy function approximation: A gradient boosting machine,” IMS 1999 Reitz Lecture, Tech. Rep., Feb 1999.
  * [5] ——, “Stochastic gradient boosting,” Stanford University, Tech. Rep., Mar 1999\.
  * [6] J. Leskovec, D. P. Huttenlocher, and J. M. Kleinberg, “Governance in social media: A case study of the wikipedia promotion process,” in _Proceedings of the 4th International Conference on Weblogs and Social Media (ICWSM)_ , Washington, DC, USA, 2010.
  * [7] D. Zhang, J. Lu, R. Mao, and J.-Y. Nie, “Time-sensitive language modelling for online term recurrence prediction,” in _In Proceedings of the 2nd International Conference on the Theory of Information Retrieval (ICTIR)_ , Cambridge, UK, 2009, pp. 128–138.
  * [8] D. Zhang and J. Lu, “What queries are likely to recur in web search?” in _Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , Boston, MA, USA, 2009\.
  * [9] J. L. Elsas and S. T. Dumais, “Leveraging temporal dynamics of document content in relevance ranking,” in _Proceedings of the 3rd International Conference on Web Search and Web Data Mining (WSDM)_ , New York, NY, USA, 2010, pp. 1–10.
  * [10] A. Kulkarni, J. Teevan, K. M. Svore, and S. T. Dumais, “Understanding temporal query dynamics,” in _Proceedings of the 4th International Conference on Web Search and Web Data Mining (WSDM)_ , Hong Kong, China, 2011, pp. 167–176.
  * [11] D. Zhang, R. Mao, and W. Li, “The recurrence dynamics of social tagging,” in _Proceedings of the 18th International Conference on World Wide Web (WWW)_ , Madrid, Spain, 2009, pp. 1205–1206.
  * [12] H. Halpin, V. Robu, and H. Shepherd, “The complex dynamics of collaborative tagging,” in _Proceedings of the 16th International Conference on World Wide Web (WWW)_ , Banff, Alberta, Canada, 2007, pp. 211–220.
  * [13] Y.-R. Lin, H. Sundaram, Y. Chi, J. Tatemura, and B. L. Tseng, “Detecting splogs via temporal dynamics using self-similarity analysis,” _ACM Transactions on the Web (TWEB)_ , vol. 2, no. 1, pp. 1–35, 2008.
  * [14] F. Abel, Q. Gao, G.-J. Houben, and K. Tao, “Analyzing temporal dynamics in twitter profiles for personalized recommendations in the social web,” in _Proceedings of the 3rd International Conference on Web Science (WebSci)_ , Koblenz, Germany, 2011.
  * [15] Y. Koren, “Collaborative filtering with temporal dynamics,” in _Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_ , Paris, France, 2009, pp. 447–456.
  * [16] A. Clauset, C. R. Shalizi, and M. E. J. Newman, “Power-law distributions in empirical data,” _SIAM Review_ , vol. 51, no. 4, pp. 661–703, 2009.