vskip=5pt

# Transforming Unstructured Text into Data with Context Rule Assisted Machine
Learning (CRAML)

Stephen Meisenbacher***Equally contributing authors. We are grateful for
support from the Economic Security Project Anti-Monopoly Fund; Loyola Rule of
Law Institute; Loyola Quinlan School of Business; and Loyola University
Chicago. We also thank Patricia Tabarani, Eric George, Steve Sauerwald, and
Chris Erickson. †††Technical University of Munich, School of Computation,
Information and Technology Peter Norlander11footnotemark: 1 ‡‡‡Loyola
University Chicago, Quinlan School of Business

###### Abstract

Abstract: We describe a method and new no-code software tools enabling domain
experts to build custom structured, labeled datasets from the unstructured
text of documents and build niche machine learning text classification models
traceable to expert-written rules. The Context Rule Assisted Machine Learning
(CRAML) method allows accurate and reproducible labeling of massive volumes of
unstructured text. CRAML enables domain experts to access uncommon constructs
buried within a document corpus, and avoids limitations of current
computational approaches that often lack context, transparency, and
interpetability. In this research methods paper, we present three use cases
for CRAML: we analyze recent management literature that draws from text data,
describe and release new machine learning models from an analysis of
proprietary job advertisement text, and present findings of social and
economic interest from a public corpus of franchise documents. CRAML produces
document-level coded tabular datasets that can be used for quantitative
academic research, and allows qualitative researchers to scale niche
classification schemes over massive text data. CRAML is a low-resource,
flexible, and scalable methodology for building training data for supervised
ML. We make available as open-source resources: the software, job
advertisement text classifiers, a novel corpus of franchise documents, and a
fully replicable start-to-finish trained example in the context of no poach
clauses.

Keywords: labor markets, data creation, text classification, hybrid system,
big data

##  1 Introduction

Advances in computational methods offer new ways to gain insight from large
volumes of unstructured text, and yet these methods each have significant
limitations, tradeoffs, and a lack clear guidelines [99]. Despite the advances
in Natural Language Processing (NLP), Machine Learning (ML), Artificial
Intelligence (AI) and more recently Deep Learning (DL), there is still “the
herculean task of finding constructs using full-text search” [79, 547].

Unstructured text lack a schema, are not standardized, have multiple formats,
and come from diverse sources [9]. Greater attention and systematic research
is needed to develop processes to create structured data from unstructured
text [45]. A lack of structured information is a barrier to understanding for
researchers and organizations: “as much as 80% of an organization’s data is
‘dark”’ [78]. For knowledge professionals, just-in-time access to domain
specific document repositories is vital, but knowledge must first be codified
[114]. Zettabytes (ZB) of new data are produced daily [20], and roughly 80% is
unstructured [67]. While the manual expert labor required for qualitatively
coding novel classification schemes is hard to scale, statistical packages,
many information systems, and quantitative social scientists expect data to be
codified and in a regular format: “tabular data – variables in columns, cases
in rows” [80].

ML and AI are potential solutions, but require structured training data. Hand-
coding by experts is required in many domains, but does not easily scale. To
address the disjunction between these, we describe a method that bridges
expert-built set of context rules scaled to classify unstructured text and
build training data for machine learning. Context Rule Assisted Machine
Learning (CRAML) is a hybrid method for structuring textual data from large-
scale corpora into datasets ready for training ML models and quantitative
research. CRAML begins with a workflow common in qualitative manual coding
research, and ends with ML models trained on text input data shaped by a
qualitative coding scheme. CRAML is useful for qualitative and quantitative
social scientists: it empowers domain experts to efficiently mine volumes of
text and scale classification schemata, and it yields tabular data that
captures the occurrence of concepts within a corpus of documents.

This paper contributes to information systems research on the design of
intelligent systems that can generate economic and social value from
unstructured data [1]. By clearly defining the problem, managing the training
data before AI is attempted, and affording users the opportunity to evaluate
and adjust both inputs and outputs, CRAML addresses major concerns identified
in research on AI [131]. This is the first paper to offer a thorough
description of a versatile method with the potential to empower advanced users
of an ordinary desktop computer to build machine learning classifiers and
rectangular datasets from unstructured text according to the users’ desired
scheme. Expert built rule-assisted machine learning models have been described
in the literature, and found to outperform benchmark methods and professional
perception in critical cases such as detecting persons in states of emotional
distress [35]. In addition to describing the method, we call and set new
standards for transparency and interpretability of research on text-to-data
machine learning, and demonstrate the high utility of this approach in three
applications.

In this methods paper, we highlight the need for the CRAML tool and
demonstrate its versatility through applications. First, we use CRAML to
perform a simple literature review that highlights current approaches to
analysis of large text corpora in five top management journals. Second, we
release machine learning classifiers with the potential to enhance labor
market efficiency, and set a new standard for transparency for research
studying job advertisement text, where underlying data includes proprietary
data obtained under license. We release ML classifiers, and the rules that
created them, to enable other researchers using job advertisement text to
replicate and interpret our approach. Last, we publish a new and massive
document corpus of mandatory franchise disclosure documents and present a
start-to-finish analysis to enable a full replication of our process. We
detect thousands more plausibly illegal “no poach clauses” that could violate
state and federal antitrust law than found in earlier research [77]. We
release a rectangular dataset that represents the prevalence of suspected no
poach clauses that restrict the ability of franchise firms to recruit
employees from other companies that belong to a franchise system. As part of
the contribution of this work, we publish everything needed to fully replicate
the no poach analysis: the original PDFs and the machine readable text of the
franchise documents, CRAML software, the manually coded inputs that creates
the structured, labeled data output, the training data, and a ML classifier to
detect suspect no poach clauses.

Section 2 reviews the well-known challenges CRAML was created to address and
illustrates the frequency of current approaches in the management literature.
Section 3 introduces and provides detailed motivation for our solution in the
context of job advertisement text. Methodological detail on CRAML and
empirical analysis of franchise no poach agreements is in Section 4. A more
technical methodology is described in Appendix LABEL:sec:appendixc. Section 5
provides future directions and discussions.

##  2 The State of the Art

Across many fields of research, methods for analyzing unstructured text are
needed. Below, we highlight literature in management, information systems,
computational social science and computer science on: machine learning (ML),
artificial intelligence (AI), natural language processing (NLP), topic
modeling and Latent Dirichilet Allocation (LDA), linguistic inquiry and word
counting (LIWC), computationally aided text analysis (CATA), and text or
document corpora. We used the CRAML software to narrow our search for relevant
literature. We obtained a corpus of 1,994 papers published from 2015-2021 in 5
top empirical management journals (Academy of Management Journal,
Administrative Science Quarterly, Journal of International Business Studies,
Organization Science, and Strategic Management Journal). Within the management
literature corpus, we searched for papers that could illustrate the use of a
method for turning unstructured text into data. In Appendix A, we provide
additional detail on how CRAML can be used in bibliographic research.

###  2.1 Word Counting Approaches (LIWC, CATA)

Computer-aided text analysis (CATA) and Linguistic Inquiry and Word Count
(LIWC) are two popular techniques used in the management literature. Experts
develop lists of words or dictionaries that correspond to an overall construct
and measure the occurrence of keywords inside specific documents [110, 109].
Researchers often use LIWC standard dictionaries to detect common constructs,
such as positive / negative sentiment in a corpus of tweets [16]. Using these
methods, researchers can also develop custom dictionaries to capture
previously unstudied phenomena within an environment: for example, to measure
how actors use specific cultural toolkits in a field of action [125]. Such
efforts are typically first driven by deduction (keyword selection) and then
interrogation of text (examining n-grams and short excerpts of text containing
keywords) [90].

We developed keywords deductively (available in Appendix A) and narrowed our
search for relevant papers using CRAML. CRAML output indicates whether each
academic paper in the corpus contains keywords that fall within a given
construct (or tag): we focused on a subset of 110 papers that contain
reference to text data or a document corpus. Figure 1 shows that text corpus
analysis and CATA along with it have grown over time; 88 of the 110 papers
that mention a text corpus also refer to CATA.

Figure 1: Counting Words: The Growth of Management Papers Using Text Data and
CATA, AI, ML, LDA, NLP

####  2.1.1 Limitations of Keyword-Based Approaches

Conceptual schemes that rely only on keywords are not very rich [46].
Constructs may not be well-captured by word frequency approaches used in CATA
and LIWC programs; these methods perform less well than ML at tasks such as
inferring personality [41]. Word counting assumes that all word occurrences
equally contribute to the measurement of a construct - that every use of the
keyword “artificial intelligence” or “machine learning” captures both papers
that use an AI or ML method, as well as those that study or mention AI and ML
phenomena. Because the method relies on exact matching and correct spellings,
CATA and LIWC will fail to recognize concepts when faced with messy data or
typos. Without the benefit of context surrounding the keywords, key meaning
and accuracy are lost to the user. When describing the frequency of terms
inside a document, CATA methods do not typically adjust for document length,
although additional steps can be taken to make adjustments [66].

While keywords can start a search for relevant concepts, rules and qualitative
coding pinpoint only the relevant text within the documents. Qualitative
researchers who seek to inductively build theory typically begin by engaging
in keyword search, but also engage in qualitative hand coding [84].

###  2.2 Unsupervised Learning (Topic Modeling and LDA)

Text clustering aims to solve the problem of information overload by
simplifying a mass of text into a smaller number of meaningful categories.
While many text clustering methods exist [10], a popular technique is topic
modeling, which aims to learn topics from an unstructured collection of text
documents. Data is often first heavily pre-processed (stemmed), stripped of
common words, and converted into a document-term matrix, which leaves only the
stems of latent keywords appearing. Latent Dirichlet Allocation (LDA) assigns
all words in a document to a topic, under the assumption that every part of
every document contributes to some topic. LDA’s versatility is that it is not
domain-specific: it can be used to study hacker forums [129], why people
retweet [56], online communities [18], and discourse on emerging technologies
like blockchain [92]. LDA can also assist in inductive theorizing [111].

Of the 110 text corpus papers in the management literature, 11 papers discuss
the use of this method, with 5 of these papers appearing in 2021. [36] use LDA
to arrive at text-based measures of corporate dissimilarity from corporate
annual reports, and use 25 crowd-sourced MTurkers to ensure themes are
reliably identified for the top 25 of 125 latent topics chosen. [58] use LDA
on annual reports and conduct a semantic network analysis to illustrate the
connections and develop grounded theories of how firms interact with the legal
environment. [73] use patent text to study the recombination in breakthrough
innovation. [112] study abstracts within a citation network. [37] use topic
models to analyze CEO text and synthesize analysis of topic models using
machine learning analysis of CEO facial expressions.

####  2.2.1 Limitation: Meaningless Output

Latent themes that emerge inductively are meaningless and un-interpretable
absent additional data (e.g., facial expression analysis to match themes
(ibid)), or expert or crowd-sourced judgement. LDA assumes that all of the
text in every document inside a text corpus is assigned to a theme. This is
unwarranted if an expert has only a niche interest in a particular phenomenon.
Papers we reviewed address this concern by limiting the scope of their corpus,
or by extensively pre-processing to remove potential noise, but this also
removes potentially relevant context and data. A second assumption of topic
modeling approaches is that one must know the number of topics that exist
within a collection of documents. This is non-trivial, particularly with
massive corpora.

Finally, topic modeling imposes an assumption that a researcher be interested
in an unbiased representation. Experts often disagree about the interpretation
of text, and a quest for unbiased (as opposed to transparent and replicable)
analysis may be misguided [97]. Ultimately, unsupervised computational methods
face difficulty producing meaningful results in the absence of human
involvement and expert interpretation [95].

###  2.3 The Modern ML Paradigm

Supervised machine learning (ML) models are widely used in many commercial
applications: to evaluate candidates for employment [38], to support managers
in giving employee feedback [120], to moderate text content on the internet,
judge the sentiment of customer-written reviews [86], and to build proprietary
datasets for financial and academic use. Widely available ML methods can train
a model to learn academic constructs and scale such models over vast volumes
of unstructured text. However, computational models of text often struggle to
provide insights that are interpretable, and may be perceived as too difficult
for low-resource users to access. Among 10 papers in the management corpus
discuss ML, and four more that discuss AI, we could not locate any that
describe the development of a new academic text classification model that is
used to construct new data from unstructured text.

While general AI and ML tools rapidly advance, there is a great need for
applications in specific domains, such as detecting banking fraud [4] or fake
websites [3]. Human-in-the-Loop or hybrid systems are increasingly
prominent.444Hybrid systems are not novel, with discussions already spanning
decades. For example, placing a human in the loop was proposed in the design
of socially intelligent agents [43]. More recently, human involvement in
security settings [40] and cyber-physical systems [107] have appeared in the
computer science literature. The idea of human-in-the-loop for AI applications
is promoted by [130] as a way to introduce responsibility into intelligent
agents, as well as to increase interpretability. Likewise, human-in-the-loop
for ML and NLP is particularly promising, both for data preprocessing and
model learning [127]. Efforts to use AI and exclude domain experts are likely
to fail, as shown in a study of how AI use in hiring decisions evolved into
hybrid practices [28]. Many AI systems cannot (and arguably should not) be
trained without human oversight, and thus such intervention is vital to
incorporating the necessary human knowledge into these systems.

Even with a human-in-the-loop approach, challenges have been noted, for
example by [128], which among others includes the ability to analyze the
impact changes with iterative human intervention. For academics in management
and information systems, incorporating the knowledge of domain experts into
the process of building structured data is essential for both deductive and
inductive study. Such domain knowledge is crucial to forming the essence of
training data, which necessarily represents some aspect of the worldview of a
human who chose to train a model from it.

For supervised ML, challenges lie not so much with the ML methods or models
themselves, but with the paradigm for ML. The process by which ML models are
built (i.e. how information is learned) is opaque at the outset. Supervised
methods in the Modern ML paradigm, and existing tools for the ML pipeline, do
not give experts the ability to easily create training data and thus train a
simple model that can scale over vast quantities of text. As discussed by [7],
much of the attention around Explainable AI (XAI) has been placed on the way
in which models behave, seeking meaning and interpretability in the outputs of
models that can often be highly opaque. If, instead, experts could easily
build and curate training datasets, there would be greater opportunity to
build interpretable models accepted by experts and members of the relevant
audience.

Labeled training data is the critical input to supervised ML models to code
future, unseen text instances. What we call the Modern ML Paradigm for
supervised learning of text data largely sees training data only as input, and
holds the interpretability and explainability of ML model outputs as a post
hoc concern. If building custom training datasets for text were easier, then
resulting models can be interpreted – and modified – at the input stage.
Expert-curated ML models could be interpretable and put into action in a
domain, and escape some of the limitations of the modern ML paradigm: black
boxes, proprietary training data, and un-reproducible, meaningless output that
exclude participation from voices without supercomputers or advanced computer
science knowledge.

####  2.3.1 Limitation: The Black Box of Supervised Learning

The Black Box refers to the mystery by which textual inputs are categorized by
machines leading to certain outputs and are rooted in increasingly large model
architectures and consequently, how computational models memorize, rather than
truly learn. In turn, one is often hard pressed to decipher the behavior of a
classifier, even with state of the art models – they simply reflect patterns
observed during training.

Trust requires understanding what the machines are doing [33]. In the modern
ML paradigm, attempts to explain the black box often come after the fact,
which [12] calls Post-hoc Explainability. Here, when faced with models that
are not interpretable by design, one may try to boost explainability after the
fact. As [12] writes, post-hoc approaches have become the _de facto_ way of
achieving some level of explainability with complex DL models.

Without understanding the data that goes into training the model, the
explainability of the modern ML paradigm is limited. In a recent survey of
explainability in supervised ML methods, [30] call for rethinking the problem
from first principles, and encourage researchers to ask What are we actually
looking for? Do we really need a black box model? For many experts, the
immediate answer is no. Even a ML model with a high degree of predictive power
is likely to be seen as undesirable if it does not produce interpretable
results [97, 95].

Underspecification can result from training models on massive volumes of data
and selecting the model(s) that achieve a desired predictive power [42].
Underspecification often leads to failure when those models are used outside
of the context in which they were trained. As [69] writes: the process used to
build most machine-learning models today cannot tell which models will work in
the real world and which ones won’t. [65] calls for greater interaction
between humans and machines and for explainability when models are faced with
new situations – a sign of mastery.

####  2.3.2 Limitation: Proprietary Training Data and Methods

Seen from a different angle, black boxes may be valuable in order to protect
trade secrets hidden behind such trained models [105]. Secrecy presents risks
in the cases where high-stakes decisions are made by the models in question,
having potentially significant societal or economic implications. Academic
users often rely on commercial data providers that give little to no insight
into their proprietary process for data creation. In our analysis of the
management literature, 13 of the 100 text analysis papers, and 117 papers in
total indicate the use of licensed, proprietary data or models. The problem
identified by [81] is that such computational methods could become the
exclusive domain of private companies and government agencies that operate
contrary to the academic commitment to openness. The inadequacy of data-
sharing paradigms for big data cast doubt regarding the veracity of ML models
[82].

Private firms and academics typically advance the state of the art by breaking
new records for the complexity of data, the size of models, and the number of
parameters in a model. Many potential contributions from researchers who lack
access to super-computing resources, the funds to buy large proprietary
datasets, or access to confidential government data are excluded [32]. Our
search for management papers that indicate proprietary data or models
illustrate stable trends over time, but when big or secret data outputs become
the criteria through which the quality of research is judged, only those with
access can compete, and there may be less attention to cumulative progress
toward answering important questions [44].

By reducing the need for humans to think, act, or belong (contribute, be
included) in the public sphere, [72] write that ML systems can constrain and
oppress, rather than achieve emancipatory goals. To deliver on the potential
of ML, improvements needed include greater sharing of data, protections for
privacy to enable more open data, intellectual property standards to reduce
transaction costs, and approaches that enable genuine replication [75]. Pre-
processing of datasets for supervised learning could be a core strength and
contribution of social scientists in analyzing text data [45].555One promising
approach is computational grounded theory: a novel framework to use
unsupervised machine learning to identify patterns, engage a human in
interpretive refinement of patterns, and use computational methods to assess
and confirm identified patterns [95]. This human-centered computational
exploratory analysis is proposed to help social scientists benefit from
unsupervised computational methods and enhance the transparency and
reproducability of content analysis [95]. Despite advances, DiMaggio’s (2015)
call for greater systematic research to develop solutions for curation
challenges and guidelines for pre-analysis and developing training data has
largely been unmet.

####  2.3.3 Limitation: Meaningful Contributions to Knowledge

In a statement that rings true almost a decade after it was written, there is
little evidence of computational social science in leading social science
field journals [124], or in the management corpus reviewed here. Computational
social science methods have advanced significantly, but it has yet to be
demonstrated how computational scientific infrastructure can be applied to
important social problems [82]. Disciplinary silos, privacy concerns, and
unreliable, non-replicable, and proprietary data hamper the potential for
computer science methods to provide meaningful insight into society’s
challenges [82].

The emergence of a computational social science apart from traditional
academic disciplines speaks to the hope that ML and AI can advance knowledge.
However, the possibilities of these technologies are lessened when such tools
are perceived as beyond the reach of ordinary users [48]. Less than a decade
old, some such initiatives, such as data science for good, have been
criticized for not giving the social sector the opportunity to design for what
we want [101]. Without tools that enable domain experts to develop their own
ML models, there is a risk that new interdisciplinary or trans-disciplinary
centers might lead to yet another siloed academic discipline as ill equipped
to solve problems as others [5]. Future opportunities for AI identified by
[21] include democratization, reducing requirements for data, and enhancing AI
explainability and transparency.666Emerging paradigms for low-resource Natural
Language Processing (NLP) seek to eliminate barriers to accessing ML
technology [71].

[122] attributes a Hyper-Focus in current ML research to three causes that
have led it to lose its connection to problems of import to the larger world
of science and society. First, the (over)use of benchmark datasets, many of
which are synthetic, has glaring limitations and still make reproducability
difficult. Only 1% of ML papers are applied to a specific domain and the rest
use benchmark datasets.777Classic examples of these datasets include the Movie
Review Dataset [87] or the Yelp Review Dataset. While use of standardized
datasets may reduce the reproducibility crisis in ML [74], there is a
tradeoff. As one might imagine, benchmark and standardized datasets have
limited application to the real world; furthermore, many benchmark datasets
are tailored for specific tasks, e.g. movie reviews are often used for
sentiment analysis. By way of contrast, [79] report that 90% of research
constructs are uncommon, suggesting that using pre-labeled standard benchmark
data for training will miss entire vast areas of domain-specific knowledge.
Secondly, the focus on abstract metrics like F-scores to determine model
quality across domains is a mirage that masks the need to focus on impact and
usefulness of a model. A final issue is the lack of follow-through, suggesting
that current computational research leaves little incentive for connection to
sustained research agendas.

A related concern is the rapidly depreciating relevance of data that many ML
models are trained on [82]. In the ML training process, models are trained in
the lab on a training dataset, yet the environment in which models are
deployed is dynamic. Big data research may apply only to specific users of a
specific portal at a specific time in which a study is conducted [68, 100].
For that reason, online studies and experiments with large response rates may
fail to produce reliable or general knowledge [27]. A static training process
may simply not reflect the real world environment or its rapid evolution [15].
The velocity of internet-based data also creates a limitation for
computational social scientists’ conclusions [93]. Solutions to these
obstacles include forming new multi-disciplinary, international journals that
permit rapid publication of rigorous quantitative descriptions of rapidly
evolving phenomena [94].

###  2.4 Seeking Explainable Tools for Social Science

To sum up the issues described in this section, each method reviewed has
strengths, limitations, and tradeoffs for experts who seek to analyze
unstructured text. For domain experts with an interest in specific phenomena,
the methods reviewed so far do not allow for expert-led, targeted extraction
of niche constructs hiding within corpora and the construction of standardized
data. For many qualitative researchers, the systematic analysis and discovery
of new patterns within domain-specific digitized records is a core
contribution – the ability to find the diamond (a single topic) in the rough
(lots of text). If qualitative hand-coded contributions could be easily
scaled, it can address some issues found in the supervised ML domain.

##  3 Context Rule Assisted Machine Learning (CRAML)

Enabling experts to create training data for niche classifiers to detect
constructs that are relevant within a specific domain would address many of
the problems identified with the modern ML paradigm, and the limitations of
unsupervised computations methods. The CRAML method we present emphasizes
explainability and has a contribution on each of the six dimensions that
explain the performance of AI systems according to [13]: the models are
specific to a single construct, the goals are clear, by limiting the chunk
size and extracting text from documents in the corpus, the training and input
data is intended to be context-specific, the output data is interpretable (a
0/1 binary classification), and the environment is also intended to be domain
specific and expert-guided. By emphasizing the “human in the loop,” CRAML
gives qualitative and quantitative researchers control over the training data
that is the key input to ML models. This offers new capabilities to control
the analysis and measurement of phenomena found in massive volumes of
unstructured text. This section briefly introduces the solution developed and
explains how the solution addresses the problems described above.

###  3.1 A Framework for Unstructured Text Analysis

CRAML software gives users the tools to manipulate keywords to extract
relevant data from massive corpora, create tags that seek to capture topics or
themes, and subsequently, develop context rule sets that codify knowledge
about the specific pieces of text that correspond to specific tags. The CRAML
process yields tabular output that matches document-level metadata with a 0/1
indicator that symbolizes the occurrence of tags in a document [80].

Figure 2 provides a high-level overview of the CRAML framework. The CRAML
process begins with the full text of a corpus of documents (Step 0). While
metadata is not required, CRAML software is designed to combine
characteristics extracted from the text with metadata at the document or
record ID level.

Figure 2: A Simple View of the CRAML Process

In step 1, the researcher chooses keywords to extract from the corpus, and
retrieves a _chunk_ : a block of text containing a user-selected number of
words surrounding the keyword. The chunk length should contain the full
_context window_ : all of the relevant context for a human to determine if the
keyword-containing chunk is relevant or not for some binary classification
schema. The user can extract all chunks from all of the documents in the
corpus, or in low-resource environments, can randomly sample from the
documents in order to perform Steps 2-4 on a smaller subset.

In step 2, the researcher analyzes the patterns within the chunks, studying
n-grams that reveal the most common phrasings surrounding the keyword.

In step 3, the researcher writes _context rules_ that elaborate an ever-more
detailed scheme of binary classifications of the chunks. We refer to these
binary classification schemata as _tags_ : each tag represents a topic or
theme that the researcher defines by recognizing patterns in the extracted
text and determining if each chunk is (tag=1), or is not (tag=0), illustrative
of the tag. There is no limit on the number of tags a researcher can define.
For each tag, there is no limit to the number of rules that can be defined
using either exact text matching or regular expressions (RegEx).

In step 4, the researcher written context rules are applied or _extrapolated_
over either the full extracted chunks or a sample of the extracted chunks to
create a training dataset for ML modeling. Extrapolation labels each chunk
with a 0 or 1 for each “tag,” or binary classification scheme, and creates a
dataset that can be used for training ML models. In step 5, common ML
algorithms use the training dataset to tag the full extracted chunks, and the
researcher identifies the best performing model. In the final step, the
results are aggregated into a document-level database of structured
information: for any document within the text corpus, the researcher knows if
the document contains a tag. Additional hand-coding by independent third-
parties can validate that the training dataset extrapolation, and the ML
results, are highly accurate.

##### Versatile tools for mixed approaches

CRAML’s pipeline incorporates a novel set of tools that provide an end-to-end
solution to the challenge of structuring unstructured text data. However,
researchers may appreciate its flexibility and multiple uses: a researcher may
wish to pursue analysis using only pieces of the CRAML process. The proposed
system is not a rigid algorithm, but rather a flexible pipeline of tools
created to implement a research framework that was developed to address a
specific research task. As [99] note, choosing between a dictionary, rules, or
supervised machine learning involves tradeoffs, and no clear guidelines. In
CRAML, particular steps and technologies can be interchanged according to the
user’s preference, skill set, and research challenge. A qualitative researcher
can quickly sample and sift through text data and experiment with alternative
classification schemes and definitions.888Emerging paradigms for low-resource
Natural Language Processing (NLP) seek to eliminate barriers to accessing ML
technology [71]. Particularly with low-resource NLP, the availability of
labeled data, i.e. text, become crucial. For another researcher, extraction
could be a pre-processing step before engaging in a computational grounded
theory project that uses unstructured NLP methods once the plausibly relevant
themes are extracted based upon keywords. A third researcher may wish to
extrapolate a full dataset without ML. A fourth researcher may wish to only
sample a massive corpus, extract context windows around keywords, and
inductively generate topics or themes with extracted text and have no desire
to perform further computational avenues of analysis.

###  3.2 A Bridge to Solving Real-World Problems

By positioning context rules as the crucial bridge between domain expert
knowledge and structured datasets that represent embodied knowledge, CRAML
provides a traceability from datasets classified by ML models back to the
context rules that lead to them – and the humans who wrote those. By placing a
focus on explainability at the first bridge from unstructured data to
structured data(sets), the second bridge from training data to trained models
becomes easier to cross. ML modeling “presumes a world of already existing
(divine or rationalistic) rules, which only need to be formalised, in order to
make sense to a machine (or an analytical philosopher for this reason)” [11].
Context rules in CRAML codify which text belongs to which topic or theme
according to the user. To the extent such rules can be articulated
consistently, then they can be extrapolated over unstructured text to create
fully structured datasets to train ML models.

Rather than presuming unbiased training data, CRAML extends the user’s
worldview to the realm of ML models via expert-created context rules.
Interpretive frameworks lend meaning to text and are a cornerstone of academic
research. Scaling methods such as coding and grounded theory over large
volumes of text contributes to theory development [117]. Rigorous manual
content analysis by qualitatitve social scientists is often geared toward the
induction of topics and themes, yielding new constructs, analytical
frameworks, or mid-level theory of some phenomenon [59].999For scientific
purposes, inductively derived analytical frameworks from text should be a)
intra-subjectively reliable, b) inter-subjectively reliable, and c) fully
reproducible [95]. This means that a single person should reach the same
coding conclusion again and again, a second person should reach the same
conclusion, and the methods, data, and analysis should be transparent enough
to permit replication. Alternatively, a quantitative or hypothesis-driven
researcher might pursue evidence of a phenomena within a text corpus, quantify
its frequency, and then analyze it.

In CRAML, the use of context rule sets serves as the foundation for
representing the knowledge of a domain expert, without the need for explicitly
annotating all unseen instances (i.e. documents). Through the extrapolation
process, the hybrid CRAML framework requires manual work only in the crucial
first stages, and leaves the remainder to automation. CRAML’s initial step is
similar to processes used in IS research: for example, to develop a
specialized ontology of hypotheses, [85] couple rule-based approaches for
extraction and word embeddings. The “human in the loop” contributes
transparent manual codifications of text, which allow for the replicable and
configurable construction of tabular datasets used either immediately for
quantitative social science analysis, or to train ML models.101010Explainable
and interpretable models can be built when users “steer learning” through
feedback and interaction [65]. [7] writes that explainability can only happen
through interaction between human and machine. Of course, the creation of such
hybrid system requires expert human labor. [105] writes that [i]nterpretable
models can entail significant effort to construct, in terms of both
computation and domain expertise. Because CRAML involves human expertise and
gives control over training data to the user, the ML model that results from
CRAML is interpretable as a product of a human-built and curated rule set that
can be published, and then contested, examined, and modified by others. This
mitigates concerns over black box ML models.

CRAML built models can have real-world applicability, because training data
and ML-built datasets can be formed by domain experts who largely address
Wagstaff’s (https://doi.org/10.48550/arxiv.1206.4656) concerns regarding
abstract metrics and lack of follow through. Empowering users to create novel,
meaningful datasets and ML models in their worldview also enables rapid
changes that can be put into action locally. This comes at a time where novel
ML or DL approaches can be severely limited by available datasets, with no
widely accepted approach to create novel ones as needed and in an efficient
manner.

As a final response to the challenges introduced in the previous section,
CRAML’s methodological transparency is intended to address several of the
problems with proprietary data and otherwise unreachable methods. With a
standard desktop computer and open source software, CRAML enables interactive
engagement with massive text corpora without prior programming knowledge
(although users will find some programming knowledge helpful and advanced
users may seek to customize the underlying code for niche purposes). At each
crucial step of the framework, CRAML saves and shows the user the underlying
data in CSV format in order to enable the interactivity essential to truly
achieving a human in the loop and records each step of the process for
replication and transparency purposes.

By making the framework as open and transparent as possible, and publishing
the code base and the underlying data used here, we provide opportunities for
users to test, harness, or modify the framework. CRAML software is available
under a under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
International License. The artifacts created and used by CRAML in the
intermediate stages are extracts and samples of the plausibly relevant text
from a document and document corpus using keywords and context windows. A user
can control what information is extracted from the corpus, and thus remove
sensitive data. Thus, CRAML can mitigate some privacy protection and sensitive
information concerns that limit access to sensitive data. Other researchers
working with big data have taken a similar approach by extracting only
relevant pieces of larger documents [113]. By

###  3.3 An Application of CRAML: Job Advertisement Text

Job seekers face difficulties finding the right jobs, and employers face
frictions in job search that represent additional recruitment costs. Search
frictions harm both workers and employers by increasing the barriers to good
matches in the labor market, and reducing competition for labor [29, 89]. A
suite of ML classifiers that added niche filters to job search engines could
benefit jobseekers, employers, and workforce agencies. With support from the
National Science Foundation, the National Labor Exchange created the NLx
Research Hub in 2021 to “increase the amount of actionable labor market
information in the U.S. to facilitate the recruitment, hiring, and training
opportunities of American workers” and “deepen partnerships between industry,
government, and academia by enhancing the infrastructure to support the
convergence of research, education, and talent pipelines.”

Workers who desire more flexible, remote jobs after COVID-19, disabled workers
seeking specific working conditions, union workers, workers with a
professional license, and veterans as well as veteran’s spouses may all be
underserved even while being preferred by specific employers. An employer can
gain a competitive advantage by hiring workers who are otherwise discriminated
against, but only in an efficient and competitive labor market [19]. Job
seekers may easily become discouraged without information on whether, for
example: a job is located near public transit, or whether an employer will
interview an ex-felon, a teen, or a non-resident on a visa. If some employers
seek to target and hire underserved audiences, but job ads never reach the
workers, then research should aim to reliably identify which jobs might be
especially relevant for a particular worker.

##### Research application

CRAML was first developed using job advertisement text data to support
research on the labor market. While the underlying text is confidential and
licensed, we describe and release rules files and nine ML classifiers that
achieved a high accuracy in detecting job characteristics and barriers to
employment under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
International License. The research implications of this work are not only
immediate research papers, but a potential to release custom-built ML
classifiers and create open tools for real-time information systems that track
changes in the labor market. Such application could improve both research and
contribute to reducing barriers to employment in job advertisements.
Ultimately, better data on employment practices obtained from job
advertisement text and other sources of text information could have a profound
impact on research capabilities in this field.111111In ongoing work supported
by the Russell Sage Foundation Future of Work Program and NLX, a job-
advertisement level SQL database built from CRAML’s classifiers using NLX data
is now operational and new classifiers are under construction.

In Appendix LABEL:sec:appendixb, we provide a list of tags and the keywords
extracted as part of an effort to identify specific barriers to employment,
the accuracy of the models built, and discuss how the broader labor market
research community can contribute to this effort.

##  4 A Step-by-Step, Replicable No Poach Classifier

We demonstrate the start-to-finish CRAML approach by applying it to an
empirical context where the resulting analysis might be meaningful,
transparent, and replicable. Toward this end, we contribute a new text corpus
of 151,708 franchise documents, and the cleaned and pre-processed text we used
in this analysis. We publish metadata that tracks which of 12,992 records a
particular document corresponds to, and includes the effective date, company
name, and unique ID. We also publish the rule sets that are the manual hand-
coded input, enabling replication and scrutiny of the results. We also publish
the training dataset and the ML model.121212The PDFs are hosted in partnership
with DocumentCloud, and will be released upon publication. On DocumentCloud,
where they can now be searched, discussed, and annotated. Additional items
will be uploaded to a scholarly repository upon publication.

###  4.1 Empirical and Data Context

“No poach” clauses in contracts are also referred to as collusive pacts or
anti-competitive restraints on employee mobility, and have drawn interest from
academics, regulators, and policy-makers. While anti-competitive language is
contained in public documents, sometimes contrary to public policy, these
documents are inaccessible for many: available only in massive, unsearchable
collections of PDF documents on state agency websites. With subtle variations
in language and no guide about where to look for these clauses, no poach
clauses are diamonds in the rough: a few sentences in hundreds of thousands of
documents with millions of pages. Indeed, such clauses were largely not known
to exist by the relevant academic and policy community of interest until the
release of a 2017 working paper published as [77]. The paper contained a
limited sample and relied on a third party data provider to identify no poach
clauses. Following the release of the working paper, eleven state attorneys
general issued a letter in 2018 demanding the practice end, many franchises
voluntarily ended the practice, and the State of Washington in 2018 first
negotiated settlements with franchise companies in which many removed their no
poach clauses.131313See Washington State Attorney General Report, Letter from
State Attorneys General. As Ashenfelter wrote, “it is instructive that the
mere revelation of collusive agreements, whether legal or not, has so quickly
provoked a strong response from both the antitrust authorities and the
franchisors whose agreements contained these no-poach clauses” (qtd. in [77]).

Both no poach and non-compete clauses are known to place downward pressure on
wages [31, 17]. Given their importance in reducing opportunities for job
mobility, there is great interest in descriptive statistics regarding the
prevalence of these clauses. Overcoming the inaccessibility of the text within
public documents was a significant challenge. We assembled a large collection
of Franchise Disclosure Documents (FDDs) from the state of California by
scraping PDF documents from the public web. These records exclude companies
that may be exempt from filing, and may be missing observations due to
limitations when accessing data from the web.141414While we were able to
obtain and match the 151,708 documents to the 12,992 records via web scraping,
we identified 16,216 franchise disclosure records from January 2013-July 2022,
and are investigating additional methods to acquire any missing data, which we
believe to be randomly distributed.

In addition, we make available the machine-readable text corpus we build from
the downloaded PDFs and the metadata that ties documents to specific company
filings. For pre-processing, we used Python’s Tika, and for files that could
not be initially read with this, we used ABBYY Fine Reader. For the California
corpus, we assembled 151,708 documents in 6.99 GB of cleaned machine-readable
text that can be traced back to a total of 12,992 franchise records. We
consider the record to be the unique ID that the state assigns to multiple
documents from a franchise on a particular date. We trace each document back
to a record with metadata that includes the name of the franchise, and the
date of the filing.

###  4.2 The Construction of Training Data

####  4.2.1 Step 1: Extract Relevant Text to Reduce Information Overload

Figure 3: First process in the CRAML Framework – Extract Data

The extraction process encompasses taking the unstructured text files and
preparing a cleaned dataset of only the text that is plausibly relevant that
is ready for hand-coding. By reducing the size of the text files being worked
with, and delimiting the extracts to the relevant text in the corpus, this
step addresses challenges of computational resource intensity, and assists in
sifting for relevant information in a large corpus. This is a manual,
iterative process – a loop that is exited when the user is satisfied that all
of the relevant text is extracted from the corpus. Figure 3 illustrates the
extraction process, which corresponds to Step 1 outlined in Figure 2. Two main
software tools (extract.py and text_ex.py) aid the user in finding and
exploring the relevant text.

We chose initial keywords hire, recruit, employ, and solicit to explore if
franchise documents contain a no poach clause. The initial keywords chosen
suggested additional ideas for tags, which in turn, suggested new ideas for
keywords. To aid in this process, a repeated N-gram exploration helped
discover the contexts in which given keywords appear the most. This
exploration prioritizes the extract, suggest new ideas for keywords, tags,
rules, context windows. Because the full context was important to explore
initially, the earliest exploratory extracts focused on the 20 words
surrounding each keyword. Following all of the processes described below, a
final context window of 13 words was selected (6 words to the left and right
of the keyword). The goal when removing irrelevant or non-essential
information is to reduce noise and build a highly focused ML classification
model. In an iterative and flexible process, more keywords were subsequently
added: for example, “poach”, “non compet”, “noncompet”, “covenant” and “not to
compete” simply restarted the process.

####  4.2.2 Step 2: From Rule Sets to (Training) Data via Extrapolation,
Testing, and Validation

After the extraction process, an expert conceptualizes and defines the desired
classifications, or tags, and builds a rule set for each tag in a manual and
iterative fashion. Figure 4 details the iterative process of Step 2 in Figure
2 that involves validation, extrapolation, and the refinement of rule sets.
This process concludes when the expert is satisfied that the extrapolated
rules are valid on a sample of the text. The rule set is then “extrapolated”
to the full extract from the corpus, yielding a structured dataset of
accurately classified chunks – a training dataset, in other words. As
described in the following section, the results of Step 2 can be used
immediately for descriptive research regarding the contents of documents, or
as a basis for training ML models.

The cycle begins with the initial creation of rule sets for each tag, which
can be done manually by an individual researcher, or team of researchers.
Constructing and designing accurate indicators requires researcher familiarity
with ontological subtleties and appropriate construct validation and scale
development methodology [126, 88]. Rules are written in a CSV file that
contains a list of rules, the priority level of each rule, and the
corresponding tag, assigned a 0 or 1. As earlier, the N-Gram tool may be
helpful – doing this prioritizes the creation of rules that will cover the
largest parts of the entire data. Priority level determines the order in which
a rule is run. A earlier rule can be over-written by a subsequent priority
rule.

Step 2 then involves the use of rule sets and the extrapolation algorithm
(extrapolate.py) to validate the rules. In essence, the extrapolation process
will convert the set of rules into an extended dataset, and will tag each
chunk according to the user’s encoding of each rule. The resulting dataset
contains the unique document identifier, the extracted chunk matching a
certain rule, and the defined encoding for this chunk. Note that this
extrapolation is done individually for each rule set and tag, thus resulting
in a training dataset for each rule set (file). A second software tool
(validate.py) relates to the validation of the data that is the output of
extrapolation. Once extrapolated, the expert tests the performance of the
rules against actual chunks that are coded by those rules. The expert can
export a strategic sample of chunks meant to ensure each rule is valid –
choosing a random sample of N examples per rule. This enables independent
hand-coders to score each chunk, which can then be re-imported to assess
accuracy and performance of the rules.

Figure 4: Second process in the CRAML Framework – Build Rules

For the franchise document data, allowing iterative exploration of keywords
and adjustments to rules, we developed a set of over 500 rules and 6 tags to
classify suspect anti-competitive clauses in the corpus. We first focused on
finding all employee “no poach” clauses. As one rule for the no poach tag
states: “you may not seek to employ or retain any employee or independent
contractor who is at any time employed by us.” When examining similar
extracted chunks, we found that many documents also contained language that
went further than prohibiting an employer from soliciting another party’s
employee, but also prohibited an employer from hiring or employing. We create
a separate tag these “no hire” clauses. Such “no hire” clauses, as in the
following example from the no hire rule set, state that: “party shall not hire
or solicit to hire any person employed then or within the preceding year by
the other party.”

Independent validation in which a third researcher hand-coded 706 chunks
indicates an initial 91% match between an independent third party and the
extrapolation of rules, suggesting a high degree of inter-rater reliability in
ability to detect characteristics of no poach clauses. While our emphasis here
is to ensure that there is strong inter-rater reliability between the CRAML
user and an independent observer in this test example, we do not claim perfect
identification of all no poach clauses. For a regulator, additional scrutiny
would be required, but the overarching goal is to identify plausibly relevant
and/or problematic no poach language. The pipeline enables any observer to
classify the documents according to any arbitrary schema. We emphasize that
CRAML here enables content validation processes at the input stage to ML.
Independent assessments of reliability are key to any conclusions.

##### Step 2 Results: A Labeled, Structured Dataset for Analysis

Step 2 yields a dataset for training a ML classification model, or, if the
underlying corpus is small enough (as is the case here), the rules can be
applied to the entire dataset and reveal which filings contain which clauses
according to the specified rules.

####  4.2.3 Step 3: Training ML Classifiers to Build Structured Datasets

While the above analysis is based on an extrapolation of rules to all chunks
in the dataset, CRAML yields data that a ML classifier can train. Although
many algorithms exists for such a training process, the CRAML framework trains
a binary classifier for each tag. In other words, each training dataset is
created from a rules file, which contains one or more tags. Accordingly, each
classifier can be traced back to a rule set to perform a 0/1 classification
for the tags included therein. The training process is described in greater
detail in Appendix LABEL:sec:appendixc.

Once the classifier(s) are trained, they can be deployed to classify the
original data, i.e. the entire unstructured text corpus, or can be used on
other sources of data. This represents the completion of the CRAML framework,
as one can now build a structured, labeled dataset from the unstructured text
of a document corpus using a ML model. This entire final process is
illustrated in Figure 5.

Figure 5: Third process in the CRAML Framework – Train Classifiers

###  4.3 Evaluating the Model

In this empirical context, we train a no poach classifier built with training
data from California. We use the classifier to build datasets for both
California, in order to obtain a comparison between the rules-based approach
and the ML approach using the same data. The “no poach” classifier is built
with training data from a 10% sample plus all of the no poach tag=1
observations in the full sample. In some cases, when the occurrence of
positive or negative tags is very low, manual augmentations to training data
such as this are required in order to receive acceptable model performance. We
obtain an accuracy score of 0.99, precision of 0.97, recall of 0.96, and an
overall F1-Score of 0.97. The harmonic mean of precision and recall (or
F1-score) is based upon a comparison between the ML output and the training
dataset. A common statistic in ML, with a maximum score of 1 when precision
and recall are perfect, and 0 when either precision or recall is zero, the F1
score suggests a high level of accuracy and recall between the training
dataset and results.

Figure 6 displays the percent of all rule-detected and ML-detected no poach
clauses in the California Corpus within each year for each record filed from
years 2013-2022 (with partial data for 2022). The pattern depicted with the
California data is similar to the rule-based analysis. It shows that from
2015-2017 over 60% of the records contained no poach clauses, after which
there was a decline until the percent stabilizes below 40%. This suggests that
the interventions that followed [77] had a large effect to decrease the
prevalence of anti-competitive no poach language in franchise documents. The
continued prevalence of these clauses, despite interventions from authorities,
requires further investigation (in another paper): while the no poach clauses
continue to exist, inspection of the text chunks reveals that while some
remain the same as before the intervention of the Washington State Attorney
General, while others now limit their applicability to more favorable
jurisdictions and others restrict their application to highly compensated
employees.

Figure 6: Analysis of Suspect No Poach Clauses in California Franchise
Documents

##### Practical Note on Model Accuracy

One noteworthy aspect of CRAML is that the above statistics reflect
performance of the model at the “chunk” level. If an expert were looking to
confirm that certain documents contain no poach clauses, the record is the
level of analysis where they would want to begin their search. Given over
150,000 documents stored in nearly 13,000 records, this could be a daunting
task. Moving to the level of the records, in California, the no poach
classifier closely matches the rule-generated results, with only 22 false
negatives and 672 false positives in 12,922 records. Thus, at the record
level, ML model achieves a 0.95 F1-score, 0.996 recall, 0.91 precision, and
accuracy of 0.946.

Further, an inspection of text chunks reveals that a significant portion of
the false positives are in fact related to other language that accompanies no
poach clauses and belong properly in the context of anti-competitive language
in these documents: jurisdictional restrictions on no poach clauses, language
that applies no poach clauses only to specific employees, and language that
creates non-compete clauses. Precision at the level of the record would be
0.04 higher (0.95) if one considers these clauses to be relevant to a search
for a broader construct of anti-competitive language. The results of CRAML are
fully traceable to choices made in the construction of a rule set for the no
poach classifier. As seen in Figure 6, the ML classifier identifies more
suspected no poach clauses then the rules, and this increases in 2021.
Additional research using this corpus is pinpointing fine-grained
characteristics of language in these documents that restrict employee
mobility.

A final note: if one were dissatisfied with the model performance and sought
one with higher recall or lower accuracy, or a more or less specific
definition of the no poach construct, it would possible to “steer learning”
and achieve a desired result by changing rule sets and thus re-shaping the
training data. If desired, a user could add or remove all jurisdiction,
narrow, and non-compete language from the no poach rule set in order to
achieve a more discriminating classifier. A user could also combine all of the
rules to make a single classifier that memorizes a larger pattern related to
all types of anti-competitive language that appear in the documents.

##  5 Discussion

The CRAML framework makes several contributions. First, the framework enables
the systematic and structured process of creating novel datasets from
unstructured text corpora. Second, the result harmonizes advanced, automated
information extraction and ML techniques with the input and expertise from
manual analysis performed by supervising researchers. The exploration and
iterative process performed during the context rule creation stage involves
work that is not only difficult to perform automatically, but also that will
arguably never come close to matching the diversity and expertise of human
experts with domain knowledge. As a result, this intermediate stage in the
framework incorporates a human element that is lacking in modern
classification frameworks.

###  5.1 Future Directions

####  5.1.1 Technical Improvements

Future research directions regarding the technical backbone of the CRAML
framework should aim to bolster the extrapolation stage of the pipeline.
Embeddings can be utilized to improve the transformation of context rules to
datasets. With embeddings added to the extrapolation method, the need for
training a ML classifier in the end may become obsolete.

A second area of future work involves building up tools to aid the domain
expert in the initial, manual-driven stages of the CRAML process. Keyword
extraction, topic modeling, automatic rule induction, and knowledge graphs
could all be useful tools to support the domain expert express his or her
worldview, as well as explore large text corpora in a richer way.

Although the focus is currently placed on ML techniques, the role of more
advanced classifiers remains an open area of investigation. Deep Learning, in
particular sequential models, could prove to be powerful in boosting the
predictive capability of models trained on CRAML-generated datasets. Of
course, this would come with the added overhead of extra training and
parameter tuning, which must also be taken into account for future endeavors.

####  5.1.2 Computational Social Science

CRAML is designed with flexibility and web-based sources of text in mind. A
plug-in was created to integrate DocumentCloud (used by journalists and
academics to post government documents) for the project related to no poach
agreements. New plug-ins could draw data from web-based sources of text such
as Twitter. Niche ML classifiers built from user-data in a specific domain
could address content moderation challenges at scale [57, 62]. To address a
concern in [101], government, civil society, and non-governmental
organizations could develop context rule sets to capture text that is relevant
to their interests and publish or license ML classifiers that provide useful
feeds and classifications of information tracking on topics of societal
concern. For academics, editors and reviewers could demand greater
transparency when claims are made about data built form text.

Concerns with proprietary data sources and privacy can be partially addressed
by the information extraction methods used here. While additional efforts are
needed, including the potential development of further tools for differential
privacy with text, the extraction process is a step toward efforts to protect
individuals against identification and protect the data owner from violation
of proprietary information and trade secrets. For example, researchers have
been barred from accessing records due to privacy and confidentiality
concerns, but with careful selection of keywords, analysts could code chunks
of extracted text and classify every document in a corpus without ever
accessing the full or confidential information.151515For example, researchers
could use CRAML software to track the 20th Century use of residential
restrictive covenants that barred Black, Jewish, Asian and other ethnic groups
from living in certain neighborhoods. County commissioners of deeds have
either barred researchers or severely limited access to digitized records due
to concerns. For one project focused on Hennepin County, see
https://mappingprejudice.umn.edu/.

Finally, researchers in the humanities and social scientists can hopefully
develop diverse and practical applications that uncover constructs and
patterns currently “hidden” in unstructured text. The CRAML framework and
software tools can be used to analyze text in any language. To the extent a
construct is intra-subjectively reliable, it could be developed by a
researcher into rules and scaled over vast amounts of unstructured text. To
the extent that inter-subjective variance is high, we hope that by decreasing
the time and effort needed to scale a framework, there will be many
opportunities for diverse voices to produce contesting interpretative schemes
that are transparent and replicable.

## Acknowledgements

We are grateful for support from the Economic Security Project Anti-Monopoly
Fund; Loyola Rule of Law Institute; Loyola Quinlan School of Business; and
Loyola University Chicago. All errors are the authors’.

## Conflict of interest

The authors have no conflicts of interest.

## Supporting Information

See our Github repository for the full code base:
https://github.com/sjmeis/CRAML_Beta.

## References

  * [1] A. Abbas, Y. Zhou, S. Deng and P. Zhang  “Text analytics to support sense-making in social media: A language-action perspective” Publication Title: MIS Quarterly  par.nsf.gov, 2018  URL: https://par.nsf.gov/servlets/purl/10064455
  * [2] A. Abbasi and H. Chen  “CyberGate: a design framework and system for text analysis of computer-mediated communication” Publisher: JSTOR  In _Mis Quarterly_ , 2008  URL: https://www.jstor.org/stable/25148873
  * [3] A. Abbasi, Z. Zhang, D. Zimbra and H. Chen  “Detecting fake websites: The contribution of statistical learning theory” Publisher: JSTOR  In _Mis Quarterly_ , 2010  URL: https://www.jstor.org/stable/25750686
  * [4] A. Abbasi, C. Albrecht, A. Vance and J. Hansen  “Metafraud: a meta-learning framework for detecting financial fraud” Publisher: JSTOR  In _Mis Quarterly_ , 2012  URL: https://www.jstor.org/stable/41703508
  * [5] Andrew Abbott  “Chaos of Disciplines” 01071  University of Chicago Press, 2001
  * [6] Daron Acemoglu and David H. Autor  “Skills, Tasks and Technologies: Implications for Employment and Earnings”  In _Handbook of Labor Economics_ Volume 4, Part B Elsevier, 2011, pp. 1043–1171  URL: http://www.sciencedirect.com/science/article/pii/S0169721811024105
  * [7] Amina Adadi and Mohammed Berrada  “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)”  In _IEEE Access_ 6 Institute of ElectricalElectronics Engineers (IEEE), 2018, pp. 52138–52160  DOI: 10.1109/access.2018.2870052
  * [8] Kiran Adnan and Rehan Akbar  “An analytical study of information extraction from unstructured and multidimensional big data”  In _Journal of Big Data_ 6.1, 2019, pp. 91  DOI: 10.1186/s40537-019-0254-8
  * [9] Kiran Adnan and Rehan Akbar  “Limitations of information extraction methods and techniques for heterogeneous unstructured big data” Publisher: SAGE Publications Ltd STM  In _International Journal of Engineering Business Management_ 11, 2019, pp. 1847979019890771  DOI: 10.1177/1847979019890771
  * [10] Charu C Aggarwal and ChengXiang Zhai  “A survey of text clustering algorithms”  In _Mining text data_ Springer, 2012, pp. 77–128
  * [11] Clemens Apprich  “Secret Agents”  In _Digital Culture & Society_ 4.1 Transcript Verlag, 2018, pp. 29–44  DOI: 10.14361/dcs-2018-0104
  * [12] Alejandro Barredo Arrieta et al.  “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI”  arXiv, 2019  DOI: 10.48550/ARXIV.1910.10045
  * [13] A. Asatiani, P. Malo, P.. Nagbøl and E. Penttinen  “Challenges of explaining the behavior of black-box AI systems” Publication Title: MIS Quarterly …  pure.itu.dk, 2020  URL: https://pure.itu.dk/portal/files/85687226/Challenges_of_Explaining_the_Behavior_of_Black_Box_AI_Systems.pdf
  * [14] David Autor  “The "Task Approach" to Labor Markets: An Overview”, 2013  URL: http://econpapers.repec.org/paper/izaizadps/dp7178.htm
  * [15] Boris Babic, I. Cohen, Theodoros Evgeniou and Sara Gerke  “When Machine Learning Goes Off the Rails”  In _Harvard Business Review_ , 2020  URL: https://hbr.org/2021/01/when-machine-learning-goes-off-the-rails
  * [16] E. Bachura, R. Valecha, R. Chen and H.. Rao  “THE OPM DATA BREACH: AN INVESTIGATION OF SHARED EMOTIONAL REACTIONS ON TWITTER.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2022  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=157194645&h=PwMopjORi%2FKLehPfc55mRiFwJERVEK2%2FxgkMVatAZ1XaULaUmen9kLXAGVFVc%2FsbtGP%2B6HiwZIk6WpAGvFa89Q%3D%3D&crl=c
  * [17] Natarajan Balasubramanian et al.  “Locked In? The Enforceability of Covenants Not to Compete and the Careers of High-Tech Workers” Publisher: University of Wisconsin Press  In _Journal of Human Resources_ 57.S, 2022, pp. S349–S396  DOI: 10.3368/jhr.monopsony.1218-9931R1
  * [18] S. Bapna, M.. Benner and L. Qiu  “Nurturing Online Communities: An Empirical Investigation.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2019  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=136245817&h=%2FCdEQ%2BgtTYGyegYaDLekqWS5d0VTXKpdjqSr%2BLlBuSyfxz3LpWhsLr8bHR8eMWffHiSdONg28JMk4lSgQYvsiw%3D%3D&crl=c
  * [19] Gary Becker  “The Economics of Discrimination”  Chicago, IL: The University of Chicago Press, 1957
  * [20] S.. Begum and F. Nausheen  “A comparative analysis of differential privacy vs other privacy mechanisms for Big Data”  In _2018 2nd International Conference on Inventive Systems and Control (ICISC)_ , 2018, pp. 512–516  DOI: 10.1109/ICISC.2018.8399125
  * [21] H. Benbya, T.. Davenport and S. Pachidi  “Artificial intelligence in organizations: current state and future opportunities” Publisher: papers.ssrn.com  In _MIS Quarterly Executive_ , 2020  URL: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3741983
  * [22] James Beniger  “The control revolution: Technological and economic origins of the information society”  Harvard university press, 2009
  * [23] Richard Biernacki  “Inside the Rituals of Social Science”  In _Reinventing Evidence in Social Inquiry: Decoding Facts and Variables_ , Cultural Sociology  New York: Palgrave Macmillan US, 2012, pp. 1–26  DOI: 10.1057/9781137007285_1
  * [24] Ann M. Blair  “Too Much to Know: Managing Scholarly Information before the Modern Age” Publication Title: Too Much to Know  Yale University Press, 2010  DOI: 10.12987/9780300168495
  * [25] Nicholas Bloom and John Van Reenen  “Measuring and Explaining Management Practices Across Firms and Countries” 00990  In _The Quarterly Journal of Economics_ 122.4, 2007, pp. 1351–1408  DOI: 10.1162/qjec.2007.122.4.1351
  * [26] Nicholas Bloom et al.  “The New Empirical Economics of Management” 00000, 2014  URL: http://www.nber.org/papers/w20102
  * [27] Valerie C. Bradley et al.  “Unrepresentative big surveys significantly overestimated US vaccine uptake” Number: 7890 Publisher: Nature Publishing Group  In _Nature_ 600.7890, 2021, pp. 695–700  DOI: 10.1038/s41586-021-04198-4
  * [28] E. den Broek, A. Sergeeva and M. Huysman  “When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2021  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=152360587&h=vkpGRMZvYJrdHNg7A%2Fy3jWu5rozlDbhUik4nr0r%2Fvd%2F6vE0LoQ3g7o76O5EPx8mavR5S1h2m9N7yv3%2BfRX7AAg%3D%3D&crl=c
  * [29] Kenneth Burdett and Dale T. Mortensen  “Wage Differentials, Employer Size, and Unemployment”  In _Intenational Economic Review_ 39.3, 1998, pp. 257–273
  * [30] Nadia Burkart and Marco F. Huber  “A Survey on the Explainability of Supervised Machine Learning”  In _Journal of Artificial Intelligence Research_ 70 AI Access Foundation, 2021, pp. 245–317  DOI: 10.1613/jair.1.12228
  * [31] Brian Callaci, Sergio Pinto, Marshall Steinbaum and Matthew Walsh  “The Effect of No-poaching Restrictions on Worker Earnings in Franchised Industries”  In _Available at SSRN_ , 2022
  * [32] David Card, Raj Chetty, Martin S. Feldstein and Emmanuel Saez  “Expanding Access to Administrative Data for Research in the United States”, 2010  DOI: 10.2139/ssrn.1888586
  * [33] Davide Castelvecchi  “Can we open the black box of AI?”  In _Nature_ 538.7623 Springer ScienceBusiness Media LLC, 2016, pp. 20–23  DOI: 10.1038/538020a
  * [34] Aaron K. Chatterji, Rodolphe Durand, David I. Levine and Samuel Touboul  “Do ratings of firms converge? Implications for managers, investors and strategy researchers” ISBN: 0143-2095 Publisher: Wiley Online Library  In _Strategic Management Journal_ 37.8, 2016, pp. 1597–1614
  * [35] M. Chau et al.  “Finding People with Emotional Distress in Online Social Media: A Design Combining Machine Learning and Rule-Based Classification.” Publication Title: MIS Quarterly  pweb.fbe.hku.hk, 2020  URL: https://pweb.fbe.hku.hk/~mchau/papers/MISQ2020.pdf
  * [36] Jaeho Choi, Anoop Menon and Haris Tabakovic  “Using machine learning to revisit the diversification–performance relationship” Number: 9 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/smj.3317  In _Strategic Management Journal_ 42.9, 2021, pp. 1632–1661  DOI: 10.1002/smj.3317
  * [37] Prithwiraj Choudhury, Ryan T. Allen and Michael G. Endres  “Machine learning for pattern discovery in management research” Number: 1 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/smj.3215  In _Strategic Management Journal_ 42.1, 2021, pp. 30–57  DOI: https://doi.org/10.1002/smj.3215
  * [38] Lisa E. Cohen and Sara Mahabadi  “In the midst of hiring: Pathways of anticipated and accidental job evolution during hiring” Number: 5 Publisher: INFORMS  In _Organization Science_ 33.5, 2022, pp. 1938–1963  DOI: 10.1287/orsc.2021.1516
  * [39] Open Science Collaboration  “Estimating the reproducibility of psychological science” 00091  In _Science_ 349.6251, 2015, pp. aac4716  DOI: 10.1126/science.aac4716
  * [40] Lorrie F Cranor  “A framework for reasoning about the human in the loop”  Advanced Computing Systems ProfessionalTechnical Association, 2008
  * [41] Andrew D. Cutler, Stephen W. Carden, Hannah L. Dorough and Nicholas S. Holtzman  “Inferring Grandiose Narcissism From Text: LIWC Versus Machine Learning” Publisher: SAGE Publications Inc  In _Journal of Language and Social Psychology_ 40.2, 2021, pp. 260–276  DOI: 10.1177/0261927X20936309
  * [42] Alexander D’Amour et al.  “Underspecification Presents Challenges for Credibility in Modern Machine Learning” arXiv: 2011.03395  In _arXiv:2011.03395 [cs, stat]_ , 2020  URL: http://arxiv.org/abs/2011.03395
  * [43] Kerstin Dautenhahn  “The art of designing socially intelligent agents: Science, fiction, and the human in the loop”  In _Applied artificial intelligence_ 12.7-8 Taylor & Francis, 1998, pp. 573–617  DOI: https://doi.org/10.1080/088395198117550
  * [44] Gerald F. Davis  “Editorial essay: What is organizational research for?” ISBN: 0001-8392 Publisher: Sage Publications Sage CA: Los Angeles, CA  In _Administrative Science Quarterly_ 60.2, 2015, pp. 179–188
  * [45] Paul DiMaggio  “Adapting computational text analysis to social science (and vice versa)” Publisher: SAGE Publications Ltd  In _Big Data & Society_ 2.2, 2015, pp. 2053951715602908  DOI: 10.1177/2053951715602908
  * [46] Vincent J. Duriau, Rhonda K. Reger and Michael D. Pfarrer  “A Content Analysis of the Content Analysis Literature in Organization Studies: Research Themes, Data Sources, and Methodological Refinements” Publisher: SAGE Publications Inc  In _Organizational Research Methods_ 10.1, 2007, pp. 5–34  DOI: 10.1177/1094428106289252
  * [47] empty  “20 Newsgroups Dataset”, 2008  URL: http://people.csail.mit.edu/jrennie/20Newsgroups/
  * [48] I. Faik, M. Barrett and E. Oborn  “How information technology matters in societal change: An affordance-based institutional logics perspective.” Publication Title: MIS quarterly  xwcbx.cn, 2020  URL: http://www.xwcbx.cn/zb_users/upload/2021/04/202104281619567936312948.pdf
  * [49] Ronen Feldman  “Techniques and applications for sentiment analysis”  In _Communications of the ACM_ 56.4, 2013, pp. 82–89  DOI: 10.1145/2436256.2436274
  * [50] Peer C. Fiss  “Building Better Causal Theories: A Fuzzy Set Approach to Typologies in Organization Research” Publisher: Academy of Management  In _Academy of Management Journal_ 54.2, 2011, pp. 393–420  DOI: 10.5465/amj.2011.60263120
  * [51] Samuel Fosso Wamba et al.  “How ‘big data’ can make big impact: Findings from a systematic review and a longitudinal case study”  In _International Journal of Production Economics_ 165, 2015, pp. 234–246  DOI: 10.1016/j.ijpe.2014.12.031
  * [52] Marion Fourcade  “The Problem of Embodiment in the Sociology of Knowledge: Afterword to the Special Issue on Knowledge in Practice”  In _Qualitative Sociology_ 33.4, 2010, pp. 569–574  DOI: 10.1007/s11133-010-9173-x
  * [53] Andreas Geiger, Philip Lenz and Raquel Urtasun  “Are we ready for autonomous driving? The KITTI vision benchmark suite” ISSN: 1063-6919  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 3354–3361  DOI: 10.1109/CVPR.2012.6248074
  * [54] Andreas Geiger, Philip Lenz and Raquel Urtasun  “Are we ready for autonomous driving? the kitti vision benchmark suite”  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2012_ , 2012, pp. 3354–3361  IEEE
  * [55] BARRY GERHART, PATRICK M. WRIGHT, GARY C. MC MAHAN and SCOTT A. SNELL  “MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?” _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1744-6570.2000.tb02418.x  In _Personnel Psychology_ 53.4, 2000, pp. 803–834  DOI: https://doi.org/10.1111/j.1744-6570.2000.tb02418.x
  * [56] H. Geva, G. Oestreicher-Singer and M. Saar-Tsechansky  “Using retweets when shaping our online persona: Topic modeling approach” Publication Title: MIS Quarterly  misq.umn.edu, 2019  URL: https://misq.umn.edu/skin/frontend/default/misq/pdf/appendices/2019/V43I2Appendices/07_14346_RA_GevaOestreicherAppendices.pdf
  * [57] Tarleton Gillespie  “Content moderation, AI, and the question of scale” Publisher: SAGE Publications Ltd  In _Big Data & Society_ 7.2, 2020, pp. 2053951720943234  DOI: 10.1177/2053951720943234
  * [58] Simona Giorgi, Massimo Maoret and Edward J.  “On the relationship between firms and their legal environment: The role of cultural consonance” Number: 4  In _Organization Science_ 30.4, 2019, pp. 803–830  DOI: 10.1287/orsc.2018.1250
  * [59] Barney G. Glaser and Anselm Strauss  “The Discovery of Grounded Theory: Strategies for Qualitative Research”  Aldine Transaction, 1967
  * [60] Gary Goertz and James Mahoney  “A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences”  Princeton University Press, 2012  DOI: 10.23943/princeton/9780691149707.001.0001
  * [61] Joseph K. Goodman, Cynthia E. Cryder and Amar Cheema  “Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples” 00473  In _Journal of Behavioral Decision Making_ 26.3, 2013, pp. 213–224  DOI: 10.1002/bdm.1753
  * [62] Robert Gorwa, Reuben Binns and Christian Katzenbach  “Algorithmic content moderation: Technical and political challenges in the automation of platform governance” Publisher: SAGE Publications Ltd  In _Big Data & Society_ 7.1, 2020, pp. 2053951719897945  DOI: 10.1177/2053951719897945
  * [63] Nikolitsa Grigoropoulou and Mario L. Small  “The data revolution in social science needs qualitative research”  In _Nature Human Behaviour_ , 2022  DOI: 10.1038/s41562-022-01333-7
  * [64] Gruss, Peter  “Open Access to Science and Culture” Publisher: American Association for the Advancement of Science  In _Science_ 303.5656, 2004, pp. 311–312  DOI: 10.1126/science.303.5656.311
  * [65] David Gunning et al.  “XAI—Explainable artificial intelligence”  In _Science Robotics_ 4.37, 2019, pp. eaay7120  DOI: 10.1126/scirobotics.aay7120
  * [66] Wei Guo, Metin Sengul and Tieying Yu  “The impact of executive verbal communication on the convergence of investors’ opinions” Number: 6 Publisher: Academy of Management  In _Academy of Management Journal_ 64.6, 2021, pp. 1763–1792  DOI: 10.5465/amj.2019.0711
  * [67] Khodor Hammoud et al.  “Personal information privacy: what’s next?”, 2019  CEUR Workshop Proceedings
  * [68] Eszter Hargittai  “Is Bigger Always Better? Potential Biases of Big Data Derived from Social Network Sites” Publisher: SAGE Publications Inc  In _The ANNALS of the American Academy of Political and Social Science_ 659.1, 2015, pp. 63–76  DOI: 10.1177/0002716215570866
  * [69] Will Douglas Heaven  “The way we train ai is fundamentally flawed”  In _MIT Technology Review_ MIT Technology Review, 2020  URL: https://www.technologyreview.com/2020/11/18/1012234/training-machine-learning-broken-real-world-heath-nlp-computer-vision/
  * [70] Michael A. Hedderich et al.  “A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios”  arXiv, 2020  DOI: 10.48550/ARXIV.2010.12309
  * [71] Michael A. Hedderich et al.  “A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios” arXiv:2010.12309 [cs] type: article, 2021  DOI: 10.48550/arXiv.2010.12309
  * [72] G.. Kane, A.. Young, A. Majchrzak and S. Ransbotham  “Avoiding an oppressive future of machine learning: A design theory for emancipatory assistants” Publication Title: MIS Quarterly  researchgate.net, 2021  URL: https://www.researchgate.net/profile/Amber-Young-7/publication/346955741_Avoiding_an_Oppressive_Future_of_Machine_Learning_A_Design_Theory_for_Emancipatory_Assistants_Forthcoming_in_MISQ_Special_Issue_on_Next-Generation_Information_Systems_Theories/links/5fd3c581299bf1408800b6b4/Avoiding-an-Oppressive-Future-of-Machine-Learning-A-Design-Theory-for-Emancipatory-Assistants-Forthcoming-in-MISQ-Special-Issue-on-Next-Generation-Information-Systems-Theories.pdf
  * [73] Sarah Kaplan and Keyvan Vakili  “The double-edged sword of recombination in breakthrough innovation” Number: 10  In _Strategic Management Journal_ 36.10, 2015, pp. 1435–1457  DOI: 10.1002/smj.2294
  * [74] Sayash Kapoor and Arvind Narayanan  “Leakage and the Reproducibility Crisis in ML-based Science”  arXiv, 2022  DOI: 10.48550/ARXIV.2207.07048
  * [75] Gary King  “Ensuring the Data-Rich Future of the Social Sciences” Publisher: American Association for the Advancement of Science  In _Science_ 331.6018, 2011, pp. 719–721  DOI: 10.1126/science.1197872
  * [76] Gary King, Robert O. Keohane and Sidney Verba  “Designing Social Inquiry: Scientific Inference in Qualitative Research” Publication Title: Designing Social Inquiry  Princeton University Press, 1994  DOI: 10.1515/9781400821211
  * [77] Alan B. Krueger and Orley Ashenfelter  “Theory and Evidence on Employer Collusion in the Franchise Sector” Publisher: University of Wisconsin Press  In _Journal of Human Resources_ , 2022, pp. 1019  DOI: 10.3368/jhr.monopsony.1019-10483
  * [78] M. Lacity and L. Willcocks  “Becoming strategic with intelligent automation” Publisher: aisel.aisnet.org  In _MIS Quarterly Executive_ , 2021  URL: https://aisel.aisnet.org/misqe/misqe_forthcoming_2021.pdf
  * [79] Kai R. Larsen and Chih How Bong  “A tool for addressing construct identity in literature reviews and meta-analyses” ISBN: 0276-7783 Publisher: JSTOR  In _Mis Quarterly_ 40.3, 2016, pp. 529–552
  * [80] David Lazer and Jason Radford  “Data ex Machina: Introduction to Big Data” _eprint: https://doi.org/10.1146/annurev-soc-060116-053457  In _Annual Review of Sociology_ 43.1, 2017, pp. 19–39  DOI: 10.1146/annurev-soc-060116-053457
  * [81] David Lazer et al.  “Computational Social Science” Publisher: American Association for the Advancement of Science  In _Science_ 323.5915, 2009, pp. 721–723  DOI: 10.1126/science.1167742
  * [82] David Lazer et al.  “Computational social science: Obstacles and opportunities” Publisher: American Association for the Advancement of Science  In _Science_ 369.6507, 2020, pp. 1060–1062  DOI: 10.1126/science.aaz8170
  * [83] John Lees-Miller, John Hammersley and R Wilson  “Theoretical maximum capacity as benchmark for empty vehicle redistribution in personal rapid transit”  In _Transportation Research Record: Journal of the Transportation Research Board_ Transportation Research Board of the National Academies, 2010, pp. 76–83
  * [84] H. Li, C. Zhang and W.. Kettinger  “DIGITAL PLATFORM ECOSYSTEM DYNAMICS: THE ROLES OF PRODUCT SCOPE, INNOVATION, AND COLLABORATIVE NETWORK CENTRALITY.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2022  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=157194642&h=UaV1oyxahBUuaam%2Bd5URs75k9QR%2Fv5el5zVHwjifuixanPd3E17h9QaT%2BQk1i8MT525Bo6dclOPWoJoZu6TgcA%3D%3D&crl=c
  * [85] J. Li, K. Larsen and A. Abbasi  “TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning” Publication Title: MIS Quarterly  par.nsf.gov, 2020  URL: https://par.nsf.gov/servlets/purl/10282376
  * [86] A. Liu, Y. Li and S.. Xu  “Assessing the Unacquainted: Inferred Reviewer Personality and Review Helpfulness.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2021  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=152360573&h=BYQ0BbcFncLJKMek1moinCCUGUPML8jWbsKeWpcd39YbEooW12Czzu4YcO%2B4DZRuIgh5fZ1KL3LsP99bJRZFjw%3D%3D&crl=c
  * [87] Andrew L. Maas et al.  “Learning Word Vectors for Sentiment Analysis”  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ Portland, Oregon, USA: Association for Computational Linguistics, 2011, pp. 142–150  URL: http://www.aclweb.org/anthology/P11-1015
  * [88] S.. MacKenzie, P.. Podsakoff and N.. Podsakoff  “Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques” Publisher: JSTOR  In _MIS quarterly_ , 2011  URL: https://www.jstor.org/stable/23044045
  * [89] Alan Manning  “Monopsony in Motion: Imperfect Competition in Labor Markets”  Princeton University Press, 2003
  * [90] Aaron F. McKenny, Jeremy C. Short and G. Payne  “Using Computer-Aided Text Analysis to Elevate Constructs: An Illustration Using Psychological Capital” Publisher: SAGE Publications Inc  In _Organizational Research Methods_ 16.1, 2013, pp. 152–184  DOI: 10.1177/1094428112459910
  * [91] John B. Miner  “Organizational Behavior 2: Essential theories of process and structure” Google-Books-ID: qqZ_FDFoDcMC  M.E. Sharpe, 2006
  * [92] S.. Miranda, D.. Wang and C.. Tian  “DISCURSIVE FIELDS AND THE DIVERSITY-COHERENCE PARADOX: AN ECOLOGICAL PERSPECTIVE ON THE BLOCKCHAIN COMMUNITY DISCOURSE.” Publication Title: MIS Quarterly  researchgate.net, 2022  URL: https://www.researchgate.net/profile/Dawei-Wang-3/publication/363261727_Discursive_Fields_and_the_Diversity-Coherence_Paradox_An_Ecological_Perspective_on_the_Blockchain_Community_Discourse/links/6313fb355eed5e4bd1457cda/Discursive-Fields-and-the-Diversity-Coherence-Paradox-An-Ecological-Perspective-on-the-Blockchain-Community-Discourse.pdf
  * [93] Kevin Munger  “The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity” Publisher: SAGE Publications Ltd  In _Social Media + Society_ 5.3, 2019, pp. 2056305119859294  DOI: 10.1177/2056305119859294
  * [94] Kevin Munger, Andrew M. Guess and Eszter Hargittai  “Quantitative Description of Digital Media: A Modest Proposal to Disrupt Academic Publishing”  In _Journal of Quantitative Description: Digital Media_ 1, 2021  DOI: 10.51685/jqd.2021.000
  * [95] Laura K. Nelson  “Computational Grounded Theory: A Methodological Framework” Publisher: SAGE Publications Inc  In _Sociological Methods & Research_ 49.1, 2020, pp. 3–42  DOI: 10.1177/0049124117729703
  * [96] Laura K. Nelson  “Leveraging the alignment between machine learning and intersectionality: Using word embeddings to measure intersectional experiences of the nineteenth century U.S. South”  In _Poetics_ 88, Measure Mohr Culture, 2021, pp. 101539  DOI: 10.1016/j.poetic.2021.101539
  * [97] Laura K. Nelson  “To Measure Meaning in Big Data, Don’t Give Me a Map, Give Me Transparency and Reproducibility” Publisher: SAGE Publications Inc  In _Sociological Methodology_ 49.1, 2019, pp. 139–143  DOI: 10.1177/0081175019863783
  * [98] Nosek B. A. et al.  “Promoting an open research culture” Publisher: American Association for the Advancement of Science  In _Science_ 348.6242, 2015, pp. 1422–1425  DOI: 10.1126/science.aab2374
  * [99] Sheela Pandey and Sanjay K. Pandey  “Applying Natural Language Processing Capabilities in Computerized Textual Analysis to Measure Organizational Culture” Publisher: SAGE Publications Inc  In _Organizational Research Methods_ 22.3, 2019, pp. 765–797  DOI: 10.1177/1094428117745648
  * [100] Juergen Pfeffer et al.  “This Sample seems to be good enough! Assessing Coverage and Temporal Reliability of Twitter’s Academic API” arXiv: 2204.02290  In _arXiv:2204.02290 [cs]_ , 2022  URL: http://arxiv.org/abs/2204.02290
  * [101] Jake Porway  “Funding Data and AI That Serve the Social Sector” Publisher: Stanford Social Innovation Review, 2022  DOI: 10.48558/YKY9-RX35
  * [102] Eric A. Posner  “How Antitrust Failed Workers”  Oxford, New York: Oxford University Press, 2021
  * [103] Charles C. Ragin  “Fuzzy-Set Social Science”  University of Chicago Press, 2000
  * [104] Joan Robinson  “The Economics of Imperfect Competition”  London, UK: MacmillanCo. ltd., 1933
  * [105] Cynthia Rudin  “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead”  arXiv, 2018  DOI: 10.48550/ARXIV.1811.10154
  * [106] Matthew J. Salganik et al.  “Measuring the predictability of life outcomes with a scientific mass collaboration” Publisher: Proceedings of the National Academy of Sciences  In _Proceedings of the National Academy of Sciences_ 117.15, 2020, pp. 8398–8403  DOI: 10.1073/pnas.1915006117
  * [107] Gunar Schirner, Deniz Erdogmus, Kaushik Chowdhury and Taskin Padir  “The future of human-in-the-loop cyber-physical systems”  In _Computer_ 46.1 IEEE, 2013, pp. 36–45
  * [108] Jeremy C. Short, Aaron F. McKenny and Shane W. Reid  “More than words? Computer-aided text analysis in organizational behavior and psychology research” ISBN: 2327-0608 Publisher: Annual Reviews  In _Annual Review of Organizational Psychology and Organizational Behavior_ 5, 2018, pp. 415–435
  * [109] Jeremy C. Short and Timothy B. Palmer  “The Application of DICTION to Content Analysis Research in Strategic Management” Publisher: SAGE Publications Inc  In _Organizational Research Methods_ 11.4, 2008, pp. 727–752  DOI: 10.1177/1094428107304534
  * [110] Jeremy C. Short, J. Broberg, Claudia C. Cogliser and Keith H. Brigham  “Construct validation using computer-aided text analysis (CATA) an illustration using entrepreneurial orientation” ISBN: 1094-4281 Publisher: Sage Publications Sage CA: Los Angeles, CA  In _Organizational Research Methods_ 13.2, 2010, pp. 320–347
  * [111] Yash Raj Shrestha, Vivianna Fang He, Phanish Puranam and Georg Krogh  “Algorithm supported induction for building theory: How can we use prediction models to theorize?” Number: 3 Publisher: INFORMS  In _Organization Science_ 32.3, 2021, pp. 856–880  DOI: 10.1287/orsc.2020.1382
  * [112] Wesley D. Sine, Arkangel M. Cordero and Ryan S. Coles  “Entrepreneurship through a unified sociological neoinstitutional lens” Number: 4 Publisher: INFORMS  In _Organization Science_ 33.4, 2022, pp. 1675–1699  DOI: 10.1287/orsc.2022.1586
  * [113] Juan Luis Suárez, Roderick E. White, Simon C. Parker and Antonio Jiménez-Mavillard  “Entrepreneurship Bias and the Mass Media: Evidence from Big Data” Publisher: Academy of Management  In _Academy of Management Discoveries_ 7.2, 2021, pp. 247–265  DOI: 10.5465/amd.2018.0177
  * [114] M. Subramani, M. Wagle, G. Ray and A. Gupta  “Capability Development through Just-in-Time Access to Knowledge in Document Repositories: A Longitudinal Examination of Technical Problem Solving.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2021  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=152360578&h=mxKtk%2FKq4njfjGHHDdAho8S7UOgB2XctCA9NAqbT%2BSCdZNKsu9pH1vb359KmdavNS4jYgkAtOcgMc6kDY5H7Yg%3D%3D&crl=c
  * [115] H. Tanriverdi and K. Du  “Corporate Strategy Changes and Information Technology Control Effectiveness in Multibusiness Firms.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2020  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=148572767&h=03cttd%2FoM64SDZ7L63F7cmv%2F%2FlO47Wg2nyeawm41dah3m%2Beh15Mt6EqAtXdjuoY0vbgea%2BlKEf6Ode9abpv58A%3D%3D&crl=c
  * [116] I. Tavory and S. Timmermans  “Two cases of ethnography: Grounded theory and the extended case method”  In _Ethnography_ 10.3, 2009, pp. 243
  * [117] Hovig Tchalian  “Microfoundations and recursive analysis: A mixed-methods framework for language-based research, computational methods, and theory development”  In _Microfoundations of institutions_ Emerald Publishing Limited, 2019
  * [118] David A. Teich  “Deep learning AI needs tools to adapt to changes in the data environment”  In _Forbes_ Forbes Magazine, 2020  URL: https://www.forbes.com/sites/davidteich/2020/07/16/deep-learning-ai-needs-tools-to-adapt-to-changes-in-the-data-environment/
  * [119] Peter A. Todd, James D. McKeen and R. Gallupe  “The evolution of IS job skills: A content analysis of IS job advertisements from 1970 to 1990” ISBN: 0276-7783 Publisher: JSTOR  In _MIS quarterly_ , 1995, pp. 1–27
  * [120] Siliang Tong, Nan Jia, Xueming Luo and Zheng Fang  “The Janus face of artificial intelligence feedback: Deployment versus disclosure effects on employee performance” Number: 9 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/smj.3322  In _Strategic Management Journal_ 42.9, 2021, pp. 1600–1631  DOI: 10.1002/smj.3322
  * [121] Chris Urmson et al.  “Autonomous driving in urban environments: Boss and the urban challenge”  In _Journal of Field Robotics_ 25.8 Wiley Online Library, 2008, pp. 425–466
  * [122] Kiri Wagstaff  “Machine Learning that Matters”  arXiv, 2012  DOI: 10.48550/ARXIV.1206.4656
  * [123] Nada Wasi and Aaron Flaaen  “Record linkage using Stata: Preprocessing, linking, and reviewing utilities” ISBN: 1536-867X Publisher: SAGE Publications Sage CA: Los Angeles, CA  In _The Stata Journal_ 15.3, 2015, pp. 672–697
  * [124] Duncan J. Watts  “Computational Social Science: Exciting Progress and Future Directions”  In _The Bridge on Frontiers of Engineering_ 43.4, 2013  URL: https://nae.edu/106118/Computational-Social-Science-Exciting-Progress-and-Future-Directions
  * [125] Klaus Weber  “A toolkit for analyzing corporate cultural toolkits”  In _Poetics_ 33.3, Culture and classification in markets, 2005, pp. 227–252  DOI: 10.1016/j.poetic.2005.09.011
  * [126] R. Weber  “CONSTRUCTS AND INDICATORS: AN ONTOLOGICAL ANALYSIS.” Publisher: search.ebscohost.com  In _MIS Quarterly_ , 2021  URL: https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=02767783&AN=153886340&h=5WyTEax6ZVemKVMOqavJBwf9%2FhDZts4Cr0F6Bk3TpMeCg%2FzVeSAQzGGdljbohKFyPdM6NLbT9DYS%2Baj%2BgQjppQ%3D%3D&crl=c
  * [127] Xingjiao Wu et al.  “A survey of human-in-the-loop for machine learning”  In _Future Generation Computer Systems_ Elsevier, 2022
  * [128] Doris Xin et al.  “Accelerating human-in-the-loop machine learning: Challenges and opportunities”  In _Proceedings of the second workshop on data management for end-to-end machine learning_ , 2018, pp. 1–4
  * [129] W.. Yue, Q.. Wang and K.. Hui  “See no evil, hear no evil? Dissecting the impact of online hacker forums” Publisher: ink.library.smu.edu.sg  In _Mis Quarterly_ , 2019  URL: https://ink.library.smu.edu.sg/sis_research/4377/
  * [130] Fabio Massimo Zanzotto  “Human-in-the-loop artificial intelligence”  In _Journal of Artificial Intelligence Research_ 64, 2019, pp. 243–252
  * [131] Zhewei Zhang, Jochem Thomas Hummel, Joe Nandhakumar and Lauren Waardenburg  “Addressing the Key Challenges of Developing Machine Learning AI Systems for Knowledge-Intensive Work” Publisher: Association for Information Systems  In _MIS Quarterly Executive_ 19.4, 2020, pp. 221–238  DOI: 10.17705/2 msqe.00035

##  Appendix A Appendix A: Using CRAML for a Literature Review

Bibliographic research often begins with search for relevant papers using an
academic search engine, but such search has a difficult time retrieving only
the papers that use a specific empirical method or type of data, for example.
To find papers that illustrate the challenges CRAML was created to address, we
analyzed a corpus of all published papers from 5 top management journals from
2015-2021 (Academy of Management Journal, Administrative Science Quarterly,
Journal of International Business Studies, Organization Science, and Strategic
Management Journal). We first assembled the corpus and PDFs in Zotero, and
then exported the metadata. We converted all PDFs from these journals to text,
minimally cleaned the text (lower case, removing punctuation, replaced
abbreviations), and used CRAML software to find relevant papers using
keywords, tags, and rules.

We deductively generated list of tags and keywords. The keywords.json file
contians the tags and keywords used to extract all keyword-containing chunks
of text. The context window was set to 6 words, yielding chunks of text of a
maximum length of 13 words. After retrieving 13 word chunks, we examined
n-grams in CRAML and developed rules files for each tag in order to classify
the chunks that were relevant to our search. The following headings present
the rules files used to classify which papers in the management literature
contain relevant concepts - keywords alone generated numerous false positives.
We note that this effort could be much further refined. Given the small size
of the corpus and the variety of tags, machine learning was not desirable or
attempted. CRAML was highly useful to iteratively narrowing a search for
relevant literature, and we stopped when the results produced the relevant
papers. As seen in the notes on the rule sets, keyword based approaches alone
are insufficient to narrowing accurately, and researchers can benefit from a
process of refinement that is context-aware and rule-based.

###  A.1 “Is Text” rules: Papers drawing from Text Corpora

These rules that define the tag _istext_ or “is text” identify papers that
refer to a text corpus or text data. In the first priority of rule-based
classification, chunks containing the keywords “text”, “repository”, “corpus”,
and “document” are initially coded 0 (not _istext_). The subsequent rule at
priority level 1 classifies all chunks containing the exact matching text
(e.g. “text analyis”) 1 (is _istext_). Given the _istext_ rules, the resulting
metadata will report the papers that discuss text data and corpora equal to 1
for _istext_. The most relevant papers were found when combining the _istext_
tag with additional tags described below.

Rule | Prio |
---|---|---

Figure 7: istext Rules

###  A.2 ML rules

To find papers that discuss supervised learning, we construct the simple _ML_
tag. To make Figure 1, we focus only on observations where _istext_ and _ML_
are both equal to 1.

Rule | Prio |
---|---|---

Figure 8: ML Rules

###  A.3 NLP Rules

We follow the same structure for the _NLP_ tag for finding papers that use
NLP. For NLP, we also use regular expressions to find any chunk containing
mention of the term “embedding” and one or more of “word”, “text”, or
“language.”

vendor TEXT effective-date DATE type TEXT text TEXT

Figure 16: Example Database Schema (for nopoach)

Using this schema, a $nopoach$ database was created where the novel, complete,
and classified dataset could be stored. An example excerpt of this database’s
contents is shown in Figure 17.

Note that some of the meta-data, including the text, was left out for
readability. In addition, we mask the firm name here in order to avoid drawing
attention to the specific detail as opposed to the results of the process. In
essence, the tag column represents the codification of the original text data
into a structured dataset. This dataset, in turn, can be used for further
research and analysis.

db2.csv

Figure 17: Random Sample from the nopoach Database