longtable mm[#1]

# A unified statistical and computational framework for ex-post harmonisation
of aggregate statistics

Cynthia A. Huang

Department of Econometrics and Business Statistics, Monash University

<EMAIL_ADDRESS>0000-0002-9218-987X

###### Abstract

Ex-post harmonisation is one of many data preprocessing processes used to
combine the increasingly vast and diverse sources of data available for
research and analysis. Documenting provenance and ensuring the quality of
multi-source datasets is vital for ensuring trustworthy scientific research
and encouraging reuse of existing harmonisation efforts. However, capturing
and communicating statistically relevant properties of harmonised datasets is
difficult without a universal standard for describing harmonisation
operations. Our paper combines mathematical and computer science perspectives
to address this need. The _Crossmaps Framework_ defines a new approach for
transforming existing variables collected under a specific measurement or
classification standard to an imputed counterfactual variable indexed by some
target standard. It uses computational graphs to separate intended
transformation logic from actual data transformations, and avoid the risk of
syntactically valid data manipulation scripts resulting in statistically
questionable data. In this paper, we introduce the _Crossmaps Framework_
through the example of ex-post harmonisation of aggregated statistics in the
social sciences. We define a new provenance task abstraction, the crossmap
transform, and formalise two associated objects, the shared mass array and the
crossmap. We further define graph, matrix and list encodings of crossmaps and
discuss resulting implications for understanding statistical properties of ex-
post harmonisation and designing error minimising workflows.

##  1 Introduction

As the availability of data continues to grow, opportunities for leveraging
conceptually related but separately collected data also increase
combinatorically. Joint analysis of data collected over multiple years, or
multiple jurisdictions under similar but distinct survey instruments is a
common and appealing opportunity for research in the social sciences. However,
harmonising and integrating existing datasets is a complex process involving
many diverse tasks. Preparing an harmonised dataset often requires data access
rights, domain expertise, statistical design and data engineering, amongst
other skills. Dataset preparation and analysis can span multiple individuals
within a team, or even multiple independent parties, due to these diverse
requirements. Unfortunately, the lack of standardised formats for documenting
multi-source datasets limiting the reusability of data preparation efforts.
Idiosyncratic approaches, particularly to the implementation of a
harmonisation strategy, can make it difficult to assess the quality and
characteristics of harmonised datasets. Details and decisions that could be
pivotal to the suitability and robustness of downstream analysis can easily
get lost in the long process of wrangling multiple datasets into a single
analysis-ready dataset.

The specific details of a harmonisation strategy are often hidden away in
custom data wrangling scripts, and only described in general terms as part of
the data preparation process. Mapping details are often relegated to
footnotes, appendices or supplementary materials, if recorded at all. For
example, Humlum (2022) harmonises and integrates Danish micro-data with
occupation codes from the 1988 and 2008 versions of the Statistics Denmark’s
Classification of Occupations (DISCO88 and DISCO08) to study interactions
between robot adoption and labour market dynamics. In the absence of an
officially published DISCO88 to DISCO08 correspondence, Humlum combines
multiple published correspondences from both the International Labour
Organisation and Statistics Denmark with relations inferred from job code
changes in the microdata. Detailed notes on the DISCO88-DISCO08 correspondence
created for and used to prepare the analysis data are not included in the main
paper or appendix, and can only be found in a separate documentation note on
the author’s website (Humlum 2021).

Even when preparation scripts and reproducible workflows are diligently
provided, the length of such scripts often increases exponentially with number
of data sources and the complexity of harmonising concepts between them.
Combined with the idiosyncrasies of different coding languages and data
manipulation tools, reproducibility alone is not sufficient to facilitate
comprehensive auditing or understanding of an integrated dataset within a
reasonable time frame and amount of effort. Identifying, documenting and
communicating key data preparation decisions is also a precondition for
answering the larger and more interesting question of how the provenance of
datasets and preprocessing decisions affects downstream analysis and
conclusions.

This paper offers a unified framework for overcoming these limitations and
designing workflows and documentation formats that facilitate auditability in
addition to reproducibility. The _Crossmaps_ framework provides abstract
conceptual and formal mathematical tools for the production, documentation and
validation of data integration workflows based on examples in the social
sciences. Compared to existing data wrangling frameworks, we focus on a much
narrower task scope. The example tasks we illustrate specifically involve
harmonising numeric data that form a shared aggregate, such as industry-level
output statistics or labour force counts by occupation, rather than other
similar harmonisation tasks such as recoding categorical variables in
individual survey responses.

We proceed by reviewing the process of ex-post harmonisation and briefly
review existing attempts to standardise documentation of harmonisation
workflows. We then formally define the abstract task of interest, the crossmap
transform, and the inputs to the operation, shared mass arrays and crossmaps.
Next, we define graph, matrix and tabular encodings of crossmaps and highlight
some advantages and utility of each encoding. From these definitions, we
discuss statistical and computational insights that arise from the _Crossmaps
Framework_. This includes the correspondence between crossmaps and the
commonly described mapping cases: one-to-one, one-to-many, many-to-one and
many-to-many; as well as how to audit existing datasets and design safeguards
for harmonisation workflows. Finally, we discuss future work and opportunities
for implementing and extending the insights presented in this paper.

##  2 Background

###  2.1 Data Harmonisation

Figure 1: Decomposition of an Ex-Post Harmonisation Process for combining two
source observations collected using different classifications. The source
observation for USA is already in the target classification, represented by
the letter index and green shading. However, the observation for AUS,
totalling 140 units, was collected in alternative “source” classification,
represented by the shape index and blue shading. Thus, in addition to any
necessary source specific cleaning steps, the AUS observation also requires a
_Crossmap Transform_ into the target “green-letter” index.

####  2.1.1 Nomenclature

The transformation and merging of related datasets into a cohesive analysis
dataset has various names including _data fusion_ , _data integration_ , or
_data harmonisation_. The diversity in terms likely reflects the appetite for
and growing practice of preparing multi-source datasets across the many
domains and applications of data science.

We focus specifically on retrospective efforts to harmonise already collected
datasets in the social sciences, and follow Kołczyńska (2022) in using the
term ex-post data harmonisation. Figure 1 illustrates a stylised example of
combining two country-specific datasets into a single ex-post harmonised
dataset.

####  2.1.2 Harmonisation Strategies

Existing literature on data harmonisation often focuses on innovations in
harmonisation strategy and validity of particular approaches (e.g. Pierce and
Schott 2012; Lohr and Raghunathan 2017). However, the ideas in this paper
arise from specific efforts to improve the ease and reliability of
transforming data using some predefined ex-post harmonisation logic. As such,
we do not directly address the design of mappings between statistical
classifications. Instead, we focus on abstracting and formalising the data
manipulation operations involved in ex-post harmonisation.

Our approach most closely relates to existing frameworks in computer science
and statistical programming for specifying and implementing data-wrangling
workflows at the domain problem level rather than in lower-level database
manipulations. The design of the _Crossmaps Framework_ is informed by domain-
specific languages and interfaces for interactive discovery and correction of
data discrepancies (e.g. Raman and Hellerstein 2000; Kandel, Paepcke, et al.
2011), and Wickham (2014)’s _Tidy Data_ principles for data wrangling and
analysis in the R language.

####  2.1.3 Ex-Post Survey Data Harmonisation

The challenges of preparing ex-post harmonised datasets are well documented in
the existing literature on survey data harmonisation (Granda, Wolf, and Hadorn
2010; Dubrow and Tomescu-Dubrow 2016; Fortier et al. 2016; Ehling 2003). The
difficulty of implementing and documenting ex-post harmonisation increases
with the number of data sources and the complexity of correspondence between
the semantically similar but distinct classification standards. The
combination of iterative and sequential steps, subjective imputation and
mapping choices, as well as technical idiosyncrasies associated with different
data storage formats and programming languages or software all contribute to
the difficulty of standardising documentation and methods.

In their study of survey data harmonisation efforts, Dubrow and Tomescu-Dubrow
(2016), reiterate earlier calls by Granda, Wolf, and Hadorn (2010) for
“development of software that standardises the documentation process”.
However, existing ex-post harmonisation guidelines predominantly focus on
survey design considerations and ensuring the comparability of measures over
the specifics of implementation. For instance, Fortier et al. (2016) relegates
data processing to being “achieved using algorithms”, followed by separate ad-
hoc quality checks and verification of said algorithms. Kołczyńska (2022)
attempts to address this gap in specific implementation guidance by proposing
the use of annotated lookup tables, also known as crosswalks. In Section
2.3.3, we contextualise their suggestion within the related concepts of schema
crosswalks and concordance tables, and illustrate how crossmaps overcome key
limitations of crosswalks.

###  2.2 Data Provenance

This work addresses the role of data provenance and access in broader
conversations about computational reproducibility and replicability (e.g. Peng
and Hicks 2021). We contribute to existing efforts to documenting the
provenance and preprocessing of datasets at different granularities. Tools
such as data information cards (e.g. Gebru et al. 2021; Pushkarna, Zaldivar,
and Kjartansson 2022), and metadata standards (e.g. Koren et al. 2022) are
designed for broad capture of data provenance information. Such tools attempt,
as far as possible, to encourage and support the full documentation of dataset
genealogy from collection, preprocessing, through to licensing and archival
availability. Extending beyond high-level dataset documentation, there exist
some attempts to capture and communicate specific preprocessing steps.

Standardised description of specific preprocessing steps is challenging due to
the wide variety of possible data alterations. Moreover, as observed by
Lucchesi et al. (2022), definitions of data preprocessing vary with audience
and context from highly specific lists of tasks, to broadly encompassing
boundaries within a longer data pipeline. Existing provenance tools such as
(Lucchesi et al. 2022; Kai Xiong et al. 2022; Wang et al. 2022) attempt to
achieve generality by comparing dataset snapshots at various points in a
preprocessing pipeline. Lucchesi et al. (2022) and Kai Xiong et al. (2022)
both trace code execution between snapshots, and attempt to illustrate the
data pipeline using glyph representations of function calls. A related
approach is visualising step-wise data pipelines as directed-acyclic-graphs
(e.g. Landau 2021).

Unfortunately, difference tracing is often not sufficient for capturing the
complexities of mapping between classifications. Harmonisation mappings are
seldom simple one-to-one functions of data frames, and in many cases input and
output data frames cannot uniquely identify the mapping used to produce the
output. For example, it should be clear multiple combinations of “blue-shape”
index and “green-letter” mappings could result in the transformed data in
Figure 1 as one-to-many redistributions can be offset by many-to-one
aggregations. The challenge of resolving ambiguity in transformations has been
mentioned multiple times in existing work on data wrangling (e.g Wickham
2014), but is often dismissed as uncommon (e.g. Niederer et al. 2018) or
impossible (e.g. Kandel, Heer, et al. 2011).

###  2.3 Existing Workflows and Toolkits

In the absence of specialised software or standard documentation formats,
researchers are encouraged to share data preparation scripts. Unfortunately,
even when available, custom harmonisation scripts can be difficult to audit or
reuse. The specific mappings used are obscured by the idiosyncrasies of the
programming language or data wrangling approach.

One approach to overcoming the difficulties of reusing scripts is the
development of generic tool for harmonisation tasks such as transformation
description and implementation. Tools vary greatly in scope and functionality.
Descriptive tools generally involves specifying and documenting harmonisation
logic and mappings between taxonomies, whilst workflow helpers aim to assist
with implementing the desired harmonisations.

####  2.3.1 Descriptive Tools

Harmonisation description tools overlap somewhat with generic data provenance
tools, but tend to focus documenting harmonisation logic. Examples include
Goerlich and Ruiz (2018), which attempts to define a domain specific language
for encoding transformations between geographic units; Denk and Froeschl
(2004) which offers a formal semantic model describing hierarchical-taxonomic
classifications and algebraic transformations between them; and Dang et al.
(2015) which offers matrix and graph visualisations of taxonomic alignments,
but does not support transformation of datasets.

####  2.3.2 Domain Specific Toolkits

Domain specific toolkits attempt to provide some combination of descriptive
and workflow functionality tailored to commonly used data sources or types. In
the social sciences domain, helpers for working with official statistics,
census and electoral data are common. For example, strayr (Mackey et al. 2023)
provides crosswalks and helper functions for transforming data to or from
statistical classifications published by Australian Bureau of Statistics
(ABS), while countrycode (Arel-Bundock, Enevoldsen, and Yetman 2018) provides
helpers for working with country names and codes.

Domain specific toolkits are clearly preferable over standalone scripts or
replication packages for facilitating the reuse of data harmonisation efforts.
Such tools often offer more detailed documentation than paper replication
packages and have the potential to develop credibility through popular
adoption. However, such packages are likely to suffer from at least some of
same comprehension issues as bespoke wrangling scripts.

####  2.3.3 Crosswalks and Lookup Based Approaches

Table 1: Example crosswalk mapping between the two, three and numeric country codes from the 2020 release of the _ISO-3166 International Standard for country codes and codes for their subdivisions_ Country | ISO2 | ISO3 | ISONumeric
---|---|---|---
Afghanistan | AF | AFG | 004
Albania | AL | ALB | 008
Algeria | DZ | DZA | 012
American Samoa | AS | ASM | 016
Andorra | AD | AND | 020

Table 2: Crossmap for recoding and distributing country statistics from | to | weight
---|---|---
BLX | BEL | 0.5
BLX | LUX | 0.5
E.GER | DEU | 1.0
W.GER | DEU | 1.0
AUS | AUS | 1.0

Crosswalks are lookup tables, which encode mappings between two related
measures. They consist of at least two columns, one with keys in the source
measure and one for the target measure. They can also include columns for
annotating the source and target measures with more descriptive labels or
other useful information. As shown in Table 1, crosswalk tables structure
recoding logic in a format that is both natural for people to read, and can
also store metadata such as extended descriptions or notes. Furthermore, as
lookup tables, they can be used to transform data without any additional
reshaping or row-wise translation into programming commands.

The terminology for crosswalks differs depending on the specific mapping task
and domain. For example, crosswalks used to harmonise values in related
variables are referred to as _correspondence_ or _concordance_ tables in
economics and official statistics (e.g. Pierce and Schott 2012; Dorner and
Harhoff 2018), while Kołczyńska (2022) uses the term _value crosswalks_. Each
row in a _correspondence_ or _concordance_ table encodes link between keys in
the equivalent code standards. The term crosswalk can also refer to lookup
tables used to collect already compatible variables from different datasets.
Such tables are referred to as _Metadata or Schema crosswalks_ in database and
computing contexts (Khan, Shafi, and Rizvi 2015; Cheney, Chiticariu, and Tan
2007, 430), while Kołczyńska (2022) refers to them as _variable crosswalks_.

Unfortunately, crosswalks only contain enough information to transform
aggregate statistics according to unambiguous one-to-one or many-to-one
relations between source and target keys. The two-column structure is unable
to support transformations where a single source key is related to multiple
targets, otherwise known as one-to-many relations. As such, crosswalk based
approaches and tools generally treat one-to-many relations as a special cases.
These special cases reintroduce the need for bespoke code, hindering the
auditing and reuse of the harmonised dataset and increasing the potential for
mistakes.

####  2.3.4 Assertive Data Validation

A common recommendation for avoiding mistakes in data preparation is adding
verification assertions into the preparation pipeline. In R, assertive
programming and data validation is supported by packages such as assertr
Fischetti (2024), pointblank Iannone and Vargas (2022) and validate van der
Loo and de Jonge (2021). As general purpose tools, the design and selection of
useful assertions is left up to the data analyst. In the case of simple
transformations sensible assertions are relatively straightforward to write.
However, designing appropriate checks for more complex correspondences and
transformations is non-trivial. For example, it is often useful to check the
number of rows in a data table matches expectations after performing a
transformation. However, when working with multiple many-to-many
transformations, it can be difficult to determine whether the transformed data
should have more or less rows than the original dataset as this will depend on
whether transformation involves more aggregating or disaggregating relations.

##  3 Crossmaps Framework

Developing tools for and understanding the statistical implications of Ex-Post
Harmonisation procedures is a multi-faceted challenge. As detailed above,
solutions for and discussion of these various facets are split across computer
science, statistics and domain-specific literatures. To the best of our
knowledge, this paper is the first attempt to directly address the
description, validation and implementation of ex-post harmonisation in a
unified manner.

The _Crossmaps Framework_ overcomes the limitations of crosswalks, whilst
retaining the benefits of lookup-based approaches, by extend crosswalks to
handle one-to-many transformations. The addition of weights to the relation
between source and target classifications facilitates a shift from comparison
based provenance and script based validation, to direct examination and
verification of data inputs and transformation logic via conditions on the
crossmap structure. As shown in Table 2, the weights can encode decisions
about how numeric values, attached to source keys, such as GDP or other
country-level statistics, are redistributed to multiple targets.

Formalisation also offers new ways to examine the statistical properties of
ex-post harmonisation. The modularised structure of crossmaps supports
standardised specification, implementation and comparison of alternative
harmonisations of a single set of source datasets. This standardised workflow
can be used to test the robustness of downstream analysis to alternative
harmonisation decisions. Furthermore, observing that crossmaps are
computational graphs, we can use graph properties to examine and quantify
imputation in ex-post harmonisation procedures.

###  3.1 Ex-Post Harmonisation Task Abstraction

Existing definitions of ex-post harmonisation tend to enumerate requirements
in a checklist style format, chronological steps or a mixture of both (Granda
and Blasczyk 2016; Fortier et al. 2016; Kołczyńska 2020). For example,
Kołczyńska (2020) defines a linear process for ex-post harmonisation as
follows: (1) concept definition, (2) data preparation, (3) harmonisation
transformation and (4) verification and documentation. We propose a new
abstract definition based on Bors et al. (2019)’s provenance task abstraction
framework:

  1. 1.

Data Collection: discovering and obtaining datasets containing harmonisable
data

  2. 2.

Source Specific Cleaning: identifying and resolving issues specific to a data
source and collection method

  3. 3.

Crossmap Transforms: transforming each source dataset into a common
measurement standard, including both the design or selection of mappings
between source and target keys and the actual data manipulation.

  4. 4.

Data Merging: merging each transformed data into a single analysis-ready
dataset.

Our definition is illustrated in Figure 1, and focuses on abstracting high
level mapping and transformation tasks, rather than describing the workflow
commonly used when producing harmonised datasets. Our definition differs from
existing definition in two significant ways.

First, we distinguish source-specific data preparation from harmonisation
focused data transformation and merging. Source-specific tasks include missing
data imputation and format conversion, as well as variable selection and
renaming, and schema matching in preparation for harmonisation. Although
downstream harmonisation strategies and analysis plans can inform source-
specific preparation, data altered in this stage will generally be suitable
for transformation into multiple reasonable target measures and combinations.

Secondly, our definition does not include a separate step for documentation
and verification of the harmonised dataset, which should instead be performed
at each stage with appropriate tools. As we will see, the crossmap structure
can unify verification and documentation into a single mathematical
abstraction. Verification of quality indicators, such as the equivalence of
numeric totals before and after transformation, will follow from satisfying
formal mathematical definitions. Documentation formats, such as tabular
summaries or graph visualisations, correspond to alternative representations
of the computational graph used to transform the data.

###  3.2 Crossmap Transforms

Figure 2: Conceptual illustration of the _Crossmaps Framework_ using the same
harmonisation shown in Figure 1. The example shared mass array data inputs and
outputs of a crossmap transform are shown either side of the crossmap input
which specifies the mapping between source and target keys. The equivalent
graph, matrix and list encodings of the crossmap are all illustrated.

We refer to the abstract operation of transforming source key-indexed values
into values indexed by a set of related target keys as a crossmap transform. A
crossmap transform operation takes source data and applies transformations
according to a weighted relation between the source and target keys and
returns data in the target standard or measure. Under the declarative data
transformation language framework of Kandel, Paepcke, et al. (2011) and
Wickham (2014), a crossmap transform is a high-level action consisting of
three lower-level data-wrangling operations: join, map/transform and
aggregation. We loosely use _transform_ 111This nomenclature is borrowed from
Raman and Hellerstein (2000), but applied at a higher level of abstraction. In
the formalism that follows, nomenclature decisions attempt to straddle
notation conventions across set theory, statistics, graph theory, linear
algebra and databases. However, pragmatics demand deviations from these
conventions in several cases. as a noun to denote a single operation, and
transformation to refer to a sequence or collection of related transforms.

The data input $A_{[\mathcal{S},\mathbf{x}]}$ and output
$A_{[\mathcal{T},\mathbf{y}]}$ of a crossmap transform are shared mass arrays.
The logic encoding input of a crossmap transform is a crossmap. Figure 2
illustrates the inputs and output of a crossmap transform, which we proceed to
formally define below:

######  Definition 3.1.

A shared mass array with index set $\mathcal{K}={\kappa_{i}:i=1\dots K}$, is
an associative array of $K$ key-value pairs, such that
$A_{[\mathcal{K},\mathbf{x}]}$ =
$\\{(\kappa_{i},x_{i}):\kappa_{i}\in\mathcal{K},x_{i}\in\mathbb{R}^{+}\\}$,
where $x_{i}=A(\kappa_{i})$ is the positive real value retrievable by the key
$\kappa_{i}$ and:

  * •

each key $\kappa_{i}$ corresponds to part of the conceptual unit defined by
the index set $\mathcal{K}$ (e.g. state in a country), and

  * •

the sum $\sum_{i=1}^{K}x_{i}$ forms a numeric mass belonging to the same unit
(e.g. GDP in each state).

We could also define $A_{[\mathcal{K},\mathbf{x}]}$ as the function
$A:\mathcal{K}\to\\{x\\}$, where $\\{x\\}$ is the set of unique $x_{i}$
values. However, defining shared mass arrays as associative arrays of key-
value pairs more closely aligns with the tabular format by which such data are
generally presented and shared.

We refer to the value transforming mapping between the source and target
measures as crossmaps, and define them as follows:

######  Definition 3.2.

A crossmap is a collection
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$ with elements
satisfying the following:

  * •

$\mathcal{S}=\\{s_{j}:j=1\dots S\\}$ and $\mathcal{T}=\\{t_{k}:k=1\dots T\\}$
are two sets, referred to as the source and target key sets respectively;

  * •

$\mathcal{R}=\\{(s_{j},t_{k}):s_{j}\in\mathcal{S}\text{ shares value with
}t_{k}\in\mathcal{T}\\}\subseteq\mathcal{S}\times\mathcal{T}$ is a binary
relation between source and target keys, such that there exists
$(s_{j},t_{k})\in\mathcal{R}$ for all source keys $s_{j}\in\mathcal{S}$; and

  * •

$\mathcal{W}=\\{w_{jk}\in(0,1]\text{ if }(s_{j},t_{k})\in\mathcal{R}:\forall
j\ \sum_{k}w_{jk}=1\\}\subseteq(0,1]^{S\times T}$, is a set of weights
representing the share of value attached to a source key to be distributed to
the target key.

Weights will only be fractional in the case of redistribution from a single
source key to multiple target key, and must total one across all pairs
originating from a given source key. The condition
$\mathcal{K}\subseteq\mathcal{S}$ in Definition 3.3 corresponds to a subtle
but clear requirement that the crossmap input to a crossmap transform must
contain mapping logic for all key-value pairs in the data input. We refer to
pairs of shared mass arrays and crossmaps which satisfy this condition as
_conformable_. Now that the required inputs and outputs are defined, we
proceed with defining the operation of interest:

######  Definition 3.3.

A crossmap transform is an operation that applies a _crossmap_
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$ to a shared
mass array $A_{[\mathcal{K},\mathbf{x}]}$, where
$\mathcal{K}\subseteq\mathcal{S}$. The operation redistributes the total
numeric mass $\sum_{i=1}^{K}x_{i}$ across a target index $\mathcal{T}$ and
returns a shared mass array:
$A_{[\mathcal{T},\mathbf{y}]}=\\{(t_{k},y_{k}):t_{k}\in\mathcal{T},y_{k}=\sum_{i:(\kappa_{i},t_{k})\in\mathcal{R}}x_{i}w_{ik}\\}$

######  Corollary 3.1.

For any valid _crossmap transform_ that applies a _crossmap_ $\mathcal{X}$ to
a _shared mass array_ $A_{[\mathcal{K},\mathbf{x}]}$, resulting in
$A_{[\mathcal{T},\mathbf{y}]}$, numeric mass is preserved through the
operation such that $\sum_{k=1}^{T}y_{k}=\sum_{i=1}^{K}x_{i}$.

###### Proof.

This follows naturally from the definition of the output shared mass array and
crossmap weights. Since $\mathcal{K}\subseteq\mathcal{S}$ in a valid crossmap
transform, $\kappa_{i}\in\mathcal{S}$ for all $i=1\dots K$. Then, by
Definition 3.2, $\exists(\kappa_{i},t_{k})\in\mathcal{R}$ for all $i=1\dots
K$. The total mass of the output array can be rewritten as
$\sum_{k=1}^{T}y_{k}=\sum_{k=1}^{T}(\sum_{i=1}^{K}x_{i}w_{ik})=\sum_{i=1}^{K}x_{i}\sum_{k=1}^{T}w_{ik}$.
Again by Definition 3.2, $\sum_{k=1}^{T}w_{ik}=1$ for all $i=1\dots K$. Thus,
$\sum_{i=1}^{K}x_{i}\sum_{k=1}^{T}w_{ik}=\sum_{i=1}^{K}x_{i}=\sum_{k=1}^{T}y_{k}$
∎

Corollary 3.1 shows that the condition $\forall j\ \sum_{k}w_{jk}=1$ in
Definition 3.2 ensures total mass is preserved, and we refer to both the
condition on the crossmap and on the data operation as the _mass-preserving
condition_.

###  3.3 Collections of Crossmaps

Let us briefly situate these definitions in the overall process of ex-post
harmonisation. In particular, note that under the above abstractions,
producing an ex-post harmonised dataset could require multiple parallel and/or
sequential crossmap transforms. For example, consider harmonising occupation
counts from multiple countries and years, where each country-year observation
is collected using a country-specific list of occupation codes, which itself
is subject to updates over time. Harmonising observations within the same
country requires mapping the time-varying occupation codes into a single
target classification, whilst harmonisation across countries requires mapping
country-year observations into a relevant target classification, such as the
International Standard Code of Occupations (ISCO). Each linkage between
classifications forms the basis for another crossmap transform.

The added complexity of managing collections of crossmaps could seem
contrived. However, the above definitions provide a mathematical basis for
implementing and validating harmonisation workflows. They set out various
explicit and implicit conditions under which a crossmap transform is feasible.
Explicit conditions include what combinations of relations and weights form
valid logic for preserving total mass when transforming numeric values from
the index they were collected under to a counter-factual target index.

###  3.4 Suitable Applications

Crossmaps can encode logic for any combination of common harmonisation tasks
including category recoding (one-to-one), value aggregating (many-to-one) and
value redistributing (one-to-many) relations. However, crosswalks are
considerably more parsimonious than crossmaps for implementing one-to-one
recodings. Categorical variables can be converted into shared mass array
through one-hot-encoding, and transformed by applying crossmaps with binary
weights between the source and target categories. However, this introduces
unnecessary data reshaping, and requires explicitly specifying weights that
are implicit in the crosswalk format.

Similarly, if the harmonisation logic involves continuous variables,
alternative functional descriptions may be more suitable. This includes cases
where the source and target key sets are uncountable by definition. For
example, consider the common task of binning income into defined ranges.
Although in practice currency is generally truncated to two decimal places,
the theoretical source key set is $\mathbb{R}^{+}$. The target key set, and
codomain of the binning function, is the set of income ranges defined in the
data preparation process. This transformation can be cast in terms of
Definition 3.3 if we restrict the source key set to income values actually
observed in the source shared mass array. However, the resulting crossmap
would be likely be much more difficult to understand compared to a rule or
function based description of the binning process.

The relative utility of crossmaps arises when documenting and implementing
more complex transformations between countable source and target index sets
such as value redistribution between geographic units, or concordance of
numeric mass between statistical classifications.

##  4 Equivalent Encodings and Features

Crossmaps can be represented in various forms for different purposes. The
computational graph encoding facilitates flexible documentation through
summary and visualisation and provides a mathematical lens for identifying
interesting characteristics of a crossmap transforms. The transformation
matrix encoding illuminates the verification properties of crossmaps by
casting crossmap transforms as linear mappings. The edge list encoding allows
crossmaps to be used directly to transform shared mass arrays via database
operations. The notation used for encodings in this section is summarised in
Figure 2. We define and discuss each encoding in turn, beginning with graph
encoding of crossmaps.

###  4.1 Graph Encoding and Provenance Documentation

Figure 3: Graph and List representations of a crossmap based on a subset of
the crosswalk between the 2022 update of the Australian and New Zealand
Standard Classification of Occupations (ANZSCO22) and the fourth iteration of
the International Standard Classification of Occupations (ISCO08) published by
the Australian Bureau of Statistics Australian Bureau of Statistics (2022)

######  Definition 4.1.

Given a crossmap
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$, let
$G=(\mathcal{S},\mathcal{T},\mathcal{R},\mathbf{B})$ be a directed bipartite
graph where:

  * •

$\mathcal{S},\mathcal{T}$ are the disjoint node sets,

  * •

$\mathcal{R}$ is the edge set, and

  * •

$\mathbf{B}\in\mathbb{R}_{+}^{P\times P}$, is the weighted adjacency matrix,
where $P=S+T$.

$G$ is the computational graph encoding of the crossmap $\mathcal{X}$ if and
only if $\mathbf{B}$ has following block structure:
$[\mathbf{0},\mathbf{C};\mathbf{0},\mathbf{0}]$, where
$\mathbf{C}=[c_{jk}:c_{jk}=c_{jk}\in\mathcal{W}>0$ if
$(s_{j},t_{k})\in\mathcal{R}$ and $b_{jk}=0$ otherwise$]$ is a $S\times T$
matrix, containing weights from $\mathcal{W}$, row-indexed by $\mathcal{S}$
and column-indexed by $\mathcal{T}$.

####  4.1.1 Lateral Mappings

The asymmetric block structure of $\mathbf{B}$ reflects the fact that, except
in the case of one-to-one renaming, weighted linkages between classifications
are lateral (i.e. one-way). Consider reversing the aggregation illustrated
from {111311, 111312, 111399} to {1111} in Figure 3. It should be clear that
defined the reverse transformation using the transpose of $\mathbf{B}$, {1111}
would have three links outgoing links with weights of one, violating the
_mass-preserving condition_. This lateral property reveals an additional
connection between crosswalks and crossmaps, whereby crosswalks can only
encode the binary relation ‘$s$ shares value with $t$’, whilst crossmaps
encode the ternary relation ‘$s$ distributes value to $t$ according to $w$’.

####  4.1.2 Visualising Harmonisation Logic

Computational graph encodings provide a natural framework for designing
interfaces for editing, auditing, exploring and communicating the logic of
complex crossmap transforms. Figure 3 is a visualisation design for one-step
crossmaps proposed in related work (Huang 2023). The proposed visualisation
leverages multiple visual channels to highlight important features relevant
for auditing and comprehension of the harmonisation logic embedded in the
crossmap. Line style, ordering, opacity and labels are used to help the viewer
focus their attention on links that warrant closer inspection. Line style is
used to highlight source codes which are part of split relations, which carry
stronger imputation assumptions relative to the solid one-to-one unique and
shared links. The layout also highlights the existence of sub-structures in
crossmaps that correspond to the commonly known mapping cases: one-to-one,
one-to-many, many-to-one and many-to-many. We discuss these special cases in
more detail in Section 5.1.

Although static visualisations are useful for understanding the general
structure of simple crossmaps, interactivity is the natural option for
visualisation of larger and more complex crossmaps. Building upon the use of
line style in Figure 3, less interesting parts of the crossmap could be hidden
or collapsed, allowing users to examine more interesting parts of a crossmap,
such as sub-graphs with fractional weights. Interactivity also provides an
avenue for non-code specification of crossmaps by domain experts, which could
be validated in real time, and used directly to transform data with minimal
additional data wrangling code.

Next, we examine the validation properties of crossmaps by casting crossmap
transforms as linear mappings.

###  4.2 Matrix Encoding and Mapping Validation

######  Definition 4.2.

The matrix encoding of
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$ is the
$S\times T$ matrix $\mathbf{C}=[c_{jk}:c_{jk}=w_{jk}\in\mathcal{W}>0\text{ if
}(s_{j},t_{k})\in\mathcal{R},\text{ and }c_{jk}=0\text{ otherwise}]$.

It should be clear that the matrix encoding is the same as the block component
$\mathbf{C}$ from the adjacency matrix $\mathbf{B}$ of the crossmap graph
encoding $G$ defined in Definition 4.1.

####  4.2.1 Linear Mappings

Linkages between statistical classification can be characterised as linear
mappings between source and target vector spaces, as shown by Hulliger (1998).
Following their approach, we characterise crossmap transforms as linear
mappings by first defining discrete vector spaces based on the source and
target sets. For a given crossmap $\mathcal{X}$, recall that the cardinality
of the source and target index sets are denoted $S$ and $T$ respectively.
First, attach an $S\times 1$ identification vector $o_{j}$ to each item
$s_{j}\in\mathcal{S}$, which has $0$ in all entries except for the $j$-th
entry which is $1$. The identification vectors $\\{o_{j}\\}$ define the source
vector space $\mathcal{O}$. Similarly attach to each target item
$t_{k}\in\mathcal{T}$ a $T\times 1$ identification vector $d_{k}$, which has
$0$ in all entries except for the $k$-th entry which is $1$. The vectors
${d_{k}}$ define the target vector space $\mathcal{D}$. Now, attach a vector
of values $\mathbf{x}=[x_{1},\dots,x_{S}]$ to each source category $s_{j}$ to
form a shared mass array $A_{[\mathcal{S},\mathbf{x}]}$. The crossmap
$\mathcal{X}$ induces a linear mapping $W:\mathcal{O}\to\mathcal{D}$, where
$W(\mathbf{x})=\mathbf{y}$ and $y_{k}=\sum_{j=1}^{S}w_{jk}x_{j}$ for
$j=1,\dots,t$. Since $\sum_{i=1}^{s}w_{ij}=1$ for all $i=1,\dots,s$, the
linear mapping $W:\mathcal{O}\to\mathcal{D}$ preserves the shared numeric
mass, and satisfies the _mass-preserving condition_. The matrix which encodes
the linear mapping $W$ is the transformation matrix encoding of the crossmap
$\mathcal{X}$.

####  4.2.2 Validation Conditions

Using the above correspondence between crossmaps and linear mappings, we
proceed to show how crossmaps are restricted by definition to only encode
valid transformation logic.

######  Corollary 4.1.

The matrix encoding $\mathbf{C}$ of crossmap $\mathcal{X}$ is row-indexed by
$\mathcal{S}$ and column-indexed by $\mathcal{T}$ and satisfies the matrix
multiplication $\mathbf{C}\ell=\ell$, where $\ell$ is a vector of ones with
length $S$.

###### Proof.

The result follows from the requirement that the sum of weights originating
from a given source key must total one for every source key in a crossmap. Let
$\mathbf{z}=\mathbf{C}\ell=[z_{j}=\sum_{k=1}^{S}c_{jk}]$, which returns the
sum of each row $j$ in $\mathbf{C}$. By Definition 4.2,
$\sum_{k=1}^{S}c_{jk}=\sum_{k:(s_{j},t_{k})\in\mathcal{R}}w_{jk}+\sum_{k:(j,k)\notin\mathcal{R}}0$.
By Definition 3.2, $\forall j\ \sum_{k:(s_{j},t_{k})\in\mathcal{R}}w_{jk}=1$.
Therefore, $\mathbf{z}=\ell$. ∎

Corollary 4.1 provides a principled way to look for data leakage in the
transformation pipeline. Data leakage in the context of crossmap transforms
refers to the unintended loss or creation of numeric value. A common check for
data leakage is comparing the aggregate totals before and after the
transformation. Unfortunately, passing this condition is a necessary but not
sufficient condition to ensure that the harmonisation operations are valid and
match the intended design. This is because there are multiple ways to re-
aggregate or re-distribute a disaggregated mass which will preserve the total
numeric mass. For example, multiple sub-industry re-groupings could preserve
the fixed total of GDP collected using some initial sub-industry
classification. Crossmaps not only flag when aggregates will not be preserved
but also facilitate straightforward location and correction of any errors.
Based on Corollary 4.1, we can see that any rows in $\mathbf{C}\ell$ not equal
to one correspond to a source key with at least one incorrectly specified
outgoing relation.

######  Proposition 4.1.

For a given crossmap transform of $A_{[\mathcal{K},\mathbf{x}]}$ by a
conformable crossmap
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$ with matrix
encoding $\mathbf{C}$ resulting in $A_{[\mathcal{T},\mathbf{y}]}$, if
$\mathcal{K}$ and $\mathcal{S}$ are identical ordered sets, then
$\mathbf{y}=\mathbf{C^{\prime}x}$ is equivalent to
$A_{[\mathcal{T},\mathbf{y}]}$, where $\mathbf{x}$ is the vector of all values
$x_{i}=A(\kappa_{i})$.

###### Proof.

Let $\mathcal{K}$ be $\mathcal{S}$ identical ordered sets with index $i=1\dots
S$. Now express $A_{[\mathcal{S},\mathbf{x}]}$ as a column vector
$\mathbf{x}=[x_{i}]_{i=1\dots S}$. Then
$\mathbf{y=C^{\prime}x}=[\sum_{i=1}^{S}c_{i}kx_{i}]_{j=1\dots S}$. By
Definition 4.2,
$\sum_{i=1}^{S}c_{ik}x_{i}=\sum_{i:(s_{i},t_{k})\in\mathcal{R}}x_{i}w_{ik}+\sum_{i:(s_{i},t_{k})\notin\mathcal{R}}x_{i}0$
for all $t_{k}\in\mathcal{T}$. Thus,
$\mathbf{y}=[y_{k}=\sum_{i:(s_{i},t_{k})\in\mathcal{R}}w_{ik}x_{i}]$ as per
Definition 3.3. ∎

Proposition 4.1 is a refinement of the crossmap transform operation in terms
of matrix multiplication. Under this matrix representation, _conformable_
crossmaps and shared mass array inputs are additionally restricted to
conformable matrix dimensions. In other words, to implement a crossmap
transform using matrix multiplication, the condition
$\mathcal{K}\subseteq\mathcal{S}$ from Definition 3.3 becomes
$\mathcal{K}=\mathcal{S}$.

###  4.3 Edge List Representation and Human-Centred Computing

######  Definition 4.3.

Given a crossmap
$\mathcal{X}=(\mathcal{S},\mathcal{T},\mathcal{R},\mathcal{W})$ with graph
encoding $G=(\mathcal{S},\mathcal{T},\mathcal{R},\mathbf{B})$, let
$E(s_{j},t_{k},w)$ be a table with primary key $(s_{j},t_{k})\in\mathcal{R}$
and attribute $w=w_{jk}\in\mathcal{W}$ such that each record represents a
weighted edge in $G$. $E$ is the edge list encoding of $\mathcal{X}$.

The edge list encoding corresponds directly to the extension on crosswalks
introduced at the start of Section 3.2 and illustrated in Table 2. The rows in
$E$ also correspond to non-zero entries in the matrix encoding $\mathbf{C}$.
If we remove the attribute $w$, the remaining primary key $(s_{j},t_{k})$
forms a crosswalk table of the form discussed in Section 2.3.3. Thus, as noted
by Hulliger (1998), $E$ is a sparse representation of the linear mapping $W$
between the source and target vector spaces.

####  4.3.1 Matrix Multiplication via Database Queries

It has been shown that properties of directed graphs can be obtained via
matrix multiplication on the edge list encoding (Zhou and Ordonez 2020).
Crossmaps are a special case of directed graphs, and as such the matrix-vector
transformation detailed in Proposition 4.1 can be implemented as the following
database query:

Listing 1 Query Implementation of Matrix-Vector Multiplication. Adapted from
Zhou and Ordonez (2020).

SELECT E.k as k, sum(E.w * S.x) as y

FROM E JOIN S AS E.j=S.j

GROUP BY E.k

For any conformable crossmap $\mathcal{X}$ and shared mass array
$A_{[\mathcal{S},\mathbf{x}]}$, Listing 1 corresponds to implementing the
crossmap transform via the following steps:

  1. 1.

For each tuple $(s_{j},x_{j})$ in $A_{[\mathcal{S},\mathbf{x}]}$, append the
attribute $t_{k}$ such that $(j,k)\in\mathcal{R}$; then

  2. 2.

For each tuple $(t_{k},s_{j},x_{j})$ multiply $x_{j}$ by
$w_{jk}\in\mathcal{W}$ to obtain $(s_{j},t_{k},x_{j},x_{j}w_{jk})$; then

  3. 3.

For each group of tuples defined by $t_{k}$, calculate the aggregate
$y_{k}=\sum_{j}x_{j}w_{jk}$ to obtain $(t_{k},y_{k})$, which corresponds to
the output $A_{[\mathcal{T},\mathbf{y}]}$.

####  4.3.2 Tidy Data Harmonisation

The tabular data structure of edge lists provide a conceptual bridge between
existing idiosyncratic practices of ex-post harmonisation and human-centred
approaches to data wrangling and analysis such as _Tidy Data_ principles and
the tidyverse suite of R packages(Wickham 2014; Wickham et al. 2019). The
correspondence between the above algorithm and Proposition 4.1, via
possibility redundant calculations, permits the specification, implementation
and storage of crossmap transform logic with only tabular data structures.

Redundancy of calculations can arise through properties of the crossmap. For
instance, for a given crossmap $\mathcal{X}$, step 1 can be thought of as
renaming source keys $s\in\mathcal{S}$ to target keys $t\in\mathcal{T}$, and
is the only necessary step when implementing categorical variable recoding.
Intuitively, this corresponds with the observation that renaming source keys
in $A_{[\mathcal{S},\mathbf{x}]}$ does not modify the values in $\mathbf{x}$.
Similarly, step 2 is not strictly necessary if $\mathcal{T}$ is a hierarchical
structure over $\mathcal{S}$, as in the case of aggregation operations. In
such as case, all weights $w_{jk}$ with be 1, and the unmultiplied values
$x_{j}$ are identical to the multiplied values $x_{j}w_{jk}$.

Now that we have established the equivalence graph, matrix and edge list
representations of crossmap transforms, we proceed to recast and examine
common mapping concepts, data quality considerations and workflow challenges
in ex-post harmonisations in terms of crossmaps. We show in the following
examples how combinations of perspectives offered by each encoding can lead to
useful practical and theoretical insights.

##  5 Conceptual and Statistical Implications

###  5.1 One-to-one, One-to-Many, Many-to-One and Many-to-Many Components

It should be clear visually that the computational graph in Figure 3 can be
partitioned into three disjoint subgraphs. The bottom subgraph corresponds to
a _many-to-one_ relationship, while the middle subgraph corresponds to _one-
to-one_ relationships between source and target keys. The remaining subgraph
contains two intersecting/overlapping _one-to-many_ relationships,
corresponding to the ancillary relationship type _many-to-many_. It should
also be clear that the _many-to-many_ subgraph introduces stronger imputation
assumptions in the transformation process than the _one-to-one_ subgraph. From
an auditing or review perspective, _one-to-many_ and _many-to-many_
redistribution weights require additional scrutiny relative to binary
relationships between source and target keys (i.e. statements of the form
$s_{j}$ shares value with $t_{k}$ without reference to weights).

Identifying and grouping disjoint subgraphs can facilitate the examination of
these two distinct types of assumptions in harmonisation strategies. The most
obvious set of disjoint subgraphs is the partition defined by the set of all
disjoint components in $G$, ignoring the direction of the edges, which
corresponds to the separate disjoint subgraphs visible in Figure 3. In larger
crossmaps, there could be hundreds of subgraph components, especially as every
one-to-one links forms a disjoint component. Thus, it is useful to define
conditions on the subgraphs which can group them into meaningful subsets.

Grouping conditions should naturally correspond to the type of relationship
between source and target keys in the subgraph. Starting first with _one-to-
one_ relationships, define a subset $\mathcal{R}^{1}$ of the relation
$\mathcal{R}$ which satisfies the binary condition “$s_{j}$ and $t_{k}$ share
value only with each other”. This condition corresponds to separating all the
comparatively trivial one-to-one renaming operations from aggregating and
disaggregating operations encoded in a crossmap. Next, consider _one-to-many_
and _many-to-one_ relationships, which are mirrors of each other. Define a
subset $\mathcal{R}^{2}$ which satisfies the exclusive OR condition “either
$s_{j}$ is connected to more than one $t_{k}$, OR $t_{k}$ is connected to more
than one $s_{j}$. Finally, let remaining components belong to the subset
$\mathcal{R}^{M}$. This subset contains _many-to-many_ relationships, which
are overlapping combinations of the previous relationship types.

The conditions for $\mathcal{R}^{1}$ and $\mathcal{R}^{2}$ can be translated
into conditions on the node degree and number of edges of each disjoint
component. Thus, the partition described above could be achieved via the
following steps:

  1. 1.

Identify disjoint components via breadth-first or depth-first search over the
vertices of the graph (Hopcroft and Tarjan 1973).

  2. 2.

Compute the number of edges in each component and group any components with
only one edge into $\mathcal{R}^{1}$.

  3. 3.

Compute the node degrees for the remaining components and group into
$\mathcal{R}^{2}$ any components with all node degrees equal to 1, except for
one node, which has a node degree equal to the number of edges in the
component.

  4. 4.

Group any remaining components into $\mathcal{R}^{M}$.

The above partition can be paired with appropriately chosen graph summary and
visualisation techniques to improve the readability and concision of
provenance documents for ex-post harmonisation datasets. For instance, since
one-to-one and many-to-one relations always have edge weights of 1, they could
be summarised in tabular form without reference to weights. Conversely, given
the complexity of many-to-many components, visualising the components in a
style similar to Figure 3 would likely be preferable over tabular
presentations.

###  5.2 Data Preprocessing Sensitivity and Robustness Checks

Crossmaps provide a conceptual link between ex-post harmonisation and existing
theoretical and applied research on data preprocessing. It is generally
accepted that empirical results should be tested for robustness under
plausible alternative model assumptions. However, as observed by Blocker and
Meng (2013), the same attention is not given to data preprocessing decisions,
despite the risk to the validity of downstream analyses. They propose a formal
framework for exploring the statistical implications of preprocessing
decisions under the banner of multi-phase inference. They formulate data
preprocessing decisions in terms of existing work on multiple imputation and
missing data (see Rubin 1976, 1996), and consider theoretical bounds on the
performance of multi-phase procedures under various scenarios.

####  5.2.1 Missing Data Imputation

Crossmap transforms can be viewed as single imputation procedures that map
indexed numeric values, (i.e. a shared mass array), into counterfactual values
indexed under an alternative index. The transformed values are counterfactual
in the sense that they correspond to an estimate or imputation of what we
would have observed if the initial source data were also collected or measured
under the target classification. For example, in the case of occupation
statistics, consider transforming data collected under the 2022 Australian and
New Zealand Standard Classification of Occupations (ANZSCO22) into the closest
International Standard Classification of Occupations (ISCO08) as illustrated
in Figure 3. The resultant data reflects a single deterministic estimate of
which ISCO08 occupation respondents to the original ANZSCO22 survey would have
selected if asked to select from the ISCO08 occupation list. The crossmap used
to map ANZSCO22 occupation codes to ISCO08 implicitly specifies the
assumptions used to impute the missing counterfactual data.

####  5.2.2 Quantifying Imputation

Compared to filling missing values for individual survey responses, imputation
in crossmap transforms exists at the aggregated level of the shared mass
array. This makes quantifying and describing the degree of imputation applied
much more difficult than counting up the number of missing values. However,
observe that crossmaps are both computational graphs and data imputation
models for the missing counterfactual target shared mass array in a crossmap
transform. Thus, graph summary techniques could be used to describe and
quantify the _potential imputation_ of a crossmap and the actual degree of
imputation applied to create a specific ex-post harmonised dataset. Here we
use the term _potential imputation_ to refer to how much a given crossmap
could modify the values in a conformable shared mass array.

Measures for potential imputation include properties such as the relative
share of each type of subgraph component defined in Section 5.1, where
crossmaps with only one-to-one components form a zero imputation baseline from
which more complex crossmaps can be compared. The actual degree of imputation
embedded in a harmonised dataset is determined by both the potential
imputation, and the actual input data. The interaction between these two
inputs to a crossmap transform determines the degree to which the output
shared mass array produced by a given crossmap transform reflects the observed
source data versus assumptions about the counterfactual world. Consider two
potential shared mass array inputs to the crossmap in Figure 3, one with the
majority of the overall shared mass in 111212, and one with the majority in
111111. In the former case, most of the mass is just re-indexed as 111212
forms a one-to-one relation with 0110. However, in the latter case, the value
for 111111 is split up between {1112, 1114, 1120}, producing an output array
with much stronger counterfactual assumptions than in the former case.

####  5.2.3 Multiple Imputation and Data Multiverses

Following similar motivations as guide multiple imputation, Steegen et al.
(2016) argue against the practice of preparing a single analysis dataset. They
observe that empirical research often takes for granted that any dataset used
in a given analysis is just one of many potential datasets that could have
been prepared from the available raw data and suggest that empirical
researchers perform _multiverse analyses_ to increase transparency and check
the robustness of their findings to alternative reasonable preprocessing
decisions. Multiverse analysis involves constructing a “data multiverse”
containing multiple reasonable preparations of the raw data, and then
calculating a resulting “multiverse of statistical results” by applying the
same downstream analysis to each alternative dataset.

The crossmaps framework offers a systematic and structured tool for extending
the principles of multiverse analysis to studies using ex-post harmonised
datasets. The dual nature of crossmaps as logic encodings and functional
inputs to crossmap transform operations avoids creating multiple data
preparation scripts. Instead, different crossmaps can be passed into a
crossmap transform workflow with a fixed collection of shared mass arrays to
generate a multiverse of ex-post harmonised datasets. In addition to
increasing the scientific reliability of studies using ex-post harmonised
datasets, multiverse analyses could provide insight for future research into
the statistical properties of ex-post harmonisation as a data preprocessing
procedure.

##  6 Computation and Design Implications

###  6.1 Understanding and Auditing Existing Scripts and Datasets

####  6.1.1 Extracting Crossmaps

We can use insights from Proposition 4.1 to extract the crossmap embedded in
existing code and to confirm the validity of the implemented transformations.
To illustrate, Listing 2 provides sample STATA code used aggregate occupation
codes into larger categories. Notice on line 7, there is an interaction
between the conditions for teacher and professional, whereby the mapping into
professional depends on teacher==0. Such interactions make it more difficult
for other data users to understand and validate the overall mapping logic.

Recall that the output of a crossmap transform corresponds to the matrix-
vector multiplication $\mathbf{C^{\prime}x}=\mathbf{y}$. The crossmap
transform embedded in Listing 2 could also be represented in this form, where
$\mathbf{C}^{\prime}$ corresponds with the STATA commands in Listing 2,
$\mathbf{x}$ is the $S$-length vector component of a shared mass array formed
from the input data occupation.dta on line 1, and $\mathbf{y}$ is the data
created by running the script. Let us replace the input vector $\mathbf{x}$
with an identification vector $o_{j}$ for the $j$-th key in the $S$ element
source index set $\mathcal{S}$. $\mathbf{C^{\prime}}o_{j}=y$ returns a
$T$-length vector with the weights for any outgoing links from $s_{j}$ to
elements in $\mathcal{T}$. It should thus be clear that we can extract the
implied crossmap by passing $n$ identity vectors, one for each source key,
through the script and combining the output data. This corresponds to
obtaining $\mathbf{C}^{\prime}\mathbf{I}=\mathbf{C}^{\prime}$, where
$\mathbf{I}$ is a $S\times S$ identity matrix.

In practice, extracting embedded crossmap logic can be complicated by the
structure of a given script. However, in the case of Listing 2, we were able
to replace the script input with an identify vector formed from the
occupation.dta, and extract a valid crossmap. Table 3 summarises some key
features of the crossmap extracted from Listing 2, and illustrates how the
validation properties implied by Corollary 4.1 can be verified using simple
summary calculations. In particular, notice that the extracted crossmap has 12
disjoint components, 11 of which are many-to-one relations with the remaining
component forming a one-to-one relation. The implied weights on all the edges
are thus 1, and the mass preserving condition $\mathbf{C}\ell=\ell$ is
trivially satisfied. We can also confirm that Listing 2 implements each
source-to-target link only once, as the number of links extracted equals the
number of unique source keys. Finally, observe that the largest grouping
assprofclerk combines 87 source keys, while smaller groupings such as
armforces and driver combine only 4 and 7 source keys respectively. This
discrepancy might warrant further investigation depending on how the
transformed data is used and/or interpreted.

Listing 2 Example STATA script for merging multiple occupations into larger
groups. Included with permission from authors.

use "occupation.dta", clear

gen farmer=0

replace farmer=1 if occupn>6000 & occupn<7000

gen teacher=0

replace teacher=1 if occupn>2400 & occupn<2500

gen professional=0

replace professional=1 if occupn>2000 & occupn<3000 & teacher==0

gen manager=0

replace manager=1 if occupn>1000 & occupn<1129

replace manager=1 if occupn>1131 & occupn<2000

gen armforces=0

replace armforces=1 if occupn<200

gen xefe=0

replace xefe=1 if occupn==1130

gen assprofclerk=0

replace assprofclerk=1 if occupn>3000 & occupn<5000

gen svcsales=0

replace svcsales=1 if occupn>5000 & occupn<6000

replace svcsales=1 if occupn>9000 & occupn<9200

gen labourer=0

replace labourer=1 if occupn>9200 & occupn<9320

gen driver=0

replace driver=1 if occupn>8320 & occupn<8330

replace driver=1 if occupn>9330 & occupn<9340

gen craftrademach=0

replace craftrademach=1 if occupn>7000 & occupn<9000 & driver==0

gen notclass=0

replace notclass=1 if occupn>9990 & occupn<10000

sum professional manager teacher assprofclerk svcsales armforces xefe ///

farmer craftrademach labourer driver notclass if p3p30_school_level==6

Table 3: Summary of Aggregation Logic based on Crossmap extracted from Listing 2 Target Key | No. Incoming Sources | Incoming Source Keys
---|---|---
assprofclerk | 87 | 3111,3112,3113,3114,3115,3116,3117,3118,3119,3121,…
craftrademach | 70 | 7111,7112,7113,7121,7122,7123,7124,7129,7131,7132,…
professional | 57 | 2111,2112,2113,2114,2121,2122,2131,2132,2133,2141,…
svcsales | 36 | 5111,5112,5113,5121,5122,5123,5131,5132,5133,5134,…
manager | 32 | 1110,1120,1141,1142,1143,1210,1221,1222,1223,1224,…
farmer | 17 | 6111,6112,6113,6114,6121,6122,6123,6124,6129,6130,…
teacher | 10 | 2410,2421,2422,2431,2432,2440,2450,2461,2462,2469
driver | 7 | 8321,8322,8323,8324,9331,9332,9333
labourer | 6 | 9211,9212,9213,9311,9312,9313
armforces | 4 | 110,120,140,190
notclass | 2 | 9998,9999
xefe | 1 | 1130

####  6.1.2 Concurrent Crossmap Transforms

The above example illustrates how crossmaps can be extracted and examined from
a script implementing a single crossmap transform. However, as mentioned in
Section 3.3, an ex-post harmonised dataset will likely involve multiple
sequential and/or concurrent crossmap transforms. Decomposing ex-post
harmonised datasets as outputs of a set of crossmap transforms provides a new
perspective for understanding properties of the overall harmonised dataset.
Figure 4 visualises a single transformation step in the ex-post harmonisation
of country-year records from the INDSTAT 4 dataset. Each tile represents the
country-year specific crossmap transform for a shared mass array of industry-
level output indexed by country-specific industry codes into 4-digit codes
from ISIC Revision 3. The colour of the tile indicates whether any recorded
values were split in the process of transforming the data into the target
schema of ISIC Revision 3, while the facets arrange the country-year
transforms by their 1996 world bank income group. Compared to long-form
explanatory notes, such visualisations offer an alternative and more
structured format for summarising and communicating publisher (e.g. country)
or observation (e.g. country-year) level variations in data quality and data
modification.

Figure 4: Summary visualisation of a set of concurrent crossmap transforms
applied to industry level output statistics collected according to country-
year specific industry codes. Each tile represents a country-year observation
of output (GDP) production in the INDSTAT4 Revision 3 Industry Level Dataset.
The colour of the tile indicates whether that country-year observation
contained industry codes and associated output values that were redistributed
to the codes in the target ISIC classification

####  6.1.3 Sequential Crossmap Transforms

In addition to the concurrent crossmap transforms above, we can also consider
examining a sequence of related crossmap transforms. Hulliger (1998) describes
mathematically how correspondence matrices, which are equivalent in definition
to the matrix encoding $\mathbf{C}$, can be combined to describe concatenated
correspondences from a source classification to a target classification via
one or multiple intermediate classifications, as well as to describe
correspondences involving changes at multiple levels of a hierarchical
classification schema. An example of the former case occurs in the
transformation of INDSTAT 4 data from Revision 4 to Revision 2 of the
_International Standard Industrial Classification of All Economic Activities
(ISIC)_.

Official concordances are available between ISIC2 and ISIC3.1, as well as
ISIC3.1 and ISIC 4, but not directly between ISIC4 and ISIC2. It should be
clear from the results in Section 4.2 that the concatenated crossmap from
ISIC4 to ISIC2 can be described by the matrix product
$\mathbf{C}_{42}=\mathbf{C}_{43}\mathbf{C}_{32}$, where the subscripts ab
indicate the source $a$ and target $b$ indexes. This may seem trivial in a
two-step transformation, but consider increasing the size of the dataset to be
transformed or the number of transformation steps. In the former case,
collapsing multiple steps eliminates intermediate computations which reduces
the time required to produce the transformed dataset. In the latter case, the
concatenated crossmap can be summarised in a style similar to Table 3, to
inspect the composition of the transformed data directly in terms of the
initial source keys, rather than as a chain of transformations.

###  6.2 Workflow Design

In addition to improving documentation, the crossmap format provide a
conceptual foundation for modular and auditable workflows for ex-post
harmonisation. The formalisation of crossmap transform operations and the
associated conformability conditions give rise to meaningful constraints and
principles for implementing ex-post harmonisation workflows. Pivoting between
matrix and database representations of crossmap transforms can help us to
design workflows and tools that insure against various implementation errors
and risks. In particular, we highlight several subtle and difficult-to-trace
programming errors that can be avoided by translating constraints from the
matrix representations into table based workflows.

Figure 5: Stylised example of a data leakage error. The crossmap shown on the
left-hand side does not contain mapping instructions for the source key
x7285!. Thus, under a naive transformation the associated value 3895 could be
lost.

Figure 6: Stylised example of a potential missing value arithmetic error. The
error results from passing a missing NA value to a valid one-to-many
relationship in the example crossmap. The implied transformation for the
missing value is a splitting of value between the target keys D6 and D7.

####  6.2.1 Data Leakage and Crossmap Coverage

We previously discussed in Section 4.2 how the structure of crossmaps and
conformability conditions theoretically preclude data leakage. Retaining these
conditions in a data wrangling workflow requires identifying and validating
properties of the data structures used to store crossmap edge lists and shared
mass arrays. For instance, the mass preserving condition corresponds to
checking the sum of weights grouped by source key in the edge list equals one.
Depending on whether weights are implemented symbolically or numerically, the
validation may be subject to some floating point tolerance. Additionally, the
conformability condition corresponds to checking that the unique set of source
keys in the edge list $E$ contains all the index keys in the shared mass
array, which we refer to as a coverage check. Recall from Listing 1 that
implementing matrix-vector multiplication involves joining the edge list $E$
with the input shared mass array on the source keys. The flexibility of
database joins means that, without a coverage check, it is possible to perform
a non-conformable crossmap transform, which could cause data leakage if the
join drops rows from the shared mass array with non-zero values. Figure 5
illustrates an scenario where data leakage could occur.

####  6.2.2 Missing Value Handling

In addition to conditions on the crossmap edge list, the structure of crossmap
transform operations give rise to implementation constraints for shared mass
arrays. Except in the case of strictly one-to-one crossmap transforms, the
presence of missing values (i.e. NA or NULL values) can lead to _missing value
arithmetic_ errors. Missing value arithmetic errors occur when we perform
programmatically valid calculations, which may not be mathematically valid.
Missing values in the crossmap edge list $E$ are precluded by definition. It
is less obvious that missing values in the input shared mass array should be
dealt with prior to performing the crossmap transform operation. However,
consider the scenario shown in Figure 6, where a NA value attached to x5555 is
split into D6 and D7, and also combined with other incoming values. It is
straightforward programmatically to implement this conformable crossmap
transform using the query in Listing 1. However, the output of the transform
varies depending on the treatment of missing values in the multiplication and
aggregation steps. If the missing value is propagating into the final sum,
other incoming values would be overwritten. Alternatively, if the calculation
of the final sum is modified to remove missing values, then the crossmap
transform implementation effectively treats NA values as zeroes. It is hard to
imagine that the former case would ever be intended, however, even when the
later case is intended, it is much clearer from a provenance perspective to
replace missing values with zeros prior to the crossmap transform.

####  6.2.3 Addition and Removal of Index Keys

The addition and removal of index keys between revisions of statistical
classifications should also be handled outside of the crossmap transform
rather than implicitly within the operation. Hulliger (1998) shows that births
and deaths of categories can be represented as columns or rows of zeroes in
the correspondence matrix $\mathbf{C}$. However, this representation conflicts
with the invariance of numeric totals across the crossmap transform, and
represents an ontological question of whether the shared mass in the source
and target classifications are comparable. Rather than combine recoding and
redistribution actions with removing or appending elements of category-indexed
variables, we suggest removing any unwanted categories prior to applying the
crossmap, and attaching any new categories after transforming the existing
data. For example, the existence of a target key without a corresponding
source key suggests that the target shared mass could be larger or smaller
than the observed shared mass. If this is the case, then the additional target
key-value pair should be added after the crossmap transform for corresponding
source-target links. This preserves cross-taxonomy transformation as a
redistribution operation, rather than one that creates or destroys numeric
mass and thus avoids unnecessary data validation challenges.

###  6.3 Computational Constraints and Interface Design

####  6.3.1 Floating Point Discrepancies

In Section 6.1, we treated crossmaps as perfect representations of
harmonisation logic embedded in existing scripts. However, as alluded to in
Section 6.2.1, the output dataset produced by applying an extracted crossmap
could differ from the output produced by the original data wrangling approach
and code. To illustrate, consider an mapping that equally distributes some
source value $x$ to three target keys. The corresponding computational graph
would have three links connecting the source key and target keys with weights
of one-third on each link. If the weights are implemented using floating point
representation, then the value assigned to each target key will be
$0.\overline{333}x$, subject to the defined floating point precision. Compare
this to implementing the value redistribution using a FOR LOOP, such that for
each of the three target keys, we write a rule that divides the source value
in thirds and assign that value to the target key. Then the resulting target
value would be $x/3$ rather than $0.\overline{333}x$. Floating point weights
also complicate verification of the _mass-preserving condition_ , by
necessitating some floating point tolerance when comparing the sum of weights
to one.

In practice, such floating point discrepancies are likely to occur in all
alternative implementations of a particular multi-source dataset. However, the
crossmap structure materialises such discrepancies multiple times in a given
workflow. Floating point inaccuracies can arise when the crossmaps are
created, as well as when they are applied to transform source datasets.
Furthermore, since the _mass-preserving condition_ must be satisfied for every
single source key, the cumulative extent of discrepancies grows with the size
of the crossmap graph, as discussed in Bauer (1974). For this reason, symbolic
representations of link weights are recommended when implementing data
structures for crossmaps.

####  6.3.2 Multipartite Graph Layouts

As shown in Section 4.1, opportunities exist to adapt and extend existing
graph visualisation tools and algorithms to realise the potential
communication and interface benefits of crossmaps. In particular, sequential
transformations are a natural match for multi-partite graph visualisation
methods. For example, Sankey layout algorithms are more suitable for the
layered structure of crossmaps relative to more general purpose network graph
layout algorithms. However, the most commonly implemented multi-layer graph
layout algorithm is the heuristic algorithm Sugiyama, Tagawa, and Toda (1981),
which does not support by default support grouping of substructures as shown
in Figure 3. As discussed in Huang (2023), Zarate et al. (2018) offer an
alternative layout algorithm that supports grouping which could be adapted to
visualising crossmaps.

####  6.3.3 Multi-Table Data Wrangling

As illustrated in Figure 4, and discussed in Section 3.3, producing harmonised
datasets often involves handling multiple data tables and crossmaps. Designing
tools and interfaces which support the management and use of multiple
crossmaps and source datasets is closely related to work on _Multi-Table Data
Wrangling_ as discussed by Kasica, Berret, and Munzner (2021). They observe in
their study of data wrangling practices by data journalists that previous
wrangling frameworks emphasise operations within a single table, but
journalists often use and combine multiple tables for their analysis. The
combine operations and merge actions in their _Multi-Table Data Wrangling_
frameworks most closely align with the operations in our proposed framework.

##  7 Conclusion, Limitations and Future Work

Ex-Post Harmonisation is an increasingly common practice as the volume and
diversity of data sources grows in the social sciences and other fields. This
paper presents a unified framework for exploring and solving the various
workflow, provenance and statistical challenges associated with ex-post
harmonised dataset. We have introduced a new task abstraction and formalised a
structure for encoding mappings used to transform aggregated statistics from
one classification standard to another. We show with multiple examples how
equivalent graph, matrix and list representations of these mappings can reveal
insights and guide novel approaches to theoretical and practical issues in ex-
post harmonisation.

The results in this paper are limited to the transforming aggregated
statistics with meaningful alternative groupings. Furthermore, as discussed
earlier in Section 3.4, the framework is most useful for complex mappings
involving countable source and target key sets. We chose a narrow task scope
to support precise mathematical abstraction and formalisation at the expense
of direct applicability to other common ex-post harmonisation task. However,
we believe the framework could be adapted or extended to other similar
workflows and tasks.

Planned future work includes developing software and interactive tools based
on the framework, and applying and testing the framework on examples from
other domains. We implement a selection of the crossmap features discussed in
this paper in the R package xmap (Huang and Puzzello 2023). The package
implements matrix, graph and edge list representations of crossmaps, tools for
specifying, validating, and applying crossmap transforms. We plan to implement
symbolic fractional weights to circumvent floating point issues and provide
helper functions for visualising and summarising crossmaps. The package is
designed to be compatible with the tidyverse suite of R packages (Wickham et
al. 2019), and is built upon the vctrs package for defining and validating
data structures in R (Wickham, Henry, and Vaughan 2023). The package is
currently in active development, and we welcome contributions and feedback
from interested parties.

## References

## References

  * Arel-Bundock, Vincent, Nils Enevoldsen, and CJ Yetman. 2018. “Countrycode: An r Package to Convert Country Names and Country Codes.” _Journal of Open Source Software_ 3 (28): 848. https://doi.org/10.21105/joss.00848.
  * Australian Bureau of Statistics. 2022. “ANZSCO - Australian and New Zealand Standard Classification of Occupations.” https://www.abs.gov.au/statistics/classifications/anzsco-australian-and-new-zealand-standard-classification-occupations/2022.
  * Bauer, Friedrich L. 1974. “Computational Graphs and Rounding Error.” _SIAM Journal on Numerical Analysis_ 11 (1): 87–96. https://www.jstor.org/stable/2156433.
  * Blocker, Alexander W., and Xiao-Li Meng. 2013. “The Potential and Perils of Preprocessing: Building New Foundations.” _Bernoulli_ 19 (4). https://doi.org/10.3150/13-BEJSP16.
  * Bors, Christian, John Wenskovitch, Michelle Dowling, Simon Attfield, Leilani Battle, Alex Endert, Olga Kulyk, and Robert S. Laramee. 2019. “A Provenance Task Abstraction Framework.” _IEEE Computer Graphics and Applications_ 39 (6): 46–60. https://doi.org/10.1109/MCG.2019.2945720.
  * Cheney, James, Laura Chiticariu, and Wang-Chiew Tan. 2007. “Provenance in Databases: Why, How, and Where.” _Foundations and Trends in Databases_ 1 (4): 379–474. https://doi.org/10.1561/1900000006.
  * Dang, Tuan, Nico Franz, Bertram Ludascher, and Angus Graeme Forbes. 2015\. “ProvenanceMatrix: A Visualization Tool for Multi-Taxonomy Alignments.” _CEUR Workshop Proceedings_ 1456 (January): 13–24.
  * Denk, M., and K. A. Froeschl. 2004. “East of Neuchatel: A Universal Model for the Representation of Statistical Taxonomy Systems.” In _Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004._ , 373–82. Santorini Island, Greece: IEEE. https://doi.org/10.1109/SSDM.2004.1311233.
  * Dorner, Matthias, and Dietmar Harhoff. 2018. “A Novel Technology-Industry Concordance Table Based on Linked Inventor-Establishment Data.” _Research Policy_ 47 (4): 768–81. https://doi.org/10.1016/j.respol.2018.02.005.
  * Dubrow, Joshua Kjerulf, and Irina Tomescu-Dubrow. 2016. “The Rise of Cross-National Survey Data Harmonization in the Social Sciences: Emergence of an Interdisciplinary Methodological Field.” _Quality & Quantity_ 50 (4): 1449–67. https://doi.org/10.1007/s11135-015-0215-z.
  * Ehling, Manfred. 2003. “Harmonising Data in Official Statistics.” In _Advances in Cross-National Comparison_ , edited by Jürgen H. P. Hoffmeyer-Zlotnik and Christof Wolf, 17–31. Boston, MA: Springer US. https://doi.org/10.1007/978-1-4419-9186-7_2.
  * Fischetti, Tony. 2024. _Assertr: Assertive Programming for r Analysis Pipelines_. https://docs.ropensci.org/assertr/ (website) https://github.com/ropensci/assertr.
  * Fortier, Isabel, Parminder Raina, Edwin R Van Den Heuvel, Lauren E Griffith, Camille Craig, Matilda Saliba, Dany Doiron, et al. 2016. “Maelstrom Research Guidelines for Rigorous Retrospective Data Harmonization.” _International Journal of Epidemiology_ , June. https://doi.org/10.1093/ije/dyw075.
  * Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. “Datasheets for Datasets.” _Communications of the ACM_ 64 (12): 86–92. https://doi.org/10.1145/3458723.
  * Goerlich, Francisco, and Francisco Ruiz. 2018. “Typology and Representation of Alterations in Territorial Units: A Proposal.” _Journal of Official Statistics_ 34 (1): 83–106. https://doi.org/10.1515/jos-2018-0005.
  * Granda, Peter, and Emily Blasczyk. 2016. “Data Harmonization.” In _Guidelines for Best Practice in Cross-Cultural Surveys_. Ann Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan.
  * Granda, Peter, Christof Wolf, and Reto Hadorn. 2010. “Harmonizing Survey Data.” In _Survey Methods in Multinational, Multiregional, and Multicultural Contexts_ , edited by Janet A. Harkness, Michael Braun, Brad Edwards, Timothy P. Johnson, Lars Lyberg, Peter Ph. Mohler, Beth-Ellen Pennell, and Tom W. Smith, 1st ed., 315–32. Wiley. https://doi.org/10.1002/9780470609927.ch17.
  * Hopcroft, John, and Robert Tarjan. 1973. “Algorithm 447: Efficient Algorithms for Graph Manipulation.” _Communications of the ACM_ 16 (6): 372–78. https://doi.org/10.1145/362248.362272.
  * Huang, Cynthia A. 2023. “Visualising Category Recoding and Numeric Redistributions.” August 12, 2023. http://arxiv.org/abs/2308.06535.
  * Huang, Cynthia A., and Laura Puzzello. 2023. “Xmap: A Principled Approach to Recoding and Redistributing Data Between Nomenclature.”
  * Hulliger, Beat. 1998. “Linking of Classifications by Linear Mappings.” _Journal of Official Statistics_ 14 (January): 255–66.
  * Humlum, Anders. 2021. “Crosswalks Between (D)ISCO88 and (D)ISCO08 Occupational Codes.” https://www.andershumlum.com/codes.
  * ———. 2022. “Robot Adoption and Labor Market Dynamics.” Rockwool Foundation Research Unit.
  * Iannone, Richard, and Mauricio Vargas. 2022. _Pointblank: Data Validation and Organization of Metadata for Local and Remote Tables_.
  * Kai Xiong, Siwei Fu, Guoming Ding, Zhongsu Luo, Rong Yu, Wei Chen, Hujun Bao, and Yingcai Wu. 2022. “Visualizing the Scripts of Data Wrangling with SOMNUS.” _IEEE Transactions on Visualization and Computer Graphics_ , January, 1–1. https://doi.org/10.1109/tvcg.2022.3144975.
  * Kandel, Sean, Jeffrey Heer, Catherine Plaisant, Jessie Kennedy, Frank van Ham, Nathalie Henry Riche, Chris Weaver, Bongshin Lee, Dominique Brodbeck, and Paolo Buono. 2011. “Research Directions in Data Wrangling: Visualizations and Transformations for Usable and Credible Data.” _Information Visualization_ 10 (4): 271–88. https://doi.org/10.1177/1473871611415994.
  * Kandel, Sean, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011\. “Wrangler: Interactive Visual Specification of Data Transformation Scripts.” In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , 3363–72. Vancouver BC Canada: ACM. https://doi.org/10.1145/1978942.1979444.
  * Kasica, Stephen, Charles Berret, and Tamara Munzner. 2021. “Table Scraps: An Actionable Framework for Multi-Table Data Wrangling From An Artifact Study of Computational Journalism.” _IEEE Transactions on Visualization and Computer Graphics_ 27 (2): 957–66. https://doi.org/10.1109/TVCG.2020.3030462.
  * Khan, Nadim Akhtar, S M Shafi, and Sabiha Zehra Rizvi. 2015. “Metadata Crosswalks as a Way Towards Interoperability:” In _Encyclopedia of Information Science and Technology_ , edited by Mehdi Khosrow-Pour, D.B.A., Third, 1834–42. IGI Global. https://doi.org/10.4018/978-1-4666-5888-2.ch177.
  * Kołczyńska, Marta. 2020. “Micro- and Macro-Level Determinants of Participation in Demonstrations: An Analysis of Cross-National Survey Data Harmonized Ex-Post.” _Methods, Data, Analyses_ 14 (1): 36. https://doi.org/10.12758/mda.2019.07.
  * ———. 2022. “Combining Multiple Survey Sources: A Reproducible Workflow and Toolbox for Survey Data Harmonization.” _Methodological Innovations_ 15 (1): 62–72. https://doi.org/10.1177/20597991221077923.
  * Koren, Miklós, Marie Connolly, Joan Lull, and Lars Vilhuber. 2022. “Data and Code Availability Standard,” December. https://doi.org/10.5281/ZENODO.7436134.
  * Landau, William Michael. 2021. “The Targets R Package: A Dynamic Make-like Function-Oriented Pipeline Toolkit for Reproducibility and High-Performance Computing.” _Journal of Open Source Software_ 6 (57): 2959.
  * Lohr, Sharon L., and Trivellore E. Raghunathan. 2017. “Combining Survey Data with Other Data Sources.” _Statistical Science_ 32 (2). https://doi.org/10.1214/16-STS584.
  * Lucchesi, Lydia R., Petra M. Kuhnert, Jenny L. Davis, and Lexing Xie. 2022\. “Smallset Timelines: A Visual Representation of Data Preprocessing Decisions.” In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 1136–53. Seoul Republic of Korea: ACM. https://doi.org/10.1145/3531146.3533175.
  * Mackey, Will, Matt Johnson, David Diviny, Matt Cowgill, Bryce Roney, William Lai, and Benjamin Wee. 2023. _Strayr: Ready-to-use Australian Common Structures and Classifications and Tools for Working with Them_. Manual.
  * Niederer, Christina, Holger Stitz, Reem Hourieh, Florian Grassinger, Wolfgang Aigner, and Marc Streit. 2018. “TACO: Visualizing Changes in Tables Over Time.” _IEEE Transactions on Visualization and Computer Graphics_ 24 (1): 677–86. https://doi.org/10.1109/TVCG.2017.2745298.
  * Peng, Roger D., and Stephanie C. Hicks. 2021. “Reproducible Research: A Retrospective.” _Annual Review of Public Health_ 42 (1): 79–93. https://doi.org/10.1146/annurev-publhealth-012420-105110.
  * Pierce, Justin R, and Peter K Schott. 2012. “A Concordance Between Ten-Digit U.S. Harmonized System Codes and SIC/NAICS Product Classes and Industries.” _Journal of Economic and Social Measurement_ 37 (1-2): 61–96.
  * Pushkarna, Mahima, Andrew Zaldivar, and Oddur Kjartansson. 2022. “Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI.” In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 1776–826. Seoul Republic of Korea: ACM. https://doi.org/10.1145/3531146.3533231.
  * Raman, Vijayshankar, and Joseph M Hellerstein. 2000. “An Interactive Framework for Data Cleaning.” UCB/CSD-0-1110. Computer Science Division (EECS): University of California.
  * Rubin, Donald B. 1976. “Inference and Missing Data.” _Biometrika_ 63 (3): 581–92. https://doi.org/10.1093/biomet/63.3.581.
  * ———. 1996. “Multiple Imputation After 18+ Years.” _Journal of the American Statistical Association_ 91 (434): 473–89. https://doi.org/10.1080/01621459.1996.10476908.
  * Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016\. “Increasing Transparency Through a Multiverse Analysis.” _Perspectives on Psychological Science_ 11 (5): 702–12. https://doi.org/10.1177/1745691616658637.
  * Sugiyama, Kozo, Shojiro Tagawa, and Mitsuhiko Toda. 1981. “Methods for Visual Understanding of Hierarchical System Structures.” _IEEE Transactions on Systems, Man, and Cybernetics_ 11 (2): 109–25. https://doi.org/10.1109/TSMC.1981.4308636.
  * van der Loo, Mark P. J., and Edwin de Jonge. 2021. “Data Validation Infrastructure for R.” _Journal of Statistical Software_ 97 (10): 1–31. https://doi.org/10.18637/jss.v097.i10.
  * Wang, April Yi, Will Epperson, Robert A DeLine, and Steven M. Drucker. 2022\. “Diff in the Loop: Supporting Data Comparison in Exploratory Data Analysis.” In _CHI Conference on Human Factors in Computing Systems_ , 1–10. New Orleans LA USA: ACM. https://doi.org/10.1145/3491102.3502123.
  * Wickham, Hadley. 2014. “Tidy Data.” _Journal of Statistical Software_ 59 (10). https://doi.org/10.18637/jss.v059.i10.
  * Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the Tidyverse.” _Journal of Open Source Software_ 4 (43): 1686. https://doi.org/10.21105/joss.01686.
  * Wickham, Hadley, Lionel Henry, and Davis Vaughan. 2023. _Vctrs: Vector Helpers_. https://vctrs.r-lib.org/.
  * Zarate, David Cheng, Pierre Le Bodic, Tim Dwyer, Graeme Gange, and Peter Stuckey. 2018. “Optimal Sankey Diagrams Via Integer Programming.” In _2018 IEEE Pacific Visualization Symposium (PacificVis)_ , 135–39. https://doi.org/10.1109/PacificVis.2018.00025.
  * Zhou, Xiantian, and Carlos Ordonez. 2020. “Matrix Multiplication with SQL Queries for Graph Analytics.” In _2020 IEEE International Conference on Big Data (Big Data)_ , 5872–73. Atlanta, GA, USA: IEEE. https://doi.org/10.1109/BigData50022.2020.9378275.