#  schlably: A Python Framework for Deep Reinforcement Learning Based
Scheduling Experiments

Constantin Waubert de Puiseau, Jannik Peters, Christian Dörpelkus, Tobias
Meisen  Institute for Technologies and Management of the Digital
Transformation University of Wuppertal Rainer-Gruenter-Str. 21, Wuppertal,
42119, NRW, Germany

###### Abstract

Research on deep reinforcement learning (DRL) based production scheduling (PS)
has gained a lot of attention in recent years, primarily due to the high
demand for optimizing scheduling problems in diverse industry settings.
Numerous studies are carried out and published as stand-alone experiments that
often vary only slightly with respect to problem setups and solution
approaches. The programmatic core of these experiments is typically very
similar. Despite this fact, no standardized and resilient framework for
experimentation on PS problems with DRL algorithms could be established so
far. In this paper, we introduce schlably, a Python-based framework that
provides researchers a comprehensive toolset to facilitate the development of
PS solution strategies based on DRL. schlably eliminates the redundant
overhead work that the creation of a sturdy and flexible backbone requires and
increases the comparability and reusability of conducted research work.

###### keywords:

Scheduling , Reinforcement Learning , JSSP , Framework

††journal: SoftwareX

## Required Metadata

## Current code version

Nr. |  Code metadata description |  Please fill in this column
---|---|---
C1 |  Current code version |  v0.1.0
C2 |  Permanent link to code/repository used for this code version |  https://github.com/tmdt-buw/schlably
C4 |  Legal Code License |  Apache License, 2.0 (Apache-2.0)
C5 |  Code versioning system used |  git
C6 |  Software code languages, tools, and services used |  Python, OpenAI Gym, RLlib, Weights and Biases
C7 |  Compilation requirements, operating environments & dependencies |  Python 3.10
C8 |  If available Link to developer documentation/manual |  https://github.com/tmdt-buw/schlably/tree/main/docs
C9 |  Support email for questions |<EMAIL_ADDRESS>
Table 1: Code metadata (mandatory)

##  1 Motivation and Significance

Production scheduling (PS) is a challenging and highly researched problem in
operations research (OR) and optimization. It deals with allocating resources
to production jobs over time to minimize criteria such as time, effort, and
cost [1]. PS problems are considered NP-hard and therefore require much
computation effort to be solved sufficiently well with increasing problem
sizes. In recent years, with increasingly powerful algorithms and
computational hardware, deep reinforcement learning (DRL) has emerged as a
promising tool to address PS problems [2, 3, 4, 5, 6, 7, 8]. DRL is a machine
learning paradigm, in which deep learning models are trained through
interaction with an environment to autonomously derive solution strategies for
sequential tasks [9].

The young research field of DRL-based PS lies at the intersection of
operations research and artificial intelligence and as such is characterized
by a heterogeneous community with varying problem-solving approaches and
technical skill sets. Yet, all empirical studies apply a very similar
experimental setup consisting of the same main software components: An
environment representing the production facility layout and logic, a
scheduling problem generator, a DRL agent algorithm, and logging as well as
evaluation tools. The difference between different experimental setups usually
lies within one or more of these components, for example by incorporating a
new DRL algorithm [10, 11, 12], interaction logic between agent and
environment [6, 3], training procedure [13], learning objective [14, 12] or
additional problem constraint [14, 15]. Regardless of large overlaps, all
researchers implement their own individual experimentation framework with the
following two consequences: Large initial ramp-up efforts when experimenting
with new methodologies or custom problem settings, and scarcity of empirical
comparisons to other works. In this paper, we address these shortcomings and
present the software framework schlably for developing and evaluating DRL-
based PS solutions. schlably provides the following contributions:

  * 1.

It is modular, so individual changes may be adapted without much overhead.

  * 2.

It works out of the box with functioning environments, data-generation
scripts, agents, logging functions, training, and testing scripts.

  * 3.

It provides benchmark datasets for different scheduling problem classes and
sizes.

  * 4.

It includes widely recognized benchmark algorithms and results.

  * 5.

It facilitates the application of algorithms designed for one problem class
and size to other problems.

schlably will accelerate the research area of DRL-based PS under real-world
conditions by lowering the barrier of entry for researchers from different
domains with different perspectives, levels of expertise, and objectives.

##  2 Background and Related Work

schlably started as a code base for experiments on a real-world inspired
scheduling problem in the context of a university research project with
industrial partners. As such, several requirements became apparent early on
that can be summarized in four general design goals. First, from the applied
industrial perspective, schlably has to offer the integration of DRL methods
and heuristics which work out of the box. Second, it also has to cover
different scheduling scenarios, e.g. including such bounded by resource
constraints. Third, from the scientific research perspective, schlably should
support detailed comparable evaluations of methods. Lastly, it has to be easy
to interact with at the code level, to enable students with limited experience
to quickly understand the topic, concepts, and implementation. The
implications of these design goals are discussed in this section.

In the following, we present an overview of related published experimental
frameworks and compare them to our design goals in schlably. In the
comparison, we include frameworks dedicated to being used by others [16, 17,
18, 19, 20] and frameworks published in a supplementary fashion along with
research papers and projects [21, 22, 23, 24, 25, 26], because in practice
both may serve as starting points for additional experiment designs. The
frameworks were found via references in scientific publications and a search
of "Scheduling Reinforcement Learning" on GitHub. We do not claim the list to
be exhaustive but are not aware of any other popular frameworks at the time of
writing this paper. Commercial or proprietary scheduling software was excluded
because license fees and other accompanying challenges introduce a major
barrier. However, we are not aware of any commercial software that provides
the tight integration of DRL and PS. Table 2 provides an overview of the
frameworks. In addition, we assessed them regarding their fulfillment of our
design goals which are formally categorized into the following four groups.

|  | [16] | [17] | [18] | [21] | [19] | [20] | [22] | [23] | [24] | [25] | [26] | schlably
---|---|---|---|---|---|---|---|---|---|---|---|---|---
B | Implemented RL-agents | ◑ | ○ | ◑ | ◑ | ○ | ● | ○ | ○ | ◑ | ○ | ◑ | ●
Implemented PDRs | ○ | ● | ◑ | ○ | ○ | ○ | ○ | ● | ○ | ○ | ● | ●
Implemented opt. solver | ○ | ● | ○ | ○ | ○ | ● | ● | ○ | ○ | ○ | ● | ●
Interface for RLLib | ○ | ○ | ● | ○ | ◑ | ○ | ○ | ○ | ○ | ○ | ○ | ●
S | Flexible Data Generation | ○ | ○ | ○ | ○ | ○ | ● | ○ | ○ | ● | ◑ | ● | ●
JSSP | ● | ● | ● | ○ | ● | ● | ● | ● | ● | ● | ● | ●
FJSSP | ○ | ○ | ○ | ● | ○ | ○ | ○ | ● | ○ | ○ | ○ | ●
Different problem types | ○ | ● | ◑ | ● | ● | ● | ◑ | ● | ● | ● | ● | ●
Resource constraint tool | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ●
L | Log achieved results | ○ | ◑ | ◑ | ◑ | ○ | ● | ○ | ○ | ◑ | ○ | ◑ | ●
Evaluate achieved results | ○ | ○ | ○ | ◑ | ○ | ● | ○ | ◑ | ◑ | ◑ | ● | ●
Visualize Gantt-Chart | ○ | ● | ○ | ● | ● | ● | ○ | ● | ● | ◑ | ○ | ●
Comparison to solver | ○ | ○ | ○ | ○ | ○ | ● | ○ | ○ | ○ | ○ | ● | ●
Comparison to PDRs | ○ | ○ | ◑ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ● | ●
C | Paper | ○ | ● | ○ | ○ | ● | ● | ○ | ○ | ● | ○ | ● | ●
README | ◑ | ● | ● | ● | ● | ● | ● | ● | ○ | ● | ● | ●
Code documentation | ○ | ● | ○ | ○ | ○ | ○ | ● | ● | ○ | ● | ◑ | ●
Easily personalizable | ○ | ○ | ● | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ○ | ●
Works out-of-the-box | ● | ◑ | ● | ○ | ○ | ● | ● | ● | ● | ○ | ● | ●
User manual in Readme | ● | ◑ | ◑ | ○ | ◑ | ● | ◑ | ◑ | ◑ | ● | ○ | ●
OpenAI Gym Env | ○ | ● | ○ | ○ | ● | ● | ○ | ○ | ● | ● | ● | ●
|  |  |  |  |  |  |  |  |  |  |  |  |

Legend: ○not fulfilled ◑half-fulfilled ●fulfilled
---

Table 2: Overview of related frameworks and their fulfillment regarding pre-
implemented benchmarks (B), scheduling instances (S), logging and evaluation
(L), and code usability (C)

### Pre-Implemented Benchmarks

Several frameworks either provide pre-implemented DRL agents and scripts for
training or the easy integration of agents from popular DRL libraries, e.g.
from StableBaselines [27] or RLlib [28]. Like [20], our goal is to enable and
facilitate both, the manual extensions of basic DRL algorithms, like DQN and
PPO, as well as the usage of powerful third-party libraries. Both options are
important to empower users to choose the appropriate approach for their
respective research interests. Additionally, for the sake of comparability, it
is crucial to provide common benchmarks in the form of popular priority
dispatching rules (PDRs), such as Shortest-Processing-Time-First, and a
flexible optimal solver that can handle several scheduling problem types. Most
frameworks only cover a few PDRs, often missing competitive ones, such as
Most-Tasks-Remaining and even random baselines, where agent actions are
sampled from a normal distribution.

### Scheduling Instance Generation

Generating new data with varying problem cases is necessary to enable
comprehensive training and testing of a DRL agent. Accordingly, a suitable
framework must implement a flexible problem instance generator. This generator
enables the user to create scheduling problems of different popular categories
(e.g. the Job Shop Scheduling Problem (JSSP) or the Flexible Job Shop
Scheduling Problem (FJSSP) [1]) instances with any combination of instance
variables, like the number of jobs, number of tasks, runtimes, and more.
Moreover, its design simplifies the integration of additional scheduling
problem types to encourage the implementation of individual, more complex, or
more specific use cases. Finally, to the best of our knowledge, this is the
first framework providing an optional resource constraint. Concretely, the
user is able to specify a required tool per operation.

### Logging and Evaluation

Logging results and evaluation metrics in a structured manner is key for quick
feedback during training runs, but also for identifying patterns in and
drawing conclusions from large-scale experiments. Our objective with schlably
is to provide extensive logging options that may be turned on and off, and
where results and models may be shared to promote collaboration on projects.
This design goal is met to a large degree by [20] from which we took much
inspiration in this regard. All other frameworks do not address this design
goal. For the evaluation of solutions generated by DRL agents, a comprehensive
framework should apply the benchmarks mentioned above and provide an overview
of the overall performance. Most of the reviewed frameworks lack functionality
in this aspect. Moreover, to get a graphical overview and to visually support
the tracking of very specific actions in a production schedule, a Gantt chart
plotter is useful for human inspection. The Gantt chart should display all
metadata of operations, e.g. the runtime or required tool, and has been found
to be helpful for debugging and evaluating DRL agents. Many other frameworks,
but not all, include a Gantt chart plotter.

### Code Usability

As usability is of utmost importance, a framework has to offer easy access
through full documentation and must include a README, user application
programming interface (API) manual, and formal functionality description.
Within the reviewed frameworks, only [20] covers all criteria. To be usable
with different skill sets our explicit goal is to enable users to start
experimenting with small design decisions by only using the configuration
files but at the same time facilitate substantial logical changes to routines
and components by means of their own implementations. For that reason, we
would favor comprehensibility over efficiency wherever a trade-off is
unavoidable. This requires a careful balance. In our opinion, all other
frameworks overemphasize one side: [20] offers many functional changes through
configuration files but at the cost of a comparatively complex software
architecture. On the other hand, all other frameworks are much smaller and
easier to get an overview of, at the expense of limited functionality. Lastly,
a framework with a claim to widespread use should stick to conventional APIs.
In the context of DRL, the most commonly used API is the OpenAI Gym [29] API.
Only half of the reviewed frameworks adhere to it.

##  3 Software Architecture

This chapter describes our framework with focus on implementation-specific
details. We are providing a general overview of the code itself, focusing on
currently existing exemplary implementations while also pointing out open
interfaces. Furthermore, we describe details regarding the main components of
schlably to demonstrate the realization of the design goals, as introduced in
Chapter 2, and to enable users to fit schlably to their needs. The overall
structure is illustrated in Figure 1. We divided the code base into six main
components, which are described below in detail. Following this component-
oriented approach, and in combination with comprehensive code documentation,
schlably adheres to the objective of the fourth design goal, which requires
easy interaction and usability at the code level.

Figure 1: Overview of schlably project and code structure.

### Data generator

The general data structure of scheduling problems, as used in schlably, is
represented by so-called instances. A user can generate infinite instances of
a scheduling problem, however, each instance is a specific configuration and
entity. The specific configuration, contained within an instance, is given by
a number of jobs, with a job simply being an encompassing logical container
consisting of individual tasks. The data_generator component incorporates the
necessary classes to generate such an instance and the individual tasks. From
the scheduling problem point-of-view, it is the centerpiece of the problem
formulation and representation. The Task class is a specifically designed data
class and its entities are the atomic units of a scheduling problem instance.
Such an instance can be created via the SPFactory, which allows the generation
of different types of scheduling problems that are given via the included
Enum. If users would like to introduce a new type of scheduling problem into
schlably, they would have to include their function in this class and add it
to the Enum. Finally, the InstanceFactory enables high-level access to the
problem factory class and manages the configuration-based creation of batches
of instances. Thus, the data_generator component realizes the foundation of
the second design goal, which requires the implementation and handling of
different scheduling scenarios.

### Environment

An environment defines the observation space, action space, and reward
strategy. Thus, it represents a simulation of the agent’s environment and
interaction dynamics and is the central piece of any DRL approach. All
schlably environments are included in the environments component. Exemplary,
we provide a simple scheduling Env as well as a derived version named
EnvIndirectAction to showcase the expandability. All environments adhere to
the Gym API and are explicitly derived from a base Gym environment. The
EnvironmentLoader class enables high-level access and management of the
different environment types and appropriate algorithms, as not all algorithms
are feasible for every environment. New environments have to be included in
this component and added to the managing EnvironmentLoader. This encapsulated
approach, in conjunction with the data_generator component, represents the
implementation of the second design goal.

### Agent

The agents component combines the heuristic functionalities, the solver, and
implementations of DRL algorithms as well as the train and test functions for
the DRL approach. Users can to integrate functionalities from other DRL
frameworks, like more extensive training procedures, model types, and learning
algorithms via pre-defined interfaces. As such, the agents component realizes
the first design goal, to support and simplify the integration of out-of-the-
box-methods as well as pre-implemented benchmarks.

### Visual generator

The component visuals_generator incorporates all classes and scripts which are
used to create visualizations of the problem instances and generated
solutions. These functionalities are intentionally isolated as different
scheduling problem environments and still share the same visualization
approach. schlably, for example, introduces a GanttChartPlotter that enables a
user to generate individual Gantt chart images (see Figure 2b) or create a GIF
of the scheduling progress. Thus, it is part of the implementation of the
third design goal.

### Utils

The utils component aggregates classes and functions which have a supporting
character for the main functionalities of schlably. Specifically, it includes
user interface components (ui_tools), data interface components
(file_handler), e.g. to load and save data, and the high-level Logger class.
Accordingly, the utils component realizes the third design goal, facilitating
logging and evaluations for comparisons.

### Code tests

All code tests that ensure the crucial functionality of the described
components are collected in the code_tests component. Up to this point, we
included multiple unit tests with a central Runner. These are also intended as
an example for users that plan to extend the code base.

##  4 Illustrative Example

To illustrate a typical use case, we consider a scenario in which an ML
engineer wants to compare the learning behavior of two PPO agents. It is also
part of our tutorial in the documentation. One agent is trained on 6x6 JSSP
instances and receives a reward based on the change in the time to complete
all tasks (i.e. the makespan) per step, as proposed in [3]. This setup is also
the default setting delivered in the framework. The other one is trained on a
3x4 tool-constrained JSSP instance and receives a zero reward per step with
the exception of the last step, where the reward is equal to the overall
achieved makespan. The remaining training parameters are kept constant. The
second training requires only minimal manual changes to the base model. These
include setting different configuration parameters, generating new data, and
changing the reward function in the base environment. Details may be found in
the documentation. The integrated interface to Weights&Biases [30] makes it
easy to compare the training curves and achieved results, as depicted in
Figure 2.

Figure 2: Comparing agent runs in Weights&Biases (screenshot from the web
interface shown on the left-hand side). a) Visualized training curves for
interpreting the learning performance of the agent. b) Gantt chart depicting
the solution of the trained agent on a selected test instance. c) Table
providing evaluation results and comparison of the trained agents and
benchmark methods on the test instances.

The described short example reflects several of our design goals. Figure 2c)
demonstrates that the agents’ performance is automatically compared to many
other benchmarks and with respect to different dimensions such as the reward
or the gap to the optimal solver. The continuous logging and graphical
depiction are visible in Figure 2a) and b). The example also showcases our
understanding of high code usability. The experiments could be defined by
changing training parameters (only a few lines in configuration files) and
minimal intended changes to the source code. Examples of the most common
changes which are intended to be coded are explained in more detailed follow-
along tutorials in the provided documentation.

##  5 Impact

schlably is useful for the entire community around PS with DRL. Compared to
other frameworks, it is particularly useful to reduce the entry barrier for
researchers from the OR or other related domains, who want to empirically
explore a new methodology for scheduling problems, and for DRL researchers who
want to test a new algorithm on a challenging and impactful problem domain. We
believe that the seamless interchangeability of problem settings offered by
schlably will also encourage researchers in the domain of PS with DRL to try
out methodologies applied to one particular problem setting (e.g. 6x6 JSSP) on
different problem settings (e.g. 11x11 tool-constrained JSSP). This has the
potential to greatly speed up the transfer of research from academic problems
to real-world problems.

In several projects where our test partners and we have used schlably, it has
significantly increased the throughput of experiments. This is achieved
because new methodological ideas can be integrated more quickly and the
results of experiments can be compared more easily. schlably facilitates the
generation of new problem instances and the training and evaluation of custom
DRL agents. Due to the various pre-implementations in the framework, such as
training and testing routines, well-known scheduling benchmarks, and
visualization of logged results, it is much easier to conduct experimental
research in DRL for PS. In addition, collaboration has become more effective
because design changes can be compared easily and the results of peers can be
viewed online through Weights&Biases. We have further experienced a
substantial increase in productivity in research projects, where new
researchers and university students, who had no prior domain knowledge and
little coding skills, had to conduct experiments on the PS domain. This, we
mainly attribute to the code documentation and modular structure, but also to
the fact that schlably is 100% written in Python and therefore runs on all
relevant operation systems.

##  6 Discussion and limitations

In its current state, schlably serves as a useful framework for empirical DRL-
based PS research. It has reached a maturity level, at which it works out-of-
the-box and, to the best of our knowledge, offers the broadest range of
different easy-to-implement design choices compared to any published
framework. schlably, on the one hand, is intended to be abstract and modular
enough to offer different instance generation, training, and testing
configurations without many lines of code. On the other hand, it is designed
to not be too interwoven in its code structure to hinder the extension with
fundamentally different features experts might find desirable. As such, the
development required a balancing act and certain compromises, which some may
see as limitations. For example, one deliberate choice was made in favor of a
class-based problem description as opposed to a vector representation. The
class-based description simplifies the search and usage of certain information
about the current state of jobs and increases code readability compared to a
vector problem representation. Hence, the choice was made between readability
and computational efficiency in favor of the former.

##  7 Conclusions

In this paper, we introduced schlably, a software framework for research on
DRL-based PS. With the release of the framework, we strive towards two main
goals: the first is to lower the entry barrier for researchers, who have
little experience with production scheduling, deep reinforcement learning
(DRL) and/or coding. The second goal is to encourage researchers already
active in the field to apply and test their methods on other problem settings,
which is largely facilitated by schlably. Both goals aim at promoting the
transfer of DRL methods to real-world scheduling applications. In the future,
we plan to include more problem settings, such as the dynamic JSSP and
stochastic properties of environments like machine breakdowns to get even
closer to real-world scenarios.

##  8 Conflict of Interest

We wish to confirm that there are no known conflicts of interest associated
with this publication and there has been no significant financial support for
this work that could have influenced its outcome.

## Acknowledgements

This research work was undertaken within the research project AlphaMES funded
by the German Federal Ministry for Economic Affairs and Climate Action (BMWK).

## References

  * [1] M. Pinedo, Scheduling: Theory, algorithms, and systems, fifth edition Edition, Springer International Publishing, 2016.  doi:10.1007/978-3-319-26580-3.
  * [2] I. Bello, H. Pham, Q. Le V, M. Norouzi, S. Bengio, Neural combinatorial optimization with reinforcement learning (2016).  doi:10.48550/ARXIV.1611.09940.
URL http://arxiv.org/pdf/1611.09940v3

  * [3] C. Zhang, W. Song, Z. Cao, J. Zhang, P. S. Tan, X. Chi, Learning to dispatch for job shop scheduling via deep reinforcement learning, Advances in Neural Information Processing Systems 33 (2020) 1621–1632.
  * [4] A. Kuhnle, J.-P. Kaiser, F. Theiß, N. Stricker, G. Lanza, Designing an adaptive production control system using reinforcement learning, Journal of Intelligent Manufacturing 32 (44) (2021) 855–876.  doi:10.1007/s10845-020-01612-y.
  * [5] T. van Ekeris, R. Meyes, T. Meisen, Discovering heuristics and metaheuristics for job shop scheduling from scratch via deep reinforcement learning, Proceedings of the Conference on Production Systems and Logistics : CPSL 2021 1 (2021) 709–718.  doi:10.15488/11231.
  * [6] V. Samsonov, K. B. Hicham, T. Meisen, Reinforcement learning in manufacturing control: Baselines, challenges and ways forward, Engineering Applications of Artificial Intelligence vol. C (112) (2022).
  * [7] C. W. de Puiseau, R. Meyes, T. Meisen, On reliability of reinforcement learning based production scheduling systems: a comparative survey, Journal of Intelligent Manufacturing 33 (4) (2022) 911–927.  doi:10.1007/s10845-022-01915-2.
  * [8] Sebastian Pol, Schirin Baer, Danielle Turner, Vladimir Samsonov, Tobias Meisen, Global reward design for cooperative agents to achieve flexible production control under real-time constraints, in: Proceedings of the 23rd International Conference on Enterprise Information Systems - Volume 1: ICEIS,, INSTICC, SciTePress, 2021, pp. 515–526.  doi:10.5220/0010455805150526.
  * [9] R. S. Sutton, A. Barto, Reinforcement learning: An introduction, second edition Edition, Adaptive computation and machine learning, The MIT Press, Cambridge, Massachusetts, London, England, 2018.
  * [10] A. Rinciog, C. Mieth, P. M. Scheikl, A. Meyer, Sheet-metal production scheduling using alphago zero, Proceedings of the Conference on Production Systems and Logistics : CPSL 2020 1 (2020) 342–352.  doi:10.15488/9676.
  * [11] M. Monaci, V. Agasucci, G. Grani, An actor-critic algorithm with deep double recurrent agents to solve the job shop scheduling problem (2021).  doi:10.48550/ARXIV.2110.09076.
URL https://arxiv.org/pdf/2110.09076

  * [12] S. Luo, Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning, Applied Soft Computing 91 (2020).  doi:10.1016/j.asoc.2020.106208.
  * [13] Z. Iklassov, D. Medvedev, R. Solozabal, M. Takac, Learning to generalize dispatching rules on the job shop scheduling, Advances in Neural Information Processing Systems 33 (2020) 1621–1632.  doi:10.48550/ARXIV.2206.04423.
URL https://arxiv.org/pdf/2206.04423

  * [14] A. H. Sakr, A. Aboelhassan, S. Yacout, S. Bassetto, Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems, Journal of Intelligent Manufacturing 1 (2021) 1–14.  doi:10.1007/s10845-021-01851-7.
URL https://link.springer.com/article/10.1007/s10845-021-01851-7

  * [15] P. C. Luo, H. Q. Xiong, B. W. Zhang, J. Y. Peng, Z. F. Xiong, Multi-resource constrained dynamic workshop scheduling based on proximal policy optimisation, International journal of production research 60 (19) (2022) 5937–5955.  doi:10.1080/00207543.2021.1975057.
  * [16] P. Tassel, M. Gebser, K. Schekotihin, Job_shop_scheduling_problem_with_reinforcement_learning, GitHub (2021).
URL
https://github.com/dmksjfl/Job_Shop_Scheduling_Problem_with_Reinforcement_Learning

  * [17] L. Zheng, L. Zijun, Y. Dai, X. Li, B. Yuan, Gymjsp, GitHub (2022).
URL https://github.com/yunhui1998/gymjsp

  * [18] Dr-ilyassPHx, Auto-rl-competition: Dynamic job shop scheduling problem challenge, GitHub (2022).
URL https://github.com/Dr-ilyassPHx/Auto-RL-Competition

  * [19] P. Tassel, P. Willms, Jssenv: An openai gym environment for the job shop scheduling problem., GitHub (2022).
URL https://github.com/prosysscience/JSSEnv

  * [20] V. Samsonov, optimization-with-rl-in-manufacturing-control, GitHub (2021).
URL https://github.com/v-samsonov/optimization-with-rl-in-manufacturing-
control

  * [21] tejaswini medi, Rl_scheduling_system, GitHub (2022).
URL https://github.com/tejaswini-medi/RL_scheduling_system

  * [22] D. Venturelli, D. Marchand, G. Rojo, job-shop-scheduling: Determine a schedule for running a set of jobs., GitHub (2015).
URL https://github.com/dwave-examples/job-shop-scheduling

  * [23] samy barrech, Flexible-job-shop-scheduling-problem, GitHub (2018).
URL https://github.com/samy-barrech/Flexible-Job-Shop-Scheduling-Problem

  * [24] C. Zhang, W. Song, Z. Cao, J. Zhan, P. Tan, X. Chi, L2d: Official implementation of paper "learning to dispatch for job shop scheduling via deep reinforcement learning", GitHub (2020).
URL https://github.com/zcaicaros/L2D

  * [25] V. Kumar, Jobschedulingrlenv: Reinforcement learning environment for job scheduling written in python., GitHub (2019).
URL https://github.com/TimeTraveller-San/JobSchedulingRLenv

  * [26] T. van Ekeris, jobshop: Deep reinforcement learning (drl) for jobshop scheduling problems (jsp) - an evaluation framework, GitLab (2020).
URL https://gitlab.com/tvanekeris/jobshop

  * [27] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, N. Dormann, Stable-baselines3: Reliable reinforcement learning implementations, Journal of Machine Learning Research 22 (268) (2021) 1–8.
URL http://jmlr.org/papers/v22/20-1364.html

  * [28] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, Ion Stoica, Rllib: Abstractions for distributed reinforcement learning, International Conference on Machine Learning (2018) 3053–3062.
URL https://proceedings.mlr.press/v80/liang18b.html

  * [29] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, W. Zaremba, Openai gym (2016).  arXiv:arXiv:1606.01540.
  * [30] L. Biewald, Experiment tracking with weights and biases (2020).
URL https://www.wandb.com/

## Current executable software version

Ancillary data table required for sub version of the executable software:
(x.1, x.2 etc.) kindly replace examples in right column with the correct
information about your executables, and leave the left column as it is.

Nr. |  (Executable) software metadata description |  Please fill in this column
---|---|---
S1 |  Current software version |  v0.1.0
S2 |  Permanent link to executables of this version |  https://github.com/tmdt-buw/schlably
S3 |  Legal Software License |  Apache License, 2.0 (Apache-2.0)
S4 |  Computing platforms/Operating Systems |  Python, OpenAI Gym, DRLlib,Weights and Biases
S5 |  Installation requirements & dependencies |  Python 3.10
S6 |  If available, link to user manual - if formally published include a reference to the publication in the reference list |  https://github.com/tmdt-buw/schlably/docs
S7 |  Support email for questions |<EMAIL_ADDRESS>
Table 3: Software metadata (optional)


# Do Acoustic Word Embeddings Capture Phonological Similarity?
An Empirical Study

###### Abstract

Several variants of deep neural networks have been successfully employed for
building parametric models that project variable-duration spoken word segments
onto fixed-size vector representations, or acoustic word embeddings (AWEs).
However, it remains unclear to what degree we can rely on the distance in the
emerging AWE space as an estimate of word-form similarity. In this paper, we
ask: does the distance in the acoustic embedding space correlate with
phonological dissimilarity? To answer this question, we empirically
investigate the performance of supervised approaches for AWEs with different
neural architectures and learning objectives. We train AWE models in
controlled settings for two languages (German and Czech) and evaluate the
embeddings on two tasks: word discrimination and phonological similarity. Our
experiments show that (1) the distance in the embedding space in the best
cases only moderately correlates with phonological distance, and (2) improving
the performance on the word discrimination task does not necessarily yield
models that better reflect word phonological similarity. Our findings
highlight the necessity to rethink the current intrinsic evaluations for AWEs.

Index Terms: acoustic word embeddings, phonological similarity, contrastive
learning, deep neural networks

##  1 Introduction

Spoken language technologies such as spoken term discovery [1, 2, 3] and
query-by-example (QbE) search [4, 5, 6] aim to capture, organize, and
facilitate access to the linguistic content of spoken documents while
abstracting away from speaker- and context-related sources of variability in
speech. To this end, researchers have developed parametric models based on
deep neural networks (DNNs) that project variable-length spoken word segments
onto speaker-invariant vector representations, known as acoustic word
embeddings (AWEs), where acoustic segments of the same word are projected
nearby in space [7, 8, 9, 10, 11]. AWEs, and their underlying vector-space
acoustic models, enable efficient indexing and retrieval of spoken content at
a scale that non-parametric template-based approaches with dynamic programming
[12, 13] have failed to deliver.

Several DNN architectures and learning objectives have been explored in the
literature to build AWEs. State-of-the-art AWE models are trained using either
contrastive objectives [14, 15] or reconstruction objectives [16, 17]. AWEs
have been used in downstream applications including ASR [18] and QbE search
[11, 19]. However, evaluating the utility of AWEs using downstream
applications is expensive and may not be always feasible. Therefore,
researchers have developed an intrinsic evaluation for AWEs based on the
acoustic word discrimination task. In this task, AWE models are evaluated
based on their ability to determine whether or not two acoustic segments
correspond to the same word type [20, 16, 9].

Furthermore, Levin et al. [7] have hypothesized that the distance in the
emergent AWE space can be interpreted as a metric of (perceptual)
dissimilarity between linguistic units (e.g., phones, syllables, words).
However, none of the previous studies has empirically (in)validated this
hypothesis with a rigorous evaluation beyond word discrimination. Although
previous studies have proposed to incorporate the pronunciation distance in
the learning objective [15, 21], the reported performance showed no
improvement on the word discrimination task, while the distance in the AWE
space has shown only a weak correlation with orthographic similarity [15].
These observations, however, are yet to be systematically investigated across
different architectures, objectives, and languages beyond English, which we
aim to address in our study.

Since AWE models have been recently adopted as cognitive models of infant
phonetic learning [22] and cross-language non-native processing [23], we argue
that more effort should be devoted to analyze and understand the emergent
embedding space to make sure it behaves as expected. In this paper, we take a
step in this direction and make the following contributions:

  1. (1)

We train AWE models with identical resources and hyperparameters and examine
the effects of, and the interplay between, the architecture and learning
objective on model performance (§2).

  2. (2)

We analyze the correlation between the distance in the embedding space and
word-form (dis)similarity, which we measure using a phonetically-informed
extension of Levenshtein distance (§3 and §4).

  3. (3)

We empirically show that while AWE models trained with contrastive objectives
outperform other models on the word discrimination task, they are poor at
capturing phonological similarity (§5).

##  2 Acoustic Word Embedding Models

Figure 1: A visual illustration of the learning objectives for the models in
our paper: (a) phone $n$-gram detection: a classification objective based on a
multi-class multi-label classifier, (b) word-to-phones: a sequence-to-sequence
objective based on decoding the phonological sequence from the acoustic word
embedding, and (c) a contrastive siamese objective with a triplet margin loss.

The core component of a neural AWE model is an acoustic encoder, which can be
formally described as a parametric function
$\mathcal{F}_{\bm{\theta}}:\mathcal{A}\xrightarrow[]{}\mathbb{R}^{D}$, where
$\mathcal{A}$ is the (continuous) space of acoustic sequences, $D$ is the
dimensionality of the embedding, and $\bm{\theta}$ are the parameters of the
function. Given an acoustic word segment represented as a temporal sequence of
$T$ spectral vectors
$\overline{\bm{a}}=(\bm{a}_{1},\bm{a}_{2},...,\bm{a}_{T})$, an embedding is
computed as
$\mathbf{x}=\mathcal{F}_{\bm{\theta}}(\overline{\bm{a}})\in\mathbb{R}^{D}$. We
experiment with convolutional and recurrent architectures for the encoder
$\mathcal{F}_{\bm{\theta}}$ and investigate three learning objectives, which
we formally describe below.

###  2.1 Phone $n$-gram Detection Objective

Following previous work [24, 25], we use a classification objective as our
neural baseline (Fig. 1–a). However, we diverge from previous approaches in
the classification target of our network. Instead of predicting the word type,
we train our model to detect phone sequences that are present in each acoustic
segment. For example, consider acoustic segments that correspond to the word
type ``Kirche''. While previous approaches trained the model to predict the
word type as an atomic unit, we first transform the phonetic sequence /k I K ç
@/ into a set of phone bigrams {`#k', `kI', `IK', `Kç', `ç@', `@#'} and
trigrams {`#kI', `kIK', `IKç', `Kç@', `ç@#'}. We then train the model to
predict the presence of each phone $n$-gram. Formally, given a vocabulary of
word types $\mathcal{V}$ where each word is associated with a phonetic
sequence $\bm{\varphi}_{1:N}=(\varphi_{1},\dots,\varphi_{N})$, we obtain a set
of all phone $n$-grams in the training data as
$\Phi=\bigcup_{v\in\mathcal{V}}\mathcal{N}(\bm{\varphi}_{1:N}^{v})$, where
$\mathcal{N}$ is a function that converts a phone sequence into phone
$n$-grams. Each phone $n$-gram in the set $\Phi$ is a prediction target in the
network where a multi-class multi-label classification head is connected to
the AWE to give the prediction output as
${\mathbf{\hat{y}}}=\sigma(\mathbf{xW}+\mathbf{b})\in[0,1]^{|\Phi|}$, where
$\sigma$ is a sigmoid activation function, $\mathbf{W}$ is a weight matrix,
and $\mathbf{b}$ is a bias vector. The objective is to minimize binary cross-
entropy loss as

$\mathcal{L}=-\Big{[}\mathbf{y}\cdot\text{log}(\mathbf{\hat{y}})+(\mathbf{1}-\mathbf{y})\cdot\text{log}(\mathbf{1}-\mathbf{\hat{y}})\Big{]}$
(1)

where $\mathbf{y}\in\\{0,1\\}^{|\Phi|}$ is the ground truth vector and $1$
indicates the presence of a phone $n$-gram in the segment and $0$ indicates
its absence. This objective has an efficiency advantage over type-based
classification approaches since the classification layer is smaller in size,
due to the fact that $|\Phi|<<|\mathcal{V}|$.

###  2.2 Word-To-Phones Objective

Our second learning objective is based on sequence-to-sequence learning
whereby the network is trained as a word-level acoustic model (Fig. 1–b).
Given an acoustic sequence $\overline{\bm{a}}$ and its corresponding phonetic
sequence $\bm{\varphi}_{1:N}$, the acoustic encoder
$\mathcal{F}_{\bm{\theta}}$ is trained to take as input $\overline{\bm{a}}$
and produce an AWE $\mathbf{x}$, which is then fed into a recurrent phonetic
decoder $\mathcal{G}$ whose goal is to generate the corresponding sequence
$\bm{\varphi}_{1:N}$. The intuition of this objective is that phonologically
similar word-forms would usually have overlapping phonetic segments, we thus
expect similar words to end up nearby in the embedding space. The objective is
to minimize a categorical cross-entropy loss at each timestep in the decoder,
which is equivalent to

$\mathcal{L}=-\sum_{t=1}^{N}{\text{log
}\mathbf{P}_{\theta}(\varphi_{t}|\bm{\varphi}_{<t},\mathbf{x})}$ (2)

where $\mathbf{P}_{\theta}$ is the probability of the phone $\varphi_{t}$ at
timestep $t$, conditioned on the previous phone sequence $\bm{\varphi}_{<t}$
and the AWE $\mathbf{x}$.

###  2.3 Contrastive Siamese Objective

The third objective we investigate in this paper is the siamese contrastive
objective [26], which has been extensively explored in the literature with
different underlying architectures [27, 14]. This objective differs from the
first two objectives in two aspects: (1) it explicitly minimizes/maximizes
relative distances between AWEs of the same/different word types, and (2)
models are trained solely on sensory input without symbolic grounding since
each word segment is paired with another segment of the same word type. Given
a matching pair of AWEs $(\mathbf{x}^{a},\mathbf{x}^{+})$, the objective is
then to minimize a triplet margin loss

$\mathcal{L}=\text{max}\Big{[}0,\mu+d(\mathbf{x}^{a},\mathbf{x}^{+})-d(\mathbf{x}^{a},\mathbf{x}^{-})\Big{]}$
(3)

where $\mathbf{x}^{-}$ is an AWE that corresponds to a different word type,
which serves as a negative sample, and
$d:\mathbb{R}^{D}\times\mathbb{R}^{D}\rightarrow[0,1]$ is the cosine distance.
This objective aims to map acoustic segments of the same word type closer in
the embedding space while pushing away segments that correspond to other word
types by a distance defined by the margin hyperparameter $\mu$. We experiment
with two different strategies for choosing the negative sample
$\mathbf{x}^{-}$: (1) we randomly sample a negative AWE from the mini-batch,
and (2) we choose the segment that minimizes the distance
$d(\mathbf{x}^{a},\mathbf{x}^{-})$, which is known as semi-hard negative
sampling [28]. In §5, we show that the negative sampling strategy has a
significant impact on model performance.

##  3 Phonological Similarity Measure

Table 1: Examples of pairwise word distances with Levenshtein distance (LD) and the phonologically weighted LD (PWLD). Word I |  | Word II | LD | PWLD
---|---|---|---|---
Orth. | IPA |  | Orth. | IPA
sicher | /z I ç / |  | Becher | /b E ç / | 2 | 0.263
| Fischer |  /f I S / | 2 | 0.368
| Lichter |  /l I ç t / | 2 | 0.632
| sitzt |  /z I ts t/ | 2 | 0.795

We adopt the phonologically weighted Levenshtein distance (PWLD) as our
measure of phonological distance, or similarity, between different word-forms
[29]. The PWLD metric extends the string-based Levenshtein distance (LD) by
conditioning the cost of phone substitutions on phonetic similarity, which can
be characterized based on the number of distinctive features shared by two
phones. PWLD captures, for example, that the German word sicher /z I ç / is
phonologically more similar to Becher /b E ç / than to sitzt /z I ts t/, even
though the pairwise LD for both pairs is 2 (see Table 1 for more examples).
However, we make three adaptations to the original PWDL to make it suitable
for our study: (1) we represent every phone in our inventory as a discrete,
multi-valued feature vector based on the PHOIBLE [30] feature set, (2) we
compute the substitution cost between phones as the Hamming distance between
their feature vector representations, and (3) we set the deletion and
insertion cost to 0.5 which is equivalent to the maximum possible substitution
cost.

##  4 Evaluation Tasks

###  4.1 Acoustic Word Discrimination

The word discrimination task mainly evaluates the ability of a model to
determine whether or not two given speech segments correspond to the same word
type. We define this task as a segment-level retrieval task [31]: given a
query segment $\overline{\bm{q}}$ and a candidate set of $k$ word segments
$\mathcal{S}={\\{\overline{\bm{s}}_{1},...,\overline{\bm{s}}_{k}\\}}$, the
goal is to rank segments in $\mathcal{S}$ in such a way that those segments
corresponding to the same word type as the query $\overline{\bm{q}}$ are
highly ranked. To this end, a vector-based search index is built by mapping
each word segment in $\mathcal{S}$ into an embedding. Then, the cosine
similarity between the embedding of the query $\overline{\bm{q}}$ and each
embedding in the search index is computed which yields a ranked list, or an
ordering $\mathcal{R}_{\bm{\theta}}$, of segments based on the cosine
similarity score. The average precision metric is used to evaluate the quality
of the ordering for a single query as

$\text{AP}=\frac{1}{|\mathcal{S}_{\bm{q}}|}\sum_{r=1}^{k}{P_{\bm{q}}(r)\times\mathcal{I}_{\bm{q}}(r)}$
(4)

where $\mathcal{S}_{\bm{q}}$ are the segments in $\mathcal{S}$ that correspond
to the query $\overline{\bm{q}}$, $P_{{\bm{q}}}(r)$ is the precision at rank
$r$, and $\mathcal{I}_{\bm{q}}(r)$ is a relevance function such that
$\mathcal{I}_{\bm{q}}(r)=1$ if the segment at rank $r$ corresponds to the same
word type as the query, or $\mathcal{I}_{{\bm{q}}}(r)=0$ otherwise. The
arithmetic average over all AP values in the test set yields the mean average
precision (mAP) metric.

###  4.2 Word Phonological Similarity

To assess whether the emerging AWE space captures word-form similarity, we
propose the word phonological similarity task. We argue that an evaluation
based on phonological similarity will be more insightful to understand the
impact of the model architecture and learning objective on the emergent
embedding space. This evaluation task works as follows: given a query segment
$\overline{\bm{q}}$ and a search index over the candidate set $\mathcal{S}$,
two ranked lists, or orderings, are produced: (1) an ordering
$\mathcal{R}_{\bm{\theta}}$ based on the cosine similarity between the AWEs,
and (2) an ordering $\mathcal{R}_{\bm{\phi}}$ based on phonological similarity
with the PWLD introduced in §3. To measure the degree of agreement between the
two orderings $\mathcal{R}_{\bm{\theta}}$ and $\mathcal{R}_{\bm{\phi}}$, we
use Kendall's $\tau$, which is a measure of rank correlation between two
ordinal variables [32]. For each query segment $\overline{\bm{q}}$ in the test
set, Kendall's $\tau$ is computed as

$\tau=1-\frac{2\times\delta(\mathcal{R}_{\bm{\theta}},\mathcal{R}_{\bm{\phi}})}{0.5\times
k(k-1)}$ (5)

where $\delta(\mathcal{R}_{\bm{\theta}},\mathcal{R}_{\bm{\phi}})$ is the
minimum number of adjacent transpositions needed to bring
$\mathcal{R}_{\bm{\theta}}$ to $\mathcal{R}_{\bm{\phi}}$. Kendall's $\tau$
coefficient takes values between $1.0$ (identical ranks) and $-1.0$ (reverse
ranks), while $0$ indicates no association between the two orderings. Note
that Spearman's correlation is not an appropriate metric for this task, as
opposed to Kendall's $\tau$ which is designed to handle tied rankings that
occur when the PWLD gives the same phonological distance for acoustic segments
corresponding to the same word type.

##  5 Experiments

###  5.1 Experimental Data

Table 2: Word-level statistics of our experimental data. | #segments per split |  #phones ($\mu\pm\text{std}$) |  duration ($\mu\pm\text{std}$)
---|---|---|---
train | valid | test
German | 45886 | 7452 | 9964 |  6.8 $\pm$ 2.2 |  .46 $\pm$ 0.2
Czech | 68596 | 9244 | 11626 |  6.7 $\pm$ 2.5 |  .50 $\pm$ 0.2

The data in our study is drawn from the GlobalPhone speech database for German
and Czech [33]. We choose these two languages due to their predictable
grapheme-to-phoneme (G2P) mapping and the availability of high quality G2P
tools. We use the Montreal Force Aligner [34] to obtain time-aligned spoken
word segments. Each acoustic word segment is parametrized as a sequence of
39-dimensional Mel-Frequency Spectral Coefficients (MFSCs) where frames are
extracted over intervals of 25ms with 10ms overlap. Table 2 shows summary
statistics of our experimental data.

###  5.2 Architectures and Hyperparameters

CNN Acoustic Encoder. We employ a 3-layer temporal convolutional network
(1D-CNN) with 256, 512, and 1024 filters and widths of 16, 32, and 48 for each
layer and keep stride step at 1. Following each convolutional operation, we
apply batch normalization, ReLU non-linearity, and dropout. We apply average
pooling to downsample the representation at the end of the convolution block,
which yields a 1024-dimensional AWE. RNN Acoustic Encoder. We employ a 2-layer
bidirectional Gated Recurrent Unit (BGRU) with a hidden state dimension of
512, which yields a 1024-dimensional AWE. We apply layer-wise dropout with a
probability tuned over $\\{0.0,0.2,0.4\\}$.

Training Details. All models in this study are trained for 100 epochs with a
batch size of 256 using the ADAM optimizer [35] and an initial learning rate
(LR) of 0.001. The LR is reduced by a factor of 0.5 if the mAP on the
validation set does not improve for 10 epochs. The epoch with the best
validation performance during training is used for evaluation on the test set.

Implementation. We build our models using PyTorch [36] and use FAISS [37] for
efficient similarity search. Our code is publicly available on
GitHub111https://github.com/uds-lsv/AWEs_phon_sim.

###  5.3 Experimental Results

Table 3: The results of our experiments for both tasks: word discrimination (mAP) and phonological similarity ($\overline{\tau}$).  Encoder Architecture |  Learning Objective |  | German |  | Czech
---|---|---|---|---|---
|  mAP $\pm$ std |  $\overline{\tau}$ $\pm$ std |  |  mAP $\pm$ std |  $\overline{\tau}$ $\pm$ std
Convolutional (1D-CNN) | PhoneDetect |  |  0.552 $\pm$ 0.33 |  0.043 $\pm$ 0.17 |  |  0.669 $\pm$ 0.31 |  0.080 $\pm$ 0.14
Word2Phones |  |  0.602 $\pm$ 0.33 | 0.127 $\pm$ 0.19 |  |  0.692 $\pm$ 0.31 |  0.150 $\pm$ 0.16
Siamese (w/ Rand Neg) |  |  0.621 $\pm$ 0.33 |  0.126 $\pm$ 0.12 |  |  0.731 $\pm$ 0.30 | 0.176 $\pm$ 0.10
Siamese (w/ Hard Neg) |  | 0.719 $\pm$ 0.32 |  0.074 $\pm$ 0.08 |  | 0.823 $\pm$ 0.27 |  0.093 $\pm$ 0.06
Recurrent (BGRU) | PhoneDetect |  |  0.652 $\pm$ 0.31 | 0.237 $\pm$ 0.14 |  |  0.730 $\pm$ 0.31 |  0.203 $\pm$ 0.11
Word2Phones |  |  0.692 $\pm$ 0.32 |  0.181 $\pm$ 0.15 |  |  0.796 $\pm$ 0.28 | 0.226 $\pm$ 0.13
Siamese (w/ Rand Neg) |  |  0.668 $\pm$ 0.34 |  0.148 $\pm$ 0.08 |  |  0.748 $\pm$ 0.31 |  0.153 $\pm$ 0.06
Siamese (w/ Hard Neg) |  | 0.757 $\pm$ 0.32 |  0.044 $\pm$ 0.05 |  | 0.842 $\pm$ 0.27 |  0.077 $\pm$ 0.04

Our results are summarized in Table 3 for both evaluation tasks. The word
discrimination task is measured by the mAP metric while the word phonological
similarity task is measured by the mean of Kendall's $\tau$ rank correlation
coefficients ($\overline{\tau}$).

Acoustic Word Discrimination. From the mAP values reported in Table 3, one can
make two high-level observations: (1) all models outperform our classifier-
based PhoneDetect baseline, which is trained to detect phone $n$-grams in the
acoustic segment, for both languages and architectures, and (2) recurrent
models outperform their convolutional counterparts, which is consistent with
the findings reported in the literature [10]. However, we also observe that
the performance of the Siamese models, which are explicitly trained to
minimize the cosine distance between segments of the same word type, largely
depends on the negative sampling strategy. For example, in the case of
recurrent Siamese models, we observe a relative improvement in the mAP score
up to 13.32% for German and 12.57% for Czech when applying semi-hard negative
sampling. Note that the Siamese recurrent models did not outperform the
Word2Phones recurrent models when the contrastive negative samples were chosen
randomly from the mini-batch. These findings highlight the importance of
negative sampling in training AWEs with contrastive objectives, which is a
matter that has not been previously investigated to the best of our knowledge.
We conclude that models trained with objectives that explicitly optimize the
distance in the AWE space outperform other models that lack this objective on
word discrimination, especially if the negative samples are chosen with a
challenging criterion.

Word Phonological Similarity. In this evaluation, we observe a positive
correlation between the distance in the embedding space and phonological
distance for all models as indicated by the positive values of the
$\overline{\tau}$ metric. Nevertheless, the correlation seems to be either
weak or moderate in the best cases. Moreover, we observe that both the
learning objective and encoder architecture have a considerable impact on the
extent to which the embedding space captures phonological similarity. For
example, the convolutional PhoneDetect models show some of the lowest
correlation scores (German $\overline{\tau}$ = 0.043 and Czech
$\overline{\tau}$ = 0.080), while their recurrent counterparts show some of
the highest correlation scores (German $\overline{\tau}$ = 0.237 and Czech
$\overline{\tau}$ = 0.203), even their training objective remains unchanged.
These findings indicate that convolutional encoders may behave like shallow
pattern detectors when trained as a classifier while recurrent encoders tend
to preserve the temporal structure of the acoustic input in their
representation. Another observation that we find surprising is the poor
performance of the Siamese models on this task given that they outperform the
other models on word discrimination. Overall, recurrent models which are
trained with symbolic grounding (namely PhoneDetect and Word2Phones) are
better at capturing word-form similarity compared to their convolutional
counterparts on the one hand, and the recurrent Siamese models on the other.

##  6 Discussion and Conclusion

Although the vast majority of previous work has been driven by the engineering
applications of AWEs, there is a growing scientific interest in using deep
neural networks as cognitive models of (human) speech processing [22, 23, 38,
39]. Therefore, we argue that this cognitively motivated direction requires us
to take a closer look at the embedding space and examine the degree to which
we can rely on the emergent distance as an estimate of (perceptual)
dissimilarity between linguistic units. In this paper, we take a step in this
direction and conduct a set of experiments where we keep the training
conditions for each model fixed and systematically study the impact of the
architecture (convolutional and recurrent) and learning objective
(classification, phonological decoding, and contrastive objectives) on the
AWEs' performance using two evaluation tasks: acoustic word discrimination and
word phonological similarity.

Our experiments demonstrate that while contrastive objectives yield AWEs with
strong discriminative performance, they fail to reflect the phonological
distance between word-forms, especially compared to AWEs which are trained
with symbolic grounding (i.e., phone sequences corresponding to words). We
hypothesize that the contrastive objective emphasizes word separability in the
embedding space which hinders the ability of the emerging distance to reflect
word similarity. Moreover, our experiments show a consistent trend with
recurrent models outperforming their convolutional counterparts in both
evaluation tasks, which we attribute to the ability of recurrent DNNs to model
the temporal nature of speech. In conclusion, our experimental findings
highlight the necessity for more diverse evaluation schemes when working with
AWEs to investigate the degree to which they produce human-like errors.
Furthermore, our work can be extended by analyzing the correlation between
embedding distances and human perceptual similarity judgments.

##  7 Acknowledgements

We thank the anonymous reviewers for their constructive comments. This
research is funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation), Project ID 232722074 – SFB 1102.

## References

  * [1] A. Jansen, K. Church, and H. Hermansky, ``Towards spoken term discovery at scale with zero resources,'' in _Proc. Interspeech_ , 2010.
  * [2] A. Jansen and B. Van Durme, ``Efficient spoken term discovery using randomized algorithms,'' in _IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU)_, 2011.
  * [3] A. Anastasopoulos, S. Bansal, D. Chiang, S. Goldwater, and A. Lopez, ``Spoken term discovery for language documentation using translations,'' in _Proc. of the Workshop on Speech-Centric Natural Language Processing_ , 2017\.
  * [4] Y. Zhang and J. R. Glass, ``Unsupervised spoken keyword spotting via segmental DTW on Gaussian posteriorgrams,'' in _IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU)_, 2009.
  * [5] A. Jansen and B. V. Durme, ``Indexing raw acoustic features for scalable zero resource search,'' in _Proc. Interspeech_ , 2012.
  * [6] F. Metze, X. Anguera, E. Barnard, M. Davel, and G. Gravier, ``The spoken web search task at MediaEval 2012,'' in _Proc. ICASSP_ , 2013.
  * [7] K. Levin, K. Henry, A. Jansen, and K. Livescu, ``Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,'' in _IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , 2013.
  * [8] S. Bengio and G. Heigold, ``Word embeddings for speech recognition,'' in _Proc. Interspeech_ , 2014.
  * [9] H. Kamper, W. Wang, and K. Livescu, ``Deep convolutional acoustic word embeddings using word-pair side information,'' in _Proc. ICASSP_ , 2016.
  * [10] S. Settle and K. Livescu, ``Discriminative acoustic word embeddings: Recurrent neural network-based approaches,'' in _IEEE Spoken Language Technology Workshop (SLT)_ , 2016.
  * [11] S. Settle, K. Levin, H. Kamper, and K. Livescu, ``Query-by-example search with discriminative neural acoustic word embeddings,'' in _Proc. Interspeech_ , 2017.
  * [12] G. Heigold, P. Nguyen, M. Weintraub, and V. Vanhoucke, ``Investigations on exemplar-based features for speech recognition towards thousands of hours of unsupervised, noisy data,'' in _Proc. ICASSP_ , 2012.
  * [13] M. De Wachter, M. Matton, K. Demuynck, P. Wambacq, R. Cools, and D. Van Compernolle, ``Template-based continuous speech recognition,'' in _Proc. ICASSP_ , 2007.
  * [14] H. Kamper, W. Wang, and K. Livescu, ``Deep convolutional acoustic word embeddings using word-pair side information,'' in _Proc. ICASSP_ , 2016.
  * [15] W. He, W. Wang, and K. Livescu, ``Multi-view recurrent neural acoustic word embeddings,'' in _Proc. ICLR_ , 2017.
  * [16] H. Kamper, M. Elsner, A. Jansen, and S. Goldwater, ``Unsupervised neural network based feature extraction using weak top-down constraints,'' in _Proc. ICASSP_ , 2015.
  * [17] H. Kamper, ``Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models,'' in _Proc. ICASSP_ , 2019.
  * [18] S. Bengio and G. Heigold, ``Word embeddings for speech recognition,'' in _Proc. Interspeech_ , 2014.
  * [19] Y. Yuan, C.-C. Leung, L. Xie, H. Chen, B. Ma, and H. Li, ``Learning acoustic word embeddings with temporal context for query-by-example speech search,'' in _Proc. Interspeech_ , 2018.
  * [20] M. A. Carlin, S. Thomas, A. Jansen, and H. Hermansky, ``Rapid evaluation of speech representations for spoken term discovery,'' in _Proc. Interspeech_ , 2011.
  * [21] Z. Yang and J. Hirschberg, ``Linguistically-informed training of acoustic word embeddings for low-resource languages.'' in _Proc. Interspeech_ , 2019.
  * [22] Y. Matusevych, T. Schatz, H. Kamper, N. Feldman, and S. Goldwater, ``Evaluating computational models of infant phonetic learning across languages,'' in _Proc. CogSci_ , 2020.
  * [23] Y. Matusevych, H. Kamper, T. Schatz, N. H. Feldman, and S. Goldwater, ``A phonetic model of non-native spoken word processing,'' in _Proc. EACL_ , 2021\.
  * [24] S. Settle, K. Audhkhasi, K. Livescu, and M. Picheny, ``Acoustically grounded word embeddings for improved acoustics-to-word speech recognition,'' in _Proc. ICASSP_ , 2019.
  * [25] H. Kamper, Y. Matusevych, and S. Goldwater, ``Multilingual acoustic word embedding models for processing zero-resource languages,'' in _Proc. ICASSP_ , 2020.
  * [26] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, ``Signature verification using a" siamese" time delay neural network,'' in _Proc. NIPS_ , 1994.
  * [27] S. Settle and K. Livescu, ``Discriminative acoustic word embeddings: Recurrent neural network-based approaches,'' in _Proc. IEEE Spoken Language Technology Workshop (SLT)_ , 2016.
  * [28] A. Jansen, M. Plakal, R. Pandya, D. P. Ellis, S. Hershey, J. Liu, R. C. Moore, and R. A. Saurous, ``Unsupervised learning of semantic audio representations,'' in _Proc. ICASSP_ , 2018.
  * [29] L. Fontan, I. Ferrané, J. Farinas, J. Pinquier, and X. Aumont, ``Using phonologically weighted levenshtein distances for the prediction of microscopic intelligibility,'' in _Proc. Interspeech_ , 2016.
  * [30] S. Moran and D. McCloy, Eds., _PHOIBLE 2.0_. Jena: Max Planck Institute for the Science of Human History, 2019\. [Online]. Available: https://phoible.org/
  * [31] M. Müller, _Fundamentals of music processing: Audio, analysis, algorithms, applications_. Springer, 2015\.
  * [32] M. Lapata, ``Automatic evaluation of information ordering: Kendall's Tau,'' _Computational Linguistics_ , vol. 32, no. 4, pp. 471–484, 2006.
  * [33] T. Schultz, N. T. Vu, and T. Schlippe, ``GlobalPhone: A multilingual text and speech database in 20 languages,'' in _Proc. ICASSP_ , 2013.
  * [34] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal Forced Aligner: Trainable text-speech alignment using Kaldi.'' in _Interspeech_ , 2017.
  * [35] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' in _Proc. ICLR_ , 2015.
  * [36] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , ``Pytorch: An imperative style, high-performance deep learning library,'' in _Proc. NeuRIPS_ , 2019\.
  * [37] J. Johnson, M. Douze, and H. Jégou, ``Billion-scale similarity search with GPUs,'' _IEEE Transactions on Big Data_ , 2017.
  * [38] J. S. Magnuson, H. You, S. Luthra, M. Li, H. Nam, M. Escabi, K. Brown, P. D. Allopenna, R. M. Theodore, N. Monto _et al._ , ``Earshot: A minimal neural network model of incremental human speech recognition,'' _Cognitive science_ , vol. 44, no. 4, p. e12823, 2020.
  * [39] A. Mayn, B. M. Abdullah, and D. Klakow, ``Familiar words but strange voices: Modelling the influence of speech variability on word recognition,'' in _Proc. EACL, Student Research Workshop_ , 2021.