# Phase Separation in Binary Fluid Mixture with Quenched Disorder

Rounak Bhattacharyya and Bhaskar Sen Gupta  Department of Physics, School of
Advanced Sciences, Vellore Institute of Technology, Vellore, Tamil Nadu -
632014, India

###### Abstract

Quenched or frozen-in, structural disorder is ubiquitous in real experimental
systems. Much of the progress is achieved in understanding the phase
separation of such systems using diffusion-driven coarsening in Ising model
with quenched disorder. But there is a paucity of research in the phase-
separation kinetics in fluids with quenched disorder. In this letter, we
present results from a detailed Molecular dynamics (MD) simulation the effects
of randomly placed localized impurities on the phase separating kinetics of
binary fluid mixture. Our system resembles the fluid imbibed into a porous
medium. We observe a dramatic slowing down in the pattern formation with
increasing pin particle concentration. The domain growth follows the power-law
with a disorder-dependent exponent. Due to the energetically favorable
positions, the domain boundary roughens which modifies the correlation
function and structure factor to a non-Porod behavior. The correlation
function and structure factor provide clear evidence that the
superuniversality does not hold in our system.

Introduction.— Research attention in statistical physics has developed rapidly
to understand the out of equilibrium phenomena over the past few decades. In
this context, there is a paramount interest to investigate the evolution of a
phase separating multi-component mixture with time, due to its application in
diverse fields, e.g., magnetic materials, colloidal mixtures, polymer
solutions, glasses, metallic alloys, binary liquids Fisher ; Stanley ; Binder
; Jones ; Bray . When a system of homogeneous multicomponent mixture is
rendered thermodynamically unstable by a swift quench inside the miscibility
gap, domains with different ordered phases form and expand with time until the
system attains a local equilibrium. The domain coarsens due to the
organization of matter, mostly dominated by diffusion and can be seen in
almost all phase separating systems Binder1 ; Siggia ; Furukawa ; Miguel ;
Tanaka ; Beysens ; Tanaka1 ; Kendon ; Puri ; Dutt ; Laradji ; Thakre ; Ahmad .
A single time dependent characteristic length $\ell(t)$ usually characterizes
the domain growth pattern or domain morphology Bray . The same can be obtained
by calculating two point equal time correlation function
$\mathrm{C}_{\psi\psi}(\vec{r},t)$, where $\vec{r}$ is the distance between
two spatial points and $t$ is the time after quench. So far, it is well
demonstrated that pattern formation or coarsening of domains is a scaling
phenomenon and exhibits the form ${C}_{\psi\psi}(\vec{r},t)=g[r/\ell(t)]$
Binder2  where $g(x)$ is the scaling function. The average domain size
$\ell(t)$ follows the power law: $\ell(t)\sim t^{\alpha}$, $\alpha$ defines
the growth exponent. The value of the exponent depends on the apposite
coarsening mechanism which drives phase separation.
Coarsening mechanism depends on the character of the system. For instance, in
solid-solid mixtures (e.g., metallic alloys), diffusion dominates the domain
growth whereas hydrodynamic effects contribute significantly in fluid-fluid
mixtures. For the earlier the growth exponent $\alpha=1/3$ which is attributed
to Lifshitz-Slyozov (LS) law Bray ; Binder3 . This diffusive regime is very
transient in fluid-fluid mixtures and we experience a prompt crossover from
diffusive to the hydrodynamic regime. Here $\alpha$ takes two distinct values,
viz., $\alpha=1$ in viscous hydrodynamic regime followed by the inertial
hydrodynamic regime with $\alpha=2/3$. The above values of growth exponent are
universal and pertinent to pure and isotropic systems Siggia ; Furukawa .
However, almost all real systems have local randomly frozen heterogeneities
known as quenched disorder. It acts as a background random potential for the
fluctuating degrees of freedom. The ever-present random disorder strongly
influences the kinetics of phase separation and the domain morphology.
Numerous studies have been performed so far to understand the diffusion-driven
coarsening in Ising systems with quenched disorder Puri1 ; Huse ; Grest ;
Srolovitz ; Puri2 ; Bray2 ; Rao ; Paul ; Henkel ; Aron . But the phase-
separation kinetics in fluids with quenched disorder is still in its infancy.
In this letter, we undertake an extensive numerical study of domain growth
dynamics of an immiscible symmetric binary fluid mixture in the presence of
quenched disorder. The disorder is introduced by randomly positioned pinning
of particles in the system. Our system resembles phase separation of fluid in
random porous materials Brochard ; Gennes1 ; Maher ; Goh ; Wiltzius  which are
of great technological interest in the industry and oil recovery process.
Numerical simulations.— For our present analysis we perform a three-
dimensional (3D) molecular dynamics (MD) simulation on a binary liquid in the
NVT ensemble. We consider a $50:50$ mixture of A and B particles at high
density $\rho=N/V=1$, where N and V represent the number of particles and
volume of the system respectively. The two species interact via Lennard-Jones
(LJ) potential

$U_{\alpha\beta}(\vec{r})=4\epsilon_{\alpha\beta}\Big{[}\Big{(}\frac{\sigma_{\alpha\beta}}{\vec{r}}\Big{)}^{12}-\Big{(}\frac{\sigma_{\alpha\beta}}{\vec{r}}\Big{)}^{6}\Big{]}$
(1)

where $\ \vec{r}=|\vec{r_{i}}-\vec{r_{j}}|$ and $\alpha,\beta\in\rm{A,B}$. To
ensure energetically favorable phase separation, the parameters in the LJ
potential are chosen as follows: $\sigma_{AA}=\sigma_{BB}=\sigma_{AB}=1.0$ and
$\epsilon_{AA}=\epsilon_{BB}=1.0,\epsilon_{AB}=0.5$. The choice of our
interaction strength corresponds to the critical temperature $T_{c}=1.42$,
outlying the possible liquid-solid and gas-liquid transition point Das . The
temperature is measured in units of $\epsilon/k_{B}$ , where $k_{B}$ is
Boltzmann’s constant. Length is measured in units of $\sigma$. For simplicity
we set the mass $m_{0}$ of all the particles and $k_{B}$ equal to unity. For
the sake of computational efficiency, the interaction potential is truncated
to zero at $r_{c}=2.5\sigma$. Periodic boundary condition is applied in all
three directions.
To incorporate the effect of quenched disorder, we introduce randomly placed
pinned (localized) particles in the system. These particles do not participate
in the dynamics and stay frozen in their respective positions. The mobile
particles interact with the pinned particles via the LJ potential. The density
of the pinned particle is chosen such that $\rho_{\rm{P}}\ll\rho_{0}$ where
$\rho_{\rm{P}}$ and $\rho_{0}$ are the densities of the pinned and not pinned
particles respectively with $\rho=\rho_{\rm{P}}+\rho_{0}$. Our simulation is
accomplished with five different pinning concentrations. We performed the MD
simulation using the Velocity Verlet algorithm Verlet  over a total of 262144
($=64^{3}$) particles for each pinning percentage. We begin our simulation by
preparing a well equilibrated homogeneous mixture at high temperature $T=10.0$
followed by a quench to $T=0.77T_{c}$ at $t=0$. Here the reduced unit of time
is taken as $\sigma\sqrt{m/\epsilon}$. Temperature is controlled by using the
Nose-Hoover thermostat (NHT) which preserves the hydrodynamic effect Nose .
Finally, the system is allowed to evolve to the thermodynamically favored
state until the complete phase separation is achieved. The ensemble average of
all the statistical quantities is obtained from 10 independent runs at
$0.77T_{c}$ starting from completely different initial configurations.
To characterize the domain morphology and study the domain growth we introduce
the two point equal time correlation function $C_{\psi\psi}(\vec{r},t)$ as
follows:

$C_{\psi\psi}(\vec{r},t)=\langle\psi(0,t)\psi(\vec{r},t)\rangle/\langle\psi(0,t)\rangle^{2}$
(2)

where the order parameter $\psi(\vec{r},t)$ is obtained as follows: we compute
the local density difference $\delta\rho=\rho_{A}-\rho_{B}$ between the two
species A and B, calculated over a box of size $2\sigma^{3}$ located at
$\vec{r}$. The $\psi(\vec{r},t)$ is assigned a value $+1$ when $\delta\rho>0$,
and $-1$ otherwise. The angular brackets indicate the statistical averaging.
We have also computed structure factor $S(\vec{k},t)$ by taking the Fourier
transform of the correlation function given by

$S(\vec{k},t)=\int d\vec{r}\hskip 2.84544ptexp(i\vec{k}.\vec{r})\hskip
2.84544ptC_{\psi\psi}(\vec{r},t)$ (3)

to investigate the structural changes of the system. Finally for the isotropic
system, spherically averaged $C_{\psi\psi}(r,t)$ and $S(k,t)$ are computed.
Numerical Results.— The effect of impurity on the phase separation dynamics is
depicted in Fig. 1. Here we show the configurations obtained from our
simulation for three different $\rho_{\rm{P}}$ at time $t=2000$. It is
observed that the kinetics of ordering slows down drastically with
$\rho_{\rm{P}}$. The pinned particles localize the domain walls in
energetically favorable positions. However, this is overcome by the thermal
energy due to sufficiently high temperature. It is worth noting in Fig. 1 that
the domain boundary becomes rough with impurity. This is evident from the 2D
cross-sectional plot of the configurations. The implication of the modified
domain boundary will be discussed shortly in terms of the correlation function
and structure factor.

Figure 1: Typical snapshots of our system for the impurity concentrations
$\rho_{\rm{P}}=$0.0, 1.0 and 2.0 at time $t=2000$ (LJ unit ) are shown in the
upper panel (left to right) at $T=0.77T_{c}$. The A, B and pinned particles
are marked as red, blue and yellow respectively. A two dimensional cross
sectional area of the configurations in the upper panel with one type of
particles A is shown in the lower panel.

In Fig. 2 we show the scaling plot for $C_{\psi\psi}(r,t)$ vs $r/\ell(t)$,
where $\ell(t)$ is the average domain size. There are quiet a few ways to
quantify the $\ell(t)$, e.g, first zero crossing of $C_{\psi\psi}(r,t)$, half-
crossing of $C_{\psi\psi}(r,t)$, inverse of the first moment of $S(k,t)$. We
have used the first method to calculate $\ell(t)$. An excellent data collapse
is observed for different times when the same disorder concentration is
considered. This suggests that even in the presence of disorder (fixed
$\rho_{\rm{P}}$), the system belongs to the same dynamical universality class
Binder3 . However, the correlation function gets modified when the
$\rho_{\rm{P}}$ changes and they do not overlap with each other when plotted
at a fixed time $t$. This is shown in Fig. 2. For the pure system the
$C_{\psi\psi}(r,t)$ shows a linear behavior at small $r$ following the Porod
law. However, in the presence of impurity, a nonlinear or cusp nature is
observed, which can be attributed to the scattering from the rough domain
boundary Gaurav . Therefore, the quench disorder results in breaking down the
Porod law in the correlation function Shaista .

Figure 2: Upper panel: the scalling plots for $C_{\psi\psi}(r,t)$ vs
$r/\ell(t)$ are shown for the pure (left) and disordered system (right). Lower
panel: we show the comparison for the same between the pure and disordered
systems.

The evolution morphology is best realized in terms of the domain size $\ell$.
In Fig. 3 we show the time dependence of $\ell(t)$ for different impurity
concentration. Our simulation is able to access the viscous hydrodynamic
regime after an initial transient period. However, the growth rate slows down
and the total time taken for the system to be completely phase-separated
increases significantly with increasing impurity. This is consistent with the
observation in Fig. 1.

Figure 3: The time evolution of the average domain size $\ell(t)$ is shown for
different $\rho_{\rm{P}}$ in the log-log scale.

To gain more insight into the disorder-dependent growth dynamics, we compute
the instantaneous slope of $\ell(t)$ with time. This is best appraised by the
instantaneous dynamical exponent $z_{\rm{eff}}=1/\alpha_{\rm{eff}}$ where
$\alpha_{\rm{eff}}=d(ln~{}\ell(t))/d(ln~{}t)$. The result is shown in Fig. 4.
In the absence of impurity, the exponent $\alpha_{\rm{eff}}\sim 1$ in the
viscous hydrodynamic regime. This is in accordance with our expectation Bray .
In the presence of quenched disorder, the $z_{\rm{eff}}$ vs $t$ curve exhibits
a flat behavior for the intermediate and long time, indicating a power-law
domain growth ($\bar{z}$) throughout the phase separation process. However,
the exponent decreases gradually with $\rho_{\rm{P}}$ due to the slowing down
of ordering. The variation of $\bar{z}$ with $\rho_{\rm{P}}$ yields
$\bar{z}\sim\rho_{\rm{P}}^{0.25}$ as shown in Fig. 4.

Figure 4: (a) The instantaneous dynamical exponent $z_{\rm{eff}}$ vs the
average domain size $\ell(t)$ is shown for different $\rho_{\rm{P}}$. (b)
$z_{\rm{eff}}$ vs $t$ (points) plotted on a log-log scale. The solid line is
the power law fitting with exponent 0.25 (see text).

In the presence of disorder, the relaxation mechanism changes from purely
energy lowering to thermally activated process, due to the creation of energy
barriers. The presence of the energy barrier slows down the relaxation
process. An important issue in this regard is the so-called superuniversality
(SU), i.e, whether the disorder-dependent spatial autocorrelation function
scales to a master curve when the length is rescaled with respect to the
domain size $\ell(t)$ Lai ; Corberi . From Fig. 2 it is conspicuous that our
system does not fall into the SU class. We also observe an algebraic domain
growth throughout the phase separation process with a disorder-dependent
exponent. This implies that the barrier size does not depend on the domain
size for a given $\rho_{\rm{P}}$.
In Fig. 5, we show the scaled structure factor $S(k,t)\ell^{-3}$ vs $\ell k$
plot for different $\rho_{\rm{P}}$ in log-log scale. Although, the decaying
part of the tail for the pure system shows the Porod law behavior $S(k,t)\sim
k^{-(d+1)}$ Binder3 , the systems with immobile impurities shows a non-Porod
behavior, $S(k,t)\sim k^{-(d+\theta)}$, where $\theta\simeq 0.5$. This value
is very close to the results observed in the 3D random field ising model
Gaurav . This substantiates the roughening of the domain interfaces with the
increasing pinned particles and the violation of SU.

Figure 5: The scaled structure factor $S(k)\ell^{-3}$ vs $\ell k$ is plot
(points) for three different $\rho_{\rm{P}}$ in the log-log scale. The
straight lines are the guideline for the Porod law $S(k)\sim k^{-d+1}$ and the
non-Porod behavior $S(k)\sim k^{-d+\delta}$ where $\delta=0.5$ (see text).

Summary and Discussion.—In summary, we have examined the kinetics of phase
separation in a binary liquid in the presence of quench disorder. The disorder
was obtained by pinning of particles at random positions in the system. We
have observed that the dynamics of ordering slows down dramatically with the
concentration of localized particles. The spatial correlation function scaled
with domain size shows disorder dependence and non-Porod behavior which can be
attributed to the roughening of domain boundaries. The pure system clearly
exhibits power-law domain growth with exponent unity in the viscous
hydrodynamic regime. In the presence of disorder, the nature of growth still
remains algebraic but the exponent decreases with impurity concentration. The
disorder localizes the domain wall and creates energy barrier which is
overcome by the thermally activation process. However, the algebraic domain
growth conforms to the domain size-independent energy barrier throughout the
phase separation process. We have also examined the structure factor for our
system. The decaying part of the structure factor gets modified in the
presence of disorder and shows non-Porod tail. Both the spatial correlation
function and the structure factor do not obey superuniversality. We believe
that our model and the results presented here offer a method to control the
phase separation dynamics in real systems by adding quenched disorder
appropriately. Studies following this line of thought will be presented
elsewhere.
Acknowledgement.—The authors acknowledge useful discussions with J. Midya.
B.S.G. acknowledges Science and Engineering Research Board (SERB), Dept. of
Science and Technology (DST), Govt. of India (no. SRG/2019/001923) for
financial support.

## References

  * (1) M.E. Fisher, Rep. Prog. Phys. 30, 615–730 (1967).
  * (2) H.E. Stanley, Introduction to Phase Transitions and Critical Phenomena (Oxford University Press, 1971).
  * (3) K. Binder, Rep. Prog. Phys. 50, 783–859 (1987).
  * (4) R.A.L. Jones, Soft Condensed Matter (Oxford University Press, 2002).
  * (5) A.J. Bray, Adv. Phys. 51, 481–587 (2002).
  * (6) K. Binder, Phys. Rev. B 15, 4425–4447 (1977).
  * (7) E. D. Siggia, Phys. Rev. A 20, 595 (1979).
  * (8) H. Furukawa, Phys. Rev. A 31, 1103–1108 (1985).
  * (9) M. San Miguel, Phys. Rev. A 31, 1001–1005 (1985).
  * (10) H. Tanaka, J. Chem. Phys. 103, 2361 (1995).
  * (11) D. Beysens, Y. Garrabos, D. Chatain, Europhys. Lett. 86, 16003 (2009).
  * (12) S. Tanaka, Y. Kubo, Y. Yokoyama, A. Toda, K. Taguchi, H. Kajioka, J. Chem. Phys. 135, 234503 (2011).
  * (13) V.M. Kendon, M.E. Cates, I. Pagonabarraga, J.C. Desplat, P. Blandon, J. Fluid Mech. 440, 147–203 (2001).
  * (14) S. Puri, B. Dünweg, Phys. Rev. A 45, R6977–R6980 (1992).
  * (15) C. Datt, S.P. Thampi, R. Govindarajan, Phys. Rev. E 91, 010101(R) (2015).
  * (16) M. Laradji, S. Toxvaerd, O.G. Mountain, Phys. Rev. Lett. 77, 2253–2256 (1996).
  * (17) A.K. Thakre, W.K. den Ohe, W.J. Briels, Phys. Rev. E 77, 011503 (2008).
  * (18) S. Ahmad, S.K. Das, S. Puri, Phys. Rev. E 82, 040107 (2010).
  * (19) K. Binder and D. Stauffer, Phys. Rev. Lett. 33, 1006 (1974); Z. Phys. B 24, 407 (1976).
  * (20) K. Binder, in Phase Transformation of Materials, edited by R. W. Cahn, P. Haasen, and E. J. Kramer, Material Science and Technology (VCH, Weinheim, 1991), Vol. 5, p. 405; Kinetics of Phase Transitions, edited by S. Puri and V. Wadhawan (CRC Press, Boca Raton, 2009).
  * (21) S. Puri, Phase Transitions 77, 469 (2004).
  * (22) D. A. Huse and C. L. Henley, Phys. Rev. Lett. 54, 2708 (1985).
  * (23) G. S. Grest and D. J. Srolovitz, Phys. Rev. B 32, 3014 (1985).
  * (24) D. J. Srolovitz and G. S. Grest, ibid. 32, 3021 (1985).
  * (25) S. Puri, D. Chowdhury and N. Parekh, J. Phys. A 24, L1087 (1991).
  * (26) A. J. Bray and K. Humayun, J. Phys. A 24, L1185 (1991).
  * (27) M. Rao and A. Chakrabarti, Phys. Rev. Lett. 71, 3501 (1993).
  * (28) R. Paul, S. Puri and H. Rieger, Europhys. Lett. 68, 881 (2004); Phys. Rev. E 71, 061109 (2005).
  * (29) M. Henkel and M. Pleimling, Europhys. Lett. 76, 561 (2006); Phys. Rev. B 78, 224419 (2008).
  * (30) C. Aron, C. Chamon, L. F. Cugliandolo and M. Picco, J. Stat. Mech., P05016 (2008); L. F. Cugliandolo, Physica A 389, 4360 (2010).
  * (31) F. Brochard and P. G. de Gennes, J. Phys. Lett. (Paris) 44, 785 (1983).
  * (32) P. G. de Gennes, J. Phys. Chem. 88, 6469 (1984).
  * (33) J. V. Maher, W. I. Goldburg, D. W. Pohl and M. Lanz, Phys. Rev. Lett. 53, 60 (1984).
  * (34) M. C. Goh, W. I. Goldburg and C. M. Knobler, Phys. Rev. Lett. 58, 1008 (1987).
  * (35) P. Wiltzius, S. B. Dierker and B. S. Dennis, Phys. Rev. Lett. 62, 804 (1989).
  * (36) S. K. Das, M. E. Fisher, J. V. Sengers, J. Horbach and K. Binder, Phys. Rev. Lett. 97, 025702 (2006).
  * (37) L. Verlet, Phys. Rev. 98, 159 (1967).
  * (38) D. Frenkel and B. Smit, Understanding Molecular Simulations: From Algorithms to Applications (Academic Press, San Diego, 2002).
  * (39) G. P. Shrivastav, S. Krishnamoorthy, V. Banerjee and S. Puri, Europhys Lett. 96, 36003 (2011).
  * (40) Shaista Ahmad, Sanjay Puri and Subir K. Das, Phys. Rev. E 90, 040302(R) (2014).
  * (41) Z. W. Lai, G. F. Mazenko and O. T. Valls, Phys. Rev. B 37, 9481 (1988).
  * (42) F. Corberi, E. Lippiello, A. Mukherjee, S. Puri and M. Zannetti, Phys. Rev. E 85, 021141 (2012).


# Idiolect: A Reconfigurable Voice Coding Assistant

Breandan Considine  McGill University
<EMAIL_ADDRESS>Nicholas Albion  Independent Developer
<EMAIL_ADDRESS>Xujie Si  University of Toronto
<EMAIL_ADDRESS>
###### Abstract

This paper presents Idiolect, an open source
111https://github.com/OpenASR/idiolect IDE plugin for voice coding and a novel
approach to building bots that allows for users to define custom commands on-
the-fly. Unlike traditional chatbots, Idiolect does not pretend to be an
omniscient virtual assistant but rather a reconfigurable voice programming
system that empowers users to create their own commands and actions
dynamically, without rebuilding or restarting the application. We offer an
experience report describing the tool itself, illustrate some example use
cases, and reflect on several lessons learned during the tool’s development.

###### Index Terms:

speech recognition, voice programming, bots

##  I Introduction

Humans are able to quickly learn new words and phrases, and apply them in a
variety of contexts. Chatbots, however, are often limited to a set of static
commands and phrases. This creates a frustrating experience for users, who
struggle to express their intent, as well as bot developers, who must
anticipate user intent and make new capabilities discoverable. This rigidity
is a common source of misalignment between user intent, author expectations
and bot capabilities.

Voice coding allows users to quickly dictate new commands and behaviors. For
example, one might say, “whenever I say open sesame, do the following action”,
thereafter, the system will perform the desired action when so instructed. Or,
“whenever I say redo thrice, repeat the last action three times”. Or, e.g.,
invoke a function in a scripting language, open a file, or perform other
manually repetitive chores.

Idiolect provides a default lexicon of phrases, but does not impose them upon
end-users. Instead, users may override the defaults with custom voice commands
on-the-fly, which are incorporated into the lexicon without delay. By shifting
the burden of adaptation from the user to the system, this frees our users to
express their intent in a more natural manner.

Primarily, Idiolect observes the following design principles: be (1) natural
to use, (2) easy to configure, (3) as unobtrusive as possible. We believe that
these principles are important for a system intended to be used by developers,
who are busy people and capable of configuring the system themselves. We also
support developers with visual and motor impairments, who may have difficulty
typing, or prefer to use a voice interface.

In the following paper, we describe Idiolect, an IDE plugin originally
developed at JetBrains in 2015 and recently updated to provide many new
features, such as deep speech recognition and dynamically-reconfigurable voice
commands, letting users verbally or programmatically express their desired
intent.

##  II Prior Work

Mary Shaw, during her 2022 SPLASH keynote [1] called for programming languages
to address the needs of “vernacular developers”. Jin Guo has also talked about
the need for programming in “ordinary people’s language”. We take their
proposals quite literally to mean that computers should be able to interpret
spoken programs, and not just written ones.

Although related, voice assistants like Siri and Alexa are not configurable
nor intended for programming. In this regard, we believe our tool is more akin
to a programmable bot with audio I/O than a dedicated voice assistant. Spoken
programs are also closely related to natural language programming, an idea
once scorned [2], but now becoming increasingly plausible.

Early attempts to build voice programming systems can be traced back at least
two decades to Leopold and Amber’s [3] (1997) work on keyboardless
programming, later revisited by Arnold and Goldthwaite [4] (2001), Begel and
Graham [5] (2005) and others. These systems allow users to write code by
speaking into a microphone, however early voice programming systems were
limited by a small vocabulary, and unlike our work, do not consider IDE
integration or reconfigurability.

Another stream of work has explored teachable voice assistants and user-
defined phrases (Chkroun & Azaria [6]). Though similar to our own, their work
targets general-purpose voice user interfaces and does not directly consider
voice programming. It also predates most of the recent progress on large
language modeling, which we consider to be a transformative enabling
technology for programming by voice.

##  III Architecture

Today, automatic speech recognition (ASR), the translation of an audio
waveform containing speech to text, is essentially a solved problem – one of
the many pretrained ASR models would work well enough for our purposes.
However, we also require realtime offline speech recognition capabilities
built on an open source deep speech pipeline, which has only recently become
possible for users running on commodity hardware.

Idiolect integrates with Vosk, a state-of-the-art deep speech system with
realtime models for various languages, which provides an open source Java API.
This allows us to provide a high-quality speech recognition experience without
requiring users to upload their personal data to a cloud service.

Users may optionally configure a built-in TTS voice from the host operating
system and a cloud-based speech recognition or synthesis service, with the
caveat that web speech requires uninterrupted internet connectivity and
introduces an additional 300-500ms of overhead latency depending on the user’s
proximity to the datacenter and other load factors.

Figure 1: Idiolect has four components: a speech and intent recognizer,
language synthesizer, and TTS service, which together faciliate a dialog with
the IDE.

Once a spoken utterance is decoded as text, Idiolect must determine the
relevant actions and entities needed to fulfill the user’s intent.
Furthermore, it may need to consider the IDE context, i.e., the current editor
state and command history, to resolve potentially ambiguous commands. For
example, the command, “open plugin menu” could refer to multiple different
menus, depending on when and how it was invoked.

The IntelliJ Platform has over $10^{3}$ possible actions. These actions are
bound to keyboard shortcuts, menu items, and toolbar buttons. The user can
also bind a voice command directly to an action, presuming the user already
knows the action’s identifier. Idiolect’s default grammar was manually curated
from the IDE action list, using the CamelCase identifier to generate a
suitable description for intent recognition.

In keeping with the principle of configurability, we allow users to define
custom grammars and bind utterances to actions using a context-free grammar.
By providing an extensible DSL, users can bind their own command patterns
using either a simple configuration file, or programmatically via the plugin
API to handle more complex usage scenarios.

Idiolect dispatches utterances to a series of recognizers using a turn-based
priority queue, in which each recognizer is given a single turn to match or
pass on each utterance. Once a command is recognized, the command is consumed
and dispatched to no subsequent recognizer thereafter, which prevents a single
command from triggering multiple actions.

The plugin first attempts to resolve an utterance using an exact lexical match
against a finite language, i.e., a lexicon of predefined commands. Highest
priority are those which control the plugin itself, enabling and disabling
speech recognition.

User-defined commands are the next highest priority. These can be matched
lexically, or more generally as a sentence in a context-free language.
Nonterminal parameters allow one to define more complex instructions, such as
“open the ⟨filename⟩[in ⟨project_name⟩]” or “jump to the ⟨nth⟩line”.

The recognizer of last resort is a large language model. Open-ended commands
which are unrecognizable to all of the previous methods are dispatched to a
service that matches the intent against a set of available actions using a
prompt, e.g., “Out of these actions: …, which one most likely fulfills the
command …?”, whose response is then dynamically invoked.

##  IV Barriers and Pathways to Usability

In our experience as plugin users and maintainers, usability challenges
typically arise in a few key areas. In the following section, we will discuss
some of those obstacles encountered while developing Idiolect and our efforts
to overcome them.

###  IV-A User Onboarding

Several users reported confusion when first installing our plugin. To address
this issue, we added a wizard that guides users through installation. Upon
first installing the plugin, a user is greeted and prompted to download the
Vosk model for recognizing their preferred natural language, defaulting to the
system locale. Once unpacked, the model is stored in the plugin configuration
directory,  /.idiolect. The user is next prompted to configure the properties
file, and bind a few voice commands. This is a one-time process, and users may
reconfigure the plugin via the settings menu at any time.

###  IV-B Plugin Observability

Failures arise in many stages of the intent recognition pipeline. Short of
fixing the error directly, often the best way to address failure is by
improving observability, so that users can diagnose an error themselves and
avoid it in the future.

Recognition failure is a common issue, often caused by transcription errors in
the ASR model, due to, e.g., noise in the audio signal, stopwords, or poor
recognition accuracy. To address this issue, we add a visual cue at the corner
of the IDE that reports the phrase transcribed in realtime, as well as the
action (if any) that was triggered on intent recognition.

###  IV-C Command Discoverability

We draw a distinction between capability discovery and intent recognition.
Users previously familiar with the IDE capabilities may wish to invoke or bind
a specific action directly, but may not know the specific action identifier.
The first and foremost way to improve discovery is through documentation,
however keeping documentation in sync with capabilities can be challenging,
and users are uninclined to read verbose documentation. To address this issue,
we autogenerate documentation for each action by preprocessing the action list
and annotating each action with a natural language description from the
IntelliJ Platform source code.

Another common scenario is when the user is unfamiliar with the IDE
capabilities, and the action they wish to perform is not supported by the IDE
directly. User awareness of IDE functionality is outside the scope of this
plugin, however, certain actions users may intend to perform are compound
actions or have no associated binding. For example “delete the method named
⟨foo⟩” requires first resolving the method, selecting the body of the text and
deleting it. Such actions can be defined, but require implementing a
handcrafted recognition handler, and are generally quite brittle. We hope to
improve scripting support for such actions in the future.

###  IV-D API Extensibility

Idiolect tries to anticipate users actions and expose default action bindings.
Some functionality, however, is best left implemeneted by downstream
developers, who are better-equipped to understand their users’ needs. In
addition to end-user configurability, Idiolect is designed to be extensible by
other IntelliJ Platform plugins, which can define additional commands,
recognition handlers and custom actions. We offer a simple message-passing API
for plugins to communicate with Idiolect, and a DSL for defining custom
grammars.

###  IV-E Error Recovery

Another common usability barrier is when transcription is accurate, but the
utterance does not correspond to an actionable command, possibly due to
stopwords, verbal fillers or extraneous text which cannot be directly parsed.
In short, if a given phrase, e.g., “open uh foo java” is received, we attempt
to repair the utterance. At the user’s discretion, or in case of ambiguity
when the phrase has multiple plausible alternatives, we can either visually or
verbally prompt the user to choose from a set of actionable phrases, e.g.,
“Did you mean ⟨a⟩open file foo.java, ⟨b⟩open folder foo/java, or ⟨c⟩something
else?”

To correct recognition errors, we apply Considine et al.’s [7] work on
Tidyparse, which supports recognition and parsing of context-free and mildly
context-sensitive grammars, and computing language edit distance. Tidyparse
implements a novel approach to error correction based on the theory of
context-free language reachability, conjunctive grammars and Levenshtein
automata. We use a SAT solver to find the smallest edit transforming a string
outside the language to a string inside the language. Only when the utterance
is one or two tokens away from a known command do we attempt a repair.

VoskAPI is also capable of returning a list of alternate utterances alongside
a confidence score for each. We use this list to determine if any recognized
utterance is sufficiently close to an actionable command in the Idiolect
grammar.

###  IV-F Build Reproducibility

One of the challenges of managing a multiplatform audio project is ensuring
reproducible builds. The IntelliJ Platform does not allow shipping OS-specific
plugins, which requires building a single artifact. In the past, updates would
result in breaking changes on Windows, macOS, or Linux, which are difficult to
debug or test from a development environment and only identified by user
acceptance testing on those platforms.

To streamline the development process and mitigate the risk of platform-
specific regressions in production, we set up an automated pipeline for
testing and deploying the plugin. We use GitHub Actions to build the plugin
for each release, and when a new commit is tagged with a release number and
merged, a changelog is automatically generated from the intervening commit
messages, then the plugin is signed and automatically uploaded to the
JetBrains plugin repository. This allows us to quickly iterate on the plugin,
test the plugin on multiple platforms, and to release new versions with
assurance.

##  V Evaluation

We conducted a preliminary experiment to evaluate recognition performance on
synthetic speech. Using a set of English TTS voices (both male and female)
included in macOS Ventura 13.1 with IntelliJ IDEA 2023.1, VoskAPI 0.3.45 and
Idiolect 1.3.3, we synthesized 100 utterances from the predefined command list
using the say command, then transcoded a 16000 kHz PCM signed 16-bit little-
endian audio recording using ffmpeg, and finally transcribe the audio using
VoskAPI to simulate a user-generated utterance. We report the average word
error rate (WER) of three ASR models trained on US English speakers: vosk-
model-en-us-0.22 (0.63), vosk-model-en-us-0.22-lgraph (0.66) and vosk-model-
small-en-us-0.15 (0.73). Results on a per-voice and per-model basis are shown
in Fig. 2 below.

Figure 2: WER for synthetic ASR using 100 commands from the default lexicon
across three US English models and seven TTS voices (lower is better).

WER is a common metric for evaluating ASR performance. We use it to estimate
overall intelligibility, but acknowledge it does not account for the quality
or diversity of natural speech.

Two primary trends are evident. First, model size has a weak, but observable
effect on recognition performance, consistent with prior work on large-
vocabulary continuous speech recognition. Based on plentiful evidence in the
scaling literature [8], we expect this trend to hold for larger models across
natural languages. Out of the three models we tested, vosk-model-small-en-
us-0.22 exhibits the lowest average WER across all voices. Second, we note the
presence of gender bias in the MacOS/Vosk TTS/ASR pipeline. Despite the
apparent intelligibility of both sets of voices, WER is substantially lower on
female voices (Flo, et al.) than their corresponding male counterparts (Eddy,
et al.). The cause of this discrepancy is unclear, but merits further
investigation.

We also evaluated the plugin using a variety of downstream metrics, such as
user downloads of the plugin over a five-year timespan, bug reports, GitHub
Insights, and feedback from the JetBrains Plugin Marketplace to help determine
user satisfaction and identify potential areas for improvement.

Figure 3: Plugin downloads stalled circa 2017 due to IDE incompatibility.

Downloads of the plugin stalled since its initial release in 2015. Originally
based on the CMUSphinx library that was later abandoned, our plugin was
downloaded over 15,964 times and 56 total bugs were filed, 14 of which were
reported by external users. Following a five-year hiatus, the authors
performed an intervention, rewrote the plugin from scratch based on a new
speech recognition engine, implemented the usability improvements described in
Sec. IV, and republished the plugin under its current name. Our assessment is
ongoing.

##  VI Threats to Validity

While voice programming may serve a niche purpose for developers, it has not
been fully validated as a general programming mechanism. Our experience
suggests frequent vocalizations may disturb programmers within earshot and we
anticipate its usefulness in colocated workspaces will thus be limited, but do
foresee some genuine use-cases, e.g., providing assistance to programmers with
visual and motor disabilities.

We also acknowledge that our system is limited by its own natural language
understanding capabilities. Speech is a comparatively inefficient medium for
conducting programming tasks and users may be hesitant to invest time
reconfiguring the system if intent recognition accuracy is low. Furthermore,
the IDE plugin is currently limited to the set of actions that can be easily
expressed using the IntelliJ Platform API.

Finally, we acknowledge that synthetic voices are, while increasingly
lifelike, an imperfect proxy for human speech. The voices available on macOS
are not a representative sample of human dialects and accents, nor we have not
yet conducted a systematic human evaluation. The lexicon used to generate
synthetic speech is also limited to the set of commands supported by the IDE,
and does not capture how the plugin would have performed in a more general-
purpose setting.

##  VII Future Work

In the future, we plan to conduct a thorough user study to better understand
the use cases for the plugin. In particular, we hope to understand the
development habits of visually and motor-impaired users to improve
accessibility, as well as broader support for other natural languages and
dialects.

The current evaluation is far from comprehensive and we intend to conduct a
more thorough human evaluation of the plugin’s intent-recognition capabilities
to better understand the tradeoffs between accuracy, intelligibility and
latency. We also hope to improve support for personalization features, such as
speaker adaptation and user-specific language models.

We also aim to improve the interaction mechanism by adding audiovisual
feedback and modal dialog commands. This functionality could be supported by a
more user-friendly configuration interface, using an embedded DSL for defining
dialog trees with a choose-your-own-adventure style modal dialog navigator.
Incorporating modal logic à la Hazel [9], by adding a visual backpack and
typing constraints would be an interesting direction to take and one we hope
to explore.

Currently, we only track download statistics, although to inform the
development of these features and better address the needs of voice
programmers, it would be helpful to collect minimal telemetry on user
utterances and custom commands.

Recent progress in machine learning has enabled the use of large language
models (LLMs) for a variety of tasks, including speech recognition, machine
translation, and text generation. An LLM trained on a large corpus of natural
language can be used to perform simple reasoning tasks, such as predicting the
most likely intent from a given utterance. For example, the utterance, “I want
to edit foo.java” is more likely to trigger, “open foo.java” than the command,
“execute foo.java”. While LLMs are currently served on the cloud, we predict
these models will become available on the edge over the next few years,
enabling real-time offline speech-to-intent recognition.

Given their increasing reasoning capabilities, we would like to further
explore the use of LLMs within the intent recognition pipeline. In particular,
we could use language models to rerank the most likely intent corresponding to
a user’s utterance, conditioned on a previous historical commands, the UI
state and other application-specific settings. Contextual-awareness would
require more careful IDE integration and enable us to more accurately predict
user intent.

##  VIII Conclusion

In this work, we presented Idiolect, a plugin for the IntelliJ Platform that
allows users to control the IDE and reconfigure the plugin using voice
commands. This addresses a common usability issue in voice UX design, where
users are unable to express their intent in a manner the system understands.
We described the plugin’s design and challenges we faced while implementing
it, presented the results of a preliminary evaluation, and discussed avenues
for future work.

On one end of the design spectrum are metaprogramming languages and software
language engineering frameworks that allow users to define their own commands
and build embedded domain specific languages (eDSLs). These systems can be
powerful, but require a high upfront investment from the user, who must design
a language, and then learn the language itself, all whilst doing their daily
job as a software engineer. On the other end are finite languages, which are
too inflexible.

In the middle of this language design spectrum however are idiolects, which
give users the freedom to create conversational domain-specific languages
according to their own idiomatic style: this is the design space which
Idiolect occupies. By targeting IDEs, whose users are typically adept
programmers, commands with more complex semantics can be expressed
programmatically, then invoked on-the-fly, sans recompilation.

We believe the intersection between conversational agents, vernacular
programming languages and programmable voice assistants to be fertile ground,
and one that is relatively unexplored in the language design space. We hope
our small contribution inspires others to explore these ideas and take steps
towards making programming more fun and accessible.

##  IX Acknowledgements

The first author wishes to thank his former colleagues Alexey Kudinkin and
Yaroslav Lepenkin, who contributed to the plugin during the original JetBrains
hackathon in 2015, and former manager Hadi Hariri for his advice and
encouragement.

## References

  * [1] M. Shaw, “Myths and mythconceptions: What does it mean to be a programming language, anyhow?” _Proceedings of the ACM on Programming Languages_ , vol. 4, no. HOPL, pp. 1–44, 2022.
  * [2] E. W. Dijkstra, “On the foolishness of natural language programming,” _Program Construction, International Summer School_ , pp. 51–53, 1979.
  * [3] J. L. Leopold and A. L. Ambler, “Keyboardless visual programming using voice, handwriting, and gesture,” in _Proceedings. 1997 IEEE Symposium on Visual Languages (Cat. No. 97TB100180)_. IEEE, 1997, pp. 28–35.
  * [4] S. C. Arnold, L. Mark, and J. Goldthwaite, “Programming by voice, VocalProgramming,” in _Proceedings of the fourth international ACM conference on Assistive technologies_ , 2000, pp. 149–155.
  * [5] A. Begel and S. L. Graham, “Spoken programs,” in _2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’05)_. IEEE, 2005, pp. 99–106.
  * [6] M. Chkroun and A. Azaria, “Lia: A virtual assistant that can be taught new commands by speech,” _International Journal of Human–Computer Interaction_ , vol. 35, no. 17, pp. 1596–1607, 2019.
  * [7] B. Considine, J. Guo, and X. Si, “Tidyparse: Real-Time Context-Free Error Correction,” in _Workshop on Live Programming_ , 2022. [Online]. Available: https://github.com/tidyparse/tidyparse
  * [8] J. Droppo and O. Elibol, “Scaling laws for acoustic models,” _arXiv preprint arXiv:2106.09488_ , 2021.
  * [9] C. Omar, D. Moon, A. Blinn, I. Voysey, N. Collins, and R. Chugh, “Filling typed holes with live GUIs,” in _Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation_ , 2021, pp. 511–525.