# Design and Optimization of Magnetic-core Solenoid Inductor for Multi-phase
Buck Converter

Madhava Sarma Vemuri  Department of Electrical and Computer Engineering
North Dakota State University
Fargo, North Dakota 58105
Umamaheswara Rao Tida Madhava Sarma Vemuri is with the Department of
Electrical and Computer Engineering, North Dakota State University, North
Dakota, United States – 58102Umamaheswara Rao Tida is with the Department of
Electrical and Computer Engineering, North Dakota State University, North
Dakota, United States – 58102. e-mail<EMAIL_ADDRESS>received: Oct 14, 2019;  Department of Electrical and Computer Engineering
North Dakota State University
Fargo, North Dakota 58105

# FDSOI Process Based MIV-transistor Utilization for Standard Cell Designs in
Monolithic 3D Integration (Accepted in SOCC 2023)

Madhava Sarma Vemuri  Department of Electrical and Computer Engineering
North Dakota State University
Fargo, North Dakota 58105
Umamaheswara Rao Tida Madhava Sarma Vemuri is with the Department of
Electrical and Computer Engineering, North Dakota State University, North
Dakota, United States – 58102Umamaheswara Rao Tida is with the Department of
Electrical and Computer Engineering, North Dakota State University, North
Dakota, United States – 58102. e-mail<EMAIL_ADDRESS>received: Oct 14, 2019;  Department of Electrical and Computer Engineering
North Dakota State University
Fargo, North Dakota 58105

###### Abstract

Monolithic Three-Dimensional Integrated Circuits (M3D-IC) has become an
attractive option to increase the transistor density. In M3D-IC, substrate
layers are realized on top of previous layers using sequential integration
techniques. Recent works in M3D-IC have demonstrated the feasibility of FDSOI
process-based M3D-IC implementations and, Metal inter-layer vias (MIVs) are
used to provide connections between the inter-layer devices. Since MIVs are
extended from bottom layer to top layer, they occupy a small area resulting in
area overhead. Additionally, a minimum separation is required to facilitate
connection between MIV and transistors which increases this overhead further.
Towards this, we studied the alternate utilization of MIV to create MIV-
transistors with varying channels. We have also presented a strategy to
extract the Spice parameters of the proposed models using level 70 spice
parameters. Finally, a standard cell based gate level comparison is presented
to compare the Power, Performance and Area (PPA) metrics of the traditional
two layer 2D FDSOI transistor implementation with the proposed models.
Simulation results from standard cell designs suggest that the proposed
methodology can reduce 18% layout area on average compared to the traditional
approach. In addition, power consumption and delay time of the standard cells
are reduced by 1% and 3% on average respectively.

##  I Introduction

In recent years, vertical integration has piqued interest in the IC design
industry. Vertical integration allows stacking active devices on top of each
other. Consequently, it improved the transistor density, and reduced the
global interconnection wire length, making it a viable alternative to
conventional transistor scaling. Previously, Vertical integration techniques
stacked processed substrate layers and, used Through-Silicon-Vias (TSVs) to
provide connections between processed layers. However, the size of TSV is
limited by the substrate die thickness, and does not scale with transistor
scaling [1, 2]. Aggressive scaling of TSV size has only resulted in TSV
diameters $>2\mu m$, which is still larger than conventional standard cell
dimension [3].

To address the limitations of TSV-based scaling techniques, Monolithic three
Dimensional Integrated circuit (M3D-IC) technology has been proposed. M3D-ICs
are realized by sequential integration of active layers, which reduced the
substrate thickness to $<0.1\mu m$. Existing works on M3D-IC process such as
CoolCubeTM technology sets a thermal budget and limit the process temperature
to $<500^{o}C$ [4, 5, 6]. The thermal budget prevents the degradation of
transistors in the subsequent bottom layers. An Inter-layer-Dielectric (ILD)
region is created to reduce the interactions between top and bottom layers. To
facilitate the interconnections between the top and bottom layers, Metal
Inter-Layer Via (MIV) is used. A thin ILD region allows increased MIV density
due to reduced Via sizes ( $<0.1\mu m$). For 14nm design rules, the estimated
MIV density is $>100$ M/mm2 [7]. Since MIVs are extended from bottom layer to
top layer, they occupy a small area resulting in area overhead. Additionally,
a minimum separation is required to facilitate connection between MIV and
transistors which increases this overhead further.

M3D-IC designs can be implemented with multiple abstraction styles such as
block-level, gate-level, and transistor-level [8, 9, 10]. In block-level
style, the functional blocks of standard cells are grouped, and placed in
multiple layers. In gate-level abstraction style standard cells are placed in
multiple layers, and the interconnects are routed. In the transistor-level
style, the transistors are partitioned into different layers. In comparison to
the previous design style, the transistor-level design allows finer
integration, since partitioning is done at a granular level. In an ideal
situation, M3D-IC offers 50% area reduction due to the 2-layer partitioning of
devices. However, only 40% area reduction is reported for transistor-level
design style, due to mismatch of n-type and p-type sizes and MIV area overhead
in the top layer [10].

Due to increasing MIV density and fine integration in transistor-level
abstraction style, the MIV overhead also increased significantly. To address
the MIV area overhead, alternate utilization has been investigated recently.
Using the MIV-transistor, the silicon overhead was reduced by 24% [11, 12].
However, the study considers the entire top substrate layer to be silicon
unlike the FDSOI Process used in the recent demonstrations [13].

The major contributions of the work are as follows:

  1. 1.

We have proposed FDSOI-based MIV-transistor models with varying channels,
including 1-channel, 2-channel and 4-channel models.

  2. 2.

For comparison purposes, we have extracted the Spice parameters from TCAD
simulations of conventional FDSOI 2D transistor and proposed MIV-transistors,
using level 70 (BSIMSOI4) Spice parameters.

  3. 3.

We have created 14 standard cells for Power, Performance and Area (PPA)
comparisons. Our results suggest that, standard cells created using 2-channel
MIV-transistors had shown a $3\%$ reduction in the overall power-delay-product
and 18% average layout area reduction compared to the traditional 2-layer
implementation.

The organization of the rest of the paper is as follows: Section I discusses
in detail about the 2-layer M3D-IC in FDSOI process. Section 2 showcases
multiple MIV-transistor models, and presents strategies for Spice parameter
extraction of the models. Section IV presents the PPA comparison and their
implication of the standard cells created using the extracted Spice transistor
model files. Finally, the concluding remarks are given in Section V.

##  II FDSOI Process

In this section, we discuss about the assumed FDSOI process in M3D design
throughout the paper. We have assumed a 2-layer M3D design where top and
bottom layers are assigned for n-type and p-type transistors respectively.
M3D-IC process with CoolCubeTM technology allows heterogeneous integration
where the bottom layer transistors can be Bulk, FDSOI, FinFET or Trigate but
the top layer transistors are realized only with FDSOI technology [14]. For
simplicity, we have assumed both the top and bottom layers are created using
FDSOI technology. The transistors in FDSOI technology are implemented over a
dielectric region known as BOX region (Buried-Oxide) [15]. To facilitate the
interconnections between the top and bottom layers, Metal-Interlayer-Vias
(MIV) are used. MIV is used as an,  1. Internal Contact  2. External Contact
as shown in Figure 1. When MIV is used as internal contact, Drain or Source
active regions is be connected internally, and no additional area overhead is
created. However, when MIV used as external contact to connect Gate region, it
creates area overhead in the top substrate layer due to the presence of MIV.
Additionally, minimum separation between the MIV and Gate region increases the
area overhead.

The illustration of the assumed process is shown in Figure 1. A thin film
undoped Silicon (Si) substrate (thickness of 7 nm) is used in the design of
transistors. The p-type and n-type active regions are modeled using the Boron
(B) and Arsenic (As) respectively. In this work, we assumed the active Source
and Drain substrate region is doped using a concentration of $n_{src}$. The
length and width of active regions are denoted by $l_{src}$ and $w_{src}$
respectively. The length of the gate region ($L_{G}$) is assumed to be 24nm.
We have used Silicon Nitride (Si3N4) to model the spacer region. All the
insulator regions including the Buried Oxide (BOX), Interlayer Dielectric
(ILD) and Interconnect Dielectric (ID) regions are modeled using Silicon
dioxide (SiO2). All the interconnect routing layers (M1 and M2), MIV and Gate
regions are modeled using Copper (Cu). The nominal value of thickness of MIV
$t_{MIV}$ is assumed to be $25nm$. The width and thickness of M1 and M2
routing layer is assumed to be $24$nm and $48nm$ respectively, based on the
7nm PDK assumptions given in [16] for estimating the parasitics of
interconnects. The size of via contact is assumed to be $24nm$. To model the
device behavior in FDSOI process, we have created the CAD models using the
Sentarus TCAD tool using the nominal values presented in Table I.
Shockley–Read–Hall (SRH) recombination model along with Fermi based statistics
and Poisson equations are used to model the carrier behavior.

Figure 1: FDSOI Process in M3D-IC TABLE I: Process and design parameters used in the study | Parameter |  Description | Value
---|---|---|---
Process | $t_{Si}$ |  Silicon Thickness |  $7$ nm
$h_{src}$ |  Height of source/drain region |  $7$ nm
$t_{ox}$ |  Thickness of oxide liner |  $1$ nm
$n_{src}$ |  Source/Drain doping |  $1\times 10^{19}$cm-3
$t_{spacer}$ |  Spacer Thickness |  $10$ nm
$t_{BOX}$ |  Buried Oxide Thickness |  $100$ nm
Design | tmiv |  MIV thickness |  $25$ nm
$l_{src}$ |  Length of Source/Drain region |  $48$ nm
$w_{src}$ |  Width of Source/Drain region |  $192$ nm
| $L_{G}$ |  Length of Gate |  $24$ nm

##  III Modeling and Design of MIV-transistor in FDSOI-Process

As discussed earlier, the MIVs connecting to the gate induces extra area
overhead and hence it needs to be addressed to realize efficient IC designs in
M3D-IC technology. In this section, we discuss the modeling and design of MIV-
transistor in M3D-IC technology, and spice extraction methodology in FDSOI
process. The minimum thickness of oxide around MIV to provide electrical
isolation to the substrate is assumed to be 1nm as shown in Figure 2(a). The
subtrate material is placed after this oxide liner to form metal-insulator-
semiconductor (MIS) structure similar to MOS transistor [11, 12]. To improve
the channel control, the gate is realized on the top of the substrate similar
to the conventional FDSOI implementation. This MIV-transistor structure can
utilize substrate effectively reducing the MIV area overhead and also the wire
length.

Various implementations of MIV-transistors specifically 1, 2 and 4-channel
models are shown in Figure 2. The 1-channel FDSOI technology based MIV-
transistor model shown in Figure 2(b) has 1 Source and 1 Drain region similar
to a traditional FDSOI transistor design. The Gate region and the MIV are
connected together without any spacing between MIV and the gate terminal.
However, the Source and Drain contacts should have minimum M1 spacing (which
we assumed to be $24$ nm). Similarly,the 2-channel model has two Source and
two Drain regions as shown in Figure 2(c) where the Source and Drain regions
should be connected together. Please note that for the basic gate designs, one
metal layer is sufficient to provide necessary interconnects similar to the
traditional FDSOI designs for 1-channel and 2-channel MIV-transistors. The
4-channel model has 2 Source and 2 Drain regions but has 4 channels as shown
in Figure 2(d). For 4-channel MIV transistor, the Source and Drain active
regions are on the either side, making it complex for routing interconnects.
Therefore, we need an additional routing resources such as more interconnect
wires to connect the Source and Drain regions respectively for the 4-channel
MIV-transistor model.

The minimum dimension for the active region of 4-channel MIV-transistor is
$48nm$, considering the smallest via size, and minimum separations between
active region and gate contacts. For similar comparisons between the proposed
MIV-transistor models, we assumed the width scaled exactly $2\times$ as we
move from 4-channel model to 2-channel model, to account for the missing
channel regions. Similarly, as we scale from 2-channel model to 1-channel
model, the width scaled $2\times$. For fair comparison, we have assumed the
equivalent width of transistor $w_{src}$ for all the models to be $192nm$.
Therefore, the 1-channel Source and Drain region width will be $192nm$. The
2-channel Source and Drain regions width will be $96nm$ for each Source and
Drain region shown in Figure 2(c). Similarly, the 4-channel MIV-transistor
active region width will be $48nm$ each for all active regions. As we can see
from Figure 2(d), there will be a total of four channels and hence the total
width will be $196nm$.

(a) Side View

(b) 1-channel MIV-transistor Top View

(c) 2-channel MIV-transistor Top View

(d) 4-channel MIV-transistor Top View

Figure 2: Proposed layouts of MIV-based devices

###  III-A Spice Extraction Initial Setup

TCAD modeling and design is not feasible for large designs due to the
requirement of significant computing resources and high simulation time.
Therefore, we have extracted the equivalent Spice model files for the proposed
MIV-transistor and the conventional FDSOI transistor without MIV presence.
These Spice models are then used to design standard cells for PPA comparison.
In this section, we present the strategy for Spice extraction of the FDSOI
technology based transistors and in the next section we discuss the simulation
results of the standard cell designs with the extracted Spice models.

We have used the Level 70 (BSIMSOI4 model) Spice parameters in the extraction
[17]. Based on our simulation results, BSIMSOI4 model has closely emulated the
FDSOI transistor behavior. We have used the Synopsys design flow such as
Sentaurus TCAD, to model the device behavior and, extracted the performance
characteristics. Later, TCAD2SPICE tool is used to fine-tune the Spice
parameters based on the device characteristics. All the simulations done were
ran on IntelTM Xeon 2.7GHz CPU with 112 cores and 512 GB RAM.

In the interest of space, the parameter description for level 70 Spice
parameters used in the extraction is not thoroughly discussed and a detailed
discussion is given in [17]. The nominal values used in extraction are given
in Table II. The parameters such as TSI, TOX, TBOX, L and W are set to similar
process values mentioned in Table I. To reduce the complexity in extraction,
we have turned-off Flags such as Gate-to-channel current model selector
(IGCMOD). The model selector for SOI (SOIMOD) is set to 2, selects an ’ideal
FD’ scenario. The other model selectors are set to default selection, and
hence are not mentioned in Table II.

TABLE II: Level 70 parameters constants and Flags used in extraction Parameter |  Description | Value
---|---|---
LEVEL |  Spice model selector | 70
MOBMOD |  Mobility model selector | 4
CAPMOD |  Flag for the short channel capacitance model | 3
IGCMOD |  Gate-to-channel tunneling current model selector | 0
SOIMOD |  SOI model selector | 2
TSI |  Silicon Thickness (m) | $7\times 10^{-9}$
TOX |  Oxide Thickness (m) | $1\times 10^{-9}$
TBOX |  Buried Oxide Thickness (m) | $100\times 10^{-9}$
L |  Channel Length (m) | $48\times 10^{-9}$
W |  Channel Width (m) | $192\times 10^{-9}$
TNOM |  Nominal Temperature (oC) | 25

###  III-B Spice Model Extraction Methodology

Figure 3: Spice Parameter Extraction

The methodology used in Spice extraction is presented in Figure 3. The device
characteristics are extracted from the TCAD device simulations for varying
voltage biases and are used in fine-tuning the Spice parameters. The
extraction was done sequentially:  1. Low Drain Extraction,  2. High Drain
Extraction, and  3. Capacitance Extraction.  In Low Drain extraction, the
device is simulated for lower Drain voltages ($V_{DS}=0.05V$) and device
characteristics, such as Source and Drain current with Varying Gate Voltages
($I_{D}\ \&\ I_{S}\ v.s\ V_{G}$) is obtained from the TCAD simulations.
Subsequently, for High Drain extraction, higher Drain voltages ($V_{DS}=1.0V$)
are used to obtain the $I_{D}\ \&\ I_{S}\ v.s\ V_{G}$ characteristics.
Additionally, for High Drain extraction, Source and Drain current with varying
Drain voltages with multiple biases ($V_{GS}=0.4V$ to $1.0V$) are used to
fine-tune parameters. For capacitance extraction, Gate capacitance with
varying Gate voltages is used to extract the capacitance-related parameters.
The methodology compared the device characteristics with the Spice
simulations, and the error is used to fine-tune the Spice parameters. A
detailed account for the parameters extracted is presented below:

  1. 1.

For Low Drain extraction, the parameters such as CDSC, U0, UA, UB, UD, UCS,
DVT0, and DVT1 are used to extract nominal values for the subsequent
extraction. The parameters such as U0, UA, UB, UD, UCS control the mobility of
the carrier in Spice models, DVT0 and DVT1 control the short channel effects.
The parameters such as U0, UA, DVT0, and DVT1 are passed to the subsequent
extraction regions for fine-tuning.

  2. 2.

For High Drain extraction, the parameters such as CDSC, CDSCD, U0, UA, VTH0,
PVAG, DVT0, DVT1, ETAB, and VSAT are extracted. CDSC, CDSCD and ETAB control
the subthreshold slope regions within the $I_{D}\ \&\ I_{S}\ v.s\ V_{G}$
characteristics. VTH0, VSAT and PVAG control the current regions above the
threshold voltage.

  3. 3.

For Capacitance extraction, the parameters such as CKAPPA, DELVT, CF, CGSO,
CGDO, MOIN, CGSL, and CGDL are used to model the capacitance behavior. CKAPPA,
CGSL, and CGDL control the lower biased regions in the capacitance
characteristics. DELVT is used to adjust the threshold voltage. CF, MOIN,
CGSO, and CGDO controlled the overall accuracy and, are fine-tuned along with
the previous parameters to improve the accuracy.

###  III-C Extraction Results

The overall performance of the spice model extraction results is presented in
Table III. We have compared the transistor characteristics obtained from the
TCAD models to the extracted spice model characteristics for n-type and p-type
transistors. The process and design parameters of proposed MIV-transistors
with 1-channel, 2-channel and 4-channel along with the FDSOI transistor
without MIV are given in Table I. The overall extraction error was under 10%
for all the cases. The MIV-transistor characteristics using the TCAD models
and spice extraction for 4-channel model is presented in Figure 4. From the
results, we can see that the extracted spice models for the FDSOI transistor
type can be used for the design of standard cells and in the next section, we
discuss about the Power-Performance-Area (PPA) metric comparison for different
standard cell implementations.

(a) $I_{D}$ v.s $V_{G}$ for n-type

(b) $I_{D}$ v.s $V_{D}$ for n-type

(c) $C_{G}$ v.s $V_{G}$ for n-type

(d) $I_{D}$ v.s $V_{G}$ for p-type

(e) $I_{D}$ v.s $V_{D}$ for p-type

(f) $C_{G}$ v.s $V_{G}$ for p-type

Figure 4: Level 70 Extraction results for 4-channel MIV-transistor TABLE III: TCAD to Spice extraction results Region | 4-channel | 2-channel | 1-channel | Traditional
---|---|---|---|---
n |  p |  n |  p |  n |  p |  n |  p
IDVG |  7.2% |  7.1% |  6.6% |  7.0% |  6.4% |  8.5% |  7.9% |  5.5%
IDVD |  3.5% |  7.2% |  3.4% |  6.8% |  3.2% |  7.5% |  3.7% |  5.2%
CV |  7.0% |  5.7% |  4.7% |  6.0% |  5.0% |  7.3% |  9.6% |  8.6%

##  IV Power, Performance and Area comparison of Standard cells

Several basic standarad cells are implemented using the Spice model files
extracted from the previous section to investigate the Power, Performance and
Area (PPA) metrics. In the standard cell design, we have assumed the bottom
layer and top layer active region to be p-type transistors and n-type
transistors respectively. To connect the top and bottom devices, MIVs (both
internal and external) are used. We have assumed 2-metal (M1 and M2)
interconnect routing layers. Additionally, to facilitate the routing-related
parasitics, we have evaluated the resistance of MIV and interconnect to be
$7\Omega$ and $3\Omega$ respectively. To account for parasitic related to
Voltage and Ground interconnects, we assumed resistance $5\Omega$ for both
cases. The internal interconnect capacitance values such as metal coupling
capacitance and fringing capacitance are ignored to limit the complexity of
the design. However, we assumed a load capacitance of 1fF for the driving
strength. Additionally, as the load capacitance increases the effect of
internal RC parasitic reduces significantly on overall power and delay
estimation [10].

The following standard cells with the assumed parasitic assumptions and the
extracted Spice models are designed in Hspice tool: AND2X1, AND3X1, AOI2X1,
INV1X1, MUX2X1, NAND2X1, NAND3X1, NOR2X1, NOR3X1, OAI2X1, OR2X1, OR3X1,
XNOR2X1 and XOR2X1. The PPA metrics comparison for standard cells such as
Delay time, Power, and Area are presented in Figure 5. For comparison
purposes, we have replaced the top n-type transistors of standard cells with
the 2D FDSOI transistor similar to the p-type with M1 spacing between MIV and
metal layers (This implementation is represented in the figure as 2D, which
has two-layer 2D FDSOI based implementation) and proposed n-type MIV-
transistors on the top layer with the 2D FDSOI p-type transistor on the bottom
layer. The n-type MIV-transistors on the top layers consist of three cases
specifically, 1-channel MIV-transistor (1-ch), 2-channel MIV-transistor
(2-ch), and 4-channel MIV-transistor (4-ch). A detailed comparison is given
below:

(a)

(b)

(c)

Figure 5: PPA comparison for the extracted models

####  IV-1 Delay Time

The average propagation delay of the outputs from Spice simulations of
standard cell designs is presented in Figure 5(a). The average delay reduced
by $3\%$, and $2\%$ for 1-channel and 2-channel MIV-transistor based cells
respectively. However, the average delay increased by $2\%$ for 4-channel MIV-
transistor based cells due to the differences in the transistor
characteristics, that is dependent on the model. The AND2X1 cell created using
4-channel model had higher delay time with $6\%$ increase compared with the 2D
implementation. The INV1X1 cell created using 2-channel model has delay time
reduced by up to $-11\%$ compared with the two-layer 2D FDSOI transistor
implementation.

####  IV-2 Power

The average power consumption of the standard cells from Spice simulations is
given in Figure 5(b). The average Power reduced by $0.5\%$, $1\%$, and $2\%$
for 1-channel, 2-channel, 4-channel MIV-transistor based cells respectively.
The INV1X1 cell created using the 2-channel model had higher power with $3\%$
increase compared with the two-layer 2D FDSOI transistor-based implementation.
The OR3X1 cell created using the 4-channel model has reduced power by up to
$3\%$ compared with the two-layer 2D FDSOI transistor-based design. The
differences in the power are also accounted due to the differences in the
transistor characteristics of the MIV-transistor models.

####  IV-3 Layout Area

For layout area comparison, we have assumed the M1 metal layer separation
specifically $24nm$. Therefore, the layout area on the top layer devices
considers the MIV with the M1 metal layer separation as the keep-out-zone. The
total layout area of the standard cells are presented in Figure 5(c)
calculated similarly to [18]. Note that this layout area is obtained by
considering the maximum layout dimensions on both top-layer and bottom-layer
implementation so that the standard cell placement treats both n-type and
p-type device layers together. From the figure, the layout area of the
standard cells is reduced by upto $9\%$, $18\%$, and $12\%$ on average for
1-channel, 2-channel, and 4-channel MIV-transistor models compared with the
standard two-layer FDSOI implementation respectively.

In addition, we can reduce the total substrate area consumption which is the
sum of the bottom layer p-type FDSOI transistor area, and the top layer n-type
transistor area considering the MIV placement by up to 31%. However, this
requires separate placement algorithms to optimize the layout area for both
layers separately, and also considers the delay and routing resources. In the
future, we plan to investigate the placement algorithms that consider the
bottom-layer and top-layer device placement separately.

In summary, the proposed MIV-transistor-based FDSOI models provide alternative
design choices where PPA metrics can be leveraged. For example, if the delay
can be leveraged and the area is limited, the 4-channel MIV-transistor-based
standard cells can reduce the area consumption by $25\%$. If the designer is
looking for overall improvement, out of all standard cells presented in the
paper, the 2-channel model had a $3\%$ reduction in the average power delay
product, and 18% overall area reduction.

##  V Conclusions

This paper studies the design of MIV-transistor in 2-layer M3D-IC using FDSOI
process. We looked into the design of MIV-transistor which utilizes the area
around MIV to create transistors. For performance metrics comparison, we
extracted the spice models of the proposed transistors using the level 70
spice parameters. 14 Gate models have been created to compare the performance
using different MIV-transistor models. Simulation results from standard cell
designs suggest that the proposed methodology can reduce 18% layout area on
average compared to the traditional approach. In addition, power consumption
and delay time of the standard cells are reduced by 1% and 3% on average
respectively.

## Acknowledgements

This work is supported by National Science Foundation under Award number –
2105164.

## References

  * [1] U. R. Tida, C. Zhuo _et al._ , “Dynamic Frequency Scaling Aware Opportunistic Through-Silicon-Via Inductor Utilization in Resonant Clocking,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 39, no. 2, pp. 281–293, 2020.
  * [2] U. R. Tida, R. Yang _et al._ , “On the Efficacy of Through-Silicon-Via Inductors,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 23, no. 7, pp. 1322–1334, 2015.
  * [3] S. K. Samal, D. Nayak _et al._ , “Monolithic 3D IC vs. TSV-based 3D IC in 14nm FinFET technology,” in _IEEE SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S)_ , 2016, pp. 1–2.
  * [4] M. Vinet, P. Batude _et al._ , “Opportunities brought by sequential 3D CoolCube™ integration,” in _2016 46th European Solid-State Device Research Conference (ESSDERC)_ , 2016, pp. 226–229.
  * [5] H. Han, R. Choi _et al._ , “Low Temperature and Ion-Cut Based Monolithic 3D Process Integration Platform Incorporated with CMOS, RRAM and Photo-Sensor Circuits,” in _IEEE International Electron Devices Meeting (IEDM)_ , 2020, pp. 15.6.1–15.6.4.
  * [6] B. Rajendran, R. S. Shenoy _et al._ , “Low Thermal Budget Processing for Sequential 3-D IC Fabrication,” _IEEE Transactions on Electron Devices_ , vol. 54, no. 4, pp. 707–714, 2007.
  * [7] A. W. Topol, D. C. L. Tulipe _et al._ , “Three-dimensional integrated circuits,” _IBM Journal of Research and Development_ , vol. 50, no. 4.5, pp. 491–506, 2006.
  * [8] S. Panth, K. Samadi _et al._ , “High-density integration of functional modules using monolithic 3d-ic technology,” in _2013 18th Asia and South Pacific Design Automation Conference (ASP-DAC)_. IEEE, 2013, pp. 681–686.
  * [9] S. A. Panth, K. Samadi _et al._ , “Design and CAD methodologies for low power gate-level monolithic 3D ICs,” in _Proceedings of the 2014 international symposium on Low power electronics and design_ , 2014, pp. 171–176.
  * [10] Y.-J. Lee and S. K. Lim, “Ultrahigh Density Logic Designs Using Monolithic 3-D Integration,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 32, no. 12, pp. 1892–1905, 2013.
  * [11] M. S. Vemuri and U. Rao Tida, “Dual-Purpose Metal Inter-layer Via Utilization in Monolithic Three-Dimensional (M3D) Integration,” in _IEEE 63rd International Midwest Symposium on Circuits and Systems (MWSCAS)_ , 2020, pp. 424–427.
  * [12] U. R. Tida and M. S. Vemuri, “Efficient metal inter-layer via utilization strategies for three-dimensional integrated circuits,” in _2020 IEEE 33rd International System-on-Chip Conference (SOCC)_. IEEE, 2020, pp. 195–200.
  * [13] M. S. Vemuri and U. Rao Tida, “Metal Inter-layer Via Keep-out-zone in M3D IC: A Critical Process-aware Design Consideration,” in _24th International Symposium on Quality Electronic Design (ISQED)_ , 2023.
  * [14] P. Batude, C. Fenouillet-Beranger _et al._ , “3DVLSI with CoolCube process: An alternative path to scaling,” in _2015 Symposium on VLSI Technology (VLSI Technology)_ , 2015, pp. T48–T49.
  * [15] T. Skotnicki and S. Monfray, “UTBB FDSOI: Evolution and opportunities,” in _2015 45th European Solid State Device Research Conference (ESSDERC)_ , 2015, pp. 76–79.
  * [16] K. Chang, K. Acharya _et al._ , “Impact and Design Guideline of Monolithic 3-D IC at the 7-nm Technology Node,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 25, no. 7, pp. 2118–2129, 2017\.
  * [17] “PrimeSim™ Continuum™ Reference Manual: MOSFET Models,” _Synopsys_ , 2021.
  * [18] C. Yan and E. Salman, “Mono3D: Open Source Cell Library for Monolithic 3-D Integrated Circuits,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 65, no. 3, pp. 1075–1085, 2018.


# CitySpec: An Intelligent Assistant System for Requirement Specification in
Smart Cities

Zirong Chen1, Isaac Li2, Haoxiang Zhang3, Sarah Preum4, John A. Stankovic2,
Meiyi Ma1
1 Vanderbilt University 2 University of Virginia 3 Columbia University 4
Dartmouth College
{zirong.chen<EMAIL_ADDRESS>{il5fq<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

An increasing number of monitoring systems have been developed in smart cities
to ensure that a city’s real-time operations satisfy safety and performance
requirements. However, many existing city requirements are written in English
with missing, inaccurate, or ambiguous information. There is a high demand for
assisting city policy makers in converting human-specified requirements to
machine-understandable formal specifications for monitoring systems. To tackle
this limitation, we build CitySpec, the first intelligent assistant system for
requirement specification in smart cities. To create CitySpec, we first
collect over 1,500 real-world city requirements across different domains from
over 100 cities and extract city-specific knowledge to generate a dataset of
city vocabulary with 3,061 words. We also build a translation model and
enhance it through requirement synthesis and develop a novel online learning
framework with validation under uncertainty. The evaluation results on real-
world city requirements show that CitySpec increases the sentence-level
accuracy of requirement specification from 59.02% to 86.64%, and has strong
adaptability to a new city and a new domain (e.g., F1 score for requirements
in Seattle increases from 77.6% to 93.75% with online learning).

###### Index Terms:

Requirement Specification, Intelligent Assistant, Monitoring, Smart City

##  I Introduction

With the increasing demand for safety guarantees in smart cities, significant
research efforts have been spent toward how to ensure that a city’s real-time
operations satisfy safety and performance requirements [1]. Monitoring
systems, such as SaSTL runtime monitoring [2], CityResolver [3], and STL-U
predictive monitoring [4], have been developed in smart cities. Figure 1 shows
a general framework of monitoring systems in smart cities. These systems are
designed to execute in city centers and to support decision-making based on
the verification results of real-time sensing data about city-states (such as
traffic and air pollution). If the monitor detects a requirement violation,
the city operators can take actions to change the states, such as improving
air quality, sending alarms to police, calling an ambulance, etc.

The monitor systems have two important inputs, i.e., the real-time data
streams and formal specified requirements. Despite that extensive research
efforts have been spent toward improving the expressiveness of specification
languages and efficiency of the monitoring algorithms, the research challenge
of how to convert human-specified requirements to machine-understandable
formal specifications has received only scant attention. Moreover, our study
(see Section II) on over 1,500 real-world city requirements across different
domains shows that, first, existing city requirements are often defined with
missing information or ambiguous description, e.g., no location information,
using words like nearby, or close to. They are not precise enough to be
converted to a formal specification or monitored in a city directly without
clarifications by policy makers. Secondly, the language difference between
English specified requirements and formalized specifications is significant.
Without expertise in formal languages, it is extremely difficult or impossible
for policy makers to write or convert their requirements to formal
specifications. Therefore, there is an urgent demand for an intelligent system
to support policy makers for requirement specifications in smart cities.

Despite the prevalence of developing models to translate natural language to
machine languages in various domains, such as Bash commands [5], Seq2SQL [6],
and Python [7], it is very challenging to develop such an intelligent system
for requirement specification in smart cities for the following reasons.
First, unlike the above translation tasks with thousands or even millions of
samples in a dataset, there is barely any requirement specification data. As a
result, traditional language models are not sufficient to be applied directly.
Moreover, the requirements usually contain city domain-specific descriptions
and patterns that existing pre-trained embeddings like BERT or GloVE cannot
handle effectively. Furthermore, requirements from different domains and
cities vary significantly and evolve over time, thus building a system that
can adapt to new domains at runtime is an open research question. Good
adaptability can increase user experience (e.g., policy makers do not have to
clarify new terms repeatedly), while one of the major challenges is validating
and filtering the new knowledge and avoiding adversarial examples online.

Figure 1: CitySpec in Smart Cities

In this paper, we target the above technical challenges, and describe
CitySpec, an intelligent assistant system for requirement specification in
smart cities. To the best of our knowledge, it is the first specification
system helping city policy makers specify and translate their requirements
into formal specifications automatically. As shown in Figure 1, CitySpec is
designed to bridge the gap between city policy makers and monitoring systems.
It enables policy makers to define their requirements by detecting missing,
inaccurate, or ambiguous information through an intelligent assistant
interface. To effectively train the translation model using a small amount of
city requirement data, CitySpec extracts city knowledge and enhances the
learning process through requirement synthesis. CitySpec can easily adapt to a
new city or domain through online learning and validation.

Contributions. We summarize the major contributions of this paper as follows:

  * •

We collect and annotate over 1,500 real-world city requirements from over 100
cities across different domains. We extract city-specific knowledge and build
a dataset of city vocabulary with 3,061 words in 5 categories.

  * •

We create an intelligent assistant system for requirement specification in
smart cities. In the system, we build a translation model and enhance it
through requirement synthesis, and develop a novel online learning framework
with validation under uncertainty.

  * •

We evaluate CitySpec extensively using real-world city requirements. The
evaluation results show that CitySpec is effective on supporting policy makers
accurately writing and refining their requirements. It increases the sentence
level accuracy of requirement specification from 59.02% to 86.64% through city
knowledge injection. It shows strong adaptability (user experience) to a new
city (e.g., F1 score in Seattle from 77.6% to 93.75%) and a new domain (e.g.,
F1 score in security domain from 62.93% to 93.95%).

Paper organization: In the rest of the paper, we describe the motivating study
of city requirement specification in Section II, provide an overview of
CitySpec in Section III, and present the technical details in Section IV. We
then present the evaluation results in Section V, discuss the related work in
Section VI and draw conclusions in Section VII.

##  II Motivating Study

TABLE I: Comparison between English requirements and formal specifications ID |  English Requirement |  Formal Specification | DLD
---|---|---|---
1 |  Sliding glass doors shall have an air infiltration rate of no more than 0.3 cfm per square foot. |  $\mathsf{Everywhere}_{\mathsf{{Sliding~{}glass~{}doors}}}~{}\mathsf{Always}_{[0,+\infty)}~{}\mathsf{air~{}infiltration~{}rate}~{}\leq 0.3~{}\mathsf{cfm/foot}^{2}$ | 59
2 |  The operation of a Golf Cart upon a Golf Cart Path shall be restricted to a maximum speed of 15 miles per hour from 8:00 to 16:00. |  $\mathsf{Everywhere}_{\mathsf{Golf~{}Cart~{}Path}}\mathsf{Always}_{[8,16]}\mathsf{Golf~{}Cart~{}speed}<15$ $~{}\mathsf{miles/hour}$ | 67
3 |  Up to four vending vehicles may dispense merchandise in any given city block at any time. |  $\mathsf{Everywhere}_{\mathsf{city~{}block}}\mathsf{Always}_{[0,+\infty)}\mathsf{vending~{}vehicles}\leq 4$ | 75

In this section, we study real-world city requirements and their formal
specification as motivating examples to discuss the demand and challenges of
developing an intelligent assistant system for requirement specification in
smart cities. We collect and annotate over 1,500 real-world city requirements
across different domains (e.g., transportation, environment, security, public
safety, indoor environments, etc.) from over 100 cities. We make the following
observations from the analysis of the requirement dataset.

Existing city requirements are often defined with missing information or
ambiguous description. In [2], the authors define essential elements for
monitoring a city requirement. Within the 1500 requirements, many requirements
have one or more missing elements. For example, 27.6% of the requirements do
not have location information, 29.1% of the requirements do not have a proper
quantifier, and 90% of the requirements do not have or only have a default
time (e.g., always) defined. Additionally, requirements often have ambiguous
descriptions that are difficult to be noticed by policy makers. For example, a
location is specified as “nearby” or “close to”. As a result, it is very
difficult or impossible for the monitoring system to monitor these
requirements properly. It indicates a high demand for an intelligent assistant
system to support the policy makers to refine the requirements.

The language difference between English specified requirements and their
formal specifications is significant. In Table I, we give three examples of
city requirements in English, their formal specification in SaSTL, and the
Damerau–Levenshtein Distance (DLD) [8] between each pair of requirements. DLD
measures the edit distance between two sequences. It shows that natural
languages are different from machine-compatible input languages. Formal
specifications usually consist of mathematical symbols, which makes the
conversion even more difficult. As shown in Table I, the average DLD from
English requirements to formal specifications is 67, which means that it
requires an average of 67 edits. As a reference, the average DLD brought by
translating these three English requirements to Latin is 64.67. It indicates
that the conversion from English requirements to formal specifications even
requires more edits than the translation of these requirements from English to
Latin. In general, building a translator from English to Latin would require
millions of samples. However, as an under-exploited domain, there is a very
limited number of well-defined requirements. Moreover, annotation of formal
specifications requires specialties in formal methods and is extremely time-
consuming. It presents major challenges for building such a translation model.

##  III System Overview

Figure 2: System Overview

CitySpec is designed to bridge the gap between city policy makers and
monitoring systems. It supports policy makers to precisely write city
requirements in English through an intelligent interface, and then converts
them to formal specifications automatically. An overview of CitySpec is shown
in Figure 2. There are four major components in CitySpec, including an
intelligent assistant Interface to communicate with policy makers (see Section
IV-B), a Requirement Synthesis component to extract city knowledge and
synthesize new requirements to build the translation model (see Section IV-C),
a Translation Model to convert city requirements to formal specifications (see
Section IV-D), and an Online Learning component to adapt the system to new
knowledge (see Section IV-E).

At runtime (as indicated by the orange arrows in Figure 2), a city policy
maker inputs a requirement in English through an intelligent assistant
interface, which sends the requirements to the translation model. The
translation model converts the requirements to a formal specification and
checks if there is any missing information or ambiguous description. The
translation model is built with injected city knowledge through requirement
synthesis at the training time and enhanced through online learning at
runtime. Next, based on the returned results from the translation model, the
intelligent interface communicates with the policy maker to acquire or clarify
the essential information. In this process, the assistant supports the policy
maker to refine the requirement until it is precisely defined and accepted by
the monitor. We present the technical details in Section IV, and develop a
prototype tool of the CitySpec system and deploy it online.

##  IV Methodology

In this section, we present the major components in CitySpec (as shown in
Figure 2). We first introduce requirement specification using Spatial-
aggregation Signal Temporal Logic (SaSTL) [2]. Then we show the design and
technical details of the intelligent assistant interface, requirement
synthesis, translation model, and online learning, respectively.

###  IV-A Requirement Specification using SaSTL

SaSTL is a powerful formal specification language for Cyber-Physical Systems.
We select it as our specification language because of its advantages of
expressiveness and monitoring for smart cities. However, CitySpec is general
and can work with other specification languages. SaSTL is defined on a multi-
dimensional _spatial-temporal signal_ as $\omega:\mathbb{T}\times
L\to\\{{\mathbb{R}\cup\\{\bot\\}\\}}^{n}$, where $\mathbb{T}=\mathbb{R}_{\geq
0}$, represents the continuous time and $L$ is the set of locations.
$X=\\{x_{1},\cdots,x_{n}\\}$ is denoted by the set of variables for each
location. The spatial domain $\mathcal{D}$ is defined as,
$\mathcal{D}:=([d_{1},d_{2}],\psi)$,
$\psi:=\top\;|\;p\;|\;\neg\;\psi\;|\;\psi\;\vee\;\psi$, where $[d_{1},d_{2}]$
defines a spatial interval with $d_{1}<d_{2}$ and $d_{1},d_{2}\in\mathbb{R}$,
and $\psi$ specifies the property over the set of propositions that must hold
in each location.

The syntax of SaSTL is given by

$\begin{array}[]{cl}\varphi:=&x\sim c\;|\ \neg\varphi\;|\
\varphi_{1}\land\varphi_{2}\;|\ \varphi_{1}\mathcal{U}_{I}\varphi_{2}\;|\
\mathcal{A}_{\mathcal{D}}^{\mathrm{op}}x\sim c\;|\
\mathcal{C}_{\mathcal{D}}^{\mathrm{op}}\varphi\sim c\\\ \end{array}$

where $x\in X$, $\sim\in\\{<,\leq\\}$, $c\in\mathbb{R}$ is a constant,
$I\subseteq\mathbb{R}_{>0}$ is a real positive dense time interval,
$\mathcal{U}_{I}$ is the _bounded until_ temporal operators from STL. The
_always_ (denoted $\square$) and _eventually_ (denoted $\lozenge$) temporal
operators can be derived the same way as in STL, where
$\lozenge\varphi\equiv\mathsf{true}\ \mathcal{U}_{I}\varphi$, and
$\square\varphi\equiv\neg\lozenge\neg\varphi$. Spatial _aggregation_ operators
$\mathcal{A}_{\mathcal{D}}^{\mathrm{op}}x\sim c$ for
$\mathrm{op}\in\\{\max,\min,\mathrm{sum},\mathrm{avg}\\}$ evaluates the
aggregated product of traces
$\mathrm{op}(\alpha_{\mathcal{D}}^{x}(\omega,t,l))$ over a set of locations
$l\in L_{\mathcal{D}}^{l}$, and _counting_ operators
$\mathcal{C}_{\mathcal{D}}^{\mathrm{op}}\varphi\sim c$ for
$\mathrm{op}\in\\{\max,\min,\mathrm{sum},\mathrm{avg}\\}$ counts the
satisfaction of traces over a set of locations. From counting operators, we
derive the _everywhere_ operator as
$\boxbox_{\mathcal{D}}\varphi\equiv\mathcal{C}_{\mathcal{D}}^{\mathrm{min}}\varphi>0$,
and _somewhere_ operator as
$\diamonddiamond_{\mathcal{D}}\varphi\equiv\mathcal{C}_{\mathcal{D}}^{\mathrm{max}}\varphi>0$.
Please refer to [2] for the detailed definition and semantics of SaSTL.

###  IV-B Interface for Intelligent Assistant

City requirements often have missing or ambiguous information, which may be
unnoticed by policy makers. It leads to the demand for human inputs and
clarification when converting them into formal specifications. Therefore, we
design an intelligent assistant interface in CitySpec serving as an
intermediary between policy makers and the translation model. It communicates
with policy makers and confirms the final requirements through an intelligent
conversation interface.

To briefly describe the communication process, users first input a requirement
in English, e.g., “due to safety concerns, the number of taxis should be less
than 10 between 7 am to 8 am”. CitySpec interface passes the requirement to
the translation model and gets a formal requirement
($\mathsf{always}_{[7,8]}\mathsf{number~{}of~{}taxi}<10$) with the keywords
including,

  * •

$\mathsf{entity}$: the requirement’s main object, e.g., “the number”,

  * •

$\mathsf{quantifier}$: the scope of an entity, e.g., “taxi”,

  * •

$\mathsf{location}$: the location where this requirement is in effect, which
is missing from the above example requirement,

  * •

$\mathsf{time}$: the time period during which this requirement is in effect,
e.g., “between 7 am to 8 am”,

  * •

$\mathsf{condition}$: the specific constraint on the entity, such as an upper
or lower bound of $\mathsf{entity}$, e.g., “10”.

As a result, CitySpec detects that the location information is missing from
the user’s requirement and generates a query for the user, “what is the
location for this requirement?” Next, with new information typed in by the
user (e.g., “within 200 meters of all the schools”), CitySpec obtains a
complete requirement.

The next challenge is how to confirm the formal specification with policy
makers. Since they do not understand the formal equation, we further convert
it to a template-based sentence. Therefore, CitySpec presents three formats of
this requirements for users to verify, (1) a template-based requirement, e.g.,
[number] of [taxi] should be [$<$] [10] [between 7:00 to 8:00] [within 200
meters of all the schools], (2) a SaSTL formula
$\mathsf{everywhere}_{\mathsf{school}\land[0,200]}\mathsf{always}_{[7,8]}\mathsf{number~{}of~{}taxi}<10$,
and (3) five key fields detected. Users can confirm or further revise this
requirement through the intelligent assistant.

When policy makers have a large number of requirements to convert, to minimize
user labor to input requirements manually, CitySpec also provides the option
for them to input requirements through a file. The process is similar to where
CitySpec asks users to provide or clarify information until all the
requirements are successfully converted through files.

###  IV-C Requirement Synthesis

The amount of city requirement dataset is insufficient to train a decent
translation model in an end-to-end manner. As we’ve discussed in Section I, it
requires extensive domain knowledge in both city and formal specifications and
is extremely time-consuming to annotate new requirements. Furthermore, a
majority of the existing city requirements are qualitatively or imprecisely
written, which cannot be added to the requirement dataset without refinement
[2]. To mitigate the challenge of small data to build a translation model, we
design a novel approach to incorporating city knowledge through controllable
requirement synthesis.

There are two main reasons why converting a city requirement to a formal
specification is challenging with a small amount of data. First, the
vocabulary of city requirements are very diverse. For example, requirements
from different cities (e.g., Seattle and New York City) or in different
domains (e.g., transportation and environment) have totally different
vocabulary for entities, locations, and conditions. Second, the sentence
structure (patterns) of requirements vary significantly when written by
different people. It is natural for human beings to describe the same thing
using sentences.

Targeting these two challenges, we first extract city knowledge and build two
knowledge datasets, i.e., a vocabulary set and a pattern set. The vocabulary
set includes five keys of a requirement, i.e., entity, quantifier, location,
time and condition. The pattern set includes requirement sentences with 5
keywords replaced by their labels. For instance, we have a requirement, “In
all buildings/$\mathsf{location}$, the average concentration/$\mathsf{entity}$
of TVOC/$\mathsf{quantifier}$ should be no more than 0.6
mg/m3/$\mathsf{condition}$ for every day/$\mathsf{time}$.”, the pattern
extracted is “In #$\mathsf{location}$, the average #$\mathsf{entity}$ of
#$\mathsf{quantifier}$ should be no more than #$\mathsf{condition}$ for
#$\mathsf{time}$.”

We extract the knowledge set from city documents besides requirements so that
we are not limited by the rules of requirements and enrich the knowledge of
our model. For example, we extract 336 patterns and 3061 phrases (530 phrases
in $\mathsf{entity}$, 567 phrases in $\mathsf{quantifier}$, 501 phrases in
$\mathsf{location}$, 595 phrases in $\mathsf{condition}$, and 868 phrases in
$\mathsf{time}$).

Next, we designed an approach to synthesizing controllable requirement dataset
efficiently. Intuitively, we can go through all the combinations of keywords
and patterns to create the dataset of requirements, which is infeasible and
may cause the model overfitting to the injected knowledge. In order to enhance
the model’s performance, we need to keep a balance between the coverage of
each keyword and the times of keywords being seen in the generation. We denote
$\lambda$ as the synthesis index, which indicates the _minimum_ number of
times that a keyword appears in the generated set of requirements. Assuming we
have $m$ set of keywords vocabularies $\\{V_{1},V_{2}\dots V_{m}\\}$ and a
pattern set as $P$, we have the total number of synthesized requirements
$\ell=\lambda\cdot\max(|V_{1}|,|V_{2}|,\dots,|V_{m}|)$. For each set of
vocabularies $V_{i}$, we first create a random permutation of $V_{i}$ and
repeat it until the total number of phrases reaches $\ell$, then we
concatenate them to an array $S_{i}$. Once we obtain $S_{1},...S_{m}$, we
combine them with pattern $P$ to generate a requirement set $R$. Refer to
Algorithm 1 for more details.

Algorithm 1 Requirement Synthesis

Input: $m$ set of keywords vocabularies $\\{V_{1},V_{2}\dots V_{m}\\}$,
Pattern $P$, synthesis index $\lambda$
Output: Set of requirements $R$

Initialize $R$ as an empty set$\\{\\}$

Let $\ell=\lambda\cdot\max(|V_{1}|,|V_{2}|,\dots,|V_{m}|)$

for $i\in 1\dots m$ do

Initialize $S_{i}$ as an empty array $S_{i}=[]$

while $|S_{i}|<\ell$ do

Create a random permutation of $V_{i}$: $P=\text{Permutate}(V_{i})$

Concatenate $P$ to $S_{i}$: $S_{i}=\text{Concat}(S_{i},P)$

end while

end for

for $j\in 1\dots\ell$ do

Combine keywords $S_{1}[j],S_{2}[j]\dots S_{m}[j]$ with Pattern $P$ to create
a requirement $r_{j}$

Add $r_{j}$ to the set of requirements $R$

end for

return $R$

###  IV-D Translation Model

The inputs of the translation model are requirements, and the outputs of this
module are formal specifications with token-level classification. We implement
the translation model with three major components, a learning model, knowledge
injection through synthesized requirements, and keyword refinement.

To be noted, CitySpec does not build its own translation model from scratch.
Instead, we tackle the limitation of the traditional language model and
improve it for city requirement translation. Therefore, CitySpec is compatible
with different language models.

In this paper, we implement four popular language models, which are Vanilla
Seq2Seq, Stanford NLP NER, Bidirectional Long Short Term Memory (Bi-LSTM) +
Conditional Random Field (CRF) and Bidirectional Encoder Representations from
Transformers (BERT) [9]. We apply our synthesized datasets with different
synthesis indexes to inject city knowledge into these language models. Then we
evaluate the improvement brought by our requirement synthesis approach by
testing the performance on real-world city requirements. We present the
detailed results and analysis in Section V.

Additionally, we find that time, negation and comparison are the most tricky
elements that affect the accuracy of the final specification detection.
Therefore, we implement another refinement component in the translation model.
In general, the $\mathsf{time}$ domain can be represented in several formats,
such as timestamps, or other formats like yyyy-mm-dd and mm-dd-yyyy. To
mitigate the confusion that various formats might bring, we apply SUTime [10]
when the $\mathsf{time}$ entity is not given by the translation model.
pyContextNLP [11] is applied to analyze whether there is a negation in the
input sentence. If there is any negation, the comparison symbol is reversed.
For instance, if there is a keyword “greater than”, the comparison symbol is
$>$. However, if the whole phrase is “is not supposed to be greater than”, and
a negation is detected, thus the final comparison is $\leq$ instead.

###  IV-E Online Learning

In general, the more clarifications are needed from the users, the worse
experience the users will have, especially if users have to clarify the same
information repeatedly. For example, if a user from a new city inputs a
location that the system fails to detect, the user will be asked to clarify
the location information. The user’s experience will drop if the system asks
him again on the second or third time seeing these words. However, the deep
learning-based translation model cannot “remember” this information at
deployment time. Thus, the first question is that how can CitySpec learn the
new knowledge online?

Meanwhile, the new information provided by users may also harm the system if
it is an incorrect or adversarial example. The second question is that how can
CitySpec validate the new knowledge before learning it permanently?

Figure 3: Online Learning

Targeting these two research questions, we design an online learning module in
CitySpec. As shown in Figure 3, it has two stages, which are short-term
learning and long-term learning. Short-term learning is designed to
accommodate the same user in one session of requirement specification with a
temporary memory. The question-answer pairs are stored temporarily. When the
same occasion occurs, the temporal cache gives instant answers and avoids more
user clarifications. Long-term learning is designed to adapt the new knowledge
to the model permanently after validating its reliability. The accepted
permanent knowledge is achieved by updating weights via back-propagation on
the extended dataset with both initial data and the new input-label pairs
stored in the temporary cache.

To keep CitySpec away from the adversarial inputs, we develop a Bayesian CNN-
based validation module in CitySpec. The model is to classify the category of
a new term with confidence with uncertainty estimation. We apply dropout
layers during both training and testing to quantify the model uncertainty [4].
The inputs of the validation model are the new terms provided by the user,
while the outputs are the corresponding keys among those five key domains with
an uncertainty level. In brief, a new term-key pair is rejected if (1) the
output from the validation function does not align with the given domain key;
(2) the validation function has low confidence in the output although it might
align with the given domain. In this way, we only accept new city knowledge
validated with high confidence.

##  V Evaluation

In this section, we evaluate our CitySpec system from five aspects, including
(1) comparing different language models on the initial dataset without
synthesizing, (2) analyzing the effectiveness of the synthesized requirements
by enhancing the models with city knowledge, (3) evaluating the performance of
the online validation model, (4) testing CitySpec’s adaptability in different
cities and domains, and (5) an overall case study. We use the city requirement
dataset described in Section II. To evaluate the prediction of keywords and
mitigate the influence caused by different lengths requirements, we choose to
use token-level accuracy (token-acc) and sentence-level accuracy (sent-acc) as
our main metrics. The token-level accuracy aims to count the number of key
tokens that are correctly predicted. The sentence-level accuracy counts the
prediction as correct only when the whole requirement is correctly translated
to a formal specification using SaSTL. Thus, sentence-level accuracy serves as
a very strict criterion to evaluate the model performance. We also provide the
results using other common metrics including precision, recall, and F-1 score.
The experiments were run on a machine with 2.50GHz CPU, 32GB memory, and
Nvidia GeForce RTX 3080Ti GPU.

###  V-A Performance of language models on the initial dataset

TABLE II: Performance of language models on the initial dataset Model | Token-Acc | Sent-Acc | F-1 Score | Precision | Recall
---|---|---|---|---|---
Vanilla Seq2Seq |  10.91 $\pm$ 0.57 % |  1.38 $\pm$ 0.49% |  24.12 $\pm$ 0.24 % |  65.58 $\pm$ 7.95 % |  14.81 $\pm$ 1.58 %
BiLSTM + CRF |  77.59 $\pm$ 0.52 % |  60.82 $\pm$ 1.22 % |  80.46 $\pm$ 0.84 % |  81.11 $\pm$ 1.38 % |  79.83 $\pm$ 7.24 %
BERT |  80.41 $\pm$ 0.07 % |  59.02 $\pm$ 0.42 % |  81.43 $\pm$ 0.01 % |  78.62 $\pm$ 0.01 % |  84.46 $\pm$ 0.01 %

As a baseline of translation model without city knowledge, we first evaluate
the performance of CitySpec using different language models, including Vanilla
Seq2Seq, pretrained Stanford NER Tagger, Bi-LSTM + CRF, and BERT on the
initial dataset. We present the results in Table II.

We make the following observations from the results. First, the overlap
between Stanford Pretrained NER Tagger prediction and vocabulary is only 9 out
of 729. The pretrained tagger tends to give locations in higher granularity.
Since this task is a city-level, more detailed location information is stated
in a lower granularity by providing the street name, building name, or
community name. The $\mathsf{location}$ domain in the pretrained tagger gives
more high-level information like city name, state name, or country name. For
example, “34th Ave in Nashville, the state of Tennessee” is annotated as
$\mathsf{location}$ in this task, however, the pretrained NER tagger gives
“Tennessee” as $\mathsf{location}$ instead.

Secondly, the testing token-acc from Vanilla Seq2Seq is 10.91% on average.
Other metrics also indicate that Vanilla Seq2Seq has trouble recognizing the
patterns in sequential keyword labeling. The Vanilla Seq2Seq model suffers
from data scarcity and has difficulty recognizing the general patterns in the
training samples due to the small size of the dataset.

Thirdly, the Bi-LSTM + CRF and BERT model achieve better performance than
other models, and BERT models often outperform other models with lower
standard deviation. However, the highest token-level accuracy achieved is
80.41%, which is still not high enough for an accuracy-prioritized task. A
different key may change the requirement entirely. For example, the “width” of
“car windshield” and the “width” of “car” focus on completely different
aspects, although the keywords “car windshield” and “car” have an only one-
word discrepancy. Meanwhile, the best sentence-level accuracy achieved is
60.82%, which means that about 40% of the requirements are falsely translated.
Assuming the policy makers fix these requirements through the intelligent
assistant interface, it is time-consuming and reduces the user experience.
Even worse, it may bring safety issues to the monitoring system without
noticing.

In summary, the results indicate that existing language models are not
sufficient to serve as the translation model for CitySpec directly. There is a
high demand for injecting city knowledge to build the translation model.

###  V-B Requirement Synthesis with City Knowledge

Figure 4: Performance improvement brought by requirement synthesis

Next, we evaluate CitySpec’s performance with our controllable synthesized
requirements. For a fair comparison, we do not use the requirements in the
testing set to synthesize requirements. We ensure that the trained model has
not seen the requirements in the testing set in either the knowledge injection
or training phases. We apply different synthesis indexes to test the effects
on the prediction performance. We present the overall results on token-level
and sentence-level accuracy in Figure 4, and F-1 scores on individual keyword
in Figure 5. In the figures, x-axis represents the synthesis index. When the
index equals to “inital”, it shows the results without synthesis data.

From the results, we find that, for BERT and Bi-LSTM, there is an overall
increase in performance in all token-level accuracy, sentence-level accuracy,
overall F-1 score, and F-1 score on keywords. For example, BiLSTM+CRF’s token-
level accuracy increases from 77.59% to 97% and sentence-level accuracy
increases from 60.82% to 81.3%, BERT’s sentence-level accuracy increases from
59.02% to 86.64%.

In summary, the results show that injecting city knowledge with synthesized
requirements boosts the translation model significantly. While improving
policy maker’s user experience with higher accuracy and less clarifications,
it also enhances the safety of the monitoring system potentially.

###  V-C Performance on Online Validation

We evaluate the validation model through simulating four different testing
scenarios: (I) randomly generated malicious input based on the permutation of
letters and symbols; (II) all street names in Nashville; (III) real city
vocabulary generated from Nashville requirements; (IV) generated float numbers
with different units.

First of all, the accuracy of validation model is very high. When the
uncertainty threshold is set to 0.5, i.e., all inputs cause an uncertainty
higher than 0.5 will be ruled out, CitySpec gives 100% success rate against
scenario I among 2,000 malicious inputs, 91.40% acceptance rate among 2,107
samples in scenario II, 92.12% acceptance rate among 596 samples in scenario
III, and 94.51% acceptance rate among 2,040 samples in scenario IV.
Additionally, we find that the validation function easily confuses
$\mathsf{entity}$ with $\mathsf{quantifier}$ if no further guidance is
offered. We look into dataset and figure out $\mathsf{entity}$ and
$\mathsf{quantifier}$ are confusing to even humans without any context
information. Take the requirement “In all buildings, the average concentration
of Sulfur dioxide (SO2) should be no more than 0.15 mg/m3 for every day.” As
an example, $\mathsf{entity}$ is “concentration” and $\mathsf{quantifier}$ is
“Sulfur dioxide (SO2)”. If the requirement is changed to “The maximum level of
the concentration of Sulfur dioxide (SO2) should be no more than 0.15 mg / m3
for every day.”, then $\mathsf{entity}$ is “maximum level” and
$\mathsf{quantifier}$ is “concentration” instead. In addition, terms like
“occupancy of a shopping mall”, “noise level at a shopping mall”, and “the
shopping mall of the commercial district” also introduce confusion between
$\mathsf{location}$, $\mathsf{entity}$ and $\mathsf{quantifier}$, since the
same token “shopping mall” can be $\mathsf{entity}$, $\mathsf{quantifier}$ or
$\mathsf{location}$ in certain cases.

The results show that the validation algorithm can effectively accept new city
knowledge, prevent adversarial inputs and safeguard online learning.
Therefore, CitySpec reduces unnecessary interactions between policy makers and
the system and increases efficiency.

Figure 5: F-1 scores on four keywords

###  V-D Adaptability to different scenarios

TABLE III: Adaptability on different cities in terms of token-level accuracy,
sent-level accuracy, and overall F-1 score

City | Seattle | Changsha | Charlottesville | Jacksonville
---|---|---|---|---
Metrics | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1
Non-adaptive w/ BiLSTM+CRF | 84.91% | 48.00% | 77.60% | 86.61% | 61.20% | 84.10% | 90.00% | 65.62% | 86.48% | 77.32% | 35.20% | 81.54%
Adapted w/ BiLSTM+CRF | 96.05% | 84.80% | 93.75% | 95.27% | 83.20% | 93.57% | 96.82% | 89.29% | 95.40% | 97.35% | 88.80% | 96.02%
Non-adaptive w/ BERT | 80.38% | 46.40% | 76.80% | 86.10% | 58.00% | 83.87% | 93.44% | 73.21% | 90.46% | 90.21% | 56.40% | 81.60%
Adapted w/ BERT | 95.10% | 80.70% | 90.28% | 97.16% | 88.40% | 96.88% | 97.53% | 87.05% | 94.31% | 96.72% | 83.20% | 92.59%
Knowledge Injected | 31 (2.34% ) | 23 (1.73%) | 53 (4.07%) | 22 (1.66%)

TABLE IV: Adaptability on different topics in terms of token-level accuracy,
sent-level accuracy, and overall F-1 score

Topic | Noise Control | Public Access | Indoor Air Control | Security
---|---|---|---|---
Metrics | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1 | TokenAcc | SentAcc | F-1
Non-adaptive w/ BiLSTM+CRF | 77.82% | 41.56% | 77.83% | 73.99% | 44.07% | 74.41% | 81.51% | 46.80% | 76.22% | 72.11% | 28.80% | 62.93%
Adapted w/ BiLSTM+CRF | 95.82% | 90.68% | 94.46% | 97.68% | 74.80% | 97.39% | 96.58% | 80.00% | 87.68% | 94.39% | 94.37% | 92.34%
Non-adaptive w/ BERT | 84.15% | 58.62% | 81.05% | 83.59% | 54.17% | 78.93% | 78.51% | 31.20% | 73.88% | 79.31% | 45.60% | 77.50%
Adapted w/ BERT | 98.07% | 88.31% | 92.98% | 97.43% | 88.75% | 95.75% | 95.31% | 74.40% | 93.60% | 95.41% | 82.80% | 93.95%
Knowledge Injected | 57 (4.38%) | 171 (14.37%) | 92 (7.31%) | 41 (3.13%)

In this section, we analyze CitySpec’s adaptability in different cities and
different domains. Different cities have different regulation focuses and
their city-specific vocabulary. For example, in the city of Nashville,
$\mathsf{location}$ names like “Music Row”, “Grand Ole Opry” will probably
never appear in any other cities. We select four cities, Seattle,
Charlottesville, Jacksonville, and Changsha, with different sizes and from
different countries as case studies. We separate the requirements of each
mentioned city and extract the city-wise vocabulary based on each city
independently. Each of four constructed pairs consists of: vocabulary I, which
is extracted from the requirements from one city only, and vocabulary II,
which is extracted from the requirements from all the cities but that specific
one city. Injected knowledge is measured using the number along with the ratio
of how much of the unique vocabulary one city causes. We augment vocabulary II
using 5 as the synthesis index and train a model on vocabulary II. As a
result, the trained model is isolated from the vocabulary information from
that one specific city. Afterward, we test the trained model performance on
the generated requirements using vocabulary I. We pick CitySpec with Bi-LSTM +
CRF and CitySpec with BERT in this scenario. We employ the validation function
to validate all vocab in vocabulary I and pass the validated ones to
vocabulary II. After that, we have a validated vocabulary including vocabulary
II and validated vocabulary I. The deployed model is fine-tuned based on the
validated vocabulary using few-shot learning.

From the results shown in Table III, we observe that (1) although CitySpec
immigrates to a completely unknown city, it is still able to provide
satisfying performance, e.g., 84.9% token-acc and 77.6% F-1 score in Seattle,
but the sent-acc tends to be low. (2) With new knowledge injected, the
performance increases significantly, e.g., Sent-Acc for Seattle increases from
48% to 84.8% with BiLSTM+CRF, and from 46.4% to 80.7% with BERT.

We also explore CitySpec’s adaptability to different topics. We choose four
topics including noise control, indoor air control, security, and public
access. The results also show that (1) even though CitySpec has not seen
vocabulary from a totally different topic, it still gives a competitive
performance; (2) online learning brings obvious improvements when adaptation
is further applied.

In summary, it indicates the capability of CitySpec in both city and domain
adaptation. It can also adapt to new requirements evolving overtime. Moreover,
with a different set of domain-specified knowledge, CitySpec can be
potentially applied to other application domains (e.g., healthcare).

###  V-E Case Study

Due to the absence of a real city policy maker, we emulate the process of
using CitySpec by taking the real-world city requirements and assuming that
they are input by policy makers. Specifically, this case study shows the
iteration of communication between CitySpec and the policy maker to clarify
the requirements. We emulate this process 20 times with 100 requirements
randomly selected from our datasets each time. The results show that the
average and maximum rounds of clarification are 0.8 and 4 per requirement,
respectively, due to the missing or ambiguous information. Averagely, 28.35%
of requirements require clarification on location. For example, “No vendor
should vend after midnight.”, CitySpec asks users to clarify the time range
for “after midnight” and the location defined for this requirement. Overall,
CitySpec obtains an average sentence-level accuracy of 90.60% (with BERT and
synthesize index = 5). The case study further proves the effectiveness of
CitySpec in city requirement specification.

##  VI Related Work

Translation Models. Researchers have developed models to translate the natural
language to machine languages in various domains, such as Bash commands [5],
Seq2SQL [6], and Python codes [7]. These translation models benefit from
enormous datasets. The codex was trained on a 159 GB dataset that contains
over 100 billion tokens. WikiSQL, which Seq2SQL was trained on, consists of
80,654 pairs of English-SQL conversions. NL2Bash [5] was trained on
approximately 10,000 pairs of natural language tasks and their corresponding
bash commands. As an under-exploited domain, there is a very limited number of
well-defined requirements. Therefore, existing translation models do not apply
to our task. This paper develops a data synthesis-based approach to build the
translation model.

Data Synthesis. Data synthesis exploits the patterns in study findings and
synthesizes variations based on those patterns. Data augmentation is a simple
application of data synthesis. Previous augmentation approaches wield tricks
like synonym substitution [12, 13] and blended approaches [14]. In the smart
city scenario, we need new data samples which fit in the smart city context.
Therefore, we extract extra knowledge from smart cities and fully exploit
semantic and syntactic patterns instead of applying straightforward tricks
like chopping, rotating, or zooming. This paper is the first work synthesizing
smart-city-specific requirements to the best of our knowledge.

Online Learning. Online machine learning mainly deals with the situation when
data comes available to the machine learning model sequentially after being
deployed. Similar to continuous learning, online learning aims to give model
accumulated knowledge and improve model performance continuously given
incoming learning samples [15, 16]. Some of the existing papers focus on
developing sophisticated optimization algorithms [17] or exploiting the
differences between new and old samples [18]. However, these papers do not
have a mechanism to detect or prevent adversarial samples online. This paper
develops a two-stage online learning process with online validation against
potential malicious inputs.

##  VII Summary

This paper builds an intelligent assistant system, CitySpec, for requirement
specification in smart cities. CitySpec bridges the gaps between city policy
makers and the monitoring systems. It incorporates city knowledge into the
requirement translation model and adapts to new cities and domains through
online validation and learning. The evaluation results on real-world city
requirement datasets show that CitySpec is able to support policy makers
accurately writing and refining their requirements and outperforms the
baseline approaches. In future work, we plan to have CitySpec used by real
city policy makers, but this is outside the scope of this paper.

## Acknowledgment

This work was funded, in part, by NSF CNS-1952096.

## References

  * [1] M. Ma, J. A. Stankovic, and L. Feng, “Toward formal methods for smart cities,” _Computer_ , vol. 54, no. 9, pp. 39–48, 2021.
  * [2] M. Ma, E. Bartocci, E. Lifland, J. A. Stankovic, and L. Feng, “A novel spatial–temporal specification-based monitoring system for smart cities,” _IEEE Internet of Things Journal_ , vol. 8, no. 15, pp. 11 793–11 806, 2021\.
  * [3] M. Ma, J. A. Stankovic, and L. Feng, “Cityresolver: a decision support system for conflict resolution in smart cities,” in _2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS)_. IEEE, 2018, pp. 55–64.
  * [4] M. Ma, J. Stankovic, E. Bartocci, and L. Feng, “Predictive monitoring with logic-calibrated uncertainty for cyber-physical systems,” _ACM Transactions on Embedded Computing Systems (TECS)_ , vol. 20, no. 5s, pp. 1–25, 2021.
  * [5] Q. Fu, Z. Teng, J. White, and D. C. Schmidt, “A transformer-based approach for translating natural language to bash commands,” in _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2021, pp. 1245–1248.
  * [6] V. Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,” _arXiv preprint arXiv:1709.00103_ , 2017.
  * [7] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman _et al._ , “Evaluating large language models trained on code,” _arXiv preprint arXiv:2107.03374_ , 2021.
  * [8] F. J. Damerau, “A technique for computer detection and correction of spelling errors,” _Communications of the ACM_ , vol. 7, no. 3, pp. 171–176, 1964\.
  * [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [10] A. X. Chang and C. D. Manning, “Sutime: A library for recognizing and normalizing time expressions.” in _Lrec_ , vol. 3735, 2012, p. 3740.
  * [11] B. E. Chapman, S. Lee, H. P. Kang, and W. W. Chapman, “Document-level classification of ct pulmonary angiography reports based on an extension of the context algorithm,” _Journal of biomedical informatics_ , vol. 44, no. 5, pp. 728–737, 2011.
  * [12] S. Kobayashi, “Contextual augmentation: Data augmentation by words with paradigmatic relations,” _arXiv preprint arXiv:1805.06201_ , 2018.
  * [13] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” _Advances in neural information processing systems_ , vol. 28, pp. 649–657, 2015.
  * [14] J. Wei and K. Zou, “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,” _arXiv preprint arXiv:1901.11196_ , 2019.
  * [15] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” _Neural Networks_ , vol. 113, pp. 54–71, 2019.
  * [16] Z. Chen and B. Liu, “Lifelong machine learning,” _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , vol. 12, no. 3, pp. 1–207, 2018\.
  * [17] E. Hazan _et al._ , “Introduction to online convex optimization,” _Foundations and Trends® in Optimization_ , vol. 2, no. 3-4, pp. 157–325, 2016.
  * [18] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.