SDR

    Software-Defined Radio
AGC

    Automatic Gain Control
GCI

    Global Coexistence Interface
ECI

    Enhanced Coexistence Interface
AFH

    Adaptive Frequency Hopping
FEM

    Front-End Module
SP3T

    Single Pole, Triple Throw
HCI

    Host Controller Interface
A2DP

    Advanced Audio Distribution Profile
SCO

    Synchronous Connection-Oriented
GPIO

    General Purpose Input Output
ASLR

    Address Space Layout Randomization
LMP

    Link Management Protocol
LCP

    Link Control Protocol
LNA

    Low-Noise Amplifier
EIR

    Extended Inquiry Response
CRC

    Cyclic Redundancy Check
RTOS

    Real-Time Operating System
QEMU

    Quick Emulator
UART

    Universal Asynchronous Receiver Transmitter
MitM

    Machine-in-the-Middle
MAC

    Media Access Control
BLE

    Bluetooth Low Energy
ACL

    Asynchronous Connection-Less
DSP

    Digital Signal Processing
SDIO

    Secure Digital Input Output
BCS

    Bluetooth Core Scheduler
ELF

    Executable and Linking Format
GIAC

    Global Inquiry Access Code
JSON

    JavaScript Object Notation
L2CAP

    Logical Link Control and Adaptation Protocol
LMP

    Link Management Protocol
PTM

    Pseudo Terminal Master
PTS

    Pseudo Terminal Slave
QEMU

    Quick Emulator
RCE

    Remote Code Execution
RFU

    Reserved for Future Use
SVC

    Supervisor Call
UART

    Universal Asynchronous Receiver Transmitter
WICED

    Wireless Internet Connectivity for Embedded Devices
MMIO

    Memory Mapped Input/Output
LE

    Bluetooth Low Energy
IoT

    Internet of Things
IDE

    Integrated Development Environment
ARM

    Advanced RISC Machine
LM

    Link Manager
RTOS

    Real-Time Operating System
DMA

    Direct Memory Access
RXDMA

    Receive Direct Memory Access
NVRAM

    Non-Volatile Random-Access Memory
ROM

    Read-Only Memory
Rx

    Receive
Tx

    Transmit
SPI

    Serial Peripheral Interface
MSP

    Main Stack Pointer
PSP

    Process Stack Pointer
RF

    Radio Frequency
BLOB

    Binary Large Object
SCO

    Synchronous Connection Oriented
UAF

    Use-After-Free
FHS

    Frequency Hop Sync
CRC

    Cyclic Redundancy Check
PDU

    Protocol Data Unit
NFC

    Near Field Communication
JPEG

    Joint Photographic Experts Group
LPE

    Local Privilege Escalation
DEP

    Data Execution Prevention
XN

    eXecute Never
LCP

    Link Control Protocol
GATT

    Generic Attribute
PoC

    Proof of Concept
EDR

    Enhanced Data Rate
MWS

    Mobile Wireless Standards
HAL

    Hardware Abstraction Layer
LR

    Link Register
eSIM

    embedded-SIM
RSP

    Remote SIM Provisioning
PC

    Program Counter
MD

    More Data
LLID

    Logical Link Identifier
SFI

    Serial Flash Interface
EEPROM

    Electrically Erasable Programmable Read-Only Memory
TOFU

    Trust On First Use
CVE

    Common Vulnerabilities and Exposure
PCIe

    Peripheral Component Interconnect Express
SECI

    Serial Enhanced Coexistence Interface
HID

    Human Interface Device
DoS

    Denial of Service
PTA

    Packet Traffic Arbitration
AWMA

    Alternating Wireless Medium Access
AP

    Access Point
ECDH

    Elliptic-Curve Diffie–Hellman
BT

    Bluetooth Classic
SSP

    Secure Simple Pairing
SC

    Secure Connections
MIC

    Message Integrity Check

# Happy MitM – Fun and Toys in Every Bluetooth Device

Jiska Classen  Secure Mobile Networking Lab, TU DarmstadtGermany
<EMAIL_ADDRESS>and  Matthias Hollick  Secure Mobile Networking Lab, TU
DarmstadtGermany<EMAIL_ADDRESS>
(2021)

###### Abstract.

Bluetooth pairing establishes trust on first use between two devices by
creating a shared key. Similar to certificate warnings in TLS, the Bluetooth
specification requires warning users upon issues with this key, because this
can indicate ongoing Machine-in-the-Middle (MitM) attacks. This paper uncovers
that none of the major Bluetooth stacks warns users, which violates the
specification. Clear warnings would protect users from recently published and
potential future security issues in Bluetooth authentication and encryption.

Bluetooth, Machine-in-the-Middle, Usable Security

††journalyear: 2021††copyright: acmlicensed††conference: 14th ACM Conference
on Security and Privacy in Wireless and Mobile Networks; June 28-July 2, 2021;
Abu Dhabi, United Arab Emirates††booktitle: 14th ACM Conference on Security
and Privacy in Wireless and Mobile Networks (WiSec ’21), June 28-July 2, 2021,
Abu Dhabi, United Arab Emirates††price: 15.00††doi:
10.1145/3448300.3467822††isbn: 978-1-4503-8349-3/21/06††ccs: Security and
privacy Mobile and wireless security††ccs: Security and privacy Security
protocols††ccs: Security and privacy Key management

##  1\. Introduction

Attacks on Bluetooth pairing often lead to two separate keys, as shown in
Figure 1. Such attacks are either enabled by vulnerabilities in the
specification and implementation (von Tschirschnitz et al., 2021; Tillmanns et
al., 2020; Claverie and Lopes, 2021) or the insecure _Just Works_ mode used by
most IoT devices and headsets (Bluetooth SIG, 2020, p. 985). In practice, an
attacker faces the following barriers:

  1. (1)

Presence during the initial pairing or forcing a new pairing.

  2. (2)

Presence in all future connections to re-encrypt traffic.

In other network protocols, such as TLS, continuous presence can be achieved
by placing a MitM on, e.g., a router close to the target. Bluetooth is used on
devices that move and have varying signal strength. A permanently successful
attacker must be omnipresent and immediately reply with a strong signal to all
connection attempts. If the attacker only fails once—which is very likely in a
mobile environment—devices under attack would use incompatible keys, resulting
in an authentication or encryption failure. According to the Bluetooth 5.2
specification, the user shall be notified of security failures (Bluetooth SIG,
2020, p. 1314). We find that all major Bluetooth stacks skip warning the user,
thereby violating the specification. Warnings are independent from the
underlying pairing method and technology, since pairing and connection
dialogues are implemented on top. We test user interfaces on a large variety
of devices, ranging from Bluetooth 2.1 + EDR to 5.2, including Bluetooth
Classic (BT) and Bluetooth Low Energy (BLE) as well as the pairing extensions
_Google Fast Pair_ and _Apple MagicPairing_ (Google Developers, 2021; Heinze
et al., 2020). More precisely, the following platforms are affected:

  * •

Both tested _Android_ flavors (_Google_ and _Samsung_) do not indicate
authentication failures to the remote device.

  * •

_Google Android_ further silently removes the pairing, which opens the door
for enforcing new pairings.

  * •

_iOS_ , _Samsung Android_ , and _Windows_ display a message that they could
not connect without explaining why, and _macOS_ as well as _Ubuntu Gnome_
indicate a failed connection via user interface button colors. The original
key stays valid.

  * •

Various gadgets do not indicate any error and keys stay valid.

$K_{AM}$$K_{AM}$, $K_{MB}$$K_{MB}$ Figure 1. Most MitM attacks result in two
separate keys.

We demonstrate stack and user interface failures against both BLE and BT,
using F  R IDA (Ravnås, 2021) to dynamically hook the _iOS_ and _Android_
Bluetooth stacks and substitute keys in Host Controller Interface (HCI)
commands. In contrast to existing tools, this allows conditional interaction
with the Bluetooth stack by altering commands and events. These scripts are
now part of the _InternalBlue_ framework (Mantz et al., 2019), as they will be
valuable for further Bluetooth-related research.

This paper is structured as follows. Section 2 explains Bluetooth pairing and
security fundamentals. Then, Section 3 continues with details on how to hook
into Bluetooth stacks to test for security issues. All identified
vulnerabilities are detailed in Section 4. Section 5 concludes this paper.

##  2\. Background

The following section explains Bluetooth pairing basics, recent attacks, and
expected failures in case of MitM presence.

###  2.1. Bluetooth Pairing Modes

Bluetooth pairing modes and warnings in user interfaces are separate
components on all stacks researched in this paper. However, security
vulnerabilities in Bluetooth pairing or encryption enable MitM attacks, and,
thus, motivate clear warnings in user interfaces.

The early BT versions had a very flawed pairing, now termed _Legacy Pairing_ ,
since it still is implemented for backwards compatibility (Shaked, Yaniv and
Wool, Avishai, 2005). BT 2.1 introduced Secure Simple Pairing (SSP), which was
formally verified (Chang and Shmatikov, 2007). Despite this verification, the
specification is unclear about certain aspects of SSP and the follow-up
encryption, resulting in various practical attacks (Hypponen and Haataja,
2007; Biham and Neumann, 2018; von Tschirschnitz et al., 2021; Antonioli et
al., 2020a, 2019; Claverie and Lopes, 2021).

With the Bluetooth specification version 4.0, BLE was introduced, featuring
low-energy connections for IoT gadgets and medical devices. Instead of using
exactly the same pairing mechanism, a lightweight pairing was added, which is
fundamentally broken and now called _LE Legacy Pairing_ (Ryan, 2013). Newer
versions use Secure Connections (SC), which are very similar to SSP. Thus,
attacks on SSP typically also apply to SC. Even worse, the key format in BT
and BLE devices is rather similar, and there is a cross-transport key
derivation for both protocol variants, which also is vulnerable (Antonioli et
al., 2020b).

Table 1. Authentication failure actions as defined by the Bluetooth 5.2 specification (Bluetooth SIG, 2020, p. 1314). Link Key Type |  Bonded |  Action
---|---|---
Combination |  $\times$ |  _Option 1:_ Automatically initiate pairing.
_Option 2, recommended:_ Notify user and ask if pairing is ok.
Combination |  ✓ |  Notify user of security failure.
Unauthenticated |  $\times$ |  _Option 1, recommended:_ Automatically initiate SSP.
_Option 2:_ Notify user and ask if SSP is ok.
Unauthenticated |  ✓ |  Notify user of security failure.
Authenticated |  $\times$ |  _Option 1:_ Automatically initiate SSP.
_Option 2, recommended:_ Notify user and ask if SSP is ok.
Authenticated |  ✓ |  Notify user of security failure.

From a user perspective, the underlying pairing mechanism is opaque. No matter
if SSP, SC, or _LE Legacy Pairing_ is used, they all feature a _Just Works_
mode, which is vulnerable to active MitM attacks (Bluetooth SIG, 2020, p.
985). They also have a _Numeric Comparison_ and _Passkey Entry_ mode, which
can prevent active MitM attacks—assuming that all known vulnerabilities are
fixed. The _Out Of Band_ mode is not affected by these attacks, since it uses
a non-Bluetooth channel for exchanging keys. Its vendor-specific
implementation may or may not have flaws, and attacks on stages after pairing
such as session key entropy reduction (Antonioli et al., 2019) apply either
way.

This attack history without possibility to secure the _Just Works_ mode led to
manufacturers implementing independent pairing solutions. _Apple_ uses so-
called _MagicPairing_ , which is undocumented but has been reverse-engineered
(Heinze et al., 2020). _Google_ has a similar protocol called _Fast Pair_
(Google Developers, 2021). Both protocols bind Bluetooth keys to cloud
accounts and share them across devices. Users only pair a headset once and
then can access it via all devices logged into the same cloud account. This is
not only more convenient but also reduces the amount of pairing attempts
during which MitMs could be present.

###  2.2. Attacks Resulting in Different Keys

The majority of attacks on Bluetooth pairing results in a setup with different
keys. This applies to all devices using _Just Works_ mode as well as attacks
downgrading a pairing to this mode (Hypponen and Haataja, 2007), mixing paring
modes (von Tschirschnitz et al., 2021), and reflecting messages (Claverie and
Lopes, 2021). Besides direct attacks on protocols, implementation details such
as a weak random number generator can also enable MitM attacks (Tillmanns et
al., 2020).

Even a successful attack on the pairing requires a MitM to be omnipresent
during all follow-up connections. Otherwise, inconsistent keys lead to
authentication and encryption errors. These errors can and should be used to
detect attack attempts according to the Bluetooth specification, as described
in the following.

####  2.2.1. Expected Authentication Failure Behavior

If authentication fails, the host terminates the connection (Bluetooth SIG,
2020, p. 1959). Moreover, the user should—in most cases—be warned (Bluetooth
SIG, 2020, p. 1314). The Bluetooth specification has a way more sophisticated
decision process, as shown in Table 1. A _Combination_ key is meant for BT and
BLE, thus, initiating either SSP or SC upon a failure is valid.
_Unauthenticated_ keys are the result of _Just Works_ mode pairing (Bluetooth
SIG, 2020, p. 1306), as it does not protect against MitM. In contrast, an
_Authenticated_ key requires _Numeric Comparison_ , _Passkey Entry_ , or _Out
Of Band_ pairing. _Bonded_ devices permanently store the keys established
during the initial pairing to create a trusted relationship.

If a key is _Unauthenticated_ and does not protect against active MitM, the
recommended option is to automatically initiate a new SSP for non-bonded
devices. This violates the trust on first use concept and MitM attacks can
successfully be launched without any user interaction. This is also the
default option for non-bonded _Combination_ keys as well as the alternative
option for non-bonded _Authenticated_ keys.

Authentication might legitimately fail if one of the devices deleted the
according key. This requires the user to manually reset a device, meaning that
the user is aware and can act accordingly.

####  2.2.2. Expected Encryption Failure Behavior

BLE devices activate encryption using the LE_Enable_Encryption HCI command
(Bluetooth SIG, 2020, p. 2322). Then, the BLE link layer tries to initiate
encryption. After finishing this step, an _Encryption Mode Change_ event is
sent to the host, indicating if encryption is on or still off. When replacing
BLE keys, this results in encryption being reported as off (Bluetooth SIG,
2020, p. 2299). The specification considers encryption failures in BLE in case
that the remote device does not support encryption (Bluetooth SIG, 2020, p.
3141), and there is no differentiation to having an invalid key. Thus, for BLE
encryption failures, the overall behavior is not specified in detail.

##  3\. Bluetooth Stack Test Framework

As explained in the following, over-the-air BT MitM setups are still rather
expensive. To facilitate testing BT and BLE security, we instead dynamically
hook into HCI.

###  3.1. Over-the-Air Setups

As of now, Bluetooth Low Energy (BLE) MitM setups can be realized with the
btlejack toolsuite and three _Micro:Bits_ (Cauquil, 2021). This makes a BLE
setup as cheap as USD 45.

There is still no similar open-source tool for Bluetooth Classic (BT), which
has a more complex modulation scheme making eavesdropping harder. Just in
2020, the first full-band BT sniffer for Software-Defined Radios and more
recent Bluetooth specifications was released (Cominelli et al., 2020). This
sniffer supports synchronization, dewhitening, decoding, as well as an
algorithm to deanonymize addresses. This setup requires two USRP B210,
totaling in approximately USD 3000. For an active MitM setup, even this
advanced testing is insufficient, since it does not support sending packets.

Nonetheless, it is reasonable to assume that strong attackers have a working
MitM setup. From a technological standpoint a MitM only needs a recent SDR
with $80\text{\,}\mathrm{MHz}\text{/}$ bandwidth—and move significant parts of
the implementation into an FPGA to fulfill real-time requirements. Given the
current progress of SDRs, it is to be expected that there will be affordable
BT MitM setups soon, meaning that devices should be proactively secured
against them.

###  3.2. Hooking into Bluetooth Stacks

Analyzing device behavior upon authentication failures due to changed keys
only requires changing the key locally on one of the paired devices. By
changing the key back and forth, it can even be tested if the original key is
still trusted after in between authentication failures. There are two options
to change a key:

  1. (1)

Change the key within the host’s file system.

  2. (2)

Substitute the key within HCI commands.

The first option usually requires to restart the Bluetooth daemon. Moreover,
changing the key on the file system while Bluetooth is still running might
corrupt the local key database.

Replacing key information within HCI commands, the second option, is more
flexible. Most Bluetooth chips only have a ROM and need to request the key
either on first usage or whenever they establish a connection, depending on
the implementation. _Apple_ _MagicPairing_ and _Google Fast Pair_ use this
property (Heinze et al., 2020; Google Developers, 2021): They manage keys
bound to cloud accounts separately, but still use the encryption mechanisms
provided by the Bluetooth chip. Thus, substituting keys as they are requested
even allows testing vendor-specific protocol additions. Key change behavior
can be tested by replacing the HCI_Link_Key_Request_Reply command for BT
(Bluetooth SIG, 2020, p. 720), respectively the LE_Enable_Encryption command
for BLE (Bluetooth SIG, 2020, p. 2322).

Note that it is not possible to use the original version of _InternalBlue_
(Mantz et al., 2019) for injecting different keys into the controller, because
the controller would ask the host for a key belonging to a specific address,
expecting a single response. Prior to the hooks we published along with this
paper, _InternalBlue_ only supported injecting commands but could not replace
contents of existing commands.

The _Linux BlueZ_ (BlueZ Project, 2021) stack stores connection properties in
separate files per connection, making the first option suitable. The HCI-based
alternative on _Android_ and _iOS_ enables us to also test vendor-specific
additions, which are not implemented on _Linux_.

####  3.2.1. Linux

On _Linux_ , keys for BT and BLE connections are stored in
/var/lib/bluetooth/mac1/mac2/info, where mac1 represents the device address of
the controller and mac2 represents the address of the paired device. After
replacing the keys, the Bluetooth daemon must be restarted to refresh
information from these files.

Hooking HCI in user space is infeasible on _Linux_. The Bluetooth daemon only
parses a management protocol, which further abstracts HCI. The _Linux_ kernel
parses this custom management protocol and translates it into HCI commands and
events. Having HCI functionality in the kernel space does not allow F  R IDA-
based hooks. If needed, more flexible hooks could be achieved by modifying the
HCI layer in /net/bluetooth in the kernel source (Linux Kernel Source Tree,
2021).

####  3.2.2. Android

The _Android Fluoride_ Bluetooth stack is open-source. The HCI implementation
is contained in the file system/bt/hci/src/hci_layer.cc (Android Open Source
Project, 2021). Commands to the controller are sent using the transmit_command
function, and events from the controller pass the filter_incoming_event
function. After compilation, these functions end up in the libbluetooth.so
binary.

Despite having source code access, recompiling the stack to swap the key would
not be a flexible solution for more generic HCI analysis. Instead, we hook the
stack using F  R IDA (Ravnås, 2021) on a rooted _Samsung Galaxy Note20 5G_
with the January 2021 patch level. On this device, libbluetooth.so does not
contain symbols. Thus, we locate the relative address of transmit_command
manually using _IDA Pro 7.5_ (Hex-Rays, 2021). Based on this initial _IDA_
database, the address of this function can be found automatically using
_BinDiff_ (zynamics, 2021) if libbluetooth.so was compiled for the same
architecture.

####  3.2.3. iOS

The closed-source _iOS_ Bluetooth stack can be reverse-engineered using debug
strings, which even contain some of the original function names. The part of
the Bluetooth stack responsible for HCI is contained in the Bluetooth daemon
bluetoothd itself instead of using a separate library. Since functions
implementing HCI play an important role, they all contain debug strings. Every
command passes the function OI_HciIfc_CopyPayload, and every incoming event is
processed by OI_HCIIfc_DataReceived.

Similar to the _Android_ setup, we jailbreak an _iPhone 8_ on _iOS 14.4_ with
_checkra1n_ (Kim Jong Cracks, 2020), and use a F  R IDA script to hook HCI.
The function names required for the functionality we need stayed the same
since various _iOS_ releases, at least since _iOS 13.5_ , meaning that it
should be easy to port the hook to future _iOS_ versions.

##  4\. Vulnerable Stacks

We find that under some circumstances controllers do not indicate
authentication failures (see Section 4.1). Even if the controller issues such
a failure, the user is not notified on all tested stacks, but they show
varying behavior as shown in Table 2 (see Section 4.2–4.8). While testing
popular peripherals, we discover that they use dangerously outdated Bluetooth
versions (see Section 4.9). We disclosed all issues to the vendors (see
Section 4.10).

Table 2. User notifications upon authentication and encryption failures due to invalid keys. Key Fault Injector |  Device Under Test |  MacBook (macOS 11.2.1, BT 5) |  Ubuntu+Gnome (Mar 21, BT 4.0) |  Windows 10 (Beta 21327, BT 4.0) |  iPhone 8 (iOS 14.4, BT 5) |  iPhone 12 (iOS 14.5 Beta, BT 5.2) |  Galaxy Note 20 5G (Jan 21, BT 5) |  Google Pixel 5 (Mar 21, BT 5.2) |  AirPods (BLE 5) |  Bose QC35 II (BT 4.2) |  MagicKeyboard (BT 3.0 + HS) |  Mini Keyboard (BT 2.1 + EDR) |  Xiaomi MI Band 2 (BLE 4.1)
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Invalid Key Effect |  |  |  |  |  |  |  |  | ● | ● | ● | ● | ●
iPhone 8 |  |  |  |  |  |  |  |  |  |  |  |  | $\times$
Samsung Galaxy Note 20 5G |  |  |  |  |  |  |  |  | $\times$ |  |  |  |
Ubuntu with Gnome |  |  |  |  |  |  |  |  | $\times$ |  |  |  | $\times$
● No indication, key stays valid.
Button color or symbol indication, key stays valid.
Connection error text message, key stays valid.
Pairing is removed without user notification.

$\times$ Connection type not supported.

###  4.1. Bluetooth Controllers and LMP

While running the attacks, we capture traces using the _Apple PacketLogger_
for _iOS_ and _macOS_ , _Wireshark_ on _Linux_ , and extract the
btsnoop_hci.log from _Android_ devices. All traces show that initiating BT
controllers issue an _Authentication Failure_ event and BLE controllers issue
an _Encryption Failure_ event. This means that the underlying Bluetooth chips
are specification-compliant.

Upon an authentication failure, the host shall terminate the connection
(Bluetooth SIG, 2020, p. 1959). The packet traces indicate that the Bluetooth
hosts indeed follow this procedure—none of the connections in Table 2
persisted after an authentication failure. Interestingly, while the BT
initiator always receives an _Authentication Failure_ via HCI, some responders
only see a disconnect event with the reason _Remote User Terminated
Connection_. We further analyze this by connecting the _Samsung Galaxy Note20
5G_ to a _Google Nexus 5_. The _Nexus 5_ is rather old, but supports Link
Management Protocol (LMP) sniffing via _InternalBlue_ (Mantz et al., 2019) and
features SSP with BT 4.1.

During the secure authentication phase, the initiator and responder can both
end the connection with an LMP_DETACH packet containing the error code
_Authentication Failure_ (Bluetooth SIG, 2020, p. 622). The LMP description of
handling authentication errors is not in line with the HCI part of the
specification. LMP is only accessible by the controller and not the host, but
if the host terminates the connection, it needs to issue an HCI command
towards the controller. If authentication fails on the initiator, the
controller correctly issues an HCI event to the host indicating an
_Authentication Failure_. Then, the host sends an HCI command to terminate the
connection, falsely using the error code _Remote User Terminated Connection_.
Thus, the follow-up LMP_DETACH packet falsely contains the same error code. As
a result, the disconnect event on the responder does not indicate an
authentication failure. This affects at least devices with _Android_ versions
_6.0.1_ – _11_.

In addition, the host shall notify the user of a security issue upon an
_Authentication Failure_ (Bluetooth SIG, 2020, p. 1314). When looking at
popular user interfaces, we avoid that _Android_ will not notify the responder
of authentication failures by testing both ends in the initiator role.

###  4.2. Android (Google)

The _Pixel 5_ on the March 2021 patch level shows the most unexpected
behavior. Upon an authentication failure, the pairing entry is deleted. This
happens without an additional explanation—the user taps the device they want
to connect to and next it disappears from the list of paired devices. Under
certain circumstances, deleting keys is legitimate. Following Table 1, the
specification states:

_“Non-bonded authenticated or unauthenticated link keys may be considered
disposable by either device and may be deleted at any time.”_ (Bluetooth SIG,
2020, p. 1314)

The grammar in the previous sentence is unclear, but we assume that non-bonded
link keys, no matter if authenticated or not, can be deleted. All devices
under test were paired with the _Pixel 5_ using the _Numeric Comparison_
method. The BT link keys were authenticated and bonded. Deleting bonded keys
enables MitM attackers to remove existing pairings with minimal user
interaction—and then launch an attack on the initial pairing. The _Nexus 5_
shows the same behavior on _Android 6.0.1_ , meaning that this issue is
consistent throughout _Google_ -flavored _Android_ versions.

(a) Samsung-flavored Android error message when the tethering AP changes its
key.

(b) iOS error message when the tethering AP changes its key.

(c) Windows error message when the tethering AP changes its key.

(d) Windows error message when the tethering AP is off.

Figure 2. Error messages with an iPhone for tethering.

###  4.3. Android (Samsung)

Using a F  R IDA-based Proof of Concept (PoC) on the _Samsung Galaxy Note20
5G_ on a January 2021 patch level, we test key change behavior over-the-air
against all devices. The _Samsung_ -flavored user interface looks differently
than the _Google_ user interface, including menu structures and texts. When a
BT connection fails due to a changed link key, the message _“Couldn’t
connect.”_ is displayed. In case the paired device supports a special protocol
or capability, this is added to the text. For example, when using tethering
with an _iPhone_ , the message _“Couldn’t connect. Turn on Bluetooth tethering
on iPhone.”_ is shown, as depicted in 2(a). This error message is the same as
when the paired device was switched off, meaning that harmless connection
issues and security-critical authentication errors are indistinguishable for
users.

_Google’s Fast Pair_ protocol substitutes pairing (Google Developers, 2021).
Only a limited subset of devices supports this protocol, such as the _Bose
QC35 II_ headphones. _Fast Pair_ boils down to setting a link key via HCI.
Thus, we can use the same PoC to replace the key. The internal logic of key
management is the same, independent from _Fast Pair_ , and the user is
displayed the same _“Couldn’t connect.”_ message.

On _Android_ , BLE devices usually require using the according app by the
vendor, since BLE is mostly used by IoT devices and has very diverse use
cases. Vendor apps do not provide insights on the current pairing state. Thus,
we use the _nRF Connect_ app (Nordic Semiconductor, 2021), which allows
connecting to BLE services and control the bonding state. The _nRF Connect_
app uses the _Android_ Bluetooth API, meaning that some error messages are
displayed by the system upon errors. For example, when changing the BLE long-
term key for encryption during the initial pairing, the message _“Couldn’t
pair with MI Band 2. Make sure that it’s ready to pair.”_ is displayed. When
changing the encryption key later on in the LE_Start_Encryption command, the
following _Encryption Mode Change_ event indicates that the encryption could
not be switched on while using the wrong encryption key. Thus, _Android_
terminates the connection but does not display any error message.

###  4.4. iOS

We hook the Bluetooth daemon with a F  R IDA-based PoC on an _iPhone 8_ on
_iOS 14.4_. After installing a Bluetooth debug profile, we can use
_PacketLogger_ to observe all HCI packets. This provides us with a powerful
debug tool for the _Apple_ ecosystem, supporting devices like _AirPods_ that
only function with other _Apple_ devices.

When connecting to a BT device, the error message is always _“Connection
Unsuccessful”_ , as shown in 2(b). This error message is the same no matter if
the other device is turned off or if the link key changed. When switching back
to the original key, connections are successful again. The same error message
is shown when porting the PoC to a jailbroken _iPhone 12_ on _iOS 14.1_ and
connecting it to another _iPhone 12_ on _iOS 14.6 Beta_ , because the issue is
anchored in the user interface and not the hardware.

_Apple_ uses _MagicPairing_ for _AirPods_ (Heinze et al., 2020). It leverages
the same mechanism as _Fast Pair_ —exchanging a cloud-based key and then
setting it via an HCI command. When substituting the key in this command, the
same error message is shown.

_iOS_ does not directly support third-party BLE devices, and does not show
them in the scan results. The _nRF Connect_ app for _iOS_ does not support
bonding. Thus, we are not able to test non-_Apple_ BLE devices on _iOS_ in a
comparable fashion.

The _Bose QC35 II_ send BLE advertisements and usually pair using BLE followed
by a cross-transport key derivation to switch to BT. However, after receiving
the first BLE advertisement from a _Bose QC35 II_ , _iOS_ requests further
information via a BT extended inquiry and directly pairs or connects using BT.

###  4.5. macOS

We test the _macOS_ stack by connecting it to devices that switch their key.
The version under test is _macOS 11.2.1_ on a _2020 MacBook Pro_. No matter if
connecting to a device via the menu bar on top or via the full settings
dialogue, buttons temporarily change their color to blue, similar to a connect
and disconnect action, for $23\text{\,}\mathrm{s}\text{/}$. As on _iOS_ , this
dialog does not support non-_Apple_ BLE devices.

###  4.6. Ubuntu with Gnome

A default _Ubuntu 20.10_ installation uses _Gnome_ as user interface on top of
the _BlueZ_ Bluetooth stack. We use a _ThinkPad X240_ with these packets:
gnome-control-center (1:3.38.3-0ubuntu1), bluez (5.55-0ubuntu1.1), and linux-
kernel (5.8.0-44-generic). Instead of hooking into the Bluetooth daemon
itself, we change keys within the file system.

For BT, only button colors change to blue for a short moment. We use the same
_ThinkPad X240_ as for the _Windows_ setup, with the only exception being the
test between _Linux_ and _Windows_ , for which we use a _ThinkPad X1 Yoga_
with BT 4.2 as _Linux_ device. When testing BLE with the _MI Band 2_ , the
initial pairing works, but even without changing the key, reconnecting later
on is not supported. Thus, we could not test the BLE behavior of this user
interface, even though the underlying _BlueZ_ stack supports arbitrary BLE
devices.

The _BlueZ_ stack is by far the most unreliable Bluetooth stack. While testing
the listed devices, we observed one crash in the kernel module and two crashes
in the Bluetooth daemon.

###  4.7. Windows

We use the most recent _Windows 10 Internal Build_ as available in March 2021.
_Windows_ has two menus that can connect to devices. First, the connect side
bar is reachable via  \+ K. This is primarily meant for audio devices and
other devices are only shown as connection information. The _Linux_ laptop is
detected as audio device. Upon a key change, audio devices in this menu show
the following message: _“That didn’t work. Make sure your Bluetooth device is
still discoverable, then try again.”_

To connect _Windows_ to one of the smartphones that change their key, we need
to pair them via the _Settings_ menu, go to _Control Panel $\rightarrow$
Hardware and Sound $\rightarrow$ Devices and Printers_, right-click the paired
smartphone, and connect to it using the _AP_ option. When connecting to a
smartphone with a changed key, the message _“An unexpected error occurred.
Please contact your system administrator.”_ is shown (see 2(c)). The
authentication failure error message is not helpful to determine the root
cause of not being able to connect to the smartphone. Interestingly, _Windows_
has the only interface where the message is different from the case of not
being able to connect to switched off device (see 2(d)).

###  4.8. Peripherals

We test various types of peripherals: headphones, keyboards, and a BLE fitness
tracker. _AirPods_ indicate connections with sounds, the _Bose QC35 II_ even
reads out the currently connected devices and pairing state, the _Xiaomi MI
Band 2_ vibrates during the initial pairing, the _Mini Keyboard_ indicates
pairing states with an LED, and the _MagicKeyboard_ lacks any kind of feedback
mechanism.

Given these limited user interaction capabilities of peripherals, notifying
the user of a security failure, as suggested by the Bluetooth specification
(Bluetooth SIG, 2020, p. 1314), requires special solutions. While error sounds
or status lights would be possible, none of these devices indicate an error
when using a wrong key. When switching back to the correct key, they accept
the connection again.

###  4.9. Outdated Bluetooth Versions

Smartphones tend to have new Bluetooth chips supporting the most recent
specification. Yet, peripherals that require less throughput have surprisingly
old chips. The device labeled as _Mini Keyboard_ is the cheapest keyboard in
the _Adafruit_ store sold for USD 12.95 (Adafruit, 2021). It is using
Bluetooth 2.1 + EDR, which has been released in 2007. At least, this keyboard
implements _Passkey Entry_ authentication. Even popular recent devices such as
standalone _MagicKeyboard_ sold by _Apple_ in 2021 only has Bluetooth 3.0 +
HS, dating back to 2009. On _Broadcom_ chips, as used in this device, the
firmware patching capabilities are rather limited (Ruge et al., 2020). Issues
that stem from the outdated Bluetooth version in this chip cannot be fixed in
software. Most likely due to usability reasons, the _MagicKeyboard_ does not
use numeric verification during wireless pairing.

Keyboards are low-throughput, meaning that old chips do not have any
noticeable effect for users. However, security of keyboards is essential—users
type confidential texts and passwords. In addition to adding warnings on
authentication failures as already required by the Bluetooth specification, we
suggest that users should be warned about outdated Bluetooth versions.

###  4.10. Responsible Disclosure

We contacted the _Bluetooth SIG_ , _Apple_ , _Google_ , and _Samsung_ on
February 27th 2021. After building further PoCs and testing more devices, we
contacted _Microsoft_ , _Bose_ , _Xiaomi_ , and _Gnome_ on March 13th. The
_Bluetooth SIG_ will address the issue. Moreover, _Apple_ , _Google_ , and
_Samsung_ will integrate warnings in a future release, but classified the
issue as feature request. _Microsoft_ stated that they will not change their
warnings. _Xiaomi_ misunderstood the report despite multiple clarifications.
_Bose_ and _Gnome_ did not reply.

##  5\. Conclusion

While many researchers looked into cryptographic aspects of Bluetooth
security, little has been done to raise the bar for practical MitM attacks.
Patching the newest cryptographic bugs within operating systems does not
structurally improve Bluetooth security, as peripherals remain outdated. Users
should be notified of security failures as proposed by the Bluetooth
specification. This would make the life of wireless attackers much harder, as
it significantly reduces attack stealthiness. In addition, users should be
warned if security-sensitive peripherals like keyboards use a 10 year old
Bluetooth version, vulnerable to various known issues. On a long-term
perspective, this would prevent vendors from selling outdated peripherals.
Such structural improvements require everyone to contribute, admit flaws, and
indicate them towards the users—even if this might be inconvenient.

###### Acknowledgements.

We thank Felix Rohrbach and Maximilian von Tschirschnitz for their feedback
and discussion, and Adrian Dabrowski for shepherding our paper. This work has
been funded by the German Federal Ministry of Education and Research and the
Hessen State Ministry for Higher Education, Research and the Arts within their
joint support of the National Research Center for Applied Cybersecurity
ATHENE.

## References

  * (1)
  * Adafruit (2021) Adafruit. 2021\.  Mini Bluetooth Keyboard – Black.  https://www.adafruit.com/product/3601.
  * Android Open Source Project (2021) Android Open Source Project. 2021\.  hci_layer.cc.  https://android.googlesource.com/platform/system/bt/+/refs/heads/master/hci/src/hci_layer.cc.
  * Antonioli et al. (2020a) Daniele Antonioli, Nils Ole Tippenhauer, and Kasper Rasmussen. 2020a.  BIAS: Bluetooth Impersonation AttackS. In _IEEE Symposium on Security and Privacy (S &P)_.
  * Antonioli et al. (2020b) Daniele Antonioli, Nils Ole Tippenhauer, Kasper Rasmussen, and Mathias Payer. 2020b.  BLURtooth: Exploiting Cross-Transport Key Derivation in Bluetooth Classic and Bluetooth Low Energy.
  * Antonioli et al. (2019) Daniele Antonioli, Nils Ole Tippenhauer, and Kasper B. Rasmussen. 2019.  The KNOB is Broken: Exploiting Low Entropy in the Encryption Key Negotiation Of Bluetooth BR/EDR. https://www.usenix.org/conference/usenixsecurity19/presentation/antonioli. In _28th USENIX Security Symposium (USENIX Security 19)_. USENIX Association, Santa Clara, CA, 1047–1061.
  * Biham and Neumann (2018) Eli Biham and Lior Neumann. 2018.  Breaking the Bluetooth Pairing: Fixed Coordinate Invalid Curve Attack.  http://www.cs.technion.ac.il/~biham/BT/bt-fixed-coordinate-invalid-curve-attack.pdf.
  * Bluetooth SIG (2020) Bluetooth SIG. 2020\.  Bluetooth Core Specification 5.2.  https://www.bluetooth.com/specifications/bluetooth-core-specification.
  * BlueZ Project (2021) BlueZ Project. 2021\.  BlueZ - Official Linux Bluetooth Protocol Stack.  http://www.bluez.org/.
  * Cauquil (2021) Damien Cauquil. 2021\.  BtleJack: a new Bluetooth Low Energy swiss-army knife.  https://github.com/virtualabs/btlejack.
  * Chang and Shmatikov (2007) Richard Chang and Vitaly Shmatikov. 2007.  Formal Analysis of Authentication in Bluetooth Device Pairing.  _FCS-ARSPA07_ 45 (2007).
  * Claverie and Lopes (2021) Tristan Claverie and José Lopes. 2021.  BlueMirror: Reflections on Bluetooth Pairing and Provisioning Protocols. In _15th IEEE Workshop on Offensive Technologies (WOOT 21)_.
  * Cominelli et al. (2020) Marco Cominelli, Francesco Gringoli, Paul Patras, Margus Lind, and Guevara Noubir. 2020.  Even Black Cats Cannot Stay Hidden in the Dark: Full-band De-anonymization of Bluetooth Classic Devices. In _IEEE Symposium on Security and Privacy (S &P)_. 534–548.  https://doi.org/10.1109/SP40000.2020.00091
  * Google Developers (2021) Google Developers. 2021\.  Google Fast Pair Service.  https://developers.google.com/nearby/fast-pair/spec.
  * Heinze et al. (2020) Dennis Heinze, Jiska Classen, and Felix Rohrbach. 2020\.  MagicPairing: Apple’s Take on Securing Bluetooth Peripherals. In _Proceedings of the 13th ACM Conference on Security and Privacy in Wireless and Mobile Networks_ (Linz, Austria) _(WiSec ’20)_. Association for Computing Machinery, New York, NY, USA, 111–121.  https://doi.org/10.1145/3395351.3399343
  * Hex-Rays (2021) Hex-Rays. 2021\.  IDA Pro.  https://www.hex-rays.com/products/ida/.
  * Hypponen and Haataja (2007) Konstantin Hypponen and Keijo MJ Haataja. 2007.  “Nino” Man-in-the-Middle Attack on Bluetooth Secure Simple Sairing. In _3rd IEEE/IFIP International Conference in Central Asia on Internet_. IEEE.
  * Kim Jong Cracks (2020) Kim Jong Cracks. 2020\.  checkra1n—iPhone 5s – iPhone X, iOS 12.3 and up.  https://checkra.in/.
  * Linux Kernel Source Tree (2021) Linux Kernel Source Tree. 2021\.  /net/bluetooth in the Linux 5.12-rc3 Kernel.  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/bluetooth?h=v5.12-rc3.
  * Mantz et al. (2019) Dennis Mantz, Jiska Classen, Matthias Schulz, and Matthias Hollick. 2019.  InternalBlue - Bluetooth Binary Patching and Experimentation Framework. In _The 17th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’19)_.  https://doi.org/10.1145/3307334.3326089
  * Nordic Semiconductor (2021) Nordic Semiconductor. 2021\.  nRF Connect for Mobile.  https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Connect-for-mobile.
  * Ravnås (2021) Ole André V. Ravnås. 2021\.  Frida - A world-class dynamic instrumentation framework.  https://frida.re/.
  * Ruge et al. (2020) Jan Ruge, Jiska Classen, Francesco Gringoli, and Matthias Hollick. 2020.  Frankenstein: Advanced Wireless Fuzzing to Exploit New Bluetooth Escalation Targets. In _29th USENIX Security Symposium (USENIX Security 20)_. USENIX Association, 19–36.  https://www.usenix.org/conference/usenixsecurity20/presentation/ruge
  * Ryan (2013) Mike Ryan. 2013\.  Bluetooth: With Low Energy Comes Low Security. In _7th USENIX Workshop on Offensive Technologies (WOOT 13)_.  https://www.usenix.org/system/files/conference/woot13/woot13-ryan.pdf
  * Shaked, Yaniv and Wool, Avishai (2005) Shaked, Yaniv and Wool, Avishai. 2005\.  Cracking the Bluetooth PIN. In _Proceedings of the 3rd International Conference on Mobile Systems, Applications, and Services_. ACM.
  * Tillmanns et al. (2020) Jörn Tillmanns, Jiska Classen, Felix Rohrbach, and Matthias Hollick. 2020.  Firmware Insider: Bluetooth Randomness is Mostly Random. In _14th USENIX Workshop on Offensive Technologies (WOOT 20)_. USENIX Association.  https://www.usenix.org/conference/woot20/presentation/tillmanns
  * von Tschirschnitz et al. (2021) Maximilian von Tschirschnitz, Ludwig Peuckert, Fabian Franzen, and Jens Grossklags. 2021.  Method Confusion Attack on Bluetooth Pairing. In _IEEE Symposium on Security and Privacy (S &P)_. IEEE Computer Society, Los Alamitos, CA, USA, 213–228.
  * zynamics (2021) zynamics. 2021\.  BinDiff.  https://www.zynamics.com/bindiff.html.

  *[MitM]: Machine-in-the-Middle
  *[BT]: Bluetooth Classic
  *[BLE]: Bluetooth Low Energy
  *[HCI]: Host Controller Interface
  *[SSP]: Secure Simple Pairing
  *[SC]: Secure Connections
  *[MitMs]: Machine-in-the-Middle
  *[SDR]: Software-Defined Radio
  *[SDRs]: Software-Defined Radio
  *[LMP]: Link Management Protocol
  *[PoC]: Proof of Concept
  *[PoCs]: Proof of Concept


# DocChecker: Bootstrapping Code Large Language Model for Detecting and
Resolving Code-Comment Inconsistencies

Anh T. V. Dau
FPT Software AI Center
Vietnam
<EMAIL_ADDRESS>
&Jin L.C. Guo
McGill University
Canada
<EMAIL_ADDRESS>
&Nghi D. Q. Bui
Fulbright University
Vietnam
<EMAIL_ADDRESS>

###### Abstract

Comments within source code are essential for developers to comprehend the
code’s purpose and ensure its correct usage. However, as codebases evolve,
maintaining an accurate alignment between the comments and the code becomes
increasingly challenging. Recognizing the growing interest in automated
solutions for detecting and correcting differences between code and its
accompanying comments, current methods rely primarily on heuristic rules. In
contrast, this paper presents DocChecker, a tool powered by deep learning.
DocChecker is adept at identifying inconsistencies between code and comments,
and it can also generate synthetic comments. This capability enables the tool
to detect and correct instances where comments do not accurately reflect their
corresponding code segments. We demonstrate the effectiveness of DocChecker
using the Just-In-Time and CodeXGlue datasets in different settings.
Particularly, DocChecker achieves a new State-of-the-art result of 72.3%
accuracy on the Inconsistency Code-Comment Detection (ICCD) task and 33.64
BLEU-4 on the code summarization task against other Large Language Models
(LLMs), even surpassing GPT 3.5 and CodeLlama.

DocChecker is accessible for use and evaluation. It can be found on GitHub and
as an Online Tool. For a more comprehensive understanding of its
functionality, a demonstration video is available on YouTube.

DocChecker: Bootstrapping Code Large Language Model for Detecting and
Resolving Code-Comment Inconsistencies

Anh T. V. Dau FPT Software AI Center Vietnam<EMAIL_ADDRESS>Jin L.C. Guo
McGill University Canada<EMAIL_ADDRESS>Nghi D. Q. Bui Fulbright University
Vietnam<EMAIL_ADDRESS>

##  1 Introduction

In the realm of Software Engineering, a critical challenge is the detection
and resolution of inconsistencies between source code and its accompanying
comments. Such inconsistencies, which may arise due to code changes not being
reflected in comments or from initially inaccurate descriptions, can lead to
significant issues in both understanding and maintaining software. An
illustrative example of this inconsistency, sourced from the CodeSearchNet
dataset, is depicted in Figure 1 Husain et al. (2019). These disparities can
cause software defects, degrade software quality, and lower developer
productivity, as highlighted in recent studies Wen et al. (2019); Tan et al.
(2012); Panthaplackel et al. (2021); Steiner and Zhang (2022). Moreover, the
prevalent issue of code-comment conflicts in widely used datasets impacts the
efficacy of code language models trained on them Sun et al. (2022); Shi et al.
(2022); Manh et al. (2023).

The advancements in Artificial Intelligence for Software Engineering (AI4SE)
and the development of sophisticated code language models present an
opportunity to address these challenges. These models, trained on extensive
datasets that include both source code and natural language comments, have
shown potential in understanding and processing code at an unprecedented scale
Wang et al. (2021); Nijkamp et al. (2022); Wang et al. (2023); Di Grazia and
Pradel (2023). However, the efficacy of these models is contingent upon the
quality of their training data, underscoring the need for accurate and
consistent code-comment pairs.

Figure 1: An example of code-comment inconsistency from the CodeSearchNet
dataset.

In this context, our work introduces DocChecker, a deep learning-based tool
specifically designed for the task of Inconsistency Code-Comment Detection
(ICCD). Leveraging the capabilities of AI4SE and the insights gained from
advancements in code LLMs, DocChecker addresses the critical need for high-
quality, consistent documentation in software development. The key idea is to
leverage an encoder-decoder framework, pre-trained on code-text pairs. This
pre-training employs a multi-faceted approach, including contrastive learning
to bootstrap code and text features, binary classification to discern
consistent from inconsistent pairs, and text generation to create coherent
comments. The backbone of this system is the UniXcoder Guo et al. (2022),
chosen for its effectiveness and efficiency in handling multi-modal content.
DocChecker is specifically designed not only to detect but also to resolve
inconsistencies between code and comments by generating replacement comments
that accurately reflect the current state of the codebase. Furthermore,
compared to state-of-the-art CodeLLMs, our method excels significantly on ICCD
and code summarization tasks. DocChecker outperforms StarCoder by 30% and
surpasses GPT-3.5 and CodeLlama by 10% in terms of accuracy, even though such
models are pre-trained on larger-scale datasets. In summary, the key
contributions of DocChecker are:

  * •

We propose DocChecker, a tool built on a code language model, jointly pre-
trained with three objectives: contrastive learning, binary classification,
and text generation.

  * •

To the best of our knowledge, DocChecker is the first AI4SE tool designed to
identify and fix inconsistent code-text pairs.

  * •

The experiments show that DocChecker attains state-of-the-art results on ICCD
and code summarization, compared to existing methods and LLMs such as
StarCoder, GPT-3.5, and CodeLlama.

##  2 Related Work

###  2.1 The Pre-trained Code Language Models

In recent years, language modeling has made significant strides, particularly
with the introduction of pre-trained TransformersVaswani et al. (2017); Devlin
et al. (2019); Radford et al. (2018). Pioneered by CodeX Xu et al. (2022),
LLMs have demonstrated remarkable success in code processing, giving rise to
code models such as StarCoder Li et al. (2023), CodeLlama Roziere et al.
(2023). CuBERT Kanade et al. (2020) and CodeBERT Feng et al. (2020) were the
first encoder-only models. CuBERT utilizes masked language modeling and next-
sentence prediction objectives to pre-train on the corpus of Python, while
CodeBERT is pre-trained on six programming languages using the replace token
detection task. Although encoder-only models achieve high performance for
code-understanding tasks, they need an additional decoder for tasks requiring
generation. On the other hand, Svyatkovskiy et al. (2020) and Xu et al. (2022)
employ GPT for the code completion task, but they need to be more suboptimal
for understanding tasks. Some others Wang et al. (2021); Guo et al. (2022);
Wang et al. (2023); Nijkamp et al. (2022) utilize the encoder-decoder
framework to support both understanding and generation tasks. In this work, we
propose DocChecker, which learns semantic relationships between code and text
to detect inconsistent code-comment pairs and generate improved summary
sentences for the replacement.

###  2.2 Code-Comment Inconsistency

The source code comment is very important in understanding the meaning of the
code function. The significance of comments aligning with the source code is
divided into two categories: inconsistent code-comment detection and comment
updates. Rabbi and Siddik (2020) measures the similarity between code
functions and comments, identifying inconsistency when the score falls below a
set threshold. Panthaplackel et al. (2021) develops a deep learning-based
approach to comprehend and establish relationships between comments and code
changes. Instead of using machine learning approaches, others propose rule-
based methods for analysis. Ratol and Robillard (2017) introduces Fraco, an
Eclipse plugin for fragile comment detection during identifier renaming, while
Shi et al. (2022) develops an automated code-comment cleaning tool for
accurate noise detection in the CodeSearchNet dataset Husain et al. (2019).
Although rule-based methods are clear and straightforward, they struggle with
new datasets and lack semantic understanding. Recent research explores
automatic comment updating, with tools like CUP Liu et al. (2021) and HebCUP
Lin et al. (2021) effective for simple changes (a single token change) but not
for complex ones. In contrast, our tool excels at detecting and updating
inconsistent code-comment pairs.

##  3 Overview of DocChecker

In this section, we describe the DocChecker tool as a Python package and
demonstrate its user interface. For full customization and detailed
documentation of DocChecker, users can reference our GitHub repository.

###  3.1 Python Package

We bundle DocChecker into an easy-to-use library that can be installed via
Pypi.

#### Input:

The user must provide their source code file as well as the corresponding
programming language. DocChecker is able to extract all the code function and
their metadata (e.g. function name) by using Tree-sister
111https://github.com/tree-sitter/tree-sitter parser. It supports 10 popular
programming languages: Java, JavaScript, Python, Ruby, Rust, Golang, C#, C++,
C, and PHP. An example of how to use DocChecker is illustrated in Figure 2.

Figure 2: Screenshot for the Input Example.

#### Output:

DocChecker returns in the form of a comprehensive list of dictionaries
corresponding in number to the received code functions, including the name of
each function in the raw code, code snippet, associated docstring, as well as
its prediction, and the recommended docstring. If a code-text pair is
considered as “Inconsistent!”, DocChecker will generate a complete docstring
to replace the old ones; otherwise, it will keep the original version. Figure
3 is a screenshot that shows the result of DocChecker’s prediction.

Figure 3: Screenshot for the Output file Example.

###  3.2 User Interface

Figure 4: Screenshot for the user interface.

We show a demo interface of DocChecker as depicted in Figure 4. It consists of
a coding field for directly entering source code or uploading existing code
files, a select widget for specifying the programming language used for their
code, and a button that serves as the trigger for starting the query process.
When the front-end receives the query result, it displays the previously
mentioned list of dictionaries.

##  4 Building Blocks of DocChecker

DocChecker is a neural architecture designed to learn from code-text pairs,
leveraging an encoder-decoder model. This section outlines the architecture of
DocChecker (see Section 4.1), the objectives guiding its pre-training (Section
4.2), and the specific setup used during pre-training (Section 4.3).
Initially, the model undergoes pre-training focusing on contrastive learning
and code-to-text generation, followed by fine-tuning for the specific
Inconsistency Code-Comment Detection (ICCD) task.

###  4.1 Architecture

Figure 5: Overview of the DocChecker framework.

DocChecker’s design is influenced by the effectiveness of pre-trained models.
Instead of building from scratch, it utilizes existing pre-trained encoder-
decoder models. For this project, we selected UniXcoder, an encoder-decoder
model Guo et al. (2022), as our backbone network due to its customizable
nature and efficient performance with relatively fewer parameters (details in
Section 6.3).

###  4.2 Pre-training Objectives

DocChecker’s pre-training involves three primary objectives:

#### Code-Text Contrastive Learning (CTC):

This aims to align the feature spaces of the code and text encoders. By
emphasizing similarities in positive code-text pairs and differentiating them
from negative pairs, we enhance model accuracy. Negative samples are generated
following the methodology in Li et al. (2021), focusing on hard negative pairs
based on contrastive similarity.

#### Binary Classification (BC):

This objective assesses the alignment between code and text. The model
distinguishes between consistent (positive) and inconsistent (negative) code-
text pairs, enhancing its ability to detect inconsistencies.

#### Text Generation (TG):

This focuses on creating a summary sentence from a given code snippet. By
training the model to optimize cross-entropy loss in an autoregressive manner,
it enhances the model’s capacity to generate coherent comments.

In addition to these objectives, DocChecker benefits from multi-task learning,
sharing the weights between the text encoder and decoder to improve text
representation. Separate, fully-connected layers are utilized to capture task-
specific differences and minimize task interference.

###  4.3 Pre-training Setup

DocChecker employs UniXcoder Guo et al. (2022), renowned for its performance
in multi-modal contexts and its unified cross-modal model. With 12 hidden
layers, 768 hidden sizes, and 3072 intermediate sizes, UniXcoder’s
architecture of 124M parameters proves ideal for our purposes.

The pre-training dataset comes from the CodeXGLUE dataset Lu et al. (2021),
encompassing six programming languages - Python, Javascript, Ruby, Go, Java,
and PHP — sourced from open-source libraries. This diverse dataset is
instrumental in ensuring the model’s proficiency across a wide range of
software engineering scenarios.

##  5 Experiment Setup

In this section, we first present the tasks and the datasets used to assess
the performance of DocChecker. Then, we describe the baselines and metrics
used for evaluation.

###  5.1 Tasks

DocChecker is evaluated for two tasks: ICCD and Code Summarization.

#### ICCD:

Our task is given a comment $C$ with a corresponding code method $M$,
determine whether the comment $C$ is semantically out of sync with the code
function $M$. To address this challenge, we utilize the post-hoc setting in
Panthaplackel et al. (2021), where the code changes that resulted in the
mismatch are unknown; Only the current version of the code snippet and the old
comment are available. This setting is similar to our work, where we want to
detect inconsistency for code-text pairs.

#### Code Summarization:

This task aims to generate a natural language summary from a given piece of
code, playing a significant role in the domain of software comprehension and
documentation. By summarizing key concepts and features into a concise format,
code summarization addresses the challenge of comprehending programming
constructs, especially as codebases continue to grow in complexity.

###  5.2 Datasets

As we assess the performance of DocChecker across two distinct tasks, we rely
on two datasets: the Just-In-Time dataset for the ICCD task and the CodeXGLUE
dataset for the code summarization task.

#### Just-In-Time Dataset:

In this dataset, each sample is the comment-method pair from 2 versions:
before and after updating $(C_{1},M_{1})$ and $(C_{2},M_{2})$ Panthaplackel et
al. (2021). In the post-hoc setting, $C=C_{1}$ and $M=M_{2}$. They assume that
the developer updated the comment because it became inconsistent as a result
of code changes; they take $C_{1}$ to be inconsistent with $M_{2}$,
consequently leading to a negative example. For positive examples, they
additionally examine cases in which $C_{1}=C_{2}$ and assume that the existing
comment has been revised to align consistently with the corresponding code
snippet. For a more reliable evaluation, they manually check to get 300 clean
examples from the test set and note it as the cleaned test set.

#### CodeXGLUE dataset:

This dataset comprises six programming languages: Python, Java, JavaScript,
Ruby, Go, and PHP. They come from publicly available open source non-fork
GitHub repositories, with each documentation representing the first paragraph.
Lu et al. (2021) employed rule-based methods to remove the redundant
information and enhance the overall quality of the dataset compared to its
original version from Husain et al. (2019).

###  5.3 Baselines:

#### Baselines for ICCD task:

We select the following existing work to compare against DocChecker for its
effectiveness on the ICCD task:

  * •

SVM Corazza et al. (2018): This bag-of-words approach classifies whether a
comment is coherent with the method using an SVM with TF-IDF vectors
corresponding to the comment and method;

  * •

Since we first pre-train DocChecker and fine-tune it for downstream tasks, we
also compare our method to two effective pre-trained models: CodeBERT Feng et
al. (2020) and CodeT5 Wang et al. (2021).

  * •

Deep-JIT: Panthaplackel et al. (2021) presents a method for detecting
inconsistencies between natural language comments and source code. With
different ways of encoding the method, they consider three types and note them
as SEQ, GRAPH, HYBRID. Deep-JIT is the existing SOTA method on the Just-In-
Time dataset.

  * •

CodeLLMs: We choose three widely-used and powerful CodeLLMs: GPT-3.5-Turbo,
StarCoder (15B) Li et al. (2023) and CodeLlama (34B) Roziere et al. (2023). We
evaluate LLMs in this task using zero-shot (0-shot) and few-shot (3-shot)
prompting. In zero-shot prompts, no examples from the Just-In-Time dataset are
provided, while in the few-shot experiment, each prompt includes three code-
text pairs along with the correct label from the dataset. Both sets of prompts
are run on all selected LLMs.

#### Baselines for Code Summarization task:

In this experiment, we focus on the fine-tuning setting and compare our method
with smaller-scale LMs, including RoBERTa Liu et al. (2019), CodeBERT Feng et
al. (2020) trained with masked language modeling; PLBART Ahmad et al. (2021)
is based on BART and pre-trained using denoising objective; CodeT5 Wang et al.
(2021), adapted from T5, takes into account important token-type information
in identifiers; and the variant of UniXcoder Guo et al. (2022) since we
utilize UniXcoder as the backbone network.

###  5.4 Metrics

#### Metrics for ICCD:

We use two common classification metrics: F1 score (w.r.t. the positive label)
and Accuracy (Acc) to report the performance of the methods.

#### Metrics for Code Summarization:

For this task, we use the smoothed BLEU-4 Lin and Och (2004) as the evaluation
metric and report the overall score of six programming languages.

Method | Cleaned Test set |  | Full Test Set
---|---|---|---
F1 | Acc |  | F1 | Acc
SVM | 53.9 | 60.7 |  | 54.6 | 60.3
$\text{Deep-JIT}_{\text{SEQ}}$ | 63.0 | 60.3 |  | 66.3 | 62.8
$\text{Deep-JIT}_{\text{GRAPH}}$ | 65.0 | 62.2 |  | 67.2 | 64.6
$\text{Deep-JIT}_{\text{HYBRID}}$ | 63.3 | 55.2 |  | 66.3 | 58.9
CodeBERT | 67.9 | 66.9 |  | 70.7 | 69.8
CodeT5 | 69.5 | 68.8 |  | 70.2 | 70.1
$\text{GPT-3.5 }_{\text{0-shot}}$ | 60.9 | 65.1 |  | 62.5 | 64.6
$\text{StarCoder }_{\text{0-shot}}$ | 43.7 | 43.1 |  | 45.2 | 43.9
$\text{CodeLlama }_{\text{0-shot}}$ | 70.2 | 68.7 |  | 62.6 | 61.8
$\text{GPT-3.5 }_{\text{3-shot}}$ | 66.4 | 67.0 |  | 66.1 | 61.4
$\text{StarCoder }_{\text{3-shot}}$ | 44.2 | 43.6 |  | 42.8 | 42.2
$\text{CodeLlama }_{\text{3-shot}}$ | 70.5 | 69.2 |  | 62.3 | 62.1
DocChecker | 73.1 | 70.7 |  | 74.3 | 72.3
Table 1: Results for post hoc settings on the Just-In-Time dataset

##  6 Evaluation Results

###  6.1 The effectiveness of DocChecker on ICCD

Table 1 presents results for all baselines under the post-hoc setting and
LLMs. In general, we find that our model can significantly outperform all of
the baselines. Despite CodeBERT and CodeT5 being pre-trained models with more
parameters, showcasing efficiency in numerous downstream tasks, their
performance is behind ours. DocChecker achieves a new SoTA of 72.3% accuracy
and 74.3% F1 score on the full test set of Just-In-Time.

On the other hand, although previous literature has empirically explored
various capabilities of LLMs in diverse natural language processing and code
generation tasks, billion-parameter LLMs such as StarCoder, GPT 3.5, and
CodeLlama still struggle with ICCD, even with the construction of various
types of prompts. In particular, DocChecker produces significant improvements
of +10% accuracy and F1 score compared to the selected LLMs.

The experiment results suggest that DocChecker benefits from using a pre-
trained deep learning model with the three above objectives. It supports that
our method effectively detects inconsistent samples in the code corpus.

###  6.2 The effectiveness of DocChecker on Code Summarization

Method | Summarization
---|---
BLEU-4
RoBERTa | 16.57
CodeBERT | 17.83
PLBART | 18.32
CodeT5-small | 19.14
CodeT5-base | 19.55
UniXcoder | 19.30
-w/o contras | 19.20
-w/o cross-gen | 19.27
-w/o comment | 18.97
-w/o AST | 19.33
-using BFS | 19.24
-using DFS | 19.25
DocChecker | 33.64
Table 2: Results on the code summarization task.

Our results on this task are shown in Table 2. DocChecker is compared to a
number of pre-trained code language models in our evaluation. Following
DocChecker’s pre-training with the three aforementioned objectives, our method
outperforms others significantly. DocChecker’s BLEU-4 score is twice as high
as that of RoBERTa and CodeBERT. Furthermore, though CodeT5-base employs a
12-layer encoder and a 12-layer decoder, which is two-fold the capacity of our
architecture, its result is significantly lower than ours. DocChecker yields
substantial improvements over CodeT5 and the backbone network UniXcoder by +13
BLEU-4 score.

###  6.3 The influence of the backbone network on DocChecker

As DocChecker functions as a framework, the selection of an encoder-decoder
model for the backbone network is flexible. In this section, we illustrate the
impact of several pre-trained models on DocChecker’s effectiveness. We
integrate CodeBERT, CodeT5, and UniXcoder as the backbone network for
DocChecker. Each chosen backbone undergoes pre-training within the DocChecker
framework and is fine-tuned on the Just-In-Time dataset. The results,
presented in Table 3, reveal that the pre-trained models exhibit improved
performance after re-pre-training compared to their original versions.
However, UniXcoder emerges as the most effective backbone model for this task,
leading us to adopt it for all our experiments.

Backbone Network | Cleaned test set |  | Full test set
---|---|---|---
F1 | Acc |  | F1 | Acc
CodeBERT | 68.2 | 67.1 |  | 71.5 | 70.4
CodeT5 | 70.1 | 69.5 |  | 71.9 | 71.5
UniXcoder | 73.1 | 70.7 |  | 74.3 | 72.3
Table 3: Results of DocChecker pre-trained with different backbone networks on
the Just-In-Time dataset.

###  6.4 Practical Application

Besides showing DocChecker’s performance, we highlight its efficacy in real-
world scenarios. We consider the widely used CodeXGlue dataset, which extracts
functions and their paired comments from Github repositories. Although this
benchmark dataset is expected to be of good quality, noise is inevitable due
to the differences in coding conventions and assumptions employed in modern
programming languages and IDEs. Using DocChecker, we can filter the
inconsistent code-comment samples within the dataset and generate new
comprehensive summary sentences for them.

Figure 6 illustrates an example of an inconsistent sample detected by
DocChecker in the CodeSearchNet dataset. The comment associated with the code
snippet is misaligned and requires an update. Beyond detection, our method
also provides a detailed summary sentence for each sample, serving as a
replacement for the outdated ones.

Figure 6: An inconsistent code-comment example collected from the CodeXGlue
dataset and our recommended docstring to replace.

##  7 Conclusion

In this paper, we present DocChecker, a tool to filter and generate
replacement comments for inconsistent code-comment pairs. The experimental
results demonstrate the effectiveness of this method compared to the SoTA
existing methods and LLMs, showcasing its applicability in both academic and
practical contexts. We have released DocChecker as an easy-to-use library,
complemented by a user-friendly interface to enhance user interaction.

## References

  * Ahmad et al. (2021) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Unified pre-training for program understanding and generation.  _arXiv preprint arXiv:2103.06333_.
  * Corazza et al. (2018) Anna Corazza, Valerio Maggio, and Giuseppe Scanniello. 2018.  Coherence of comments and method implementations: a dataset and an empirical investigation.  _Software Quality Journal_ , 26:751–777.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Di Grazia and Pradel (2023) Luca Di Grazia and Michael Pradel. 2023.  Code search: A survey of techniques for finding code.  _ACM Comput. Surv._ , 55(11).
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  Codebert: A pre-trained model for programming and natural languages.
  * Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022.  Unixcoder: Unified cross-modal pre-training for code representation.  pages 7212–7225.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _CoRR_ , abs/1909.09436.
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Learning and evaluating contextual embedding of source code.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org.
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023\.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_.
  * Lin et al. (2021) Bo Lin, Shangwen Wang, Kui Liu, Xiaoguang Mao, and Tegawendé F. Bissyandé. 2021\.  Automated comment update: How far are we? In _2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_ , pages 36–46.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  Orange: a method for evaluating automatic evaluation metrics for machine translation.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 501–507.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu et al. (2021) Zhongxin Liu, Xin Xia, David Lo, Meng Yan, and Shanping Li. 2021.  Just-in-time obsolete comment detection and update.  _IEEE Transactions on Software Engineering_ , pages 1–1.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. 2021.  CodeXGLUE: A machine learning benchmark dataset for code understanding and generation.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Manh et al. (2023) Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi D. Q. Bui. 2023.  The vault: A comprehensive multilingual dataset for advancing code understanding and generation.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_.
  * Panthaplackel et al. (2021) Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney. 2021\.  Deep just-in-time inconsistency detection between comments and source code.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 427–435.
  * Rabbi and Siddik (2020) Fazle Rabbi and Md Saeed Siddik. 2020.  Detecting code comment inconsistency using siamese recurrent network.  In _Proceedings of the 28th International Conference on Program Comprehension_ , ICPC ’20, page 371–375. Association for Computing Machinery.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Ratol and Robillard (2017) Inderjot Kaur Ratol and Martin P Robillard. 2017.  Detecting fragile comments.  In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pages 112–122. IEEE.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_.
  * Shi et al. (2022) Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. 2022.  Are we building on the rock? on the importance of data preprocessing for code summarization.  In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 107–119.
  * Steiner and Zhang (2022) Theo Steiner and Rui Zhang. 2022.  Code comment inconsistency detection with bert and longformer.  _arXiv preprint arXiv:2207.14444_.
  * Sun et al. (2022) Zhensu Sun, Li Li, Yan Liu, Xiaoning Du, and Li Li. 2022.  On the importance of building high-quality training datasets for neural code search.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 1609–1620.
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  Intellicode compose: Code generation using transformer.  ESEC/FSE 2020, page 1433–1443.
  * Tan et al. (2012) Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens. 2012.  @tcomment: Testing javadoc comments to detect comment-code inconsistencies.  In _2012 IEEE Fifth International Conference on Software Testing, Verification and Validation_ , pages 260–269.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2023) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.
  * Wen et al. (2019) Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2019.  A large-scale empirical study on code-comment inconsistencies.  In _2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC)_ , pages 53–64. IEEE.
  * Xu et al. (2022) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A systematic evaluation of large language models of code.  In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ , MAPS 2022, page 1–10, New York, NY, USA. Association for Computing Machinery.


# Lightweight and Interpretable Left Ventricular Ejection Fraction Estimation
using Mobile U-Net

###### Abstract

Accurate LVEF measurement is important in clinical practice as it identifies
patients who may be in need of life-prolonging treatments. This paper presents
a deep learning based framework to automatically estimate left ventricular
ejection fraction from an entire 4-chamber apical echocardiogram video. The
aim of the proposed framework is to provide an interpretable and
computationally effective ejection fraction prediction pipeline. A lightweight
Mobile U-Net based network is developed to segment the left ventricle in each
frame of an echocardiogram video. An unsupervised LVEF estimation algorithm is
implemented based on Simpson’s mono-plane method. Experimental results on a
large public dataset show that our proposed approach achieves comparable
accuracy to the state-of-the-art while being significantly more space and time
efficient (with 5 times fewer parameters and 10 times fewer FLOPS).

##  1 Introduction

Left ventricular ejection fraction (LVEF) is one of the most important metrics
in assessing cardiac function. LVEF represents the difference in left-
ventricular end-systolic volume and left-ventricular end-diastolic volume [1].
Accurate calculation of LVEF is important in clinical practice as it
identifies patients who may be in need of life-prolonging treatments. It can
also be used as a prognostic factor to predict adverse outcomes in individuals
due to cardiovascular disease [1].

Clinicians can estimate LVEF manually by analyzing 2D echocardiogram videos
across multiple cardiac cycles. Echocardiogram studies offer several
advantages compared to other medical imaging modalities as they are low cost
and non-invasive. However, echocardiogram imaging also suffers several
limitations including complex analyses during evaluation, high operator
subjectivity, wide operation ranges and observer variability [2].

Automated LV segmentation in echocardiogram studies is a challenge due to
heart structure motion during the cardiac cycle, respiratory interference and
noise/artifacts inherent to ultrasound imaging [2]. Additionally, large
labeled medical datasets are uncommon and most segmentation methods must
account for limited data. Recently proposed frameworks include active
appearance models, level sets and computer vision based frameworks [3]. Many
deep learning based left ventricular segmentation applications have also been
developed.

Multi-domain regularized fully CNNs have shown success in segmenting
anatomical structures in ultrasound images. [4]. U-Net architectures have also
been applied to LV segmentation tasks, however it is not guaranteed that they
will produce anatomically plausible masks across entire videos. [5].

Simpson’s method is the recommended procedure for manually assessing LVEF in a
single plane echocardiogram videos [1]. The method requires accurate
measurements of LV areas throughout a cardiac cycle, averaged over multiple
heart beats. This calculation can be time-consuming for clinicians as it
requires precise annotations on multiple cardiac frames, which are often noisy
and contain artifacts. Many methods have been proposed in recent literature to
automate this process.

Zhang et al. propose a model to interpret echocardiogram studies which
implements a U-Net architecture for LV segmentation and estimates LVEF using
Simpson’s method. [6]. Jafari et al. propose a similar method, in which the LV
and landmarks of interest are segmented using a modified U-Net with
adversarial training [3]. Other proposed architectures use deep learning for
both segmentation and LVEF estimation. Ouyang et al. propose a model which
uses a DeepLabv3 framework to segment the LV and a 3D CNN for LVEF
estimation[7]. A simplified version of this framework is proposed by
Lagopoulos et al. in which a regression model is trained to estimate LVEF
using geometric features of segmentation masks [8]. These frameworks perform
on par with human experts, however they are computationally complex and often
contain hundreds of millions of parameters. This means that they may not be
able to run on point of care ultrasound devices, where computational resources
are limited [3].

Attention based methods have also been proposed to estimate LVEF[9] [10] [11].
Although these ”black-box” methods perform well they are not considered
interpretable because they have no intermediate steps that can be verified by
a clinician [12]. Fully autonomous frameworks are currently not recommended in
clinical practice because they lack transparency.

This paper presents a deep learning based framework to automatically estimate
LVEF from an entire 4-chamber apical echocardiogram video. The proposed
framework aims to be computationally efficient while providing visually
interpretable intermediate results for clinical transparency. We propose a
lightweight Mobile U-Net based network with tracking to segment left ventricle
in each frame of an echocardiogram video. An LVEF estimation algorithm is
implemented based on clinical methods. Experimental results on a large public
dataset show that our proposed approach achieves comparable accuracy to the
state-of-the-art while being space and time efficent.

##  2 Methodology

###  2.1 Left Ventricular Segmentation

Accurate segmentation of the LV is important for estimating LVEF and assessing
cardiac function. The presence of a segmentation component also adds increased
interpretability to LVEF estimation frameworks. A frame level left ventricular
segmentation network was created and trained on the ground truth end-diastolic
(ED) and end-systolic (ES) tracings from the EchoNet dataset[13].

####  2.1.1 Mobile U-Net

Traditional U-Net architectures has approximately 30 million parameters and
have shown exceptional performance on many medical image segmentation tasks. A
U-Net model consists of two components; a contracting (encoder) pathway and an
expanding (decoder) pathway [14]. The contracting path is made up of a generic
convolutional network architecture which is responsible for producing feature
maps at various resolutions. The expanding pathway up-samples the features and
concatenates them with the corresponding feature map from the contracting
path. To create a network with fewer parameters than a traditional U-Net, the
contracting pathway in the proposed framework was replaced by a lightweight
MobileNetV2 based architecture illustrated in Figure 1.

Fig. 1: Full Mobile U-Net architecture for left ventricular segmentation.
$F$:filters, $S$: stride, $K$: kernel size

Mobile U-Net architectures have shown comparable performance to U-Net with
fewer than half of the parameters [15][16]. The encoder pathway is composed of
a convolutional layer followed by 7 Bottleneck Residual blocks, as described
in [17]. The number of filters, kernel size, expansion factor and stride for
each layer are kept the same as described in [17]. The decoder portion of the
U-Net architecture was maintained. The final Mobile U-Net segmentation
framework is composed of approximately 12 million parameters.

####  2.1.2 Mask Track

Fig. 2: Given an input frame $X[n]$ and a mask estimate from the previous
frame $M[n-1]$, MaskTrack predicts a refined mask for the current frame $M[n]$

Though we have access to a large dataset of echocardiogram videos, annotated
masks are available for less than 1% of the training frames. Instead of
focusing on LV segmentation as a per-frame problem, we attempt to treat the
problem as a video object segmentation task in which a single mask is tracked
across an entire echocardiogram video. We propose modifying Mobile U-Net to
behave as a MaskTrack architecture based on [18]. MaskTrack is a guided
instance segmentation framework that can be appended to any convolutional
network. The model learns to leverage the previous mask with the spatial
information in the input frame to produce smooth video segmentation, shown in
Figure 2.

Any existing convolutional network which segments three channel (RGB) images
can be converted to a MaskTrack framework by expanding it to take RGB+mask
input. The only change made to Mobile U-Net is an added channel in the first
convolutional layer of the network, which takes the mask channel as input. The
model is trained on frames with annotated ground truths. A simulated previous
frame is generated for each input image by applying random affine
transformations and non-rigid deformations to the annotation. These
transformations aim to simulate the movement of the left-ventricle across
frames.

Fig. 3: Data augmentation for an LV ground truth mask

Affine transforms consist of random scaling $\pm$10% of the original mask size
and translation of $\pm$10% of the frame size. Non-rigid deformations are
performed via thin-plate spline (TPS) mapping. The thin-plate spline warp is
approximated by sampling a subset of 5 points on the contour and randomly
shifting them $\pm$ 10% of the mask width and height. The TPS coefficients are
calculated and then applied to the remaining points on contour [19]. Figure 3
shows the warps applied to a random mask. Thin plate spline transformations
are well suited to LV data augmentation as they have been used to model
biomedical images and have been used successfully in data augmentation
frameworks for MRI images [20] [21].

###  2.2 Unsupervised Estimation of LVEF

The LVEF framework estimates ejection fraction based on the predicted masks
from the segmentation network. LV volume is calculated for each frame using
Simpson’s monoplane (area length) method. Beat analysis is used to identify
individual cardiac cycles from LV volumes and estimate the ejection fraction
for each cardiac cycle. The final LVEF estimation is calculated as the average
of EF across multiple heartbeats.

####  2.2.1 Volume Estimation

The area length method (Equation 1) is used to estimate left ventriclar volume
based on the geometry of the segmentation mask. The length and area of the
mask is calculated for each segmentated left ventricle. The volume estimation
framework implements the algorithm described by Moradi et al. which imitates
the process that clinicians use to find the long axis length in 4-chamber
echocardiograms [22] [1]. For each frame the procedure is as follows:

  1. 1.

Identify contour of mask from the segmentation network.

  2. 2.

Find the minimum enclosing triangle containing the contour using O’Rourkes
algorithm [23].

  3. 3.

For each triangle vertex find the closest point on the contour. The two
closest points on the contour mark the mitral valve annulus. The remaining
point marks the LV apex

  4. 4.

The midline is defined as the line between the apex point and the midpoint of
the mitral valve annulus.

  5. 5.

The midline is intersected with the contour.

  6. 6.

LV length is defined as the distance between the apex and the intersection
point.

  7. 7.

The LV volume estimated using the ellipsoid model equation [1]. Where $A$ and
$L$ represent the mask area and LV length respectively. These values are both
measured in pixels.

$Volume=\frac{8A^{2}}{3\pi L}$ (1)

Figure 4 illustrates this procedure on a sample frame in the EchoNet Dynamic
Dataset.

Fig. 4: Cyan: mask contour. Green: minimum enclosing triangle. Blue: bounds of
the mitral valve annulus. Red: LV apex. Yellow: intersection of midline and
contour. Magenta: LV length measurement

####  2.2.2 Beat Analysis

Ejection fraction is predicted for the entire video by averaging multiple
beat-level EF estimations. This allows the framework to accept videos of
varying length containing any number of cardiac cycles. The volumes for an
input clip are passed through a median filter to remove outliers. Peaks and
troughs are identified in the filtered volume signal, demonstrated in Figure
5. Peaks indicate maximum LV volume (end-diastole) and indicate minimum LV
volume (end-systole). Each cardiac cycle is defined between adjacent troughs
and the end-diastolic volume and end-systolic volumes are determined. Ejection
fraction for each cycle ($k$) is calculated using Equation 2.

$EF_{k}=\frac{V_{ED}-V_{ES}}{V_{ED}}$ (2) $EF=\frac{1}{N}\sum_{k=1}^{N}EF_{k}$
(3)

The final estimate of ejection fraction for the video is the mean of the
ejection fraction in each beat, Equation 3. This process is similar to the
beat-to-beat assessment method described in Ouyang et. al [7], however this
method calculates each EF directly based on the segmentation volumes rather
than averaging EF volumes estimated by an external network. This concept
aligns with clinical guidelines which recommend averaging ejection fraction
across multiple beats [24].

Fig. 5: a) Volume signal with five identified cardiac cycles. b) Volume signal
with three identified cardiac cycles

##  3 Experiments and Results

###  3.1 Dataset

The EchoNet Dynamic dataset was used to train and evaluate the proposed
framework [13]. It consists of 10 030 apical-4 chamber echocardiogram videos
of healthy hearts and various pathologies. The videos vary in length, frame
rate and image quality. Each video has a set of associated measurements
obtained by a clinician, including end-diastolic volume, end- systolic volume
and LVEF. Tracings of the left ventricle are provided for the ED and ES frames
of one cardiac cycle in each video. The dataset is divided into training,
validation and testing subsets, with 75% of videos in the training set and
12.5% of data in the validation and test set respectively. The distribution of
data is similar in each cohort and dataset summary statistics are reported in
[13].

###  3.2 Results

Several experiments were conducted to assess the effectiveness of each
component in the proposed framework. MU-Net outperformed MUNet + MaskTrack in
all evaluation methods.

Table 1: Metrics comparing MU-Net and MU-Net + MaskTrack Framework | LVEF MAE | DSC
---|---|---
MU-Net | 6.61 | 0.905
MU-Net + MaskTrack | 8.24 | 0.850

####  3.2.1 Segmentation

Dice coefficent (DSC) was used as a metric to assess the segmentation
frameworks. Dice coefficent is a pixel-wise measure of agreement between a
predicted mask ($X$) and ground-truth ($Y$). The formula is given by:

$DSC=\frac{2|X\cap Y|}{|X|+|Y|}$ (4)

A ground truth tracing is provided on one systolic and diastolic frame for
each video in the EchoNet Dynamic dataset. This results in a test set of 2552
frames to assess the segmentation performance of the network. Table 1 shows
the DSC for MU-Net and MU-Net + MaskTrack. MU-Net + Mask Track has a lower
average DSC on the ground truth frames than MU-Net, likely because error
accumulates over time. Well performing examples of MaskTrack produce visually
smoother video segmentation than MU-Net (Figure 6), however the performance on
intermediate frames cannot be assessed because ground truth frames are only
provided for end diastole and end systole. Figure 7 illustrates that MU-Net +
Mask Track produces a smoother volumes graph than MU-Net.

Fig. 6: Top: LV segmentations using MU-Net. Bottom: LV segmentations using
Mobile U-Net + Mask track

Fig. 7: Volumes signals for Mobile U-Net and Mobile U-Net + MaskTrack

####  3.2.2 Ejection Fraction Estimation

The performance of the framework was assessed by taking the mean-absolute-
error between the predicted ejection fraction and the ground truth for each
video. Mean absolute error is taken as:

$MAE=\frac{1}{N}\sum_{k=1}^{N}|\text{EF predicted}_{k}-\text{EF true}_{k}|$
(5)

The best performance was achieved by MU-Net when estimating LVEF as the mean
of the calculated ejection fraction for each identified cardiac cycle.

####  3.2.3 LVEF Classification

In practice a clinician will often classify LVEF measurements into the ranges:
reduced LVEF (rEF; LVEF $<$ 0.40), moderately reduced LVEF (mrEF; LVEF
0.40-0.49) and preserved LVEF (pEF; LVEF $>$ 0.50). Clinicians are required to
make these classifications when diagnosing and treating heart failure [24]. To
gauge the performance of the model in assigning an echocardiogram video to an
LVEF range, the proposed framework was applied to an LVEF classification task.
A label was assigned to each video in the test set based on the predicted
ejection fraction.

Table 2: Classification performance of the proposed frameworks. $F1_{\mu}$: F1
score obtained by micro-averaging based on class frequency. $F1_{m}$: F1 score
obtained by taking the unweighted mean of all per-class F1 scores

Framework | $F1_{\mu}$ | $F1_{m}$ | $\text{Recall}_{m}$ | $\text{Precision}_{m}$
---|---|---|---|---
MU-Net | 0.828 | 0.621 | 0.593 | 0.671
MU-Net + MaskTrack | 0.777 | 0.538 | 0.517 | 0.578

Table 2 displays the results of LVEF classification using MU-Net and MU-
Net+MaskTrack. The Mobile U-Net framework without MaskTrack performed best.
This framework was successful at predicting the rEF and pEF classes, with F1
scores of 0.92 and 0.65 respectively. The mrEF class had a far worse F1 score
of 0.29. From inspection of the confusion matrix (Table 3) we can see that
mrEF cases are often mislabeled as pEF by the proposed framework.

Table 3: Confusion matrix for the LVEF classification using Mobile U-Net | Predicted Class
---|---
|  pEF rEF mrEF
True Class | pEF | 932 | 6 | 53
rEF | 28 | 86 | 46
mrEF | 76 | 11 | 38

###  3.3 Comparison to State of the Art

The proposed method was compared with six other LVEF estimation frameworks.
Each framework in Table 4 recorded test metrics on the Echo-Net Dynamic
dataset. Though the best performance of the proposed model is lower than the
other frameworks, the reported MAE (6.61%) is within interobserver variation
(which ranges from 7.9% to 13.9%).

The proposed framework is more space and time efficient than existent
architectures. Additionally, many proposed solutions implement a ”black-box”
approach, where LVEF is estimated directly from an echo clip with no
intermediate steps [9][10][11]. Though these methods produce accurate results,
they may be difficult for clinicians to interpret. The proposed framework
outputs the segmented mask video, allowing a clinician to verify the EF
estimation by examining the accuracy of the masks.

Table 4: Comparison of the proposed framework to other LVEF estimation
frameworks.

Model | MAE | RMSE | Params | FLOPS
---|---|---|---|---
Ouyang et al.[7] | 4.05 | 5.32 | 71.1M | 91.97G
Lagopolous et al.[8] | 5.08 | 6.92 |  39.9M | -
Reynaud et al.[9] | 5.95 | 8.38 | - | 130.0G
Fazry et al.[10] | 5.72 | 7.63 | 49.7M | -
Muhtaseb et al.[11] | 3.95 | 5.17 | - | 19.61G
Blaivas et al.[25] | 8.08 | 11.98 | 138.4M | -
MU-Net | 6.61 | 8.91 | 12.38M | 2.06G

##  4 Conclusion

In this work we propose a lightweight framework for estimating LVEF in
echocardiogram videos. The proposed framework provides visually interpretable
intermediate results, compared to existing ”black-box” models. Though the
proposed framework has higher MAE than some existent architectures it contains
approximately 12 million parameters, while have hundreds of millions of
parameters. Our lightweight model has the potential to be used in point of
care ultrasound settings, where computational resources may be limited.

Though the MaskTrack module did not improve the results of Mobile U-Net, it
produced visually smooth video segmentation. In the future we would like to
experiment with both the MU-Net and Masktrack on other biomedical video
datasets. The code from this work is available at:
https://github.com/megmuldoon/lvef_estimation.

##  5 Compliance with ethical standards

This study was conducted retrospectively using human subject data made
available in open access by [13].

##  6 Acknowledgments

No funding was received for conducting this study. The authors have no
relevant financial or non-financial interests to disclose

## References

  * [1] Ateet Kosaraju et al.,  Left Ventricular Ejection Fraction,  StatPearls Publishing, Treasure Island (FL), 2022.
  * [2] Jia Zhou et al.,  “Artificial intelligence in echocardiography: detection, functional evaluation, and disease diagnosis,”  Cardiovascular Ultrasound, vol. 19, no. 1, Aug. 2021.
  * [3] Mohammad H. Jafari et al.,  “Automatic biplane left ventricular ejection fraction estimation with mobile point-of-care ultrasound using multi-task learning and adversarial training,”  International Journal of Computer Assisted Radiology and Surgery, vol. 14, no. 6, pp. 1027–1037, Apr. 2019.
  * [4] Hao Chen et al.,  “Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images,” 2016.
  * [5] Erik Smistad et al.,  “2d left ventricle segmentation using deep learning,”  in 2017 IEEE International Ultrasonics Symposium (IUS). Sept. 2017, IEEE.
  * [6] Jeffrey Zhang et al.,  “Fully automated echocardiogram interpretation in clinical practice,”  Circulation, vol. 138, no. 16, pp. 1623–1635, Oct. 2018.
  * [7] David Ouyang et al.,  “Video-based AI for beat-to-beat assessment of cardiac function,”  Nature, vol. 580, no. 7802, pp. 252–256, Mar. 2020.
  * [8] Athanasios Lagopoulos and Dimitrios Hristu-Varsakelis,  “Measuring the left ventricular ejection fraction using geometric features,”  in 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS), 2022, pp. 1–6.
  * [9] Hadrien Reynaud et al.,  “Ultrasound video transformers for cardiac ejection fraction estimation,” 2021.
  * [10] Lhuqita Fazry et al.,  “Hierarchical vision transformers for cardiac ejection fraction estimation,”  in 2022 7th International Workshop on Big Data and Information Security (IWBIS). Oct. 2022, IEEE.
  * [11] Rand Muhtaseb and Mohammad Yaqub,  “EchoCoTr: Estimation of the left ventricular ejection fraction from spatiotemporal echocardiography,”  in Lecture Notes in Computer Science, pp. 370–379. Springer Nature Switzerland, 2022.
  * [12] Navid Hasani et al.,  “Trustworthy artificial intelligence in medical imaging,”  PET Clinics, vol. 17, no. 1, pp. 1–12, Jan. 2022.
  * [13] David Ouyang et al.,  “Echonet-dynamic: a large new cardiac motion video data resource for medical machine learning,”  2019\.
  * [14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Lecture Notes in Computer Science, pp. 234–241. Springer International Publishing, 2015.
  * [15] Junfeng Jing, Zhen Wang, Matthias Rätsch, and Huanhuan Zhang,  “Mobile-unet: An efficient convolutional neural network for fabric defect detection,”  Textile Research Journal, vol. 92, no. 1-2, pp. 30–42, May 2020\.
  * [16] Muhammad Usman Saeed et al.,  “RMU-net: A novel residual mobile u-net model for brain tumor segmentation from MR images,”  Electronics, vol. 10, no. 16, pp. 1962, Aug. 2021.
  * [17] Mark Sandler et al.,  “Mobilenetv2: Inverted residuals and linear bottlenecks,”  2018\.
  * [18] Anna Khoreva et al.,  “Learning video object segmentation from static images,” 2016.
  * [19] Gianluca Donato and Serge Belongie,  “Approximate thin plate spline mappings,”  in Computer Vision — ECCV 2002, pp. 21–31. Springer Berlin Heidelberg, 2002.
  * [20] Fred L. Bookstein,  “Principal warps: Thin-plate splines and the decomposition of deformations,”  IEEE Transactions on pattern analysis and machine intelligence, vol. 11, no. 6, pp. 567–585, 1989.
  * [21] Adan Lin, Junhao Wu, and Xuan Yang,  “A data augmentation approach to train fully convolutional networks for left ventricle segmentation,”  Magnetic Resonance Imaging, vol. 66, pp. 152–164, Feb. 2020.
  * [22] Shakiba Moradi et al.,  “MFP-unet: A novel deep learning based approach for left ventricle segmentation in echocardiography,”  Physica Medica, vol. 67, pp. 58–69, Nov. 2019.
  * [23] Joseph O'Rourke et al.,  “An optimal algorithm for finding minimal enclosing triangles,”  Journal of Algorithms, vol. 7, no. 2, pp. 258–269, June 1986.
  * [24] Roberto M. Lang et al.,  “Recommendations for cardiac chamber quantification by echocardiography in adults: An update from the american society of echocardiography and the european association of cardiovascular imaging,”  Journal of the American Society of Echocardiography, vol. 28, no. 1, pp. 1–39.e14, Jan. 2015.
  * [25] Michael Blaivas and Laura Blaivas,  “Machine learning algorithm using publicly available echo database for simplified “visual estimation” of left ventricular ejection fraction,”  World Journal of Experimental Medicine, vol. 12, no. 2, pp. 16–25, Mar. 2022.