# MCQUIC - A Multicast Extension for QUIC

Max Franke  TU Berlin  Jake Holland  Akamai  Stefan Schmid  TU Berlin &
Fraunhofer SIT

###### Abstract

Mass live content, such as world cups, the Superbowl or the Olympics, attract
audiences of hundreds of millions of viewers. While such events were
predominantly consumed on TV, more and more viewers follow big events on the
Internet, which poses a scalability challenge: current unicast delivery over
the web comes with large overheads and is inefficient. An attractive
alternative are multicast-based transmissions, however, current solutions have
several drawbacks, mostly related to security and privacy, which prevent them
from being implemented in browsers.

In this paper we introduce a multicast extension to QUIC, a widely popular
transport protocol standardized by the IETF, that solves several of these
problems. It enables multicast delivery by offering encryption as well as
integrity verification of packets distributed over multicast and automatic
unicast fallback, which solves one of multicasts major obstacles to large
scale deployment. It is transparent to applications and can be easily utilized
by simply enabling an option in QUIC. This extension is soley focused on the
transport layer and uses already existing multicast mechanisms on the network
layer.

##  1 Introduction

A big part of today’s network traffic is caused by multiple users
simultaneously accessing the same content. In these scenarios, the predominant
method of delivery over unicast can be highly inefficient. Using multicast
delivery instead would reduce load on many parts of the network
infrastructure, especially content delivery servers and ISP networks, by
removing the need to send the same exact packet multiple times to different
receivers.

Two concrete examples for cases where unicast delivery is reaching its limit
are live streaming and game downloads. As of May 2023, the highest traffic
peak that Akamai, one of the worldwide largest CDN providers, recorded across
its network was 250 terabit per second [Tec22]. A 4k livestream has a bitrate
of around 40 Mbps [Cli21]. This means that 6.25 million concurrent viewers are
enough to fully utilize Akamai’s available capacity. This is less than 2% of
the average viewership of e.g. the EURO finals [UEF20]. This is not even
considering an increase in bitrate due to HDR or 8k. Similarly, new game
releases can cause huge bursts of traffic. One such release is GTA V, as the
fastest selling game of all time it sold over 11 million copies on the first
day [Mak13]. With a file size close to a 100GB [Bhu20], it would take all of
Akamais capacity almost 10 hours to handle the downloads on its launch day.
Since its release, file size has also continuously grown, now reaching up to
231GB for certain games [Aku21]. Both of these cases illustrate that already
today unicast delivery is becoming unsustainable, while traffic demand is
certain to further increase. This might soon lead to an inflection point where
increases in link capacities can no longer keep up with demand. Utilizing
multicast for delivery of the described types of content would offer a
solution for the mentioned scalability problems.

While multicast is widely used for intra-domain use cases, such as mDNS
[CK13], its inter-domain applications are very limited. Even though there
exists a multicast backbone, called the MBONE, it is mostly comprised of
educational networks such as Internet2 and GÉANT, while carrier and ISP
networks usually disable generic native inter-domain multicast. This was and
is largely due to a lack of legitimate applications that use multicast, a lack
of privacy and security [HT00, Kru98, CGI+99] when compared to unicast, and
the large amounts of overhead required to create and maintain any-source
multicast (ASM) trees [DLL+00].

Two recent developments have eased these issues. For one, ASM has been
officially deprecated [ACGE20] for inter-domain use cases, removing much of
the complexity related to finding multicast sources. On the other hand, the
traditional protocols used to maintain multicast trees, most notably PIM
[FHH+16], are being replaced by BIER [WRD+17] which provides a way to route
multicast packets without the need to maintain multicast trees and thus state
on each router. Furthermore, automatic multicast tunneling (AMT) [Bum15]
provides a way to bridge networks that do not support native multicast. As
such, the time might have come to make another attempt to deploy multicast
delivery to the web.

One of the critical pieces that are now missing is a way to enable multicast
reception on end user devices. Most current multicast transport protocols,
such as NORM or FLUTE, focus on reliable file transfers and are thus not
suitable for transmission of live content. Furthermore, since most users use
browsers for web content, finding a way to receive multicast packets in them
would be advantageous. However, browsers rightfully place very high bars when
it comes to user privacy and security. As such, using bare, native multicast
would not be feasible for a variety of reasons, which we will further explain
in Section 3.

To bridge this gap, this paper introduces an extension to the QUIC transport
protocol [IT21], which has recently been specified and is implemented in most
of the popular browsers. It provides many advantages compared to TCP, such as
built-in encryption through TLS, the prevention of head of line blocking by
the option to use multiple streams in parallel or support for mobility by
having the option to keep a connection alive even when a client migrates
networks and changes its IP address. Some of the mechanisms used by MCQUIC are
reused from the Multipath Extension for QUIC that is also currently in the
process of being standardized.[LMC+23] It should be noted that the extension
uses already existing mechanisms and technologies to realize multicast on the
network layer and is exclusively focused on the transport layer.

###  1.1 Our contributions

Utilizing QUIC’s features, in particular built in packet protection and
multiple streams, we introduce MCQUIC, an extension to QUIC that combines both
unicast and multicast to act as a new way to enable multicast reception
support in browsers, which due to a lack of UDP support in them has so far not
been possible.

Our extension allows clients to use encrypted multicast that is protected
against injection of packets by third parties. This provides the option to
trust packets received via multicast on public networks. It can be enabled
with minimal changes to application layer implementations and offers automatic
fallback to unicast QUIC in case the network does not support multicast.

The challenges we will specifically address are:

  * •

Prevention of injection of packets by third parties

  * •

Guaranteeing security and privacy under the Internet threat model

  * •

Guaranteeing compatibility with networks that don’t support multicast by
enabling automatic unicast fallback

###  1.2 Organization

We will first give a motivating example and basic design overview over our
extension in Section 2. Following this in Section 3, we will explain in more
detail how it solves the issues and challenges associated with traditional
multicast. In Section 4 we will then take a closer look at MCQUIC, including
some of the new frames it introduces and the design decisions behind them. We
will list more use cases in Section 5 and provide some related work in section
6. Finally, Section 7 will conclude our work. MCQUIC is still in ongoing
development, the most up-to-date version of its specification can be found on
GitHub111https://github.com/GrumpyOldTroll/draft-jholland-quic-multicast.

##  2 Motivation and design overview

In this section we will give an example to motivate why MCQUIC is a useful
addition to QUIC before giving some high level information on its design
principles.

###  2.1 Motivating example

One of the main potential use cases for MCQUIC is the streaming of live media
in browsers. As mentioned before, more and more people shift their content
consumption from traditional (cable) TV to live streaming. As such, the number
of people that are going to watch live sports digitally in the US alone is
expected to increase from 57.5 million in 2021 to over 90 million by 2025
[Ver21]. This often either occurs in browsers directly or in browser based
apps on smart TVs or devices like Chromecast or AppleTV. As such, it was a
core design requirement for our extension to be able to be used in browsers.
This is also one of the reasons that we choose QUIC as a starting point since
it is, as mentioned, available in all major distributions such as Chromium
(and thus Chromium based browsers like Chrome or Edge), Firefox and Safari.
Importantly, most browsers for security and privacy reasons do not offer a UDP
API which means that using traditional, UDP based, multicast was not an
option. Additionally, using the HTTP-based request response semantics that are
common today are not efficient for multicast. In these, each video segment has
to be specifically requested by the client. This would cause a great amount of
unnecessary load when used with multicast delivery as those requests have to
be sent over unicast and handled separately for each client. Unfortunately,
server push has been recently deprecated from Chromium and gQUIC and is thus
no longer an option. Instead, we envision using WebTransport, the successor to
WebSockets. It allows servers to push data to browsers without it first having
to be requested by the client.

Figure 1 shows a simple deployment using MCQUIC. Big data packets, such as
video segments, are carried over the red multicast channel, which means the
server only has to send them once and they are only transmitted once on each
network segment, while the other three colors represent unicast QUIC
connections used for control and integrity. This includes the client sending
Acknowledgements for each packet received via multicast. As these are
comparatively small amounts of data, compared to the actual content that gets
carried on multicast, the overhead incurred on the server and network is much
lower than sending everything over unicast. Additionally, the unicast
connections could be used to transmit data that is client specific, such as
subtitles for different languages or other auxiliary information. To make sure
this data does not head of line block any video frames, it uses a different
stream ID within QUIC. The unicast connection is also used to verify the
integrity of all packets sent over multicast, guaranteeing to the client that
they come from the intended source. Should any of the clients lose multicast
connectivity e.g. due to a network change, the server will notice this by a
lack of reception of Acknowledgements. As a result, the server might decide to
deliver the data packets over unicast instead of multicast to this specific
client. This guarantees that clients are able to receive all data, even in
cases where no native multicast reception is available. This mechanic is some
what comparable to AMT in that it also makes multicast available in networks
that don’t support native reception.

Figure 1: Schematic overview of a server and three clients using multicast
QUIC. The red connection represents multicast, used to carry big data packets
such as video frames, while the connections in the other 3 colors represent
unicast connections, used only for control.

###  2.2 Design overview

MCQUIC has two main components:

  * •

A regular unicast QUIC connection

  * •

One or more multicast QUIC channels

The QUIC connection gets established like it usually does. It goes through the
TLS handshake and cert chain to authenticate the server to the client and
exchange the keying material required to encrypt the connection from there on
out. The exact handshake process, as many other parts of QUIC, are beyond the
scope of this document and can be looked up in RFC 9000 [IT21] and 9001 [TT21]
respectively. During the handshake, server and client also exchange transport
parameters that are used to specify capabilities and limits of each peer.
These include a list of available cipher suites, the max idle timeout, the max
ack delay etc.

MCQUIC introduces two new transport parameters, one server and one client
specific. These are used to negotiate support for multicast. The client will
also use the parameter to communicate its own requirements and limitations
related to multicast to the server.

The main design addition that MCQUIC brings is that, along with the normal
unicast QUIC connection, clients can join one or more multicast QUIC channels.
These are unidirectional data streams from server to client that carry data
that is relevant for several clients at the same time. A server can instruct a
client to join a specific channel that carries content that is desirable for
the client to receive. Whether or not a channel gets joined is up to the
client, it may refuse the joining of a channel for any reason, e.g. if it
deems that doing so would violate its own bandwidth limits. If a client does
decide to join a channel, it will indicate this to the server by sending a
relevant frame over the unicast connection.

For clients to then receive multicast packets, they have to join an IP
multicast group. There are two major flavors of multicast, the before
mentioned and now deprecated ASM, and source-specific multicast (SSM), which
is the only type supported by our extension. This decision was made as ASM
requires different semantics and additional complexity when compared to SSM.
As the main intended use case for MCQUIC lies in inter-domain applications, it
was deemed that the additional design considerations required were not worth
the effort at this time. The mechanisms for joining a group rely on the MLD
[CV04] and IGMP [CDF+02] protocols. Any packets sent over multicast are
encrypted and a hash for each packet gets securely transferred to allow the
client to verify that it has indeed been sent by the original source and not
injected by a third party. The verification of the packets requires the
calculation of the hash by the receiver, the overheard incurred by this
depends on which hash function is used. After verifying the integrity of a
packet received over multicast, the client will handle it like it would handle
a regular unicast packet. As such, to the application layer, it is transparent
whether or not a packet was received over multicast or unicast.

This completes the brief overview of the extension. Before going into more
detail on its specification, in the next section we will first give an
overview of the how these mechanisms solve the issues with traditional
multicast mentioned in the introduction.

##  3 Challenges solved

As mentioned previously, the main issues traditional multicast deployments
face are related to security, privacy and a lack of support for native
multicast in many networks. In this section we will give a more detailed
explanation on these issues and how our approach solves them.

###  3.1 Security and privacy

MCQUIC allows decryption keys to be distributed over the unicast connection.
This allows packets sent over multicast to be protected by the same AEAD
mechanism as regular unicast QUIC packets. As such, third parties are not able
to gain any information (apart from fingerprinting) on what content is being
carried on a particular multicast channel. However, as the same keying
material is used by all receivers, an attacker could potentially gain access
to the content if it can get a hold of the keys from any of the legitimately
subscribed receivers. This is one of the reasons why it is important to
regularly update the keying material through MC_KEY frames. It is up to the
server to make sure only legitimate clients are sent MC_KEY frames. By
utilizing the same encryption mechanisms that are used by other QUIC
implementations, we can also make use of any already existing hardware
offloading capabilities for cryptographic functions.

###  3.2 Prevention of packet injection

As multicast senders usually have no knowledge of number, much less identity,
of receivers and there is no secured two way communication between them, there
is no way for receivers to verify that a packet that arrived on a multicast
channel has indeed been sent by the intended sender and not by a third party
spoofing its IP address.

In multicast QUIC, we use identity frames that are anchored to a secure
unicast QUIC connection, which required the sender to authenticate itself
through the usual TLS cert chain procedure. The integrity frames use standard
cryptographic hashes that are resistant against collision attacks. As such,
receivers are able to verify each packet received over multicast against its
corresponding hash. If they match it can be assured that the packet was sent
by the intended sender and not injected by a third party.

###  3.3 Fallback to unicast

Currently, most networks do not support generic native multicast. As such,
apps that want to utilize multicast always have to worry about implementing a
fallback mechanism on the application layer. With MCQUIC, applications only
have to indicate if they wish to enable support for multicast. From there, the
QUIC implementation takes care of the rest. If multicast is not supported by
the clients network, it can automatically fallback to unicast and act like a
regular QUIC connection. This fallback is, depending on the QUIC
implementation, transparent to the application as it will always only see
delivered data packets.

##  4 Detailed design

In this section we will give more details on some of the mechanics and design
decisions of MCQUIC. We will first discuss some of the general QUIC mechanisms
and how we adapt or change them to enable our extension before taking a look
at some of the new frames we introduce.

###  4.1 Client limits

During connection establishment, both client and server can include an
additional multicast transport parameter that indicates to the peer that
multicast is supported. The clients parameter includes additional details,
such as a list of supported hash and encryption algorithms and whether or not
IPv4, IPv6, or both are supported for multicast reception. These parameters
can be later updated by the client by sending an MC_LIMITS frame to the server
that contains the same parameters.

There are several reasons as to why we need separate transport parameters
instead of using the ones already negotiated for unicast QUIC. Even though
there might be support for both IPv4 and IPv6 via unicast, the client might be
aware that due to a lack of MLD support, only IPv4 multicast is supported. In
this case, disallowing IPv6 multicast from the get go saves time when the
server would otherwise instruct the client to join an IPv6 multicast channel
only to find out no reception is possible.

The list of hash and encryption algorithms might also differ from unicast QUIC
as it can be assumed that only less sensitive data will be carried over
multicast. This means that there might be an option or desire to use weaker
algorithms, such as SHA-1, to reduce the size of the checksums carried in
integrity frames. While it is true that SHA-1 is no longer considered to be
entirely collision resistant [LP20], it might be a worthwhile trade off to
still use it in certain cases where it is highly unlikely that an attacker can
create a collision in time or where a collision would not pertain to any
critical or sensitive data. In either case, it is always recommended to use
collision resistant algorithms to prevent any such risks from occurring at
all.

Also included in the client side transport parameter are the limits the client
places on all multicast channels combined. These include the maximum number of
announced or simultaneously joined channels, but most importantly the maximum
aggregated rate across all channels. As the data carried over multicast is
often of a steady and pre-determined rate, e.g. in video streams the bitrate
is known beforehand, the server can reliably take restrictions of clients into
consideration when assigning which channels to join. It might determine that
the maximum allowed rate is not high enough to receive a 4K stream, which gets
carried on one channel, so instead instructs the client to join a different
channel that carries the stream in HD. More sophisticated models could make
use of layered video [HHWH15] or similar to provide each client with an
optimal utilization of its available bandwidth.

###  4.2 Multicast channels

In QUIC, clients and servers assign (multiple) connection IDs to each
connection. These are included in each sent packet. This mechanic is used to
enable mobility by making a connection identifiable even if the 5-tuple
changes. For example, any packet sent from server to client contains a client
selected ID so the client can associate it with a connection. In multicast
QUIC channels, we use the connection ID field in packets and replace it with a
channel ID that has the same format. This allows clients to associate each
packet with a multicast channel, instead of a connection. Connection and
channel IDs share the same space. For clients to learn that an ID is a channel
ID and not a connection ID, the server sends channel announce frames that
contain, among other parameters, channel IDs of available multicast channels.

These channels are abstractions that represent a source specific multicast
channel along with additional parameters such as the AEAD algorithm used for
header and packet protection. The secret for the header algorithm is also
included and remains static during the lifetime of a channel while the secret
for packet protection is sent separately in a KEY frame. Channels exist
independent of any single unicast connection and may be joined by many
clients. Their lifetime can range from a few seconds to days or even weeks.
Channels share the stream ID space with each other and the unicast connection.
This allows for the retransmission of lost packets over a different channel or
unicast. Contrarily, each channel has its own packet number space to reduce
the coordination required between channels. In addition to the ANNOUNCE and
KEY frames, the server will also send INTEGRITY frames that include hashes for
each packet sent over the multicast channel. These are used to authenticate
packets and as the hash algorithms used are secure against collision attacks,
this mechanism protects against the injection of packets into the multicast
channel by third parties.

A server will then at some point send a JOIN frame that will indicate to the
client that it is should now join the specified channel. If it decides to do
so, it will achieve this by e.g. sending an IGMP or MLD report which triggers
the joining of the SSM channel. If the network supports multicast and packets
start to arrive, the client will send a STATE frame to the server indicating
this. As the entire multicast process is server driven, the server might
decide that if a specific timeout passes without receiving a STATE frame, that
the clients network does not support multicast. In that case, it can fall back
to unicast and send the data packets over the regular QUIC connection [GD22].

Any packets arriving on a multicast channel get acknowledged by the client
over the unicast connection. This is necessary as the channels are strictly
unidirectional from server to client. This also means that only specific
frames, such as STREAM or DATAGRAM frames, can be sent on them. To reduce load
and improve scalability, acknowledgements can be bundled and don’t need to
happen immediately. Lost packets may be retransmitted. This can occur over
both unicast or multicast, depending on what the server deems more efficient.

###  4.3 Congestion and flow control

As there is no regular congestion or flow control mechanism on channels, the
client specifies a maximum allowed aggregated rate across all channels.
Depending on this rate, the server decides on which channels are suitable for
the client to join to make sure that all joined channels combined stay under
the clients limits. Each channel ANNOUNCE frame also includes the maximum rate
of data that is expected to be sent on that channel. If the client thinks that
joining a channel would violate its limits, it can refuse the join. A client
can also update its allowed maximum aggregated rate, along with other
parameters, during the lifetime of a connection by sending a LIMITS frame.
Both server or client may decide that the client should leave a joined channel
for a variety of reasons. These include, but are not limited to, high
persistent loss on the channel, the end of the channels lifetime, or excessive
amounts of spurious traffic (that may be injected by an attacker).

####  4.3.1 Flow control

The usual mechanisms used for flow control in QUIC are too strict for use in
multicast channels without modification. The multicast extension gives the
client a new responsibility to be able to robustly handle multicast packets
that would exceed its MAX_DATA without aborting the connection, either by
increasing its MAX_DATA as needed to keep up with received multicast packets
or by dropping the packet and leaving the channel (resulting in unicast
fallback), for clients that cannot do so. As there are potentially many
clients joined on a single channel, the server can not adjust its sending of
data by limits imposed by any individual one. Additionally, the new transport
parameter is used by clients to set their limits and change them by sending
new Limit frames. The server has to make sure that the client stays within
those limits by having it either join or leave channels appropriately.

####  4.3.2 Congestion control

The server is aware of any multicast packet loss experienced by the client in
the form of missing MC_ACK frames. The server should make sure that clients
leave channels that experience heavy sustained loss. If several clients that
are subscribed to the same channels experience loss of the same packets, this
might indicate an underlying issue further upstream. Contrarily, if a client
experiences loss on several channels at once it might indicate an issue on its
own end. As such, it should react by lowering its max rate parameter.

###  4.4 New frames

Our extension introduces a total of 9 new frames. Some of these behave similar
to already existing frames, such as the MC_ACK frame that has the same
structure as a regular ACK frame with an added field for the channel ID to
identify for which channel a packet gets acknowledged. We will now give
explanations for the need of some of the new frames.

####  4.4.1 MC_ANNOUNCE

As previously mentioned, servers use Announce frames to send clients
information that are required to later join multicast channels. Servers should
send these frames for any channel the client could potentially be asked to
join in the future. The reason for this is that the IDs used for channels
share their space with the connection IDs used by the client. As such there is
a potential for a collision that should be cleared up by the client before it
is asked to join the channel by the server. Another advantage of decoupling
the announce from the join is that it allows for less overhead in cases when a
client has to temporarily leave a channel, i.e. because it suffered a
degradation of connection quality.

####  4.4.2 MC_JOIN, LEAVE, RETIRE and STATE

These 4 frames are used to handle state changes for announced channels. Join,
leave and retire are sent by the server to instruct the client to switch the
server into the specified state, retire indicating that the client should drop
any stored state associated with that channel. State frames are sent by the
client to the server to indicate that the state change was successful. Figure
2 shows the transition diagram for a QUIC channel from a clients point of
view.

Figure 2: Transition state diagram for a channel from a client’s perspective.

####  4.4.3 MC_INTEGRITY

integrity frames are essentially a list of hashes for packets that are
received via multicast. Each frame also includes the packet number for the
packet the hash list starts from. The first integrity frame can act as the
root of a Merkle tree. As such, it has to be transmitted using the unicast
connection, as only that is initially secured by TLS against injection of
packets by third parties. However, to reduce the unicast overhead even
further, any subsequent frames can be included in packets sent over multicast
as their integrity has already been verified by previous frames, either by
ones included in earlier packets sent via multicast of the original unicast
frame. The exact calculation of the hashes is of course dependent on the
chosen hash algorithm. As mentioned, to reduce the size of the hashes, it
could be considered to use algorithms that are less secure against collision
attacks, especially for integrity frames that are being sent over unicast.

####  4.4.4 MC_KEY

The MC_KEY frame is used to send a secret. The secret is then used to derive
the IV and key in the same way the TLS traffic key is calculated. While
instead of using a KDF a key could be transmitted directly, as the secret is
only used to derive one key and not multiple ones, there are several reasons
for using this approach. For one, the secret itself can be smaller than the
key as the KDF extends it. This results in reduced overhead as the KEY frames
can be smaller. Additionally, many of the built in decryption functions in
various QUIC (software and hardware) implementations are based on secrets and
KDFs instead of plain keys. This means that there is less adaptation required
when implementing multicast support and optimizations can be more easily
carried over.

The server is periodically sending out secrets from which new keys will be
derived. The reason this is done this way instead of the usual QUIC key
rotation utilizing the key phase is to prevent the sharing of secrets to
unauthorized subscribers of the multicast channel. If the secret remained
constant for the entire lifetime of a channel, a single bad acting client
could leak it and compromise the channel going forward.

For clients that are newly joining a channel, the key has to be distributed
via unicast. However, similar to integrity frames, subsequent key updates
could also be done via the multicast channel itself to further reduce unicast
overhead. To provide forward secrecy, key updates are occasionally required to
be done over unicast to prevent any clients that are no longer connected via
unicast from accessing the content of the channel.

To allow the client to know which secret is relevant for which packet, each
Key frame also includes a packet number, indicating starting from which packet
the key has to be used. A key remains active until a new key with a higher
packet number is received. As the packet number is part of the protected
portion of the QUIC header, the header protection key has to remain the same
for the lifetime of the channel as clients would otherwise not be able to
decrypt the header and thus not know when to apply a new key.

##  5 A live streaming use case

One of the main use cases we envision for MCQUIC is the streaming of live
content. One possible deployment would utilize Twitchs WARP [Cur22] protocol
that can be used to stream video over QUIC. In it, each video segment is sent
on a separate QUIC stream. Streams rise in priority to make sure newer
segments get transferred first in case of congestion. We can combine this
mechanism with MCQUIC by having several channels for the different offered
stream qualities. Clients would be told to join the channel which is most
suitable, according to the limits they specified to the server. Since each
video segment is its own stream, clients do not have to worry about receiving
data on a stream that has already been active before they joined. Thus, they
don’t have to consider an already existing Stream offset when joining a
channel. It should be noted that for browser based use cases for MCQUIC, the
common request-response semantics no longer apply. As such, for browser
deployment, WebTransport[Vas22] push or similar would be required.

The given example is just the most basic use case. It is conceivable that in
the future mechanisms like layered video could be used to combine several
channels to achieve a higher overall quality while still allowing for each
channel to be useable on its own. The IETF has only very recently started to
focus on media over QUIC and we hope to also be able to contribute to this
effort in the future.

##  6 Related work

As mentioned in the introduction, there recently has been an influx of work on
multicast.

One major ongoing effort is the TreeDN [GLA23] project championed by Juniper.
It tries to solve a similar problem of delivery of live video content via
multicast. However, instead of aiming at a browser based implementation it
mainly makes use of AMT and the AMT module of the VLC media player. It
provides the ability to stream videos from the MBONE to domestic networks that
do not support native multicast reception by going through an AMT relay that
is deployed on the MBONE. It also provides a content portal, similar to
YouTube or Twitch, on which content can be found and streamed. Finally, it
includes a service that can automatically translate a unicast stream to
multicast and ingest it into the MBONE.

The second major ongoing work relating to multicast is the mentioned BIER
[WRD+17], which tries to change the way multicast routing is done. Instead of
the traditional approach of constructing multicast trees, it works by
separating the Internet into multiple BIER domains. These could take the form
of a company network, an autonomous system or even an entire ISPs network. If
a multicast packet enters a BIER domain through an ingest router, it gets
tagged with IDs for all routers, either inside the domain or on its edge, it
has to be delivered to. This means that routers no longer have to keep state
for multicast trees as all the necessary information for routing is included
in the BIER header of each individual packet. BIER has been implemented and
evaluated in P4 [MLM21]. The downside to this approach is that unlike PIM, it
is not a general purpose protocol but instead requires a different mechanism
for the tagging of packets at the ingress router depending on the use base,
e.g. there are BIER extensions for IS-IS [GPAZ18], OSPF [PNW+18], multicast
VPN [RSA+19] etc.

In general, the advantages of using multicast for content delivery have been
well known. Research in this area reaches from multicast in mobile networks
[JN02] over using overlay networks to connect multicast islands [JCC09] to
multimedia conferencing with multicast [McC99].

##  7 Conclusion

In this paper we introduced a new extension to the QUIC transport protocol
that enables the delivery of (mass media) content via multicast. It overcomes
many of the obstacles traditional multicast implementations and deployments
face by utilizing a base unicast QUIC connection as a security anchor for
multicast channels. It provides the well established scalability benefits of
multicast while only adding a small amount of overhead. Integrity of delivered
packets is guaranteed as well as fallback to unicast delivery in networks that
don’t support multicast.

There are many opportunities for future research related to MCQUIC. For one,
it is the first time multicast senders can get telemetry information from all
their receivers. This information can be used to optimize the management of
channels as well as when to switch between unicast and multicast. Finding good
heuristics or models to determine these configurations is part of possible
future work. On the other hand, enabling multicast delivery for media content
allows to reevaluate other technologies that have so far been deemed
unsuitable or inefficient in unicast scenarios. One example for this is
scalable video encoding, which sees no use currently but might be a good way
to combine multiple multicast channels into one media stream.

## References

  * [ACGE20] Mikael Abrahamsson, Tim Chown, Lenny Giuliano, and Toerless Eckert.  Deprecating Any-Source Multicast (ASM) for Interdomain Multicast.  RFC 8815, August 2020.
  * [Aku21] ANTHONY Akugbe.  15 biggest pc games by file size, ranked.  https://gamerant.com/pc-games-file-size-hd-space-biggest-huge/, 04 2021.
  * [Bhu20] Rahul Bhushan.  Gta v download size.  https://www.sportskeeda.com/esports/gta-v-download-size, 08 2020\.
  * [Bum15] Gregory Bumgardner.  Automatic Multicast Tunneling.  RFC 7450, February 2015.
  * [CDF+02] Bradley Cain, Dr. Steve E. Deering, Bill Fenner, Isidor Kouvelas, and Ajit Thyagarajan.  Internet Group Management Protocol, Version 3.  RFC 3376, October 2002.
  * [CGI+99] Ran Canetti, Juan Garay, Gene Itkis, Daniele Micciancio, Moni Naor, and Benny Pinkas.  Multicast security: A taxonomy and some efficient constructions.  In IEEE INFOCOM’99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No. 99CH36320), volume 2, pages 708–716. IEEE, 1999.
  * [CK13] Stuart Cheshire and Marc Krochmal.  Multicast DNS.  RFC 6762, February 2013.
  * [Cli21] Scott Cliburn.  4k bitrate video: The outlook for 2021 and beyond.  https://www.zype.com/blog/4k-bitrate-video, 06 2021.
  * [Cur22] Luke Curley.  Warp - Segmented Live Media Transport.  Internet-Draft draft-lcurley-warp-01, Internet Engineering Task Force, July 2022.  Work in Progress.
  * [CV04] Luis Costa and Rolland Vida.  Multicast Listener Discovery Version 2 (MLDv2) for IPv6.  RFC 3810, June 2004.
  * [DLL+00] Christophe Diot, Brian Neil Levine, Bryan Lyles, Hassan Kassem, and Doug Balensiefen.  Deployment issues for the ip multicast service and architecture.  IEEE network, 14(1):78–88, 2000.
  * [FHH+16] Bill Fenner, Mark J. Handley, Hugh Holbrook, Isidor Kouvelas, Rishabh Parekh, Zhaohui (Jeffrey) Zhang, and Lianshu Zheng.  Protocol Independent Multicast - Sparse Mode (PIM-SM): Protocol Specification (Revised).  RFC 7761, March 2016.
  * [GD22] James Gruessing and Spencer Dawkins.  Media Over QUIC - Use Cases and Requirements for Media Transport Protocol Design.  Internet-Draft draft-gruessing-moq-requirements-03, Internet Engineering Task Force, November 2022.  Work in Progress.
  * [GLA23] Lenny Giuliano, Chris Lenart, and Rich Adam.  TreeDN- Tree-based CDNs for Live Streaming to Mass Audiences.  Internet-Draft draft-ietf-mops-treedn-00, Internet Engineering Task Force, January 2023.  Work in Progress.
  * [GPAZ18] Les Ginsberg, Tony Przygienda, Sam Aldrin, and Zhaohui (Jeffrey) Zhang.  Bit Index Explicit Replication (BIER) Support via IS-IS.  RFC 8401, June 2018.
  * [HHWH15] Yongkai Huo, Cornelius Hellge, Thomas Wiegand, and Lajos Hanzo.  A tutorial and review on inter-layer fec coded layered video streaming.  IEEE Communications Surveys & Tutorials, 17(2):1166–1207, 2015\.
  * [HT00] Thomas Hardjono and Gene Tsudik.  Ip multicast security: Issues and directions.  In Annales des télécommunications, volume 55, pages 324–340. Springer, 2000.
  * [IT21] Jana Iyengar and Martin Thomson.  QUIC: A UDP-Based Multiplexed and Secure Transport.  RFC 9000, May 2021.
  * [JCC09] Xing Jin, Kan-Leung Cheng, and S-H Gary Chan.  Island multicast: combining ip multicast with overlay data distribution.  IEEE transactions on multimedia, 11(5):1024–1036, 2009.
  * [JN02] Christophe Jelger and Thomas Noel.  Multicast for mobile hosts in ip networks: Progress and challenges.  IEEE Wireless Communications, 9(5):58–64, 2002.
  * [Kru98] Peter S Kruus.  A survey of multicast security issues and architectures.  Technical report, NAVAL RESEARCH LAB WASHINGTON DC, 1998.
  * [LMC+23] Yanmei Liu, Yunfei Ma, Quentin De Coninck, Olivier Bonaventure, Christian Huitema, and Mirja Kühlewind.  Multipath Extension for QUIC.  Internet-Draft draft-ietf-quic-multipath-04, Internet Engineering Task Force, March 2023.  Work in Progress.
  * [LP20] Gaëtan Leurent and Thomas Peyrin.  $\\{$SHA-1$\\}$ is a shambles: First $\\{$Chosen-Prefix$\\}$ collision on $\\{$SHA-1$\\}$ and application to the $\\{$PGP$\\}$ web of trust.  In 29th USENIX Security Symposium (USENIX Security 20), pages 1839–1856, 2020.
  * [Mak13] Eddie Makuch.  Guinness claims gtav day-one sales total 11.2 million copies.  https://www.gamespot.com/articles/guinness-claims-gtav-day-one-sales-total-112-million-copies/1100-6415455/, 10 2013.
  * [McC99] Steven McCanne.  Scalable multimedia communication using ip multicast and lightweight sessions.  IEEE Internet Computing, 3(2):33–45, 1999.
  * [MLM21] Daniel Merling, Steffen Lindner, and Michael Menth.  Hardware-based evaluation of scalable and resilient multicast with bier in p4.  IEEE Access, 9:34500–34514, 2021.
  * [PNW+18] Peter Psenak, Nagendra Kumar Nainar, IJsbrand Wijnands, Andrew Dolganow, Tony Przygienda, Zhaohui (Jeffrey) Zhang, and Sam Aldrin.  OSPFv2 Extensions for Bit Index Explicit Replication (BIER).  RFC 8444, November 2018.
  * [RSA+19] Eric C. Rosen, Mahesh Sivakumar, Sam Aldrin, Andrew Dolganow, and Tony Przygienda.  Multicast VPN Using Bit Index Explicit Replication (BIER).  RFC 8556, April 2019.
  * [Tec22] Akamai Technologies.  Oops, we did it again, May 2022.
  * [TT21] Martin Thomson and Sean Turner.  Using TLS to Secure QUIC.  RFC 9001, May 2021.
  * [UEF20] UEFA.  UEFA uefa euro 2020 impresses with 5.2 billion cumulative global live audience.  https://www.uefa.com/insideuefa/about-uefa/news/026d-132519672495-56a014558e80-1000--uefa-euro-2020-impresses-with-5-2-billion-cumulative-global-liv/, 2020\.
  * [Vas22] Victor Vasiliev.  The WebTransport Protocol Framework.  Internet-Draft draft-ietf-webtrans-overview-04, Internet Engineering Task Force, July 2022.  Work in Progress.
  * [Ver21] Paul Verna.  Us sports video 2021.  https://www.insiderintelligence.com/content/us-sports-video-2021, 2021.  Accessed: 2023-01-06.
  * [WRD+17] IJsbrand Wijnands, Eric C. Rosen, Andrew Dolganow, Tony Przygienda, and Sam Aldrin.  Multicast Using Bit Index Explicit Replication (BIER).  RFC 8279, November 2017.


# Unraveling Adversarial Examples against Speaker Identification - Techniques
for Attack Detection and Victim Model Classification

###### Abstract

Adversarial examples have proven to threaten speaker identification systems,
and several countermeasures against them have been proposed. In this paper, we
propose a method to detect the presence of adversarial examples, i.e., a
binary classifier distinguishing between benign and adversarial examples. We
build upon and extend previous work on attack type classification by exploring
new architectures. Additionally, we introduce a method for identifying the
victim model on which the adversarial attack is carried out. To achieve this,
we generate a new dataset containing multiple attacks performed against
various victim models. We achieve an AUC of 0.982 for attack detection, with
no more than a 0.03 drop in performance for unknown attacks. Our attack
classification accuracy (excluding benign) reaches 86.48% across eight attack
types using our LightResNet34 architecture, while our victim model
classification accuracy reaches 72.28% across four victim models.

##  1 Introduction

Neural networks have revolutionized various domains, exhibiting remarkable
performance for a range of speech related tasks, such as automatic speech
recognition [1] or speaker recognition [2]. However, the susceptibility of
these networks to adversarial attacks [3, 4, 5] has raised significant
concerns regarding their robustness and reliability [6, 7]. Adversarial
attacks exploit vulnerabilities in neural networks, causing misclassification
or erroneous behavior due to (usually) small imperceptible perturbations to
the input data. Applied to speech, this takes the form of an (almost)
imperceptible noise that can drastically change the outputs of a given system.
These attacks can be either white-box attacks (when the network model
parameters are available to the adversary) or black-box attacks (when the
network model parameters are not available to the adversary). Usually, white-
box attacks are more powerful and take comparatively shorter time to generate
than black-box attacks which rely on querying the model multiple times to
generate an attack. With the widespread availability of pre-trained models,
white-box attacks are feasible and increase the threat they represent. In this
work, we focus on speaker recognition systems. Concerned with the threat of
these systems, a range of countermeasures [8, 9, 10] have emerged. These
countermeasures can be classified into two broad categories - Proactive
defenses that make the models more robust and Reactive countermeasures that
filter out adversarial attacks. [11, 12] introduced a novel countermeasure
strategy designed to classify between different attacks, applied on speaker
identification and verification systems.

In this paper, we extend this work by improving multiple parts of the
classification system. Our proposed approach leverages insights from
adversarial attack mechanisms to develop a comprehensive defense framework.
Through extensive experimentation and analysis, we demonstrate the efficacy of
our method in accurately detecting the presence of an attack and classifying
the type of adversarial attack. Additionally, we propose a method to identify
the victim model on which the attack was performed. Toward this end, we
generate a new dataset of attacks computed on a range of speaker
identification networks. By bridging the gap between attack detection and
attack classification, our work represents a significant step towards building
more resilient and trustworthy machine learning systems in the face of
evolving adversarial threats.

Our main contributions can be summarized as:

  * •

Development of a binary adversarial attack detection system, complemented by
an ablation study that explores the impact of omitting different types of
adversarial attacks during training.

  * •

Exploration of new architectures for attack classification system.

  * •

Introduction of a new dataset featuring different adversarial attacks
targeting four different victim models.

  * •

Utilization of the above dataset to classify amongst victim models

Section 2 will present in detail the previous classification system this work
is based on, as well as the adversarial attacks we choose to focus on. Then,
Section 3 presents the dataset used for all our experiments, followed by
Section 4, presenting the various threat models that will be studied here:
what attacks, norms, victim models will be considered, and the building of the
datasets that contains all those threat models. Section 5 presents our
classification and detection experiments, which results are shown in Section
6, then discussed in Section 7

##  2 Related Work

This section presents previous works related to adversarial attack detection
and classification.

###  2.1 Adversarial attacks

The range of existing adversarial attacks is increasingly expanding, however
in this paper, we choose to focus on a set of existing un-targeted white-box
adversarial attacks that has been tested against speaker recognition and
verification systems [11, 12]. The reason for this choice is that white-box
attacks are more powerful and take comparatively shorter time to generate than
black-box attacks which rely on querying the model multiple times to generate
an attack. Since, our focus is on attack classification, the compute would be
significant. We focus on three adversarial attacks: the Fast Gradient Sign
Method (FGSM), Projected Gradient Descent (PGD) and Carlini-Wagner attack
(CW).

#### Fast Gradient Sign Method

[4] was the first adversarial attack proposed, simply projecting an input
toward the direction of the highest gradients to get the most impact on the
outputs from the least modification of the inputs. Iterative-FGSM(iter-fgsm)
extends the basic FGSM attack by applying the FGSM iteratively. Instead of
crafting a single perturbation, the input is perturbed multiple times in small
steps, and the gradient is computed again based on the perturbed input from
the previous step.

#### Projected Gradient Descent

[3] then built on FGSM by introducing an iterative method of gradient descent,
normalizing the outputs to limit the projection to a $L_{p}$ norm bound each
step.

#### Carlini-Wagner

[5] later reformulated the task as a min-max optimization problem: how to
maximize the change in the output of the system while minimizing the input
variation to a $L_{p}$ norm.

###  2.2 Adversarial attack detection and classification

Adversarial attack detection is the task of binary classification between
benign and adversarial attacks. This countermeasure can be thought of as a
filter or a gatekeeper i.e. allowing only benign signals to pass. On the other
hand, adversarial attack classification is a task of multiclass classification
i.e. the problem of identifying the specific attack. While multiple works [13,
14, 15, 16] have been carried out for detection task, the classification task
has been addressed by just a few papers[11, 17]. Both papers used an x-vector
network for representation learning to classify adversarial attacks. While
[11] used the adversarial example directly, [17] proposed using a denoiser to
extract the adversarial perturbation and used it for classification. The
denoiser used was a conditional GAN (CGAN) [18, 19] generative model with
Multi-Resolution Short-time Fourier Transform (MRSTFT) loss [20].

The previous works in adversarial attack detector focus usually on
thresholding techniques. The authors of [14] extended the ParallelWaveGAN
Vocoder used for preprocessing in [21] and used the difference between the
resynthesized and actual scores to construct a detector for speaker
verification systems. [13] proposed using a VGG-like network as a binary
classification detector for speaker verification. [15] proposed a detector
using two speaker verification models with a robustness gap (one model is a
state-of-the-art model known to the adversary and not robust to adversarial
attacks, while the other model is not known to the attacker and relatively
robust against attack. This gap leads to score inconsistency that can be
detected by a simple one-class classifier - minimum covariance determinant
(MCD). [16] proposed Minimum Energy in High FrEquencies for Short Time (MEH-
FEST) detector that can detect FakeBob [22] attacks.

##  3 Dataset

In this paper, we use for all our experiments the Voxceleb1 [23] and VoxCeleb2
[24] datasets, which contain respectively the voices of a total of 1,251 and
5,996 celebrities, used generally for training and testing speaker
verification systems. The composition of each set is detailed in Table 1

Table 1: VoxCeleb1 and 2 datasets distribution. Set | # Speakers | # utterances
---|---|---
VoxCeleb1-dev | 1,211 | 148,642
VoxCeleb1-test | 40 | 4,874
VoxCeleb2-dev | 5,994 | 1,092,009

Those utterances are recorded from various multimedia sources, and collected
on YouTube, with speakers mainly from the U.S, U.K, Germany, India, and
France, speaking English for the most part.

We use VoxCeleb2-dev to train all the victim models on a speaker
identification task. Those victim models are then evaluated for speaker
verification, using three partitions of the VoxCeleb1-test set: the Original
(O) partition, the Extended (E) and the Hard (H). Usually in speaker
verification experiments, The VoxCeleb2-dev set is used for training, and
VoxCeleb1 is used for testing as there is no overlap of speakers. However,
because we are evaluating for speaker recognition task, we would need
overlapping speakers. Hence, we split the VoxCeleb2-dev into two parts
VoxCeleb2-dev-train and VoxCeleb2-dev-test. We ensured that this split is 90%
train-10% test per speaker. The VoxCeleb1-dev-train set is used for training
and validation of speaker recognition victim models and VoxCeleb2-dev-test is
used as the test set.

##  4 Threat Models

This section presents the various threat models that are considered in this
paper. We consider three untargeted white-box adversarial attacks (PGD, FGSM,
and CW) using various $L_{p}$ norms ($p\in\\{0,1,2,\infty\\}$), against a set
of speaker identification systems. Since the systems are under attack, they
are called Victim models.

###  4.1 Victim Models

For the architecture of our victim models, we consider a set of four models:

  * •

ResNet34[25, 26] (resnet34)

  * •

LightResNet34[27, 11] (lresnet34)

  * •

Frequency-wise Squeeze Excitation ResNet34[28] (fwseresnet)

  * •

ECAPA-TDNN[29] (ecapatdnn)

For reference, when the same models are used for speaker verification and
trained on Voxceleb2-dev[24] and tested on all partitions of VoxCeleb1-test
[23], the performances in Equal Error Rate (EER) of those models, evaluated on
Voxceleb1 (Original, Extended, and Hard) are presented in Table 2

Table 2: Performances in EER of the victim models evaluated on Voxceleb1-O, Voxceleb1-H and Voxceleb1-E, using a Cosine distance. Model |  |  $\downarrow$ EER |
---|---|---|---
| Voxceleb1-O | Voxceleb1-E | Voxceleb1-H
resnet34 | 0.68% | 0.86% | 1.56%
lresnet34 | 1.59% | 1.69% | 2.84%
fwseresnet | 0.73% | 0.80% | 1.58%
ecapatdnn | 0.62% | 0.83% | 1.65%

###  4.2 Single Victim Model Dataset

Figure 1: Schematic of the computation of the SingleVM and the MultiVM
datasets.

Our first dataset, the Single Victim Model (SingleVM) follows the same recipe
as the one used in [17]. The original VoxCeleb2 development set was split into
two parts in the ratio 90% to 10% per speaker. We attack victim model
lresnet34, on a wide range of attacks (CW, FGSM, FGSM iterative, and PGD),
with various norms ($L_{0}$, $L_{1}$, $L_{2}$, and $L_{\infty}$). For each of
those attacks, we select a random 25% split of the VoxCeleb1-dev dataset,
compute the attack against the victim model, and keep the attacks that
succeeded. The signature victim models are trained using VoxCeleb2-dev-train,
and VoxCeleb2-dev-test is used to test speaker classification tasks. We keep
the associated benign utterances for each successful attack, which yields an
additional 108,000 utterances. The test set is kept seperate. The distribution
of utterances per attack is presented in Table 3.

Table 3: Distribution of the attacks generated for the Single Victim Model
(SingleVM) dataset, train and test splits. The victim model used is always the
Light ResNet34 here.

Attack | Norm |  Abbrev. | SingleVM-train | SingleVM-test
---|---|---|---|---
CW | $L_{0}$ | cw-l0 | 31,000 | 1679
CW | $L_{2}$ | cw-l2 | 34,000 | 1774
CW | $L_{\infty}$ | cw-linf | 34,000 | 1774
FGSM | $L_{2}$ | fgsm | 24,000 | 1429
Iterative FGSM | $L_{2}$ | iter-fgsm | 18,000 | 1034
PGD | $L_{1}$ | pgd-l1 | 16,000 | 916
PGD | $L_{2}$ | pgd-l2 | 16,000 | 895
PGD | $L_{\infty}$ | pgd-linf | 17,000 | 966
All attacks |  |  | 190,000 | 8,788
All benign |  |  | 108,000 | 5,669
Total |  |  | 298,000 | 14,457

###  4.3 Multiple Victim Model Dataset

As one victim model would not be enough to classify between victim models, we
generate a second dataset, the Multiple Victim Model dataset (MultiVM),
against the 4 victim models presented in Section 4.1. We only compute the
following 4 attacks: CW with $L_{2}$ norm, PGD with $L_{2}$ norm and
$L_{\infty}$ norm, and FGSM to limit the size of the dataset and
experimentation time, however, this dataset can be easily extended to include
all the norms as in SingleVM.

This new dataset is computed the same way as the SingleVM but for each of the
victim models. It is ensured that adversarial examples corresponding to benign
utterances used for training and validation of the victim network are kept for
training and validation of the attack classifier network. The test set is kept
separate. The number of utterances for the generated dataset split into the
train, validation, and test sets is shown in Table 4. Figure 1 illustrates the
making of both threat model datasets from Voxceleb2-dev set.

Figure 2: Pipeline of inference to predict the label linked to a given
utterance $x^{\prime}$ of the SingleVM or the MultiVM dataset, for the three
experiments proposed. Figure 3: Score distribution of the MultiVM-test data,
for each system trained on all but one attack. The benign utterances are
always in filled light blue, the attack that was removed from the train set is
filled while the others are emptied. Table 4: Data distribution for the
Multiple Victim Model (MultiVM) dataset. Numbers indicate the number of
utterances. The total dataset has 127,261 utterances across the four victim
models.

| Train Set | Validation Set | Test Set
---|---|---|---
ecapatdnn | 27,149 | 3,232 | 1,582
fwseresnet | 25,514 | 3,026 | 1,521
resnet | 25,715 | 3,064 | 1,522
lresnet | 29,719 | 3,514 | 1,703
Total | 108,097 | 12,836 | 6,328

Table 5: AUC, Accuracy and EER of our various attack detection experiments, for the Single Victim Model dataset (SingleVM), for the Multiple Victim Model Dataset (MultiVM), and our ablation studies on the Multiple Victim Model Dataset, using all but one attack, evaluated on the whole dataset, on all but the attack, and only on the benign and the attack. Trained on | Tested on |  $\uparrow$ AUC |  $\uparrow$ Accuracy |  $\downarrow$ EER
---|---|---|---|---
SingleVM-train | SingleVM-test | 0.982 | 93.7% | 6.34%
MultiVM-train | MultiVM-test | 0.961 | 91.1% | 8.89%
| MultiVM-test | 0.953 | 89.3% | 10.67%
MultiVM-train except cw-l2 |  MultiVM-test except cw-l2 | 0.979 | 92.9% | 7.13%
| only benign and cw-l2 | 0.916 | 85.3% | 14.68%
| MultiVM-test | 0.968 | 91.9% | 8.06%
MultiVM-train except fgsm |  MultiVM-test except fg sm | 0.965 | 91.4% | 8.63%
| only benign and fgsm | 0.983 | 95.5% | 4.46%
| MultiVM-test | 0.955 | 91.1% | 8.94%
MultiVM-train except pgd-l2 |  MultiVM-test except pgd-l2 | 0.954 | 91.3% | 8.67%
| only benign and pgd-l2 | 0.956 | 90.5% | 9.50%
| MultiVM-test | 0.967 | 92.1% | 7.92%
MultiVM-train except pgd-linf |  MultiVM-test except pgd-linf | 0.964 | 91.5% | 8.51%
| only benign and pgd-linf | 0.980 | 94.9% | 5.12%

###  4.4 Attack classification models

The attack classification networks were built with the same architectures, but
the embedding dimension was empirically set to 10. We explore two
architectures:

  * •

LightResNet34, the same architecture as in [17] and also one of the victim
model used here.

  * •

ECAPA-TDNN [29], a popular architecture using attention, to improve the
performances of our model.

##  5 Experiments

Our experiments follow a similar structure. For a given sample $x^{\prime}$
(that could be benign or adversarial), we can first use a pre-trained denoiser
$D$ (same a [17] and as briefly described in Section 2.2) to predict an
estimation of the clean associated sample $\hat{x}$, which is then used to
estimate the adversarial noise $\hat{\delta}$. We then train a classifier $C$
to predict a given class either from the estimated adversarial noise
$\hat{\delta}$ or directly from the attacked signal $x^{\prime}$. Our
experiments explore three directions:

  1. 1.

What are the performances of our pipeline when used only for attack detection
(as a binary classifier)?

  2. 2.

Does the architecture of the classifier play a role in the attack
classification?

  3. 3.

Can we classify the victim model on which the attacks were computed?

The following subsections delve into these directions. All experiments are
performed using the Hyperion toolkit111https://github.com/hyperion-
ml/hyperion.

###  5.1 Attack Detection

This experiment explores the potential of the best of our classifiers to be
retrained for a binary task: attack vs benign classification. We train a
LightResNet34 on our Single Victim Model dataset, using a pre-trained denoiser
to estimate the adversarial noise $\hat{\delta}$, and find out the best
performances we could get on 8 different attacks. Then, we train the same
architecture on our Multiple Victim Model dataset to compare the results on a
comparatively limited set of attacks. We chose these attacks because they
represent the family well and we wanted to avoid confusion between the norms
to see whether the model learns the underlying victim models better.

We also study the impact of having one unknown attack, by training 4
classifiers, each ignoring one of the four attacks from our Multiple Victim
Model dataset, and look at the performance classifications on the unknown
attack. The performances are evaluated in terms of Area Under the Curve (AUC),
Accuracy, and Equal Error Rate (EER) and presented in Section 6.1

###  5.2 Different architectures for classifier

Our first experiment explores changes in architectures for the classifier,
facing an attack that was always performed against the same victim model, a
LightResNet34. Because we want to compare the impact of a different
classifier, we choose to not use the denoiser to compare performances here, as
we don’t want any interference in the process.

We compare two architectures with our Single Victim Model dataset:

  * •

LightResNet34, the same architecture as in [17] and also the one of the victim
model used here.

  * •

ECAPA-TDNN [29], a popular architecture using attention, to improve the
performances of our model.

For both experiments, we train our classifier using the train split of our
Single Victim Model dataset, to predict the attack of each utterance.The
accuracy of attack classification is detailed and discussed in Section 6.2.

###  5.3 Victim Model Classification

Our third experiment targets the classification of the victim model on which
the adversarial attack was computed. We train a LightResNet34 classifier on
the adversarial utterances of our Multiple Victim Model dataset to classify
between the 4 victim models. The accuracy of victim model classification is
detailed and discussed in Section 6.3.

##  6 Results

This section presents the results of our three experiments along with their
associated conclusions.

###  6.1 Attack Detection

Table 5 presents the results in AUC, Accuracy and EER of our various attack
detection experiments.

Table 5 shows good classification results on the SingleVM dataset, but a
significant degradation once trained the MultiVM, which is understandable as
the latest only contains a subset of attacks and has comparatively less data.
We can also see that removing one of the attacks present only marginally
impacts the binary classification results.

To further understand how the ablation impacts the distribution of the scores,
we plot the distribution of the scores for the MultiVM dataset in each of the
cases in Figure 3.

We can see in Figure 3 score distribution of the MultiVM-test data, for each
system trained on all but one attack. It can be seen that removing one attack
has a low effect on the general distribution of the scores.

Figure 4: Normalized confusion matrix (%) using LResNet34 as classifier for
SingleVM dataset Figure 5: Normalized confusion matrix (%) using ECAPA-TDNN as
classifier for SingleVM dataset

###  6.2 Different architectures for classifier

We observe significant differences in classification performance when
different architectures are used for attack classification for the same victim
model. We use SingleVM dataset and vary the architecture for the classifier
network (LResNet, ECAPA-TDNN) as described in the following sections. The
overall classification accuracy for LResNet is 86.48 % and the confusion
matrix is shown in Figure 4. The overall classification accuracy for ECAPA-
TDNN is 84.79% and the confusion matrix is shown in Figure 5 The overall
accuracy of the two classifiers does not differ by a very significant margin
(absolute difference of 1.69%). We compared the correctly classified accuracy
(diagonals of the confusion matrices) of the two architectures and plotted
them in Table 6. The accuracies are quite close to each other and ECAPA-TDNN
is only better for PGD-L2 attack which can also be seen from comparing the
TSNE figures Figure 6 for LResNet and Figure 7. We can see in both matrices
that the main confusion is always between PGD $L_{1}$ and PGD $L_{2}$ attacks.

Table 6: Comparison between the two classifiers for victim model
classification in speaker recognition.

| lresnet | ecapatdnn
---|---|---
cw-l0 | 99.1 | 98.9
cw-l2 | 84.2 | 84.2
cw-linf | 96 | 92.9
fgsm | 94.5 | 90.9
iter-fgsm | 85.9 | 84.5
pgd-l1 | 91.7 | 74.4
pgd-l2 | 42.1 | 54.0
pgd-linf | 80.3 | 78.4

Figure 6: TSNE for LResNet34 architecture as attack classifier on SingleVM
dataset Figure 7: TSNE for ECAPA-TDNN architecture as attack classifier on
SingleVM dataset

###  6.3 Victim Model Classification

For the victim model classification, we obtain a classification accuracy of
72.28% for the 4 models of MultiVM test. The confusion matrix is shown in
Table 7 and the corresponding TSNE plot is in Figure 8. It can be clearly seen
that the adversarial attacks on ecapatdnn can be detected with the highest
accuracy. The corresponding TSNE also shows the clear separation of ECAPA-TDNN
embeddings from the rest of the embeddings. We suspect this is because the
architecture of ecapatdnn is very different from the other three (fwseresnet,
resnet34, lresnet34) which fall under the same broad category of the ResNet34.

Table 7: Normalized confusion matrix (%) for victim model classification in
speaker recognition.

| ecapatdnn | fwseresnet | resnet34 | lresnet34
---|---|---|---|---
ecapatdnn | 92.0 | 1.9 | 1.4 | 4.7
fwseresnet | 2.9 | 59.5 | 23.9 | 13.7
resnet34 | 1.5 | 19.3 | 54.7 | 24.5
lresnet34 | 1.5 | 5.3 | 12.2 | 81.1

Figure 8: TSNE for victim model classification on MultiVM dataset

##  7 Conclusion

In this study, we have undertaken a thorough examination of adversarial
attacks targeting speaker verification systems, aiming to fortify the
resilience of such systems against malicious intrusions. Our investigation
builds upon existing defense and classification methodologies while
innovatively extending them to encompass both attack detection and
classification of adversarial attacks and victim models.

Our findings demonstrate the robustness and effectiveness of our framework,
showcasing its capability to accurately detect the presence of adversarial
attacks with an impressive AUC of up to 0.982. Furthermore, our classification
accuracy of 86.48% over eight classes for the LResNet34 classifier underscores
the efficacy of our approach in discerning the type of attack employed.
Notably, the introduction of our novel handcrafted dataset, comprising various
attacks inflicted upon different victim models, has significantly enhanced our
ability to detect and classify a broader spectrum of attacks, achieving a
commendable classification accuracy of 72.28% over four classes.

Looking forward, our research opens up promising avenues for further
exploration and advancement. Refinements and optimizations of our defense
framework hold the potential to bolster its effectiveness in real-world
scenarios, particularly by broadening the scope to encompass a wider array of
victim models and diverse attack parameters. Moreover, we aspire to expand our
repertoire to include black-box attacks and adaptive attacks, thereby
enriching the diversity of threat models considered and fortifying our
defenses against an even broader spectrum of adversarial intrusions. By
continuously evolving and refining our methodologies, we strive to propel the
field of adversarial defense forward, ensuring the robustness and reliability
of speaker identification systems in the face of evolving security threats.

##  8 Acknowledgments

This research has been supported by DARPA RED
https://www.darpa.mil/program/reverse-engineering-of-deceptions under contract
HR00112090132

## References

  * [1] Sadeen Alharbi, Muna Alrazgan, Alanoud Alrashed, Turkiayh Alnomasi, Raghad Almojel, Rimah Alharbi, Saja Alharbi, Sahar Alturki, Fatimah Alshehri, and Maha Almojil,  “Automatic speech recognition: Systematic literature review,”  IEEE Access, vol. 9, pp. 131858–131876, 2021.
  * [2] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [3] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  “Towards deep learning models resistant to adversarial attacks,”  arXiv preprint arXiv:1706.06083, 2017.
  * [4] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  arXiv preprint arXiv:1412.6572, 2014.
  * [5] Nicholas Carlini and David Wagner,  “Towards evaluating the robustness of neural networks,”  in 2017 ieee symposium on security and privacy (sp). Ieee, 2017, pp. 39–57.
  * [6] Guangke Chen, Zhe Zhao, Fu Song, Sen Chen, Lingling Fan, Feng Wang, and Jiashui Wang,  “Towards understanding and mitigating audio adversarial examples for speaker recognition,”  IEEE Transactions on Dependable and Secure Computing, 2022.
  * [7] Qi Wang, Jie Jiang, Jinyuan Mo, Qiuyu Kong, Yishan Li, and Suru Feng,  “A survey of adversarial attacks on deep neural network in computer vision,”  in 2022 IEEE 8th International Conference on Computer and Communications (ICCC). IEEE, 2022, pp. 1879–1886.
  * [8] Hao Tan, Le Wang, Huan Zhang, Junjian Zhang, Muhammad Shafiq, and Zhaoquan Gu,  “Adversarial attack and defense strategies of speaker recognition systems: A survey,”  Electronics, vol. 11, no. 14, 2022.
  * [9] Jiahe Lan, Rui Zhang, Zheng Yan, Jie Wang, Yu Chen, and Ronghui Hou,  “Adversarial attacks and defenses in speaker recognition systems: A survey,”  Journal of Systems Architecture, vol. 127, pp. 102526, 2022.
  * [10] Hadi Abdullah, Kevin Warren, Vincent Bindschaedler, Nicolas Papernot, and Patrick Traynor,  “Sok: The faults in our asrs: An overview of attacks against automatic speech recognition and speaker identification systems,”  in 2021 IEEE symposium on security and privacy (SP). IEEE, 2021, pp. 730–747.
  * [11] Jesús Villalba, Yuekai Zhang, and Najim Dehak,  “x-vectors meet adversarial attacks: Benchmarking adversarial robustness in speaker verification.,”  in Interspeech, 2020, pp. 4233–4237.
  * [12] Jesús Villalba, Sonal Joshi, Piotr Żelasko, and Najim Dehak,  “Representation learning to classify and detect adversarial attacks against speaker and speech recognition systems,”  Interspeech, 2021.
  * [13] Xu Li, Na Li, Jinghua Zhong, Xixin Wu, Xunying Liu, Dan Su, Dong Yu, and Helen Meng,  “Investigating robustness of adversarial samples detection for automatic speaker verification,”  Interspeech, 2020.
  * [14] Haibin Wu, Po-Chun Hsu, Ji Gao, Shanshan Zhang, Shen Huang, Jian Kang, Zhiyong Wu, Helen Meng, and Hung-Yi Lee,  “Adversarial sample detection for speaker verification by neural vocoders,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 236–240.
  * [15] Zhiyuan Peng, Xu Li, and Tan Lee,  “Pairing weak with strong: Twin models for defending against adversarial attack on speaker verification.,”  in Interspeech, 2021.
  * [16] Zesheng Chen,  “On the detection of adaptive adversarial attacks in speaker verification systems,”  IEEE Internet of Things Journal, 2023.
  * [17] Sonal Joshi, Saurabh Kataria, Jesús Villalba, and Najim Dehak,  “Advest: Adversarial perturbation estimation to classify and detect adversarial attacks against speaker identification,”  arXiv preprint arXiv:2204.03848, 2022.
  * [18] Mehdi Mirza and Simon Osindero,  “Conditional generative adversarial nets,”  arXiv preprint arXiv:1411.1784, 2014.
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial nets,”  Advances in neural information processing systems, vol. 27, 2014.
  * [20] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim,  “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6199–6203.
  * [21] Sonal Joshi, Jesús Villalba, Piotr Żelasko, Laureano Moro-Velázquez, and Najim Dehak,  “Study of pre-processing defenses against adversarial attacks on state-of-the-art speaker recognition systems,”  IEEE Transactions on Information Forensics and Security, vol. 16, pp. 4811–4826, 2021.
  * [22] Guangke Chen, Sen Chenb, Lingling Fan, Xiaoning Du, Zhe Zhao, Fu Song, and Yang Liu,  “Who is real bob? adversarial attacks on speaker recognition systems,”  in 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 694–711.
  * [23] A. Nagrani, J. S. Chung, and A. Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  in INTERSPEECH, 2017.
  * [24] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [25] Jesús Villalba, Nanxin Chen, David Snyder, Daniel Garcia-Romero, Alan McCree, Gregory Sell, Jonas Borgstrom, Leibny Paola García-Perera, Fred Richardson, Réda Dehak, et al.,  “State-of-the-art speaker recognition with neural network embeddings in nist sre18 and speakers in the wild evaluations,”  Computer Speech & Language, vol. 60, pp. 101026, 2020.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [27] Hossein Zeinali, Shuai Wang, Anna Silnova, Pavel Matějka, and Oldřich Plchot,  “But system description to voxceleb speaker recognition challenge 2019,”  arXiv preprint arXiv:1910.12592, 2019.
  * [28] Jenthe Thienpondt, Brecht Desplanques, and Kris Demuynck,  “Integrating frequency translational invariance in tdnns and frequency positional information in 2d resnets to enhance speaker verification,”  in Interspeech2021. ISCA, 2021, pp. 2302–2306.
  * [29] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  Interspeech, 2020.