# Authenticated partial correction over AV-MACs:
toward characterization and coding
††thanks: This material is based upon work supported by the National Science
Foundation under Grant No. 2107488.

Duncan Koepke, Michaela Schnell, Madelyn St.Pierre, Allison Beemer  Department
of Mathematics, University of Wisconsin-Eau Claire
Eau Claire, WI, United States

###### Abstract

In this paper we study $\gamma$ partial correction over a $t$-user arbitrarily
varying multiple-access channel (AV-MAC). We first present necessary channel
conditions for the $\gamma$ partially correcting authentication capacity
region to have nonempty interior. We then give a block length extension scheme
which preserves positive rate tuples from a short code with zero probability
of $\gamma$ partial correction error, noting that the flexibility of $\gamma$
partial correction prevents pure codeword concatenation from being successful.
Finally, we offer a case study of a particular AV-MAC satisfying the necessary
conditions for partial correction.

###### Index Terms:

arbitrarily varying multiple-access channel, capacity region, authentication,
partial correction

##  I Introduction

An arbitrarily varying multiple-access channel (AV-MAC) combines random noise
with adversarial action over a channel with multiple senders and a single
receiver. Classical communication over AV-MACs has been studied in a variety
of works, with [1, 2, 3, 4] focusing on the capacity region in the two-user
case. The combination of these works establish the communication capacity
region, notably showing that the region has nonempty interior if and only if
the channel does not have a set of channel symmetrizability properties.
Symmetrizability, defined for point-to-point AVCs in [5], indicates that the
adversary can reliably trick the receiver into decoding in error.

While symmetrizability characterizes the communication capacity of an AVC, the
analogous condition of overwritability governs the (keyless) authentication
capacity of such a channel [6]. Overwritability indicates that an adversary is
not only able to trick the receiver into an erroneous message estimate, but
that they are able to do so while remaining undetected. In [7], Beemer et al.
formalize an extension of overwritability to the AV-MAC, in a similar vein to
the extension of symmetrizability for communication. They show that the
capacity region for (keyless) authentication over an AV-MAC is equal to that
for communication with no adversary, provided that the channel is not
overwritable to any degree.

Other work related to authentication over an AV-MAC includes work on MACs with
byzantine users. In [8, 9], Sangwan et al. consider a byzantine user in a two-
user MAC, proving results on the authenticated communication capacity region.
Indeed, the inner bound of the authentication capacity region for two users
over an AV-MAC in [7] was accomplished by fixing the byzantine user in an an
extension to three users of the scheme of [8]. In general, however, the
question of a byzantine user in a MAC is distinct from an AV-MAC, where the
adversary’s identity is known a priori.

In the present work, we give the extensions of authentication results from [7]
to the case of an arbitrary number of users, then quickly turn our attention
the idea of $\gamma$ partial correction. Partial correction over an AV-MAC was
introduced in [7] to bridge the gap between total correction and
authentication. In contrast to pure authentication, $\gamma$ partial
correction requires that a $\gamma$ fraction of users’ messages be decoded
correctly, even if the remainder be discarded. The particular users who are
decoded accurately may change with each transmission: that is, the subset of
users to be decoded is not fixed ahead of time. When $\gamma=0$, this reduces
to authentication, while with $\gamma=1$ the goal becomes classical
communication.

To our knowledge, partial correction over an AV-MAC has only been studied in
[7]. However, we note that there may be a connection to list decoding over AV-
MACs (see e.g. [10, 11]), wherein partial correction would require that
elements in the output list match on a certain number of users. In [7], the
authors focus on the two-user case, and give some initial results showing that
it is possible for a channel to have a $\gamma$ partially correcting
authentication capacity region with nonempty interior. Here, we extend these
results to an arbitrary number of users. We give a set of necessary
symmetrizability/overwritability conditions for $\gamma$ partial correction,
present a case study for a particular channel satisfying these conditions, and
provide a scheme to extend block length that preserves positive rate tuples.

Necessary background and notation is introduced in Section II. In Section III,
we present a set of necessary channel conditions for partial correction
capacity regions with nonempty interior. Section IV discusses a general method
for extending the block length of a short block length code tuple with
desirable partial correction properties, and Section V provides a case study
of the construction of these short codes for a particular channel. Section VI
concludes the paper.

##  II Preliminaries

Let $[n]:=\\{1,2,\ldots,n\\}$, and let $\textrm{supp}(\mathbf{x})\subseteq[n]$
denote the support of a length-$n$ vector $\mathbf{x}$. Capital letters (e.g.
$X$) will denote random variables, script letters the alphabets they are taken
from ($\mathcal{X}$), and lower case letters their realizations ($x$).

Our setting will be a $t$-user AV-MAC, where $t\geq 2$. More specifically, a
$t$-user AV-MAC is defined by a distribution $W_{Y|X_{1}\cdots X_{t}S}$, where
legitimate channel inputs $X_{j}$ are taken from alphabet $\mathcal{X}_{j}$
for each $j\in[t]$, the adversary’s choice of channel state is
$S\in\mathcal{S}$, and the channel output is given by $Y\in\mathcal{Y}$. In
our model, we assume the adversary has full knowledge of the channel
statistics and all user encoding strategies, but that $S$ is independent of
the particular message sequence transmitted in any given time instance. We
begin by extending the definitions of [7].

######  Definition II.1.

An $(M_{1},\ldots,M_{t},n)$ authentication code for a $t$-user AV-MAC is given
by encoders $f_{1},\ldots,f_{t}$ and decoder $\phi$:

$\displaystyle f_{i}$ $\displaystyle:[M_{j}]\to\mathcal{X}_{j}^{n},\ 1\leq
j\leq t$ (1) $\displaystyle\phi$
$\displaystyle:\mathcal{Y}^{n}\to\left([M_{1}]\cup\\{0\\}\right)\times\cdots\times\left([M_{t}]\cup\\{0\\}\right),$
(2)

where an output of “0” in any coordinate indicates adversarial interference.

We will sometimes directly discuss the codebooks
$C_{j}=f_{j}(M_{j})\subseteq\mathcal{X}_{j}^{n}$ in later sections. In this
paper, we will be concerned primarily with correcting some portion of the
users’ messages, even if others must be discarded due to adversarial
interference.

######  Definition II.2.

Let $\gamma\in(0,1)$. We say that an $(M_{1},\ldots,M_{t},n)$ authentication
code for a $t$-user AV-MAC is $\gamma$ partially correcting if, with high
probability in $n$, we can correct at least $\lceil\gamma t\rceil$ of the $t$
messages.

We observe that the case where $\gamma=0$ reduces to the classical notion of
authentication for an AV-MAC à la [8, 7, 9], while $\gamma=1$ bridges the gap
to total correction of all user messages. The use of the open interval in
Definition II.2 excludes the cases where no messages are corrected, or all
are; neither of these is partial correction. It is straightforward that if an
authentication code is $\gamma$ partially correcting, then it is $\lambda$
partially correcting for all $0<\lambda<\gamma$.

Let $\phi^{-1}(A)\subseteq\mathcal{Y}^{n}$ represent the set of channel
outputs which decode to some element $(i_{1},\ldots,i_{t})$ in the set $A$
under the decoder $\phi$, and let $\phi^{-1}(A)^{c}$ be the complement in
$\mathcal{Y}^{n}$ of this set. Let $\mathbf{x}_{j}(i):=f_{j}(i)$ denote the
length-$n$ encoding of message $i$ by user $j$. Correspondingly, we let
$\mathbf{i}$ denote a tuple of transmitted messages from
$[M_{1}]\times\cdots\times[M_{t}]$, and $\mathbf{x}(\mathbf{i})$ its encoding
under $(f_{1},f_{2},\ldots f_{t})$. Given a tuple of transmitted messages,
$\mathbf{i}$, and adversarial state $\mathbf{s}$, where
$\mathbf{s}=\mathbf{s}_{0}$ denotes that the no-adversary state sequence, we
define the probability of $\gamma$ partial correction error for the
authentication code $(f_{1},\ldots,f_{t},\phi)$ by:

$e_{\gamma}(\mathbf{i},\mathbf{s}_{0})=W(\phi^{-1}(\\{\mathbf{i}\\})^{c}\mid\mathbf{x}(\mathbf{i}),\mathbf{s}_{0}),$
(3)

and, when $\mathbf{s}\neq\mathbf{s}_{0}$,

$e_{\gamma}(\mathbf{i},\mathbf{s})=W(\phi^{-1}(A_{\mathbf{i}})^{c}\mid\mathbf{x}(\mathbf{i}),\mathbf{s}),$
(4)

where $A_{\mathbf{i}}=\\{\hat{\mathbf{i}}:\hat{i}_{j}\in\\{0,i_{j}\\}\text{
for }j\in[t],|\textrm{supp}(\mathbf{\hat{\mathbf{i}}})|\geq\lceil\gamma
t\rceil\\}$. That is, $A_{\mathbf{i}}$ is the set of decoded sequences that
match sent message tuple $\mathbf{i}$ on every nonzero entry, and have at
least a $\gamma$ fraction of nonzero entries. We will assume that each message
in $[M_{1}]\times\cdots\times[M_{t}]$ is transmitted with equal probability,
so that the average probability of error for a given adversarial choice of
$\mathbf{s}$ is:

$e_{\gamma}(\mathbf{s})=\frac{1}{M_{1}\cdots
M_{t}}\sum_{\mathbf{i}}e_{\gamma}(\mathbf{i},\mathbf{s}).$ (5)

We say that a rate tuple $(R_{1},\ldots,R_{t})\in\mathbb{R}_{\geq 0}^{t}$ is
achievable for $\gamma$ partial correction if there exists a sequence of
$(2^{R_{1}n},\ldots,2^{R_{t}n},n)$ codes such that
$\max_{\mathbf{s}}e_{\gamma}(\mathbf{s})$ approaches $0$ with increasing block
length $n$. As in a point-to-point AVC or the two-user AV-MAC case,
$\text{argmax}_{\mathbf{s}}e_{\gamma}(\mathbf{s})$ is the adversary’s best
chance of inducing a decoding error.

The ($t$-dimensional) authentication capacity region
$\mathscr{C}_{\text{auth}}$ and the $\gamma$ partially correcting
authentication capacity region, $\mathscr{C}_{\text{auth},\gamma}$, are the
closures of the sets of achievable rate tuples for each respective goal, where
the former is realized when $\gamma=0$. Let $\mathscr{C}$ denote the
communication capacity region in the no-adversary setting (i.e.,
$\mathbf{s}=\mathbf{s}_{0}$, $\gamma=1$). We say that a capacity region has
nonempty interior if it contains a point such that all coordinate values are
positive.

Critical to authentication and partial correction are the concepts of
symmetrizability [5] and overwritability [6]: channel conditions which
determine whether a channel is amenable to these types of communication.
Below, we give extensions to the original point-to-point definitions to a
$t$-user AV-MAC:

######  Definition II.3.

Let $t\geq 2$ and $m\in[t]$. A $t$-user AV-MAC $W_{Y|X_{1}\cdots X_{t}S}$
(denoted by $W$) is $X_{i_{1}}\times\cdots\times X_{i_{m}}$-symmetrizable if
there exists $P:=P_{S|X_{i_{1}}\cdots X_{i_{m}}}$ such that for all
$x_{i_{1}},\ldots,x_{i_{m}},x_{i_{1}}^{\prime},\ldots,x_{i_{m}}^{\prime},y$,

$\displaystyle\sum_{s}P(s\mid x_{i_{1}}^{\prime},\ldots,$ $\displaystyle
x_{i_{m}}^{\prime})W(y|x_{i_{1}},\ldots,x_{i_{m}},s)=$
$\displaystyle\sum_{s}P$ $\displaystyle(s\mid
x_{i_{1}},\ldots,x_{i_{m}})W(y|x_{i_{1}}^{\prime},\ldots,x_{i_{m}}^{\prime},s).$

The case $t=2$ results in the symmetrizability conditions of [2], which along
with [1, 3] showed that (lack of) symmetrizability completely characterizes
when the AV-MAC communication capacity region $\mathscr{C}$ has (non)empty
interior.

######  Definition II.4.

Let $t\geq 2$ and $m\in[t]$. A $t$-user AV-MAC $W_{Y|X_{1}\cdots X_{t}S}$
(denoted by $W$) is $X_{i_{1}}\times\cdots\times X_{i_{m}}$-overwritable if
there exists $P:=P_{S|X_{i_{1}}\cdots X_{i_{m}}}$ such that for all
$x_{i_{1}},\ldots,x_{i_{m}},x_{i_{1}}^{\prime},\ldots,x_{i_{m}}^{\prime},y$,

$\displaystyle\sum_{s}P(s\mid
x_{i_{1}}^{\prime},\ldots,x_{i_{m}}^{\prime})W(y|x_{i_{1}},$
$\displaystyle\ldots,x_{i_{m}},s)=$ $\displaystyle
W(y|x_{i_{1}}^{\prime},\ldots,x_{i_{m}}^{\prime},s_{0}).$

Again, the case of $t=2$ reduces to previous results: it was shown in [7] that
(lack of) overwritability completely characterizes when the authentication
capacity region $\mathscr{C}_{\text{auth}}$ has (non)empty interior.

For brevity, we will say that a channel is $m$-symmetrizable (resp.,
-overwritable) if there exists some subset of $m$ users $i_{1},\ldots,i_{m}$
such that the channel is $X_{i_{1}}\times\cdots\times X_{i_{m}}$-symmetrizable
(resp., overwritable).

##  III Necessary conditions for nonempty interior

Previous work completely classified the authentication capacity region
$\mathscr{C}_{\text{auth}}$ for the case of two users, and established
necessary conditions for nonempty interior of the $\gamma=0.5$ partially
correcting authentication capacity region $\mathscr{C}_{\text{auth},0.5}$ in
the same setting [7]. In this section, we extend these results to more than
two users. Because the authentication rate region is not the primary topic of
this paper, and the results extend in a straightforward way to more users, we
omit the following proof pertaining to $\mathscr{C}_{\text{auth}}$; this
result extends Lemma III.6 and Theorem III.7 of [7].

######  Theorem III.1.

A $t$-user AV-MAC is $m$-overwritable for some $m\in[t]$ if and only if
$\mathscr{C}_{\text{auth}}$ has empty interior. Otherwise,
$\mathscr{C}_{\text{auth}}=\mathscr{C}$.

The following result on $\mathscr{C}_{\text{auth},\gamma}$ can be seen by
observing that any $\gamma$ partially correcting authentication code is
simultaneously an authentication code. The result extends Lemma IV.2. of [7].

######  Lemma III.2.

For any $t$-user AV-MAC and $\gamma\in(0,1)$,
$\mathscr{C}_{\text{auth},\gamma}\subseteq\mathscr{C}_{\text{auth}}$.

Theorem III.1 and Lemma III.2 together imply that non-$m$-overwritability (for
all $m\in[t]$) is a necessary condition for $\mathscr{C}_{\text{auth},\gamma}$
to have nonempty interior. Next, we give another necessary condition for
nonempty interior of $\mathscr{C}_{\text{auth},\gamma}$. Namely, the channel
can only be symmetrizable up to the number of users we need not correct to
achieve $\gamma$ partial correction. While this result extends Theorem IV.3 of
[7], its proof contains more subtlety than the two-user case, so we include it
here in full.

######  Theorem III.3.

Let $\gamma\in(0,1)$. If a $t$-user AV-MAC $W_{Y|X_{1}\cdots X_{t}S}$ is
$m$-symmetrizable for any $m>t-\lceil\gamma t\rceil$, then
$\mathscr{C}_{\text{auth},\gamma}$ has empty interior.

###### Proof.

Let $\gamma\in(0,1)$. Suppose $W:=W_{Y|X_{1}\cdots X_{t}S}$ is
$X_{i_{1}}\times\cdots\times X_{i_{m}}$-symmetrizable, where $m>t-\lceil\gamma
t\rceil$. Without loss of generality, let $i_{j}=j$, so that the coordinates
in question are the first $m$, and let $P:=P_{S|X_{1}\cdots X_{m}}$ be an
adversarial distribution satisfying the property of Definition II.3. Consider
a sequence of $(M_{1},\ldots,M_{t},n)$ codes, with $M_{j}:=2^{R_{j}n}$ where
$R_{j}>0$ for $j\in[t]$. Let $\mathbf{x}(\mathbf{i})$ be the encoding of
message vector $\mathbf{i}$, and let $\mathbf{v}_{a}^{b}$ denote coordinates
$a$ through $b$ of a vector $\mathbf{v}$. Define
$A_{\mathbf{i}}:=\\{\mathbf{\hat{i}}:\hat{i}_{j}\in\\{0,i_{j}\\}\text{ for
}j\in[t],|\textrm{supp}(\mathbf{\hat{\mathbf{i}}})|\geq\lceil\gamma
t\rceil\\}$, as in Section II. Finally, define $M:=(M_{1}\cdots
M_{m})(M_{1}\cdots M_{t})$. Then, $\max_{\mathbf{s}}e_{\gamma}(\mathbf{s})$ is
bounded below by the expected value of $e_{\gamma}(\mathbf{s})$ over $S$:

$\displaystyle\geq\sum_{\mathbf{s}}\left(\frac{1}{M_{1}\cdots
M_{m}}\sum_{\mathbf{i}_{1}^{m}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{i})_{1}^{m})\right)e_{\gamma}(\mathbf{s})$
(6)
$\displaystyle=\frac{1}{M}\sum_{\mathbf{i}_{1}^{m},\mathbf{k},\mathbf{s}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{i})_{1}^{m})e_{\gamma}(\mathbf{k},\mathbf{s})$
(7)
$\displaystyle\geq\frac{1}{M}\sum_{\mathbf{i}_{1}^{m},\mathbf{k},\mathbf{s}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{i})_{1}^{m})W(\phi^{-1}(A_{\mathbf{k}})^{c}\mid\mathbf{x}(\mathbf{k}),\mathbf{s})$
(8)
$\displaystyle=\frac{1}{M}\sum_{\mathbf{i}_{1}^{m},\mathbf{k},\mathbf{s}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{k})_{1}^{m})W(\phi^{-1}(A_{\mathbf{k}})^{c}\mid\mathbf{x}(\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}),\mathbf{s})$
(9)

where Equations (7) and (8) follow by definition, and (9) from
symmetrizability.

Now, we consider the sets $A_{\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}}$ and
$A_{\mathbf{k}}$. If $i_{j}\neq k_{j}$ for all $j\in[m]$, then these two sets
are disjoint: indeed, any decoded message tuple with support of size at least
$\lceil\gamma t\rceil$ must contain at least one coordinate from the first $m$
(recall that $m>t-\lceil\gamma t\rceil$). In other words,
$\phi^{-1}(A_{\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}})\subseteq\phi^{-1}(A_{\mathbf{k}})^{c}$
when $i_{j}\neq k_{j}$ for all $j\in[m]$. Using that $R_{j}>0$, and thus that
the set of $\mathbf{i}_{1}^{m}$’s such that $i_{j}\neq k_{j}$ for fixed
$\mathbf{k}$ is nonempty,

$\displaystyle\geq\frac{1}{M}\sum_{\begin{subarray}{c}\mathbf{k},\mathbf{s}\\\
\mathbf{i}_{1}^{m}:{i}_{j}\neq
k_{j}\end{subarray}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{k})_{1}^{m})\cdot$ (10)
$\displaystyle\quad\quad\quad\quad\quad\quad\quad
W(\phi^{-1}(A_{\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}})\mid\mathbf{x}(\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}),\mathbf{s})$
(11)
$\displaystyle=\frac{1}{M}\sum_{\begin{subarray}{c}\mathbf{k},\mathbf{s}\\\
\mathbf{i}_{1}^{m}:{i}_{j}\neq
k_{j}\end{subarray}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{k})_{1}^{m})\left(1-e_{\gamma}(\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t},\mathbf{s})\right)$
(12)
$\displaystyle=\frac{1}{M}\sum_{\begin{subarray}{c}\mathbf{s},\mathbf{k}_{1}^{m}\end{subarray}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{k})_{1}^{m})\sum_{\begin{subarray}{c}\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t}:\\\
i_{j}\neq
k_{j}\end{subarray}}\left(1-e_{\gamma}(\mathbf{i}_{1}^{m}\mathbf{k}_{m+1}^{t},\mathbf{s})\right)$
(13)
$\displaystyle\geq\frac{1}{M}\sum_{\mathbf{s},\mathbf{k}_{1}^{m}}P(\mathbf{s}\mid\mathbf{x}(\mathbf{k})_{1}^{m})\left({\prod_{a=1}^{m}(M_{a}-1)\prod_{b=m+1}^{t}M_{b}}-e_{\gamma}(\mathbf{s})\right)$
(14)
$\displaystyle\geq\left(\frac{\prod_{a=1}^{m}(M_{a}-1)\prod_{b=m+1}^{t}M_{b}}{M_{1}\cdots
M_{t}}-\max_{\mathbf{s}}e_{\gamma}(\mathbf{s})\right)\frac{\sum_{\mathbf{k}_{1}^{m}}1}{M_{1}\cdots
M_{m}}$ (15)
$\displaystyle=\frac{\prod_{a=1}^{m}(M_{a}-1)\prod_{b=m+1}^{t}M_{b}}{M_{1}\cdots
M_{t}}-\max_{\mathbf{s}}e_{\gamma}(\mathbf{s})$ (16)

Altogether,

$\max_{s}e_{\gamma}(\mathbf{s})\geq\frac{\prod_{a=1}^{m}(M_{a}-1)\prod_{b=m+1}^{t}M_{b}}{2M_{1}\cdots
M_{t}}.$ (17)

The lower bound approaches 0.5 in $n$ given that $R_{j}>0$ for $j\in[m]$,
bounding $\max_{s}e_{\gamma}(\mathbf{s})$ away from zero. We conclude that it
is not possible that all $R_{j}$’s, $j\in[m]$, were positive. Thus,
$\mathscr{C}_{\text{auth},\gamma}$ has empty interior. ∎

As in the two user case, it is possible that a channel is not overwritable in
any sense, but is $m$-symmetrizable for some $m>t-\lceil\gamma t\rceil$; in
this case, Theorem III.3 tells us that $\mathscr{C}_{\text{auth},\gamma}$ has
empty interior, even while $\mathscr{C}_{\text{auth}}=\mathscr{C}$ may not.
Furthermore, the proof of Theorem III.3 shows something slightly stronger than
what is stated in the theorem: the projection of
$\mathscr{C}_{\text{auth},\gamma}$ onto any $m$ symmetrizable coordinates
($m>t-\lceil\gamma t\rceil$) must have empty interior.

##  IV Block Length Extension Scheme

In this section, we present a method for extending the block length of a
$\gamma$ partially correcting authentication code whose probability of
$\gamma$ partial correction error is equal to zero. We note that unlike the
classical ($\gamma=1$) message correction case, simple concatenation of such a
code will not automatically achieve the same rate as that of the original
codebook: this is due to the fact that the particular $\lceil\gamma t\rceil$
users whose messages can be corrected in each time instance may vary. To adapt
to our scenario, we use a concatenated code with the inner code tuple equal to
a $\gamma$ partially correcting code with probability of $\gamma$ partial
correction error equal to zero, and outer codes $C_{\varepsilon,j}$ given by a
point-to-point codes designed for an induced erasure channel with a power
constrained adversary. This induced channel is described in detail later in
this section. A simplified version of this scheme appears in [7] for a
particular two-user AV-MAC (a channel that is extended to more users in the
case study of Section V). Here, we extend the scheme to the case of an
arbitrary number of users. We note that our scheme is not channel-dependent
beyond the assumption that such an inner code exists.

Suppose we have a set of $t$ codebooks, each of block length $n$, that have
correction capability $\gamma:=\frac{u}{t}$ (with zero probability of $\gamma$
partial correction error).111For fixed $t$ and integer $1\leq u<t$, we take
$\frac{u-1}{t}\leq\gamma<\frac{u}{t}$ and “round” it to $\frac{u}{t}$. This
will not affect the number of users correctable due to the ceiling function on
$\lceil\gamma t\rceil$. If each user concatenates $r$ codewords from their
codebook, at least $u$ users are correctable in any given time instance, while
the remainder will be deemed to be in erasure. Notice that at most $t-u$ users
experience erasure in each of the $r$ time instances. Our outer codes,
$C_{\varepsilon,j}$, must protect against these erasures for at least $u$ of
the users. To gain an understanding of how this induced erasure channel
functions, consider the following example:

######  Example IV.1.

Consider a set of three codebooks that have partial correction capability
$\gamma=\frac{2}{3}$ and block length $n$, where
$C_{i}:=f_{i}(M_{i})=\\{c_{i1},c_{i2}\\}$ for $i\in[3]$. To extend the block
length, we will use outer codes
$C_{\varepsilon,j}\subseteq\mathbb{F}_{2}^{6}$. For example, let
$C_{\varepsilon,j}=\\{100101,011101,000101,111010\\}$ for all $j\in[3]$. Let 0
be replaced by the first codeword in each of the $C_{j}$’s and 1 be replaced
by the second codeword in each of the $C_{j}$’s. For example the first
codebook would become

$\displaystyle C_{1}^{\prime}=\\{c_{12}c_{11}$ $\displaystyle
c_{11}c_{12}c_{11}c_{12},\ c_{11}c_{12}c_{12}c_{12}c_{11}c_{12},$
$\displaystyle c_{11}c_{11}c_{11}c_{12}c_{11}c_{12},\
c_{12}c_{12}c_{12}c_{11}c_{12}c_{11}\\}$

with block length $6n$ and rate $\frac{2}{6n}=\frac{1}{3n}$. Because the
$C_{j}$’s can correct $\gamma t=2$ of the three users, there will be a maximum
of one erasure in each time instance. An example erasure pattern is given
below, where an erasure is represented by $\varepsilon$. Note that with this
erasure pattern, we can still correct two of the three users’ messages, and
thus achieve the goal of $\gamma$ partial correction.

User 1 | $\varepsilon$ | $\varepsilon$ | $\varepsilon$ | $\varepsilon$ | $\varepsilon$ | $\varepsilon$
---|---|---|---|---|---|---
User 2 | $c_{22}$ | $c_{21}$ | $c_{21}$ | $c_{22}$ | $c_{21}$ | $c_{22}$
User 3 | $c_{31}$ | $c_{32}$ | $c_{32}$ | $c_{32}$ | $c_{31}$ | $c_{31}$

The above erasure pattern example suggests that the adversary’s best strategy
will be to spread their efforts across enough users, but not any more than
needed, in order to deter $\gamma$ partial correction. Intuitively, the
adversary should choose to target $t-u+1$ users to have the most erasures per
affected user while not leaving $u$ users with zero erasures. The optimal
strategy is formalized in the proof of the following lemma.

######  Lemma IV.2.

Let $t\geq 2$ and $1\leq u<t$. If a $\gamma=\frac{u}{t}$ partially correcting
codebook tuple (with error probability zero) is concatenated $r$ times, at
least $u$ users will experience at most $\lfloor\frac{r(t-u)}{t-u+1}\rfloor$
total erasures.

###### Proof.

In each time instant, the adversary can attempt to erase $t-u$ users’ symbols.
If they are always successful, there are a total of $r(t-u)$ erasures across
all users and all time instances. Suppose that the adversary has full control
over which users will experience erasures, and they choose to restrict these
erasures to $\Gamma$ of the $t$ users. First, suppose $\Gamma<t-u+1$. In this
case, there are at least $u$ users that have zero erasures, and we are done.
Now, let $\Gamma\geq t-u+1$. The average number of erasures per targeted user
is $\frac{r(t-u)}{\Gamma}$.

We claim that at least $\Gamma-(t-u)$ users have at most
$\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ erasures. Suppose not, and that at least
$\Gamma-[\Gamma-(t-u)]+1=t-u+1$ users have strictly more than
$\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ erasures. If $\frac{r(t-u)}{t-u+1}$ is an
integer, the total number of erasures across all users and time instances is
strictly bounded below by $(t-u+1)\frac{r(t-u)}{t-u+1}=r(t-u)$. If it is not
an integer, the total number of erasures would be bounded below by
$(t-u+1)\lceil\frac{r(t-u)}{t-u+1}\rceil>r(t-u)$. Both cases contradict that
the total number of erasures is equal to (at most) $r(t-u)$.

Thus, at least $\Gamma-(t-u)$ users have at most
$\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ erasures. The $t-\Gamma$ non-targeted
users have zero erasures. Thus, at least $t-\Gamma+(\Gamma-(t-u))=u$ users
have at most $\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ erasures. ∎

This upper bound on the number of erasures for some subset of $u=\lceil\gamma
t\rceil$ users is tight if the adversary may choose which users to target, and
if they are able to reliably erase their targeted users. Both are advantageous
assumptions for the adversary; we note that they will not always be the case
(see Section V for a channel case study without the latter property).

######  Example IV.3.

Consider the codebooks of Example IV.1 with $\gamma=\frac{u}{t}=\frac{2}{3}$.
First consider the case were the adversary targets all users equally. A
possible erasure pattern is given below:

User 1 | $\varepsilon$ | $\varepsilon$ | $c_{11}$ | $c_{12}$ | $c_{11}$ | $c_{12}$
---|---|---|---|---|---|---
User 2 | $c_{22}$ | $c_{21}$ | $\varepsilon$ | $\varepsilon$ | $c_{21}$ | $c_{22}$
User 3 | $c_{31}$ | $c_{32}$ | $c_{32}$ | $c_{32}$ | $\varepsilon$ | $\varepsilon$

In this case, the decoder needs to be able to correct two erasures (per user)
in order to correct two of the three users.

Next, we look at the case were the adversary focuses their efforts on
$t-u+1=2$ users. Per Lemma IV.2, this is the adversary’s optimal strategy.

User 1 | $\varepsilon$ | $\varepsilon$ | $\varepsilon$ | $c_{12}$ | $c_{11}$ | $c_{12}$
---|---|---|---|---|---|---
User 2 | $c_{22}$ | $c_{21}$ | $c_{21}$ | $\varepsilon$ | $\varepsilon$ | $\varepsilon$
User 3 | $c_{31}$ | $c_{32}$ | $c_{32}$ | $c_{32}$ | $c_{31}$ | $c_{32}$

Here, the decoder needs to correct three erasures in order to correct two of
the three users.

######  Remark IV.4.

According to Lemma IV.2, if we wish to design $C_{\varepsilon,j}$ with
probability of $\gamma$ partial correction error equal to zero,
$d_{min}(C_{\varepsilon,j})\geq\lfloor\frac{r(t-u)}{t-u+1}\rfloor+1$ for each
$j\in[t]$.

Remark IV.4 addresses the requirement of perfect correction of
$C_{\varepsilon,j}$. Allowing for some vanishing decoding error probability,
we turn to the capacity of the induced erasure AVC.

######  Lemma IV.5.

Let $\gamma\in(0,1)$, $n\geq 1$, $t\geq 2$, and $W:=W_{Y|X_{1}\cdots X_{t}S}$
be a $t$-user AV-MAC. Suppose a $\gamma$ partially correcting authentication
code $(M_{1},\ldots,M_{t},n)$ exists for $W$ such that $M_{j}:=2^{R_{j}n}>1$
for all $j\in[t]$ and the probability of $\gamma$ partial correction error is
equal to zero. Then, $\mathscr{C}_{\text{auth},\gamma}$ has nonempty interior.

###### Proof.

Choose a user $j\in[t]$, and define a deterministic erasure AVC as follows:
let $\mathcal{X}=[M_{j}]$, $\mathcal{Y}=[M_{j}]\cup\\{\varepsilon\\}$, and
$\mathcal{S}=\\{s_{0},s_{1}\\}$. Then, $y=x$ if $s=s_{0}$, and $y=\varepsilon$
if $s=s_{1}$. Let $u:=\lceil\gamma t\rceil$. Importantly, the adversary is
power-constrained so that in a length-$r$ transmission they may choose at most
$\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ coordinates to be equal to $s_{1}$; the
remainder must be equal to $s_{0}$. There are no constraints on the legitimate
user’s channel input. This channel mimics the worst-case scenario for (at
least) $u$ users in the above-described concatenation scheme: each has at most
$\lfloor\frac{r(t-u)}{t-u+1}\rfloor$ erasures, and in our induced channel we
assume that any time the adversary attempts to erase a user they can do so
successfully. We refer the reader to Theorem 3 of [5] (and the forthcoming
full version of this work) to verify that this channel has positive capacity.

It remains to explain why this implies an achievable positive rate tuple for
the original AV-MAC. For each user $j\in[t]$, let a code sequence
$(2^{Q_{j}r},r)$ achieve the capacity of the erasure AVC. Using the
concatenation scheme described earlier in this section, with the existing
zero-error $\gamma$ partial correction code as inner code, we achieve rate
$Q_{j}R_{j}>0$ for user $j$. ∎

The case study in [7] calculates the exact capacity of an induced erasure AVC,
which is dependent on the AV-MAC studied there (and extended in Section V), as
well as the specific choice of inner code for that channel. There, when the
adversary acted they had a 0.5 probability of erasing, as opposed to the
guaranteed erasure assumed in the proof of Lemma IV.5. In other words, we
believe it is possible to be more specific about the values of the positive
rate tuples achievable using our extension scheme. We plan to address this
question, as well as the question of whether a less stringent inner code may
be utilized, in a full version of this work.

##  V Zero Probability of Partial Correction Error Codes Case Study

In this section, we turn to a particular $t$-user channel satisfying the
necessary conditions of Section III for authenticated partial correction. We
will work to construct short block length codes with zero probability of
$\gamma$ partial correction error, with the knowledge that such codes can be
extended using the scheme of Section IV. To define the channel, let
$\mathcal{X}_{1}=\cdots=\mathcal{X}_{t}=\\{0,1\\}$,
$\mathcal{S}=\\{0,1,2,\ldots,\ell\\}$ for some $\ell\geq 1$, and
$\mathcal{Y}=\\{0,1,\ldots,t+\ell\\}$, where $Y=X_{1}+X_{2}+\cdots+X_{t}+S$.
The no-adversary state is given by $s_{0}=0$. For ease of notation, we will
denote this channel by $W^{+}_{t,\ell}$. Observe that $W^{+}_{t,\ell}$ is
deterministic given a choice of state.

###  V-A Necessary conditions are satisfied

We first verify that $W^{+}_{t,\ell}$ satisfies the necessary overwritability
and symmetrizability conditions established in Section III.

######  Lemma V.1.

$W^{+}_{t,\ell}$ is not $m$-overwritable for any $m\in[t]$.

###### Proof.

Let $W:=W^{+}_{t,\ell}$ and $m\in[t]$. Toward contradiction, assume the
channel is $m$-user overwritable in the first $m$ coordinates, and let $P$ be
the distribution guaranteed by the definition of overwritability. Let
$x_{1}^{\prime}=\cdots=x_{m}^{\prime}=1$, $x_{1}=\cdots=x_{t}=0$, and $y=0$.
Then,

$\displaystyle\sum_{s=0}^{\ell}P(s|\underbrace{1,\ldots,1}_{m})W(0$
$\displaystyle|\underbrace{0,\ldots,0}_{t},s)=$ $\displaystyle
W(0|\underbrace{\overbrace{1,\ldots,1}^{m},\overbrace{0,\ldots,0}^{t-m}}_{t},0)$
(18)

On the left hand side, $W(0|0,\ldots,0,s)=1$ if and only if $s=0$; otherwise
it is zero. Therefore, the left side is equal to $P(0|1,\ldots,1)$. On the
right side, $W(0|1,\ldots,1,0,\ldots,0,0)=0$. Thus, $P(0|1,\ldots,1)=0$.

A similar argument with $x_{1}^{{}^{\prime}}=\cdots=x_{m}^{{}^{\prime}}=1$,
$x_{1}=x_{2}=\cdots=x_{m}=1$, $x_{m+1}=\cdots=x_{t}=0$, and $y=m$ yields
$P(0|1,\dots,1)=1$. This is a contradiction because $P(0|1,\ldots,1)$ cannot
be equal to both 0 and 1. Therefore, the channel is not $m$-user overwritable.
∎

Recall that the other necessary condition for a $t$-user channel to be
$\gamma$ partially correcting is that the channel is not $q$-user
symmetrizable for any $q\geq t-\lceil\gamma t\rceil$. The following
establishes allowed values of $\gamma$ given the adversary’s power constraint
$\ell$.

######  Lemma V.2.

Let $\ell\leq t$. Then $W^{+}_{t,\ell}$ is $q$-user symmetrizable for any
subset of $q$ users exactly when $q\leq\ell$.

###### Proof.

Let $W:=W^{+}_{t,\ell}$. First we show that the channel is $q$-user
symmetrizable for $q\leq\ell$. Consider the following probability
distribution: $P(s|x_{1},...,x_{q})=1$ if $\sum_{i=1}^{q}x_{i}=s$, and $0$
otherwise. Notice that because each $x_{i}\in\\{0,1\\}$, we have
$0\leq\sum_{i=1}^{q}x_{i}\leq q$. Because $q\leq\ell$, for a fixed choice of
$x_{i}$’s, $P(s|x_{1},\ldots,x_{q})=1$ for exactly one choice of $s$. Using
this distribution in Definition II.3, the left hand size yields
$W\left(y|x_{1},\ldots,x_{t},\sum_{i=1}^{q}x_{i}^{\prime}\right)$, and the
right hand size becomes
$W\left(y|x_{1}^{\prime},\ldots,x_{q}^{\prime},x_{q+1},\ldots,x_{t},\sum_{i=1}^{q}x_{i}\right)$.
The sum of the inputs will be the same on both sides, so either both sides are
equal to 1 if the sum of the inputs is equal to $y$ or both sides are equal to
0 if the sum of the inputs is not equal to $y$. Therefore, the channel is
$q$-user symmetrizable for the first $q$ users. Notice that permuting the
coordinates will not change the argument, so we have shown that the channel is
$q$-user symmetrizable for any subset of $q$ users.

Now we will show the channel is not $q$-user symmetrizable when $\ell+1\leq
q\leq t$. By way of contradiction, assume the channel is $q$-user
symmetrizable for such a $q$. Let $P$ be the distribution guaranteed by
Definition II.3, and let $z\in S$ (note that $z\leq\ell<t$). We claim that
$P(z|0,...,0)=0$. Assume $x_{1}^{\prime}=\ldots=x_{q}^{\prime}=0$,
$x_{1}=\ldots=x_{t-z}=1$, $x_{t-z+1}=\ldots=x_{t}=0$, and $y=t$. Then, the
left hand side of the definition is the sum over $s\in\mathcal{S}$ of

$P(s|\underbrace{0,\ldots,0}_{q})W(t|\underbrace{\overbrace{1,\ldots,1}^{t-z},{0,\ldots,0}}_{t},s)$
(19)

and the right hand side is equal to the sum over $s\in\mathcal{S}$ of

$P(s|\underbrace{\overbrace{1,\ldots,1}^{\min\\{t-z,q\\}},{0,\ldots,0}}_{q})W(t|\underbrace{\overbrace{0,\ldots,0}^{q},\overbrace{1,\ldots,1}^{((t-z)-q)^{+}},{0,\ldots,0}}_{t},s)$
(20)

where we use $\lambda^{+}$ to denote $\max\\{\lambda,0\\}$. In Equation (19),
$W(t|1,\ldots,1,0,\ldots,0,s)=1$ if and only if $s=z$. Therefore, (19) is
equal to $P(z|0,\ldots,0)$. In Equation (20),
$W(t|0,\dots,0,1,\dots,1,0,\ldots,0,s)=1$ if and only if $t=(t-z-q)^{+}+s$.
Since $s\leq\ell<q\leq t$, it follows that $s-q<0$, and also $s<t$. Thus,
$(t-z-q)^{+}+s<t$, and (20) equals 0. Therefore, $P(z|0,\ldots,0)=0$ for all
$0\leq z\leq\ell$, a contradiction. Therefore, no such $P$ exists and the
channel is not $q$-user symmetrizable for $\ell+1\leq q\leq t$. ∎

Lemmas V.1 and V.2 together establish the following:

######  Theorem V.3.

The channel $W_{t,\ell}^{+}$ satisfies the necessary conditions for $\gamma$
partial correction established by Theorem III.1, Lemma III.2, and Theorem
III.3.

###  V-B Zero probability of $\gamma$ partial correction characterization

To aid in our discussion of code construction, we next introduce notation that
will help explain when a codebook is $\gamma$ partially correcting with zero
probability of error for $W_{t,\ell}^{+}$. Let $C_{j}:=f_{j}(M_{j})$ denote
the block length $n$ codebook of user $j$, so that each $\mathbf{x}\in C_{j}$
is equal to $f_{j}(i)$ for some $i\in M_{j}$. Then, define the following
multisets:

$\displaystyle A_{0}$
$\displaystyle:=\left\\{\mathbf{u}=\sum_{j=1}^{t}\mathbf{x}_{j}\ \biggr{|}\
\mathbf{x}_{j}\in C_{j}\right\\}$ (21) $\displaystyle A_{1}$
$\displaystyle:=\left\\{\mathbf{u}=\mathbf{s}+\sum_{j=1}^{t}\mathbf{x}_{j}\
\biggr{|}\ \mathbf{x}_{j}\in C_{j},\
\mathbf{s}\in\mathcal{S}\setminus\\{\mathbf{s}_{0}\\}\right\\}$ (22)

In other words, the set $A_{0}$ is the set of all possible channel outputs
(with multiplicity) when the adversary does not act, and $A_{1}$ is the set of
all outputs when the adversary does act.

######  Lemma V.4.

Over the channel $W_{t,\ell}^{+}$, a codebook $t$-tuple is $\gamma$ partially
correcting with zero probability of $\gamma$ partial correction error if and
only if all of the following hold:

  1. (1)

$A_{0}\cap A_{1}=\emptyset$;

  2. (2)

The elements of $A_{0}$ are unique;

  3. (3)

For each $\mathbf{w}\in A_{1}$ that appears with multiplicity, there exists
some subset $J=\\{j_{1},j_{2},\ldots,j_{\lceil\gamma t\rceil}\\}\subseteq[t]$
such that if $\mathbf{s}+\sum_{j=1}^{t}\mathbf{x}_{j}=\mathbf{w}$ and
$\mathbf{s}^{\prime}+\sum_{j=1}^{t}\mathbf{x}^{\prime}_{j}=\mathbf{w}$, then
$\mathbf{x}_{i}=\mathbf{x}^{\prime}_{i}$ for all $i\in J$.

###### Proof.

Condition (1) ensures that the decoder can reliably distinguish between the
case where the adversary has acted and the case where they have transmitted a
sequence of all zeros ($\mathbf{s}_{0}$). Taken together with (2), we have
$e_{\gamma}(\mathbf{i},\mathbf{s}_{0})=0$ for every message tuple
$\mathbf{i}$. If either condition fails,
$e_{\gamma}(\mathbf{i},\mathbf{s}_{0})>0$.

With (1), condition (3) establishes that $e_{\gamma}(\mathbf{i},\mathbf{s})=0$
when $\mathbf{s}\neq\mathbf{s}_{0}$: if there are repeated elements in
$A_{1}$, we are guaranteed to be able to correct $\lceil\gamma t\rceil$ of the
messages, even if the others must be discarded. If condition (3) fails,
$e_{\gamma}(\mathbf{i},\mathbf{s})>0$. ∎

With this characterization in hand, we turn to short codebook design
strategies. For the remainder of the paper, we will focus on the codebooks
$C_{j}:=f_{j}(M_{j})$; thus, we will will discuss codebook tuples of the form
$(C_{1},\ldots,C_{t})$.

###  V-C Two users

Here we present necessary conditions for $\gamma=0.5$ partially correcting
codebook pairs over $W_{2,1}^{+}$ with zero probability of $\gamma$ partial
correction error, and bound the sizes of these codebooks for fixed block
length.

######  Example V.5.

The codebook pair $C_{1}=\\{011,100\\}$, and $C_{2}=\\{010,101\\}$ is $0.5$
partially correcting over $W_{2,1}^{+}$ with zero probability of $\gamma$
partial correction error. This example was explored extensively in [7].

The each codebook of Example V.5 has the property that codeword supports are
not contained in one another. We find that this is true in general for partial
correction over $W_{t,\ell}^{+}$.

######  Theorem V.6.

Let $(C_{1},\dots,C_{t})$ be a codebook tuple with $C_{j}\in\\{0,1\\}^{n}$ for
$j\in[t]$. If, for some $j\in[t]$, $\mathbf{x},\mathbf{y}\in C_{j}$ with
$\mathbf{x}\neq\mathbf{y}$ and $\text{\em supp}(\mathbf{x})\subseteq\text{\em
supp}(\mathbf{y})$, then the codebook tuple $(C_{1},\dots,C_{t})$ is not
$\gamma$ partially correcting with zero probability of $\gamma$ partial
correction error over $W_{t,\ell}^{+}$ for any $\gamma\in(0,1)$.

###### Proof.

Let $\gamma\in(0,1)$. Suppose that for some $j\in[t]$ there exist
$\mathbf{x}\neq\mathbf{y}\in C_{j}$ such that
$\textrm{supp}(\mathbf{x})\subseteq\textrm{supp}(\mathbf{y})$. Let
$\textrm{supp}(\mathbf{y})\setminus\textrm{supp}(\mathbf{x})$ be the support
of the adversarial contribution $\mathbf{s}$. Notice that
$\mathbf{s}\in\\{0,1\\}^{n}\subseteq\mathcal{S}^{n}$, and
$\mathbf{s}+\mathbf{x}=\mathbf{y}+\mathbf{0}$. Thus $A_{0}\cap
A_{1}\neq\emptyset$. By Lemma V.4, the result follows. ∎

Let the partially ordered set (poset) $\mathcal{P}([n])$ be the power set of
$[n]$ together with the partial order defined by set inclusion. Elements of
the poset can alternatively be thought of as vectors in $\\{0,1\\}^{n}$;
elements whose supports are contained in one another are then related under
the partial order. Theorem V.6 states that each codebook of a $\gamma$
partially correcting (with zero probability of $\gamma$ partial correction
error) codebook tuples are antichains (sets of unrelated elements) in
$\mathcal{P}([n])$. The following corollary is a direct consequence of Theorem
V.6.

######  Corollary V.7.

Suppose the codebook tuple $(C_{1},\dots,C_{t})$ is $\gamma$ partially
correcting with zero probability of $\gamma$ partial correction error over
$W_{t,\ell}^{+}$ for some $\gamma\in(0,1)$. If $|C_{j}|>1$, then
$\mathbf{0},\mathbf{1}\notin C_{j}$.

While each individual codebook must be an antichain, the disjoint union of
codebooks need not be. Indeed, Example V.5 has two (disjoint) related pairs
across codebooks. The following places a limit on such support containment.

######  Theorem V.8.

Let $(C_{1},C_{2})$ be a $0.5$ partially correcting codebook pair with zero
probability of $\gamma$ partial correction error over $W_{2,1}^{+}$. Provided
$|C_{1}|,|C_{2}|>1$, $C_{1}\cap C_{2}=\emptyset$ and $C_{1}\sqcup C_{2}$ has
at most two pairs of related codewords having the property that the
intersection of these pairs is empty.

###### Proof.

Let $(C_{1},C_{2})$ be a $0.5$ partially correcting codebook pair with zero
probability of $\gamma$ partial correction error over $W_{2,1}^{+}$ and
$|C_{1}|,|C_{2}|>1$. To show that the codebooks are disjoint, suppose
$\mathbf{x}\in C_{1}\cap C_{2}$. Let $\mathbf{a}\neq\mathbf{x}\in C_{1}$ and
$\mathbf{b}\neq\mathbf{x}\in C_{2}$. Note that
$\mathbf{a},\mathbf{b},\mathbf{x}\in\mathcal{S}^{n}$. Observe that if either
[$\mathbf{x}_{1}=\mathbf{x}$, $\mathbf{x}_{2}=\mathbf{b}$,
$\mathbf{s}=\mathbf{a}$], or [$\mathbf{x}_{1}=\mathbf{a}$,
$\mathbf{x}_{2}=\mathbf{x}$, $\mathbf{s}=\mathbf{b}$], the channel output is
$\mathbf{x}+\mathbf{b}+\mathbf{a}$. This violates condition (3) of Lemma V.4.
We have thus established that $C_{1}\cap C_{2}=\emptyset$.

Next, we will show that if there are two related pairs in the union, they have
a particular structure. Let
$\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{y}_{1},\mathbf{y}_{2}\in C_{1}\sqcup
C_{2}$ be distinct such that $\mathbf{x}_{1}\leq\mathbf{x}_{2}$ and
$\mathbf{y}_{1}\leq\mathbf{y}_{2}$. Making use of Theorem V.6, and without
loss of generality, we have two cases: either
$\mathbf{x}_{1},\mathbf{y}_{1}\in C_{1}$, or $\mathbf{x}_{1},\mathbf{y}_{2}\in
C_{1}$.

Suppose $\mathbf{x}_{1},\mathbf{y}_{1}\in C_{1}$ and
$\mathbf{x}_{2},\mathbf{y}_{2}\in C_{2}$. Let
$\mathbf{a},\mathbf{b}\in\mathcal{S}^{n}$ be such that
$\mathbf{x}_{1}+\mathbf{a}=\mathbf{x}_{2}$ and
$\mathbf{y}_{1}+\mathbf{b}=\mathbf{y}_{2}$. Then,

$\mathbf{x}_{1}+\mathbf{y}_{2}+\mathbf{a}=\mathbf{x}_{2}+\mathbf{y}_{2}=\mathbf{y}_{1}+\mathbf{x}_{2}+\mathbf{b},$
(23)

contradicting condition $(3)$ of Lemma V.4. Thus, it must be the case that the
two smaller elements, $\mathbf{x}_{1}$ and $\mathbf{y}_{1}$, must be in
distinct codebooks.

Now suppose that
$\mathbf{x}_{1},\mathbf{x}_{2},\mathbf{y}_{1},\mathbf{y}_{2},\mathbf{z}_{1},\mathbf{z}_{2}\in
C_{1}\sqcup C_{2}$ are distinct elements with
$\mathbf{x}_{1}\leq\mathbf{x}_{2},\ \mathbf{y}_{1}\leq\mathbf{y}_{2},\
\mathbf{z}_{1}\leq\mathbf{z}_{2}$. Based on our above structural argument,
$\mathbf{x}_{1}$, $\mathbf{y}_{1}$, and $\mathbf{z}_{1}$ must pairwise belong
to different codebooks, an impossibility. ∎

Theorem V.8 implies an upper bound on the size of the disjoint union:

######  Theorem V.9.

Let $(C_{1},C_{2})$ be a $0.5$ partially correcting codebook pair with zero
probability of $\gamma$ partial correction error over $W_{2,1}^{+}$, with
$|C_{1}|,|C_{2}|>1$. Then $|C_{1}\sqcup
C_{2}|\leq\binom{n}{\lceil\nicefrac{{n}}{{2}}\rceil}+2$.

###### Proof.

From Sperner’s theorem, the size of a largest antichain in $\mathcal{P}([n])$
is $\binom{n}{\lceil\nicefrac{{n}}{{2}}\rceil}$. From Theorem V.8,
$C:=C_{1}\sqcup C_{2}$ will have the form of an antichain with at most two
additional elements. The result follows. ∎

###  V-D Three or more users

With the goal of constructing short codebook tuples with zero probability of
$\gamma$ partial correction error over $W_{t,\ell}^{+}$ for $t\geq 3$, in this
section we develop equivalent conditions for (1) and (3) of Lemma V.4 which
are easier to check computationally. In particular, we will reinterpret these
conditions in terms of differences of sums of legitimate codewords. This
allows us to avoid actually constructing $A_{1}$, and to instead check
conditions on the elements of $A_{0}$ (i.e. sums of codewords).

In each of the results of this section, we will consider two such sums:
$\mathbf{u}=\sum_{j=1}^{t}\mathbf{x}_{j}$ and
$\mathbf{v}=\sum_{j=1}^{t}\mathbf{y}_{j}$, where (not necessarily distinct)
$\mathbf{x}_{j},\mathbf{y}_{j}\in C_{j}$ are vectors of length $n$ for each
$j\in[t]$.

######  Lemma V.10.

Let $(C_{1},\ldots,C_{t})$ be a codebook tuple for the channel
$W_{t,\ell}^{+}$, where $\ell\geq 1$. There exist distinct $\mathbf{u,v}\in
A_{0}$ such that $\mathbf{u}-\mathbf{v}$ is in
$\\{0,\ldots,\ell\\}^{n}=\mathcal{S}^{n}$ if and only if $A_{0}\cap
A_{1}\neq\emptyset$.

###### Proof.

Suppose the vector difference
$\mathbf{u}-\mathbf{v}\in\\{0,\ldots,\ell\\}^{n}$ for some choice of
$\mathbf{u,v}\in A_{0}$. In this case, $\mathbf{u}-\mathbf{v}$ is an element
in $\mathcal{S}^{n}$ not equal to $\mathbf{s}_{0}$; call this difference
$\mathbf{s}$. Then, $\mathbf{u}=\mathbf{v}+\mathbf{s}$. We then see that
$\mathbf{u}\in A_{1}$ and $\mathbf{u}\in A_{0}$. This means that $A_{0}\cap
A_{1}\neq\emptyset$. On the other hand, let $\mathbf{u}$ be an element of
nonempty $A_{0}\cap A_{1}$. Since $\mathbf{u}\in A_{1}$, it must be the case
that $\mathbf{u}=\mathbf{v}+\mathbf{s}$ for some $\mathbf{v}\in A_{0}$,
$\mathbf{s}\neq\mathbf{s}_{0}$. Then,
$\mathbf{u}-\mathbf{v}\in\mathcal{S}^{n}$, and we are done. ∎

The following two examples illustrate Lemma V.10.

######  Example V.11.

Consider the set of three codebooks with block length $n=6$ given below:

$\displaystyle C_{1}$ $\displaystyle=\\{100110,110110\\}$ (24) $\displaystyle
C_{2}$ $\displaystyle=\\{111010,100101\\}$ (25) $\displaystyle C_{3}$
$\displaystyle=\\{011111,001010\\}$ (26)

We claim that $A_{0}\cap A_{1}\neq\emptyset$ over $W_{3,\ell}^{+}$ for this
codebook. Let $\mathbf{u},\mathbf{v}\in A_{0}$ be given by

$\displaystyle\mathbf{u}$ $\displaystyle=100110+111010+011111=222231$ (27)
$\displaystyle\mathbf{v}$ $\displaystyle=110110+100101+001010=211221$ (28)

Then, $\mathbf{u}-\mathbf{v}=222231-211221=011010$. Here, we can see that
$\mathbf{u}-\mathbf{v}$ is an element in $\mathcal{S}^{n}$ for any $\ell\geq
1$. Observe that $\mathbf{u}=\mathbf{v}+\mathbf{s}$ when $\mathbf{s}=011010$,
so that $A_{0}\cap A_{1}\neq\emptyset$ and condition (1) of Lemma V.4 fails.

######  Example V.12.

Consider the set of three codebooks with block length $n=6$ given below:

$\displaystyle C_{1}$ $\displaystyle=\\{011010,100101\\}$ (29) $\displaystyle
C_{2}$ $\displaystyle=\\{010110,101001\\}$ (30) $\displaystyle C_{3}$
$\displaystyle=\\{001101,110010\\}$ (31)

We claim that $A_{0}\cap A_{1}=\emptyset$ over $W_{3,\ell}^{+}$ for this
codebook. Consider the following choice of $\mathbf{u}$ and $\mathbf{v}$ as an
example:

$\displaystyle\mathbf{u}$ $\displaystyle=011010+010110+001101=022221$ (32)
$\displaystyle\mathbf{v}$ $\displaystyle=100101+101001+110010=311112$ (33)

Then,

$\displaystyle\mathbf{u}-\mathbf{v}$
$\displaystyle=022221-311112=(-3)1111(-1)$ (34)

Here we can see that $\mathbf{u}-\mathbf{v}$ is not an element in
$\mathcal{S}^{n}$. All values of $\mathbf{u}-\mathbf{v}$ can be looped through
for this channel to show that $A_{0}\cap A_{1}=\emptyset$.

Next, we rephrase condition (3) of Lemma V.4 in terms of elements of $A_{0}$.

######  Lemma V.13.

Let $(C_{1},\ldots,C_{t})$ be a codebook tuple for the channel
$W_{t,\ell}^{+}$, where $\ell\geq 1$, and let $\gamma\in(0,1)$. Let
$\mathbf{u,v}\in A_{0}$ such that $\mathbf{u}=\sum_{j=1}^{t}\mathbf{x}_{j}$
and $\mathbf{v}=\sum_{j=1}^{t}\mathbf{y}_{j}$ and
$\mathbf{x}_{j}\neq\mathbf{y}_{j}$ for at least $t-\lceil\gamma t\rceil+1$
values of $j\in[t]$. If the maximum entry of $|\mathbf{u}-\mathbf{v}|$ is at
most $\ell$, condition (3) of Lemma V.4 fails.

###### Proof.

Let $(C_{1},\ldots,C_{t})$ be a codebook tuple for the channel
$W_{t,\ell}^{+}$, where $\ell\geq 1$, and let $\gamma\in(0,1)$. Let
$\mathbf{u,v}\in A_{0}$ such that $\mathbf{u}=\sum_{j=1}^{t}\mathbf{x}_{j}$
and $\mathbf{v}=\sum_{j=1}^{t}\mathbf{y}_{j}$ and
$\mathbf{x}_{j}\neq\mathbf{y}_{j}$ for at least $t-\lceil\gamma t\rceil+1$
values of $j\in[t]$. If the maximum entry of $|\mathbf{u}-\mathbf{v}|$ is
$\leq\ell$, then $|\mathbf{u}-\mathbf{v}|\in\mathcal{S}^{n}$. Thus,
$|\mathbf{u}-\mathbf{v}|=\mathbf{s}$ and $u_{i}-v_{i}=\pm s_{i}$ for each
$i\in[n]$. Letting $\mathbf{s}_{1}$ equal $\mathbf{s}$ on coordinates where
$u_{i}-v_{i}$ is negative, and zero elsewhere, and $\mathbf{s}_{2}=\mathbf{s}$
when $u_{i}-v_{i}$ is positive, and zero elsewhere, we have
$\mathbf{u}+\mathbf{s}_{1}=\mathbf{v}+\mathbf{s}_{2}$. Condition (3) fails
because the sums cannot match on any subset of $\lceil\gamma t\rceil$
codewords.

∎

The following example illustrates Lemma V.13.

######  Example V.14.

Consider the set of three codebooks with block length $n=6$ given below:

$\displaystyle C_{1}$ $\displaystyle=\\{011010,100101\\}$ (35) $\displaystyle
C_{2}$ $\displaystyle=\\{010110,101001\\}$ (36) $\displaystyle C_{3}$
$\displaystyle=\\{010101,101010\\}$ (37)

We claim that condition (3) of Lemma V.4 fails over $W_{3,\ell}^{+}$ when
$\gamma=\frac{1}{3}$ or $\gamma=\frac{2}{3}$. Let $\mathbf{u},\mathbf{v}\in
A_{0}$ be

$\displaystyle\mathbf{u}$ $\displaystyle=100101+010110+101010=211221$ (38)
$\displaystyle\mathbf{v}$ $\displaystyle=011010+101001+010101=122112$ (39)

It can be seen that $\mathbf{u}$ and $\mathbf{v}$ differ on all three users.

$\displaystyle|\mathbf{u}-\mathbf{v}|$ $\displaystyle=|211221-122112|=111111$
(40)

and $\mathbf{u}-\mathbf{v}=1(-1)(-1)11(-1)$ such that
$\mathbf{u}+011001=\mathbf{v}+100110$. Thus, condition (3) fails due to the
fact that there is a repeated element in $A_{1}$ for which no user codewords
match.

We observe that condition (2) of Lemma V.4 is straightforward to check on
$A_{0}$ alone. Combining all checks on $A_{0}$ described in this section, we
can algorithmically loop through the possible combinations of differences of
elements of $A_{0}$ to test whether a codebook triple is a candidate for
partial correction with zero probability of $\gamma$ partial correction error.
Notably, the check of Lemma V.13 is necessary (for some such
$\mathbf{u},\mathbf{v}$) for failure of condition (3) when there are three
users and $\gamma=\frac{2}{3}$. In fact, the codebook given in Example V.12 is
a good codebook triple for $W_{3,1}^{+}$ with $\gamma=\frac{2}{3}$. The
extension scheme of Section IV can thus be used to achieve positive rate
triples with arbitrary block length.

##  VI Conclusion

In this paper, we gave necessary (non-)symmetrizability and
(non-)overwritability conditions for nonempty interior of the $\gamma$
partially correcting authentication capacity region over a $t$-user AV-MAC. We
presented a scheme to extend the block length of a strong short block length
code, showing that the resulting extension can maintain the positive rates of
the short code. Finally, we examined the particular AV-MAC denoted
$W_{t,\ell}^{+}$, deriving structural results and bounds for zero $\gamma$
partial correction error codes over this channel. Ongoing and future
directions include sufficiency of the aforementioned necessary channel
conditions for partial correction, refinement of our block length extension
scheme, and alternative paths toward inner bounds on the $\gamma$ partially
correcting authentication capacity region.

## Acknowledgment

We would like to thank Shuqi (Ariel) Liu for her previous work on establishing
a version of Lemma V.4 for $W_{2,1}^{+}$ and the disjointness portion of
Theorem V.8.

## References

  * [1] J.-H. Jahn, “Coding of arbitrarily varying multiuser channels,” _IEEE Trans. on Inf. Theory_ , vol. 27, no. 2, pp. 212–226, 1981.
  * [2] J. A. Gubner, “On the deterministic-code capacity of the multiple-access arbitrarily varying channel,” _IEEE Trans. on Inf. Theory_ , vol. 36, no. 2, pp. 262–275, 1990.
  * [3] R. Ahlswede and N. Cai, “Arbitrarily varying multiple-access channels. I. Ericson’s symmetrizability is adequate, Gubner’s conjecture is true,” _IEEE Trans. on Inf. Theory_ , vol. 45, no. 2, pp. 742–749, 1999.
  * [4] U. Pereg and Y. Steinberg, “The capacity region of the arbitrarily varying mac: with and without constraints,” in _IEEE Int’l Symp. on Inf. Theory_ , 2019, pp. 445–449.
  * [5] I. Csiszar and P. Narayan, “The capacity of the arbitrarily varying channel revisited: positivity, constraints,” _IEEE Trans. on Inf. Theory_ , vol. 34, no. 2, pp. 181–193, 1988.
  * [6] O. Kosut and J. Kliewer, “Authentication capacity of adversarial channels,” in _IEEE Inf. Theory Workshop (ITW)_ , 2018, pp. 1–5.
  * [7] A. Beemer, E. Graves, J. Kliewer, O. Kosut, and P. Yu, “Authentication and partial message correction over adversarial multiple-access channels,” in _IEEE Conf. on Comm.s and Network Sec._ , 2020, pp. 1–6.
  * [8] N. Sangwan, M. Bakshi, B. K. Dey, and V. M. Prabhakaran, “Multiple access channels with adversarial users,” in _IEEE Int’l Symp. on Inf. Theory_ , 2019, pp. 435–439.
  * [9] N. Sangwan, M. Bakshi, B. K. Dey, and V. M. Prabhakaran, “Byzantine multiple access channels–part II: Communication with adversary identification,” _arXiv preprint 2309.11174_ , 2023.
  * [10] S. Nitinawarat, “On the deterministic code capacity region of an arbitrarily varying multiple-access channel under list decoding,” _IEEE Trans. on Inf. Theory_ , vol. 59, no. 5, pp. 2683–2693, 2013.
  * [11] N. Cai, “List decoding for arbitrarily varying multiple access channel revisited: List configuration and symmetrizability,” _IEEE Trans. on Inf. Theory_ , vol. 62, no. 11, pp. 6095–6110, 2016.


# Event Camera Demosaicing via Swin Transformer and Pixel-focus Loss

Yunfan Lu, Yijie Xu, Wenzong Ma, Weiyu Guo, Hui Xiong
AI Thrust, Hong Kong University of Science and Technology (Guangzhou)
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent research has highlighted improvements in high-quality imaging guided by
event cameras, with most of these efforts concentrating on the RGB domain.
However, these advancements frequently neglect the unique challenges
introduced by the inherent flaws in the sensor design of event cameras in the
RAW domain. Specifically, this sensor design results in the partial loss of
pixel values, posing new challenges for RAW domain processes like demosaicing.
The challenge intensifies as most research in the RAW domain is based on the
premise that each pixel contains a value, making the straightforward
adaptation of these methods to event camera demosaicing problematic. To end
this, we present a Swin-Transformer-based backbone and a pixel-focus loss
function for demosaicing with missing pixel values in RAW domain processing.
Our core motivation is to refine a general and widely applicable foundational
model from the RGB domain for RAW domain processing, thereby broadening the
model’s applicability within the entire imaging process. Our method harnesses
multi-scale processing and space-to-depth techniques to ensure efficiency and
reduce computing complexity. We also proposed the Pixel-focus Loss function
for network fine-tuning to improve network convergence based on our discovery
of a long-tailed distribution in training loss. Our method has undergone
validation on the MIPI Demosaic Challenge dataset, with subsequent analytical
experimentation confirming its efficacy. All code and trained models are
released here: https://github.com/yunfanLu/ev-demosaic.

##  1 Introduction

Figure 1: Contemporary design of an actual event camera sensor (Hybridevs
sensor), featuring red, green, and blue pixels for outputting RGB RAW signals.
Black pixels in the lower right corner of the green and red areas are
designated for event signal output, and white pixels do not emit any signals.
The demosaicing task aims to convert a RAW image with RGB signals and black
holes (a) into a full-color image with three RGB channels (b).

The event camera [8, 59, 53], with its low latency ($<100\mu s$), high dynamic
range ($>120dB$), high temporal resolution ($>1000fps$), and efficient power
consumption, has garnered significant interest for enhancing computational
imaging in applications, _e.g_., video frame interpolation [46, 35, 28],
super-resolution [15, 29], deblurring [52, 44, 14], and high dynamic range
[38, 32]. These works are realized in the RGB domain, based on the premise
that the camera sensor can simultaneously and seamlessly deliver RGB images
and events. These RGB images are obtained by RAW image processing approaches.
Specifically, RAW domain approaches convert RAW images, where each pixel
contains only one type of color information with noise, into full-color images
with all three RGB color information with high-quality [13]. Filling in the
missing color information is known as demosaicing [31, 53], a core component
of RAW domain image process.

However, the transformation of RAW to RGB faces significant challenges due to
the existing event sensor chip design technology. Specifically, event cameras
produce RAW images where specific pixels are absent, as illustrated in Fig. 1.
These missing pixels emit event signals, not RGB signals, resulting in
incomplete pixel values in the RAW output. This absence of pixel values poses
challenges for traditional RAW domain processing approaches, such as
demosaicing, because the pixels emitting event signals cause the RAW image to
lose a quarter of its red and blue color information. To effectively address
the challenge of enhancing downstream RGB images/videos with event guidance,
it is imperative to transform incomplete RAW images into high-quality RGB
full-color images without losing information. Moreover, minimizing accumulated
errors in this transformation process is essential for improving the quality
of inputs for downstream tasks [12].

RAW image process methods [10, 31, 43, 57, 17, 37, 1, 27] are predicated on
traditional sensor technology, wherein each pixel is capable of capturing a
color signal. These methods can be divided into model-based [10, 31, 43, 57]
and learning-based methods [17, 37, 1]. Learning-based approaches have
attracted substantial interest due to the powerful fitting capabilities and
robust generalizability of neural networks. Inspired by the evolution of
network architectures, these methods primarily aim to integrate and innovate
neural network designs for RAW to RGB conversion mapping. Many convolutional
neural networks (CNNs) [17, 37, 1] were designed as the backbone for tasks
such as demosaicing and denoising of RAW images. For instance, PyNet [12] has
designed a multi-scale, multi-resolution CNN network architecture for
processing RAW into RGB. More recently, benefits from the enhanced contextual
modeling abilities and broader receptive fields from Vision Transformers (ViT)
[7]. Many Transformer-based methods are proposed for image processing [42, 26,
49, 55, 16, 51]. For example, RSTCANet [51] employs the Swin-Transformer [22,
9] to the demosaicing, incorporating global residual connections. However,
RSTCANet [51] stacks Swin-Transformer layers [22, 24, 25] without down-sample
and multi-scale leads to higher computational complexity while failing to
provide the network with a sufficiently large field of view.

Based on these considerations, we employ the Swin-Transformer-based backbone
and a pixel-focus loss function for event camera demosaicing. Our motivation
is threefold: (1) Scalability: The Swin Transformer is a widely used and
powerful foundational model in the RGB domain. Adapting it to the RAW domain
could bridge foundational modeling across RAW and RGB imaging tasks. (2)
Efficiency: RAW domain methods are upstream of RGB domain processes and
underpin all computer vision tasks. Therefore, RAW domain methods need to be
sufficiently efficient to support downstream applications in the real world.
(3) Training Effectivity: A long-tail distribution of training loss was
identified for the demosaicing task. Consequently, the pixel-focus loss was
designed to facilitate a two-stage training process, enhancing the network’s
performance.

To achieve scalability, we refine the standard operators from the Swin-
transformer [22, 9] while avoiding customizations to enhance its portability.
To ensure efficiency, we initially employ the space-to-depth [5] method to
reduce the network’s resolution and design a network structure akin to U-Net
[39], achieving multi-scale and multi-resolution capabilities. This structure
allows for a broader field of view with fewer layers. For effective training,
we devised a two-stage loss function to fine-tune the network after completing
the first training phase with Charbonnier loss [18].

Our method underwent testing on the MIPI Demosaic Challenge dataset [53, 34]
of the CVPR 2024 Workshop, demonstrating its applicability and performance.
Subsequently, we conducted additional analytical experiments to evaluate its
robustness and adaptability. These tests solidified the method’s effectiveness
in various scenarios, clearly illustrating the superiority of our approach in
addressing the intricacies of demosaicing in the RAW domain. Furthermore, we
believe this work will inspire applications in the RAW domain and catalyze
enhancements across multiple RAW-based tasks, fostering a new wave of
innovation.

##  2 Related Works

Modern digital cameras capture light, producing images with individual color
channels (_e.g_., red, green, or blue) for each pixel [30]. To compensate for
the absence of color information, demosaicing is devised to reconstruct a
full-color image from a single-channel RAW image [20]. In addition, owing to
the wave-particle duality of light and the instability induced by dark
currents in electronic devices, noise is a pervasive issue in the pixels of
RAW images [3]. Consequently, the processes of denoising and demosaicing
frequently occur concurrently. Our paper focuses on the RAW domain processing
of event cameras, prioritizing the demosaicing task due to its unique
characteristics from the event camera sensor design.

Figure 2: Visual results of two images at different stages of training.
Example (I) displays an image with less edge and texture, featuring extensive
areas of sky and lake, while example (II) presents an image rich in edge and
texture, including animal fur and splashing water. For these two examples,
four groups of reconstruction results are shown under varying PSNR values,
along with different maps and difference distributions. Here, ”difference”
refers to the absolute value of discrepancies compared to the ground truth. As
PSNR increases, the differences exhibit a long-tailed distribution.

###  2.1 Camera RAW Image Demosaicing

Demosaicing approaches can be categorized into two main groups: (1) model-
based methodologies [10, 31, 43, 57], which rely on mathematical models and
spatial-spectral image priors for image reconstruction and (2) learning-based
methodologies [17, 37, 1, 12, 23, 50, 45, 24], which leverage process mappings
learned from extensive datasets of ground-truth images and corresponding
mosaic counterparts. These techniques use different neural networks, _e.g_.,
CNNs and Transformers, to learn complex mappings between mosaic images and
their corresponding full-color images. While CNN architectures have been
widely used in learning-based demosaicing methods, they are limited in fixed-
size receptive fields of convolution kernels and global context awareness
compared with Transformer [51]. As a result, recent advancements in
Transformer architectures, particularly the Swin-Transformer [51], have shown
promise in addressing these challenges and improving the performance of
learning-based demosaicing approaches.

Figure 3: Overview of the event camera demosaicing method. The input RAW image
is first preprocessed using space-to-depth and 1$\times$1 convolution
operations. The encoder then extracts multi-scale features using Swin
Transformer blocks with the shifted window mechanism. The decoder mirrors the
encoder’s structure and incorporates skip connections to recover spatial
details. Finally, the reconstruction module generates the output RGB image.
(a) Encoder block architecture. (b) Shifted window mechanism for cross-window
interactions. (c) Decoder block architecture.

###  2.2 Transformer-based Imaging Processes

Transformer have been employed in many imaging processes, _e.g_., image/video
super-resolution [54, 29], deblurring [56]. Remarkably, Swin-Transformer [21,
9, 22] utilizes the shifted window mechanism to capture long-range
dependencies in images, enabling effective aggregation of information from
distant spatial locations, presenting impressive performance in vision tasks
like super-resolution [6, 19], video deblurring [4], and video frame
interpolation [26, 9]. Besides, Swin-Transformer’s hierarchical architecture
partitions input images into smaller patches, which are then processed through
multiple transformer blocks, facilitating the learning of both local and
global features. For example, [22] utilizes a sequence of residual Swin-
Transformer blocks for deep feature extraction, demonstrating leading-edge
performance across various image super-resolution tasks. Consequently,
inspired by these successful works [9, 22, 4] we also employ Swin-Transformer
as a backbone to leverage the demosaicing.

###  2.3 Image Reconstruction Loss Functions

A series of works prefer Charbonnier Loss [18] as the loss function to train
their neural network for image reconstruction [58, 48]. Nevertheless, areas
characterized by numerous high-frequency details, _e.g_. edges and textures,
merit increased attention compared to low-frequency and smooth regions that
are easily recoverable during the training process, as shown in Fig. 2. In
[23], an adaptive-threshold edge loss is introduced to tackle this challenge,
which adaptively adjusts the edge detection threshold for different image
patches based on their edge density, allowing the model to focus more on
regions with rich edge details during training. However, the loss in [23]
needs to divide the image into specific patches according to their edge
density, demanding a series of complicated thresholds and cross-entropy loss
calculations. Consequently, we offer a facilitated approach called Pixel Focus
Loss to optimize the model to capture subtle differences effectively.

##  3 Methods

This section presents the details of our event camera demosaicing method. As
illustrated in Fig. 3, our method leverages the strengths of the Swin-
Transformer [22] and the U-Net architecture [39] to take the RAW image with
missing pixel values as input and aims to reconstruct a high-quality RGB
image. Our framework consists of five key components: (1) preprocessing 3.1
with a space-to-depth operation [40] and a $1\times 1$ convolution, (2)
encoding 3.2 with Swin-Transformer and shifted window mechanism, (3) decoding
3.3 with mirrored encoding modules, (4) reconstruction 3.4 with mirrored
preprocessing modules, and (5) loss functions 3.5 with exquisite designs.

Figure 4: Loss functions visualization. (a) (b) and (c) refer to Charbonnier
and pixel-focus loss with the power and the exponential function,
respectively. The line charts loss functions within the 0-1 range and their
gradients. It also provides a magnified view of the 0 to 0.1 interval to
observe the characteristics of different loss functions better when dealing
with long-tail distributions.

###  3.1 Preprocessing

The preprocessing module aims to transform the input RAW image into a suitable
representation for the subsequent encoding stages while reducing the
computational complexity. Given the input RAW image
$I_{RAW}\in\mathbb{R}^{H\times W\times 1}$, we apply a space-to-depth
operation [5] with a factor of $s$ to reduce the spatial resolution to
$(H/s)\times(W/s)$ and increase the channel dimension to $s^{2}$. This
operation effectively reduces the model complexity, as the computational cost
is linear with respect to the number of channels and quadratic concerning the
spatial resolution [22]. Subsequently, a $1\times 1$ convolution is employed
to generate the feature $F_{0}\in\mathbb{R}^{(H/s)\times(W/s)\times C}$ from
the RAW image. To incorporate positional information, we add positional
embeddings $E_{pos}\in\mathbb{R}^{(H/s)\times(W/s)\times C}$ to $F_{0}$. The
positional embeddings [47] are computed using a sinusoidal function:

$\begin{cases}E_{pos}(2i)&=\sin\left(p/10000^{2i/C}\right)\\\
E_{pos}(2i+1)&=\cos\left(p/10000^{2i/C}\right),\end{cases}$ (1)

where $p$ represents the position index and $i$ is the dimension index. The
resulting preprocessed feature representation is obtained as
$F_{0}^{{}^{\prime}}=F_{0}+E_{pos}$.

###  3.2 Encoding

The encoding module aims to extract multi-scale features and capture long-
range dependencies. We adopt a U-Net-like architecture [39] with the Swin-
Transformer [22] as the backbone. The encoding module consists of four stages,
each containing a Swin-Transformer block followed by a down-sample layer. The
Swin-Transformer block comprises a Layer Normalization (LN) layer [2], a
Window-based Multi-head Self-Attention (W-MSA) module [24], and a Multi-Layer
Perceptron (MLP). The W-MSA module, illustrated in Fig. 3 (a), performs self-
attention within local windows of varying sizes, allowing the model to capture
multi-scale features and structural details at different granularities. The
Shifted Window mechanism, illustrated in Fig. 3 (c), is employed in
alternating Swin-Transformer blocks to facilitate cross-window interactions
and enhance the model’s representational power. After each Swin-Transformer
block, a down-sample layer is applied to half the resolution of feature maps.
This multi-scale architecture enables the model to process information at
multiple scales while progressively reducing the spatial resolution. Each
downsampling operation reduces the computational complexity by a factor of 4
while quadrupling the receptive field, enabling the model to capture a large
view field.

Figure 5: Visualized results of our method and compared method - RSTCANet
[51]. Comparison methods 1, 2, and 3, respectively represent the processing
directly on the original RAW, processing after converting the original RAW
into Bayer Pattern, and the results after fine-tuning RSTCANet [51].

###  3.3 Decoding

The decoding module aims to gradually upsample the feature maps and recover
the spatial resolution of the output image. It follows a symmetric structure
to the encoding module, consisting of four stages. Each decoding stage
contains a decoder block, as depicted in Fig. 3 (b), followed by an up-sample
layer. The decoder block comprises an LN layer, a W-MSA module, and an MLP, as
illustrated in Fig. 3 (d). The W-MSA module in the decoder block operates
similarly to its counterpart in the encoder block, capturing local
dependencies within windows. An up-sample layer is employed after each decoder
block to increase the spatial resolution. Furthermore, skip connections are
introduced between corresponding encoder and decoder stages to facilitate the
flow of information and aid in recovering fine-grained details. The multi-
scale architecture of the decoding module enables the model to gradually
refine the reconstructed image while incorporating features from different
scales, leading to improved demosaicing performance.

###  3.4 Reconstruction

The reconstruction module aims to generate the final output RGB image from the
upsampled feature maps produced by the decoding module. To achieve this, we
apply a depth-to-space operation [41] to recover the resolution. This step is
crucial for maintaining image quality and minimizing distortions introduced
during preprocessing. The depth-to-space operation rearranges the features and
increases the spatial resolution by a factor of $s$, restoring the original
spatial resolution of the input image. Finally, we employ a $1\times 1$
convolution to map the high-dimensional features to the three RGB channels
with shape $H\times W\times 3$.

###  3.5 Loss function

To train our demosaicing network, we employ a two-stage training approach. In
the first stage, we use the Charbonnier loss [18] for pre-training. The
Charbonnier loss is a commonly used loss function in image processing as
shown:

$\mathcal{L}_{Charbonnier}=\frac{1}{N}\sum\limits_{i=1}^{N}\sqrt{\left(I_{RGB}^{(i)}-I_{GT}^{(i)}\right)^{2}+\epsilon^{2}},$
(2)

where $I_{RGB}^{(i)}$ and $I_{GT}^{(i)}$ represent the $i$-th reconstructed
RGB image pixel and it’s corresponding in the ground-truth image,
respectively, $N$ is the total number pixels of images, and $\epsilon$ is a
small constant (_e.g_., $1e-3$) added to improve the robustness of the loss
function to outliers [18]. Evident from Fig. 2 (I), the Charbonnier loss
effectively reduces the difference with larger values during the pre-training
stage.

Figure 6: Visualization for output of fin-tuned models with different loss
functions. The number values represent PSNR/SSIM.

Upon closer examination of the Difference Distribution in Fig. 2 (II), we
observe that it comprises two main components: high frequency, _e.g_., edge
areas, at low difference values and low frequency, _e.g_., smooth areas, at
high difference values. This observation suggests that while the model
efficiently learns to restore smooth blocks such as backgrounds, it has yet to
handle edge areas fully. This shortfall is primarily due to edge differences,
highlighting the importance of increasing the gradient magnitude for edge.

In response to this need, we explored two forms of Pixel Focus Loss
($\mathcal{L}_{pf}$) to capture edge-related differences better. One version
of the Pixel Focus Loss is described as a piecewise function:

$\mathcal{L}^{p}_{pf}=\begin{cases}\left(d/a\right)^{g}\cdot b&,0<d<a\\\
(d-1)(1-b)/(1-a)+1&,a\leq d\leq 1,\\\ \end{cases}$ (3)

where $d$ represents the difference value, $a$ is a threshold parameter, and
$b$ and $g$ are scaling factors that control the gradient magnitude. We also
introduced another version of Pixel Focus Loss defined as:

$\mathcal{L}_{pf}^{e}=e^{\lambda d}-\lambda d-1,$ (4)

where $\lambda$ serves as a hyperparameter, both versions address the issue,
providing a comprehensive approach to fine-tuning the model’s ability to
capture edge areas. We utilize these two Pixel Focus Loss to fine-tune the
pre-trained model obtained from the first stage, enhancing the demosaicing
performance by emphasizing gradients for edge-related differences. In the
experiments section, we delve into the impact of different forms of Pixel
Focus Loss and the effects of varying hyperparameters on the experimental
results, providing a detailed analysis of our findings. Combining the two-
stage training approach, utilizing the Charbonnier loss for pre-training and
the proposed Pixel Focus loss for fine-tuning, enables our network to learn
high-quality RGB image reconstruction.

Table 1: Comparison of our method with the RSTCANet method [51] on the MIPI-
Challenge Demosaic dataset. RSTCANet 1, 2, and 3 denote different processing
strategies.

Case | Methods | Params ($M$) | PSNR | SSIIM
---|---|---|---|---
$\\#1$ | RSTCANet1 | 7.19 | 13.2477 | 0.3590
$\\#2$ | RSTCANet2 | 7.19 | 15.7721 | 0.4234
$\\#3$ | RSTCANet3 | 7.19 | 36.2691 | 0.9659
$\\#4$ | Our-Small | 6.22 | 37.3272 | 0.9738
$\\#5$ | Our-Large | 9.40 | 37.4117 | 0.9756

Figure 7: Visualization of results from model different size. The number
values represent PSNR and SSIM, respectively.

##  4 Experiments

Dataset: Our experiments are based on the Demosaic for HybridEVs Camera
dataset at MIPI-Challenge 2024 [34], comprising 900 RAW-Color image pairs with
around $2000\times 1500$ resolution. In this dataset, 800 RAW-Color pairs are
designated for training. 50 color images are allocated for validation and
another 50 for testing. Note that the validation and test sets were not
released during the competition phase. Consequently, we adapted our approach
by utilizing 760 images from the training set and designated 40 images for
testing. Within the validation set, 26 pairs of color images are available for
local quantitative testing.

Implementation Details: Our experiments were conducted using PyTorch [36] on a
server with an Intel(R) Xeon(R) Platinum 8378A CPU and one NVIDIA A800 GPU.
The training batch size is one. Each training iteration employs random crop
augmentation with patches sized at $640\times 640$. In the first training
phase, we utilized the Charbonnier Loss [18], training for 500 epochs with a
learning rate starting from $1e-4$ and decreasing to $0$ following a cosine
function. In the second training phase, we applied the pixel-focus Loss,
training for 200 epochs with a learning rate initiating from $1e-5$ and
diminishing to $0$. To achieve faster training speeds, we employed mixed
precision techniques [33] facilitated by PyTorch.

Evaluation: PSNR and SSIM [11] were utilized as quantitative evaluation
metrics. Qualitative results are demonstrated through the visualization of
difference maps and difference distributions. Given the high performance of
demosaicing methods, directly observing differences between images can be
challenging; difference map visualization facilitates addressing this issue.

###  4.1 Comparison Experiments:

We benchmark our method against the publicly available RSTCANet [51], which is
a pioneer in applying the Transformer architecture to demosaicing tasks. In
contrast to our method, RSTCANet [51] is tailored exclusively for demosaicing
under the Bayer pattern and, consequently, is not inherently equipped to
address scenarios with missing pixel values. RSTCANet also falls short in
addressing the missing values associated with defects in RAW images.
Consequently, employing the RSTCANet method often results in images plagued
with noise, as depicted in Fig. 5.

Table 2: Ablation for the model size. Params denote the model’s parameters,
measured in millions. Depth indicates the count of Transformer layers within
each block

Case | Model Size | Params ($M$) | Depth | PSNR | SSIIM
---|---|---|---|---|---
$\\#1$ | Tiny | 4.62 | 2 | 36.3044 | 0.9696
$\\#2$ | Small | 6.22 | 4 | 37.3272 | 0.9738
$\\#3$ | Medium | 7.81 | 6 | 37.3798 | 0.9751
$\\#4$ | Large | 9.40 | 8 | 37.4117 | 0.9756

Table 3: Ablation for the Loss Function.

Case |  Loss Function |  PSNR |  SSIM
---|---|---|---
$\\#1$ |  w/o |  37.4117 |  0.9756
$\\#2$ |  $\mathcal{L}_{Charbonnier}$ |  37.4402 |  0.9758
$\\#3$ |  $\mathcal{L}_{pf}^{p}$ |  37.8164 |  0.9767
$\\#4$ |  $\mathcal{L}_{pf}^{e}$ with $\lambda=1$ |  37.8360 |  0.9765
$\\#5$ |  $\mathcal{L}_{pf}^{e}$ with $\lambda=1.1$ |  37.9656 |  0.9770

###  4.2 Ablation and Analytical Experiments:

Ablation for the model depth and size: The ablation study indicates a
progressive improvement in image reconstruction quality with increasing model
depth, as shown in Tab. 2 and Fig. 7. While the initial increase from a Tiny
to a Small model shows a substantial rise in quality metrics, the growth
tapers as the depth extends to Medium and Large. This pattern suggests a
diminishing return on enhancing PSNR and SSIM values with deeper networks,
implying an optimal balance between depth and performance.

Ablation for the loss function: The Tab. 3 presented delineates an ablation
study examining refining loss functions, particularly in mitigating the
challenges posed by long-tail distributions encountered during the latter part
of the initial training phase. A secondary training phase spanning 200 epochs
was implemented to counteract this issue, utilizing a suite of four variant
loss functions. The benchmark was set using the Charbonnier loss function.
Contrastive analyses were conducted with a power loss function and two
exponential loss functions with $\lambda$ weights set at $1$ and $1.1$, as
delineated in the second and fourth rows of Tab. 3. Notably, excessively high
$\lambda$ values, such as $2$, have been observed to induce instability within
the network. Evaluating the impact on PSNR and SSIM scores reveals that
alterations to the loss function can significantly affect model performance.
The exponential loss functions, particularly with a $\lambda$ value of $1.1$,
surpassed the performance of the baseline Charbonnier function. These insights
imply that meticulous adjustment of the loss function parameters can
effectively overcome optimization hurdles in the advanced stages of training,
thereby improving the fidelity of the reconstructed images, as shown in Fig.
6.

##  5 Conclusion

This paper employs Swin-Transformer and U-Net architecture tailored for the
demosaicing task within the CVPR 2024 MIPI-Challenge. A pixel focus loss was
designed for a two-stage training to facilitate efficient training. Our model
demonstrates advantages over transformer-based methods for event camera
demosaicing. The model and the proposed loss function hold the potential to
inspire future research in the field of demosaicing and beyond.

Acknowledgment: This work was partially supported by Guangzhou-HKUST(GZ) Joint
Funding Program (Grant No.2023A03J0008), Education Bureau of Guangzhou
Municipality, Guangdong Science and Technology Department, and the Foshan
HKUST Projects (FSUST21-FYTRI01A).

## References

  * A Sharif et al. [2021] SM A Sharif, Rizwan Ali Naqvi, and Mithun Biswas.  Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 233–242, 2021.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Brooks et al. [2019] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11036–11045, 2019.
  * Cao et al. [2022] Mingdeng Cao, Yanbo Fan, Yong Zhang, Jue Wang, and Yujiu Yang.  Vdtr: Video deblurring with transformer.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 33(1):160–171, 2022.
  * Cheng et al. [2019] Xinjing Cheng, Peng Wang, and Ruigang Yang.  Learning depth with convolutional spatial propagation network.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(10):2361–2379, 2019.
  * Choi et al. [2023] Haram Choi, Jeongmin Lee, and Jihoon Yang.  N-gram in swin transformers for efficient lightweight image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2071–2081, 2023.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Gallego et al. [2020] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(1):154–180, 2020.
  * Geng et al. [2022] Zhicheng Geng, Luming Liang, Tianyu Ding, and Ilya Zharkov.  Rstt: Real-time spatial temporal transformer for space-time video super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 17441–17451, 2022.
  * Hirakawa and Parks [2005] Keigo Hirakawa and Thomas W Parks.  Adaptive homogeneity-directed demosaicing algorithm.  _Ieee transactions on image processing_ , 14(3):360–369, 2005.
  * Hore and Ziou [2010] Alain Hore and Djemel Ziou.  Image quality metrics: Psnr vs. ssim.  In _2010 20th international conference on pattern recognition_ , pages 2366–2369. IEEE, 2010.
  * Ignatov et al. [2020] Andrey Ignatov, Luc Van Gool, and Radu Timofte.  Replacing mobile camera isp with a single deep learning model.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 536–537, 2020.
  * Ignatov et al. [2022] Andrey Ignatov, Grigory Malivenko, Radu Timofte, Yu Tseng, Yu-Syuan Xu, Po-Hsiang Yu, Cheng-Ming Chiang, Hsien-Kai Kuo, Min-Hung Chen, Chia-Ming Cheng, et al.  Pynet-v2 mobile: Efficient on-device photo processing with neural networks.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , pages 677–684. IEEE, 2022.
  * Jiang et al. [2020] Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu.  Learning event-based motion deblurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3320–3329, 2020.
  * Jing et al. [2021] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao.  Turning frequency to resolution: Video super-resolution via event cameras.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7772–7781, 2021.
  * Kim and Kim [2023] Jungwoo Kim and Min H Kim.  Joint demosaicing and deghosting of time-varying exposures for single-shot hdr imaging.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12292–12301, 2023.
  * Kokkinos and Lefkimmiatis [2018] Filippos Kokkinos and Stamatios Lefkimmiatis.  Deep image demosaicking using a cascade of convolutional residual denoising networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 303–319, 2018.
  * Lai et al. [2018] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang.  Fast and accurate image super-resolution with deep laplacian pyramid networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(11):2599–2613, 2018.
  * Li et al. [2022] Bingchen Li, Xin Li, Yiting Lu, Sen Liu, Ruoyu Feng, and Zhibo Chen.  Hst: Hierarchical swin transformer for compressed image super-resolution.  In _European conference on computer vision_ , pages 651–668. Springer, 2022.
  * Li et al. [2008] Xin Li, Bahadir Gunturk, and Lei Zhang.  Image demosaicing: A systematic survey.  In _Visual Communications and Image Processing 2008_ , pages 489–503. SPIE, 2008.
  * Li et al. [2023] Yong Li, Naipeng Miao, Liangdi Ma, Feng Shuang, and Xingwen Huang.  Transformer for object detection: Review and benchmark.  _Engineering Applications of Artificial Intelligence_ , 126:107021, 2023.
  * Liang et al. [2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1833–1844, 2021.
  * Liu et al. [2020] Lin Liu, Xu Jia, Jianzhuang Liu, and Qi Tian.  Joint demosaicing and denoising with self guidance.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2240–2249, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Liu et al. [2022] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3202–3211, 2022.
  * Lu et al. [2022a] Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia.  Video frame interpolation with transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3532–3542, 2022a.
  * Lu et al. [2022b] Yunfan Lu, Yiqi Lin, Hao Wu, Yunhao Luo, Xu Zheng, and Lin Wang.  All one needs to know about priors for deep image restoration and enhancement: A survey.  _arXiv preprint arXiv:2206.02070_ , 2022b.
  * Lu et al. [2023a] Yunfan Lu, Guoqiang Liang, and Lin Wang.  Learning inr for event-guided rolling shutter frame correction, deblur, and interpolation.  _arXiv preprint arXiv:2305.15078_ , 2023a.
  * Lu et al. [2023b] Yunfan Lu, Zipeng Wang, Minjie Liu, Hongjian Wang, and Lin Wang.  Learning spatial-temporal implicit neural representations for event-guided video super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1557–1567, 2023b.
  * Lyashenko et al. [2021] Vyacheslav Lyashenko, Svitlana Sotnik, and Volodymyr Manakov.  Modern cad/cam/cae systems: brief overview.  2021\.
  * Malvar et al. [2004] Henrique S Malvar, Li-wei He, and Ross Cutler.  High-quality linear interpolation for demosaicing of bayer-patterned color images.  In _2004 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , pages iii–485. IEEE, 2004.
  * Messikommer et al. [2022] Nico Messikommer, Stamatios Georgoulis, Daniel Gehrig, Stepan Tulyakov, Julius Erbach, Alfredo Bochicchio, Yuanyou Li, and Davide Scaramuzza.  Multi-bracket high dynamic range imaging with event cameras.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 547–557, 2022.
  * Micikevicius et al. [2017] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  _arXiv preprint arXiv:1710.03740_ , 2017.
  * MIPI Challenge 2024 [2024] MIPI Challenge 2024.  Mobile intelligent photography and imaging workshop 2024.  https://mipi-challenge.org/MIPI2024/, 2024.
  * Paikin et al. [2021] Genady Paikin, Yotam Ater, Roy Shaul, and Evgeny Soloveichik.  Efi-net: Video frame interpolation from fusion of events and frames.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1291–1301, 2021.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Qian et al. [2022] Guocheng Qian, Yuanhao Wang, Jinjin Gu, Chao Dong, Wolfgang Heidrich, Bernard Ghanem, and Jimmy S Ren.  Rethinking learning-based demosaicing, denoising, and super-resolution pipeline.  In _2022 IEEE International Conference on Computational Photography (ICCP)_ , pages 1–12. IEEE, 2022.
  * Rebecq et al. [2019] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza.  High speed and high dynamic range video with an event camera.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(6):1964–1980, 2019.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_ , pages 234–241. Springer, 2015.
  * Sajjadi et al. [2018] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown.  Frame-recurrent video super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6626–6634, 2018.
  * Shi et al. [2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1874–1883, 2016.
  * Shi et al. [2022] Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang.  Video frame interpolation transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17482–17491, 2022.
  * Su [2006] Chung-Yen Su.  Highly effective iterative demosaicing using weighted-edge and color-difference interpolations.  _IEEE Transactions on Consumer Electronics_ , 52(2):639–645, 2006.
  * Sun et al. [2022] Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu Ye, Kaiwei Wang, and Luc Van Gool.  Event-based fusion for motion deblurring with cross-modal attention.  In _European conference on computer vision_ , pages 412–428. Springer, 2022.
  * Syu et al. [2018] Nai-Sheng Syu, Yu-Sheng Chen, and Yung-Yu Chuang.  Learning deep convolutional networks for demosaicing.  _arXiv preprint arXiv:1802.03769_ , 2018.
  * Tulyakov et al. [2021] Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza.  Time lens: Event-based video frame interpolation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16155–16164, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2022] Yusheng Wang, Yunfan Lu, Ye Gao, Lin Wang, Zhihang Zhong, Yinqiang Zheng, and Atsushi Yamashita.  Efficient video deblurring guided by motion magnitude.  In _European Conference on Computer Vision_ , pages 413–429. Springer, 2022.
  * Wu et al. [2023] Wei Wu, Shuming Hu, Pengxiang Xiao, Sibin Deng, Yilin Li, Ying Chen, and Kai Li.  Video quality assessment based on swin transformer with spatio-temporal feature fusion and data augmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1846–1854, 2023.
  * Xing and Egiazarian [2021] Wenzhu Xing and Karen Egiazarian.  End-to-end learning for joint image demosaicing, denoising and super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3507–3516, 2021.
  * Xing and Egiazarian [2022] Wenzhu Xing and Karen Egiazarian.  Residual swin transformer channel attention network for image demosaicing.  In _2022 10th European Workshop on Visual Information Processing (EUVIP)_ , pages 1–6. IEEE, 2022.
  * Xu et al. [2021] Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu.  Motion deblurring with real events.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2583–2592, 2021.
  * Yaqi et al. [2024] Wu Yaqi, Fan Zhihao, Chu Xiaofeng, Ren Jimmy S., Li Xiaoming, Yue Zongsheng, Li Chongyi, Zhou Shangcheng, Feng Ruicheng, Dai Yuekun, Yang Peiqing, Loy Chen Change, et al.  Mipi 2024 challenge on demosaic for hybridevs camera: Methods and results.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2024.
  * Yue et al. [2016] Linwei Yue, Huanfeng Shen, Jie Li, Qiangqiang Yuan, Hongyan Zhang, and Liangpei Zhang.  Image super-resolution: The techniques, applications, and future.  _Signal processing_ , 128:389–408, 2016.
  * Zeng et al. [2023] Haijin Zeng, Kai Feng, Shaoguang Huang, Jiezhang Cao, Yongyong Chen, Hongyan Zhang, Hiep Luong, and Wilfried Philips.  Msfa-frequency-aware transformer for hyperspectral images demosaicing.  _arXiv preprint arXiv:2303.13404_ , 2023.
  * Zhang et al. [2022] Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Björn Stenger, Ming-Hsuan Yang, and Hongdong Li.  Deep image deblurring: A survey.  _International Journal of Computer Vision_ , 130(9):2103–2130, 2022.
  * Zhang and Wu [2005] Lei Zhang and Xiaolin Wu.  Color demosaicking via directional linear minimum mean square-error estimation.  _IEEE Transactions on Image Processing_ , 14(12):2167–2178, 2005.
  * Zhao et al. [2016] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz.  Loss functions for image restoration with neural networks.  _IEEE Transactions on computational imaging_ , 3(1):47–57, 2016.
  * Zheng et al. [2023] Xu Zheng, Yexin Liu, Yunfan Lu, Tongyan Hua, Tianbo Pan, Weiming Zhang, Dacheng Tao, and Lin Wang.  Deep learning for event-based vision: A comprehensive survey and benchmarks.  _arXiv preprint arXiv:2302.08890_ , 2023.