# Polynomial Invariant of Molecular Circuit Topology

Alireza Mashaghi111Leiden Academic Centre for Drug Research, Faculty of
Mathematics and Natural Sciences, Leiden University, 2311 Leiden, The
Netherlands and Roland van der Veen222Bernoulli Institute, Department of
Mathematics, Faculty of Science and Engineering, University of Groningen, 9712
Groningen, The Netherlands<EMAIL_ADDRESS>
###### Abstract

The topological framework of circuit topology has recently been introduced to
complement knot theory and to help in understanding the physics of molecular
folding. Naturally evolved linear molecular chains, such as proteins and
nucleic acids, often fold into 3D conformations with critical chain
entanglements and local or global structural symmetries stabilised by
formation contacts between different parts of the chain. Circuit topology
captures the arrangements of intra-chain contacts within a given folded linear
chain and allows for the classification and comparison of chains. Contacts
keep chain segments in physical proximity and can be either mechanically hard
attachments or soft entanglements that constrain a physical chain. Contrary to
knot theory, which offers many established knot invariants, circuit invariants
are just being developed. Here, we present polynomial invariants that are both
efficient and sufficiently powerful to deal with any combination of soft and
hard contacts. A computer implementation and table of chains with up to three
contacts is also provided.

circuit topology; knot theory; invariant; folding; polymer

##  1 Introduction

Linear polymers are an important subset of macromolecules with critical roles
in living organisms and are used in engineering applications [Scalvini et
al.(2020)Scalvini, Sheikhhassani, Woodard, Aupic, Dame, Jerala, and Mashaghi].
A linear polymer is a molecular chain made of units, so called monomers. By
changing the chemical properties of these monomers, one can make polymers with
different physicochemical properties. This fundamental concept in chemistry
has led to emergence of many synthetic polymers with various applications in
medicine and industry. Living organisms however generate a wide range of
linear polymers using a limited set of monomer chemistries. This is because
biomolecular chains, such as proteins and nucleic acids, typically fold into
3D conformations which give the molecules new properties. Proteins can form
various folded structures at various scales and with different symmetries by
forming intra-chain contacts; proteins may also form knots and slipknots
[Jamroz et al.(2015)Jamroz, Niemyska, Rawdon, Stasiak, Millett, Sulkowski, and
Sulkowska, Flapan et al.(2019)Flapan, He, and Wong, Scalvini et
al.(2020)Scalvini, Sheikhhassani, Woodard, Aupic, Dame, Jerala, and Mashaghi].
Inspired by biology, chemists have only recently started synthesizing folded
molecular chains [Kyoda et al.(2019)Kyoda, Yamamoto, and Tezuka]. Molecular
engineering typically uses a bottom-up approach which involves synthesizing
basic fold units and then connecting them to generate complexity. Generating
complex folded linear chains requires advancements in our synthetic
methodology as well as suitable conceptual mathematical framework for
characterization and comparison of topological complexity. The latter inspired
the development of molecular circuit topology, a framework that categorizes
the arrangement of contacts in a folded linear chain [Mashaghi(2021), Mashaghi
et al.(2014)Mashaghi, van Wijk, and Tans].

Circuit topology is inspired by physics of polymers and recognises that
contacts restrain dynamics of a chain and keep segments in close proximity
[Heidari et al.(2020)Heidari, Schiessel, and Mashaghi]. Topological
arrangement of the contacts is closely related to kinetics of their formation
[Heidari et al.(2020)Heidari, Schiessel, and Mashaghi, Mugler et
al.(2014)Mugler, Tans, and Mashaghi]. Furthermore, chain entanglement may also
restrain a physical chain and effectively stabilise certain folds. Let us
consider a folded linear chain. To describe a folded chain in space there are
roughly two aspects to address: First, intra-chain contacts or bonds that turn
the chain into a special type of graph. Second, the chain sits in three
dimensional space in a certain way, allowing it to form knots and tangles.
Both the bonding and the tangling can constrain the chain. The first type, we
call hard contacts (or H-contacts), while the second type we will describe in
terms of soft contacts (or S-contacts). Circuit topology uses a uniform
language to categorise the arrangement of hard and soft contacts.

In this article, we propose a precise mathematical model for folded chains
called $H$-tangle diagrams. Tangles are a commonly used bottom-up approach to
knot theory [Conway(1970)] and in this work we extend this approach to include
hard contacts. An example of a $H$-tangle appears in Figure 1. $H$-tangles
take into account both hard contacts (shown in black) and soft contacts where
the chain constrains itself by a clasp. We describe an algorithm to assign to
each $H$-tangle diagram $D$ a certain polynomial called
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(D)$. For example
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ applied to the
example in Figure 1 is

$\displaystyle
h_{1}t^{6}-2h_{1}t^{5}-h_{1}t^{4}+6h_{1}t^{3}-h_{1}h_{2}t^{3}+h_{2}t^{3}-6h_{1}t^{2}-2h_{2}t^{2}-2h_{1}t^{-2}+10h_{1}h_{2}t^{-2}+$
$\displaystyle
2h_{2}t^{-2}+h_{1}t^{-3}+13h_{2}t^{-3}-4h_{1}h_{2}t^{-4}-11h_{2}t^{-4}+h_{1}h_{2}t^{-5}-3h_{2}t^{-5}+5h_{2}t^{-6}-h_{2}t^{-7}+$
$\displaystyle-3h_{1}t+5h_{1}h_{2}t+h_{2}t-2h_{1}t^{-1}-5h_{1}h_{2}t^{-1}-11h_{2}t^{-1}+7h_{1}-5h_{1}h_{2}+5h_{2}+$
$\displaystyle-t^{6}+2t^{5}-2t^{4}+7t^{2}-8t^{-2}+3t^{-4}-t^{-5}-10t+9t^{-1}+2.$

The polynomial $\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$
is a topological invariant of the folded chain in the sense that if $H$-tangle
diagrams $D$ and $D^{\prime}$ represent the same folded chain, then
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(D)=\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(D^{\prime})$.
In [Ceniceros et al.(2021)Ceniceros, Elhamdadi, and Mashaghi, Golovnev and
Mashaghi(2020)] other invariants of folded chains were proposed. The
difference is that
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ is easier to
compute in practice while still being sufficiently powerful to distinguish
many different configurations. A computer implementation of our algorithm
appears in the Appendix A.

Figure 1: A knotted folded chain with three soft contacts and two hard
contacts.

Even though folded chains can take many forms, we can always bring them in a
form such as that in Figure 1 called a generalized circuit topology diagram
[Golovnev and Mashaghi(2020)]. The chain runs horizontally from left to right,
except for making a detour to make a soft or hard contact with another piece
of the chain in a controlled way. In the case of knots a similar form appears
in [Mostovoy and Polyak(2019)]. In the Appendix A we list all circuit diagrams
with at most three contacts together with their corresponding value of
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$. There do exist
several circuit topology diagrams for the same folded chain but in combination
with our invariant this provides a first step towards classification of folded
chains of low complexity.

The plan of the paper is to first describe some basic notions in circuit
topology using hard contacts only. Historically, circuit topology was first
introduced to classify such circuits. Next we review the knot theory
techniques that are fundamental to handling soft contacts, this time ignoring
hard contacts. We set up the Alexander polynomial invariant for knots and
tangles in the form of $\Gamma$ calculus. To include hard contacts in our
tangle language we develop a theory of $H$-tangles that is similar to that of
singular knots. The invariant
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ is then
introduced as a suitable extension of $\Gamma$ calculus to $H$-tangles.
Finally we present a special class of $H$-tangle diagrams especially suited
for discussion of circuit topology and tabulate
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ on those.

##  2 Circuit Topology

Perhaps the most fundamental aspect of a folded chain is the way it bonds to
itself. In this section, we focus on this aspect only, ignoring for now how it
sits in space. In other words we study the folded chain as an abstract
directed graph, see for example the chain $\mathcal{F}$ in Figure 2.

Figure 2: The chain $\mathcal{F}$ with five hard contacts.

To describe such chains in general we enumerate the spots where the chain
bonds to itself in order of appearance as one walks along the chain. Assuming
each spot bonds with precisely one other spot there must be an even number of
such spots that we can number $1,2,\dots,2n$. In the example $\mathcal{F}$, we
chose $n=5$ as there are five bonds. Each bond is described by the pair of
integers $\\{i,j\\}$ indicating the two locations involved. We often refer to
such bonds as hard contacts when we want to emphasize the distinction with the
knotting behaviour described in later sections.

Any folded chain is then described precisely by a partition of the integers
$1,\dots 2n$ into $n$ disjoint pairs. For example we describe $\mathcal{F}$ by

$\mathcal{F}=\\{\\{1,4\\},\\{2,7\\},\\{3,5\\},\\{6,9\\},\\{8,10\\}\\}$

.

In listing such examples our convention is to order the list of pairs
lexicographically and refer to any bond by the spot where it first appears as
one walks along the chain. In the mathematical literature such structures
appear under the name _chord diagrams_ [Chmutov et al.(2012)Chmutov, Duzhin,
and Mostovoy].

At this point, we should emphasize that, contrary to the picture shown, the
description $\mathcal{F}$ is completely abstract. In the picture the bond
$\\{1,4\\}$ appears to pass in front of bond $\\{2,7\\}$ but this is not part
of our mathematical description of the graph. So far the space around the
graph has not been taken into account so there is no notion of behind or in
front. Modelling and describing how abstract folded chains interact with
ambient space is the subject of the next sections as it is considerably more
difficult.

The starting point of circuit topology is to summarise the properties of
folded chains in terms of these pairwise relationships of contacts. A pair of
contacts can interact in precisely three ways that are shown in Figure 3
called $\mathcal{S}=\\{\\{1,2\\},\\{3,4\\}\\}$ (series),
$\mathcal{P}=\\{\\{1,4\\},\\{2,3\\}\\}$ (parallel) and
$\mathcal{X}=\\{\\{1,3\\},\\{2,4\\}\\}$ (cross).

Figure 3: The three basic configurations in circuit topology: Series, Parallel
and Cross.

For more complicated chains, such as $\mathcal{F}$, we can record the
relationship between each pair of contacts, ignoring the rest of the contacts.
For $\mathcal{F}$, we can describe the results in a table like the one given
below. The order of first appearance along the chain together with the
pairwise information is sufficient to reconstruct the whole chord diagram.

$\mathcal{F}$ |  {1,4} |  {2,7} |  {3,5} |  {6,9} |  {8,10}
---|---|---|---|---|---
{1,4} |  0 |  X |  X |  S |  S
{2,7} |  X |  0 |  P |  X |  S
{3,5} |  X |  P |  0 |  S |  S
{6,9} |  S |  X |  S |  0 |  X
{8,10} |  S |  S |  S |  X |  0
|  |  |  |  |

The matrix of partial relationships is more concise and was shown to be a good
way to describe the behaviour of folded chains in practice [Mashaghi(2021),
Mashaghi et al.(2014)Mashaghi, van Wijk, and Tans].

Later in the paper, Section 7, we will use the above notation to introduce
generalized circuit topology diagrams where each pair comes with a subscript
$0,1$ or $-1$. If the subscript is $0$ then the contact is hard as above,
while the $\pm 1$ indicate a soft contact in the form of a left or right
handed clasp. For example the diagram that appears in Figure 1 is described as
$\\{\\{1,4\\}_{1},\\{2,7\\}_{-1},\\{3,5\\}_{0},\\{6,9\\}_{0},\\{8,10\\}_{-1}\\}$

The circuit topology description of folded chains can thus be generalised to
also include knotting and tangling effects, see [Golovnev and Mashaghi(2020)].
We will see that any knotted folded chain can be described by a similar set of
pairs of integers partitioning $1,2\dots 2n$ together with an index to specify
the nature of the contact: hard or soft.

##  3 Knots and Tangles

Even when a chain does not bond to itself it can still be entangled, and
provided we keep the endpoints fixed, such entanglement can constrain the
chain as much as hard contacts would. There is a large literature on knot
theory that deals with this special case and in the next two sections we
briefly review some aspects that are necessary for our purposes. The reader is
warned that we use slightly non-standard versions of the usual definitions. We
start with an example and a definition of tangle diagrams.

A tangle diagram is a finite directed graph whose edges are a disjoint union
of oriented open paths called strands. Each strand carries a distinct label.
Strands are allowed to meet only in special vertices called crossings that
look like like one of the models shown in Figure 4.

Figure 4: From left to right: the positive crossing $X_{t,b}$, the negative
crossing $X^{-1}_{t,b}$ and the standard crossingless strand $1_{i}$.

For a tangle diagram $D$, we often list the labels of its strands as a
subscript. For example the diagram $D_{i,j}$ shown in Figure 5 has two strands
named $i$ and $j$. In drawing pictures the labels are sometimes suppressed.

The simplest tangle diagram is perhaps the crossing-less strand with label
$i$, denoted by $1_{i}$. Next the positive crossing $X_{t,b}$ with $t$
labeling the strand that is on top and $b$ marking the strand that is at the
bottom. Likewise the negative crossing with top strand $t$ and bottom strand
$b$ is denoted $X^{-1}_{t,b}$, see Figure 4.

Any more complicated tangle diagram can be assembled from the above examples
using the fundamental operations on diagrams shown in Figure 6: disjoint union
and strand merging. This provides a convenient notation for tangle diagrams
allowing us to compute and discuss them recursively.

Figure 5: Left: The tangle diagram $D_{i,j}$ with two strands called $i$ and
$j$. $D$ can be assembled from the three crossings shown on the right by
merging strands $0,2,4,6$ and $1,3$ as explained below. Figure 6: Left:
disjoint union of diagrams $D,E$. Right: merging strands $e,s$ of diagram $D$.

First we denote by $DE$ the disjoint union of diagrams $D$ and $E$. Second,
denote by $D//m^{e,s}_{r}$ the diagram obtained from $D$ by merging the end of
strand $e$ with the start of strand $s$, labelling the resulting strand $r$.
By merging two strands we mean connecting them without introducing new
crossings. $D$ is also assumed to have strands labelled $e,s$ and no strand
labelled $r$. Mergings may not always be defined but when the ends are close
together merging them does make sense.

For example the two strand tangle diagram $D_{i,j}$ shown in Figure 5 is
denoted by

$D_{i,j}=X^{-1}_{6,0}X_{1,4}X_{2,3}//m^{0,2}_{0}//m^{0,4}_{0}//m^{0,6}_{i}//m^{1,3}_{j}.$
(1)

The special case where a tangle diagram has a single strand is called a knot
diagram, see, for example, Figure 7. Often the ends of a knot diagram are
required to be connected to form a loop but we prefer to leave them open.

Figure 7: The trefoil knot $\mathcal{T}_{i}$ viewed as a tangle with a single
strand labelled $i$. The numbers next to the crossings suggest how
$\mathcal{T}_{i}$ can be merged from three crossings.

A pair of tangle diagrams is said to be equivalent if they can be made equal
by repeatedly making local replacements as shown in Figure 8. These
replacements are commonly known as Reidemeister moves. The theorem of
Reidemeister [Burde and Zieschang(2003)] asserts that tangle diagrams
represent the same knot if and only if they are equivalent in the above sense.
There are many more variants of the moves shown in the figure but these can
all be realized as combinations of the ones shown [Polyak(2010)].

Finally we introduce a short-hand notation to avoid long formulas

$D//m^{a,b,c,d,\dots}_{k}=D//m^{a,b}_{k}//m^{k,c}_{k}//m^{k,d}_{k}//\dots.$
(2)

For example the diagram for the trefoil knot $\mathcal{T}_{i}$ shown in Figure
7 can be described as

$\mathcal{T}_{i}=X_{1,4}X_{5,2}X_{3,6}//m^{1,2,3,4,5,6}_{i}.$ (3)

Figure 8: The Reidemeister moves. Two tangles that locally differ as in one of
the pictures are considered to be equivalent.

##  4 Gamma-Calculus and the Alexander Polynomial

The fundamental difficulty of studying knots in terms of their diagrams is
that there are many diagrams that represent the same knot. This is made
precise in terms of the Reidemeister moves introduced in the previous section,
see Figure 8. To obtain information about the knot using a diagram we need to
compute a quantity that does not change when a Reidemeister move is applied.
Such quantities are known as knot invariants.

In this section, we introduce one of the most useful knot invariants, called
the Alexander polynomial. There are many ways to describe and compute the
Alexander polynomial. Here we choose a recursive definition in terms of tangle
diagrams that fits well with our notation for tangle diagrams. It is known as
$\Gamma$ calculus developed by Bar–Natan, see [Bar-Natan and Selmani(2013),
Vo(2018)] and [Bar-Natan(2015)] (Section 9) for more context.

[$\Gamma$-calculus] For a tangle diagram $D$ whose strands are labelled by set
$L$ define $\Gamma(D)=(\omega,A)$ where $A=\sum_{i,j\in L}A_{ij}r_{i}c_{j}$
and for each $i,j\in L$ the coefficients $A_{ij}$ and $\omega$ are rational
functions in $t$.

  1. 1.

$\Gamma(1_{i})=(1,r_{i}c_{i})$.

  2. 2.

$\Gamma(X^{\pm}_{ij})=(1,r_{i}c_{i}+(1-t^{\pm 1})r_{i}c_{j}+t^{\pm
1}r_{j}c_{j})$.

  3. 3.

For $(\omega,A)$ as above define

$(\omega,A)//\mu^{e,s}_{k}=\Big{(}(1-A_{es})\omega,A+\frac{(\partial_{r_{e}}A)(\partial_{c_{s}}A)}{1-A_{es}}\Big{)}|_{\begin{subarray}{c}r_{e},c_{s}\mapsto
0\\\ r_{s},c_{e}\mapsto r_{k},c_{k}\end{subarray}}$ (4)

Then $\Gamma(D//m^{e,s}_{k})=\Gamma(D)//\mu^{e,s}_{k}$.

  4. 4.

If $\Gamma(D^{\prime})=(\omega^{\prime},A^{\prime})$ then
$\Gamma(D,D^{\prime})=(\omega\omega^{\prime},A+A^{\prime})$.

Here, $\partial_{r_{e}}A$ means the partial derivative of $A$ with respect to
the variable $r_{e}$ and the notation $r_{e},c_{s}\mapsto 0$ means we should
set both $r_{e}$ and $c_{s}$ to $0$ and $r_{s}$ to $r_{k}$ and $c_{e}$ to
$c_{k}$. To see how this works in practice let us compute $\Gamma(D_{i,j})$
where $D_{ij}$ is the tangle shown in Figure 5. It is efficient to start with
the two-crossing tangle $K_{v,j}=X_{1,4}X_{2,3}//m^{1,3}_{j}//m^{2,4}_{v}$ and
then build $D_{ij}$ by attaching the third crossing to $K$ as follows (see
also Figure 9)

$D_{i,j}=X^{-1}_{6,0}K_{v,j}//m^{0,v}_{0}//m^{0,6}_{i}.$ (5)

Figure 9: Constructing the tangle $D$ from the tangle $K$ and a single
crossing.

To compute $\Gamma(K_{v,j})$ we first compute $\Gamma(X_{1,4}X_{2,3})=$

$(1,r_{1}c_{1}+(1-t)r_{1}c_{4}+tr_{4}c_{4}+r_{2}c_{2}+(1-t)r_{2}c_{3}+tr_{3}c_{3})=(\omega,A),$

using parts (2) and (4) of Definition 4. Then part (3) with $e,s=1,3$ tells us
we should compute
$(\partial_{r_{1}}A)(\partial_{c_{3}}A)=(c_{1}+(1-t)c_{4})((1-t)r_{2}+tr_{3})$
and $A_{13}=0$ so

$\Gamma(X_{1,4}X_{2,3}//m^{1,3}_{j})=\big{(}1,A+(c_{1}+(1-t)c_{4})((1-t)r_{2}+tr_{3})\big{)}|_{\begin{subarray}{c}r_{1},c_{3}\mapsto
0\\\ r_{3},c_{1}\mapsto r_{j},c_{j}\end{subarray}}=$
$(1,(1-t)r_{2}c_{j}+r_{2}c_{2}+(1-t)^{2}r_{2}c_{4}+tr_{j}c_{j}+t(1-t)r_{j}c_{4}+tr_{4}c_{4})=(1,B).$

Applying part (3) once more with $e,s=2,4$ we find $B_{24}=(1-t)^{2}$ and
$(\partial_{r_{2}}B)(\partial_{c_{4}}B)=((1-t)c_{j}+c_{2}+(1-t)^{2}r_{2})((1-t)^{2}r_{2}+t(1-t)r_{j}+tr_{4})$
so

$\Gamma(K_{v,j})=\Gamma(X_{1,4}X_{2,3}//m^{1,3}_{j}//m^{2,4}_{v})=$
$(2t-t^{2},B+\frac{((1-t)c_{j}+c_{2}+(1-t)^{2}c_{4})((1-t)^{2}r_{2}+t(1-t)r_{j}+tr_{4})}{2t-t^{2}})_{\begin{subarray}{c}r_{2},c_{4}\mapsto
0\\\ r_{4},c_{2}\mapsto r_{v},c_{v}\end{subarray}}$
$=(2t-t^{2},tr_{j}c_{j}+\frac{((1-t)c_{j}+c_{v})(t(1-t)r_{j}+tr_{v})}{2t-t^{2}})$
$=(2t-t^{2},\frac{r_{j}c_{j}+(1-t)r_{j}c_{v}+(1-t)r_{v}c_{j}+r_{v}c_{v}}{2-t}).$

Now we are in a position to compute $\Gamma(D_{ij})$ in just three more steps,
taking disjoint union with $X^{-1}_{6,0}$ and merging twice:
$\Gamma(X^{-1}_{6,0}K_{vj})=$

$(2t-t^{2},r_{6}c_{6}+(1-t^{-1})r_{6}c_{0}+t^{-1}r_{0}c_{0}+\frac{r_{j}c_{j}+(1-t)r_{j}c_{v}+(1-t)r_{v}c_{j}+r_{v}c_{v}}{2-t})=(\omega^{\prime},A^{\prime}).$

Since $A^{\prime}_{0,v}=0$ and
$(\partial_{r_{0}A^{\prime}})(\partial_{c_{v}A^{\prime}})=\frac{t^{-1}c_{0}}{2-t}((1-t)r_{j}+r_{v})$
we find

$\Gamma(X^{-1}_{6,0}K_{vj}//m^{0,v}_{0})=(2t-t^{2},r_{6}c_{6}+(1-t^{-1})r_{6}c_{0}+\frac{r_{j}c_{j}+(1-t)r_{0}c_{j}+t^{-1}c_{0}((1-t)r_{j}+r_{0})}{2-t}).$

2

Finally, merging for the last time, the determined reader will find
$\Gamma(D_{ij})=$

$\Gamma(X^{-1}_{6,0}K_{vj}//m^{0,v}_{0}//m^{0,6}_{i})=(2t-t^{2},(1-t^{-1})r_{i}c_{i}+\frac{r_{j}c_{j}+t^{-1}c_{i}(1-t)r_{j}+(1-t)c_{j}r_{i}+t^{-1}c_{i}r_{i}}{2-t}).$

2

While instructive to do by hand once, these calculations tend to get a little
tedious, so we recommend the reader to make use of a computer algebra package
to do the calculations. See the Appendix A for a description of the algorithm
in Mathematica. As the computations are conceptually very simple (only five
lines of code!) it should be easy to implement the same algorithm in any other
suitable language.

Another interesting example is that of the trefoil
$\mathcal{T}_{i}=X_{1,4}X_{5,2}X_{3,6}//m^{1,2,3,4,5,6}_{i}$. In precisely the
same way as above we compute

$\Gamma(\mathcal{T}_{i})=(t^{3}-t^{2}+t,r_{i}c_{i}).$ (6)

We summarize the main properties of the $\Gamma$ calculus in the following:

[$\Gamma$ is a tangle invariant generalizing Alexander [Bar-Natan(2015)]] Up
to multiplication by a factor $\pm t^{k}$ in the $\omega$-part, $\Gamma$ is an
invariant of tangles. For tangles $K_{i}$ with one strand (i.e., knots) we
have $\Gamma(K_{i})=(\Delta_{K}(t),r_{i}c_{i})$, where $\Delta_{K}$ is the
Alexander polynomial of knot $K$.

To understand how the factors $\pm t^{k}$ arise we note that the Alexander
polynomial of knots has the same ambiguity. For $\Gamma$ these factors arise
because $\Gamma$ does not give the same value on both sides of a Reidemeister
1 move. For example $\Gamma(1_{i})=(1,r_{i}c_{i})$ while
$\Gamma(X_{12}//m^{12}_{i})=(t,r_{i}c_{i})$, see Figure 10.

Figure 10: Writing one of the Reidemeister 1 moves algebraically.

The other Reidemeister moves can likewise be written algebraically and it can
be checked that $\Gamma$ yields the same result on both sides of each of the
tangle equations representing Reidemeister 2 and 3. For example, Reidemeister
3 is written as:

$X_{12}X_{43}X_{56}//m^{14}_{1}//m^{25}_{2}//m^{36}_{3}=X_{16}X_{23}X_{45}//m^{14}_{1}//m^{25}_{2}//m^{36}_{3},$
(7)

and it is easy to check (by computer) that indeed,

$\Gamma(X_{12}X_{43}X_{56}//m^{14}_{1}//m^{25}_{2}//m^{36}_{3})=\Gamma(X_{16}X_{23}X_{45}//m^{14}_{1}//m^{25}_{2}//m^{36}_{3}).$
(8)

##  5 $H$-Tangle Diagrams

After warming up with knots we are now ready to introduce the topological
framework for discussing folded chains: $H$-tangles. These are an extension of
the tangles from the previous sections where the strands may now bond to each
other, forming what we call hard contacts or $H$-contacts. The local structure
of a $H$-contact is shown in Figure 11 (Left).

Figure 11: Left: The vertex or hard contact in a $H$-tangle diagram with two
strands labelled $u$ and $d$. Middle: a singular crossing in singular knots.
Right: a generalized $H$-vertex.

A $H$-tangle diagram is the same as a tangle diagram except that one more type
of vertex is allowed called a $H$-contact, see Figure 11 (Left). Each
$H$-contact carries a label. Our notation for the $H$-vertex is $H^{v}_{u,d}$
where $u$ is the strand pointing upwards in the picture and $d$ is the other
strand. The superscript $v$ denotes the label of the vertex.

One convenient way of labelling the $H$-contacts on a chain is by their first
occurrence along the strand. When the $H$-contacts are not explicitly labeled,
this labeling is assumed. An example of a $H$-tangle with one strand already
appeared in Figure 1 in the introduction. Another example with one strand and
two hard contacts appear in Figure 12.

Figure 12: The $H$-tangle diagram $E$ with two $H$-contacts labelled $A,B$.

The notion of $H$-tangle diagrams is very similar to that of singular knots
and tangles [Bataineh et al.(2018)Bataineh, Elhamdadi, Hajij, and Youmans] but
not exactly the same. The difference is in how the strands pass through the
vertex as is seen by comparing Figure 11 Left ($H$-vertex) and Middle
(singular vertex). We also note that more complicated vertices can be modelled
similarly to the proposed $H$-vertex, see for example, Figure 11 Right, but we
will leave that for future work.

Most of the theory for the usual tangle diagrams as developed in the previous
sections is still valid. For example, $H$-tangle diagrams can be assembled
from the basic ones using disjoint union and merging. These operations are
defined in precisely the same way as in the tangle case.

[+15]For example, coming back to our discussion of circuit topology the basic
configurations $\mathcal{S},\mathcal{P},\mathcal{X}$ from Figure 3 can now be
interpreted as $H$-tangles as follows:
$\mathcal{S}_{i}=H_{1,2}H_{3,4}//m^{1,3,4,2}_{i}$ and
$\mathcal{P}_{i}=H_{1,2}H_{3,4}//m^{1,2,3,4}_{i}$. The cross $\mathcal{X}_{i}$
is a little more involved as it involves crossings and there are in fact
several options, one of which is:

$\mathcal{X}_{i}=H_{12}H_{34}X_{12,5}X^{-1}_{14,7}X_{16,9}X^{-1}_{10,11}//m^{1,5,7,3,9,11,10,12,2,14,16,4}_{i}.$
(9)

The example $E$ in Figure 12 is likewise expressed as:

$E_{i}=H^{A}_{10,4}H^{B}_{3,7}X_{6,1}X^{-1}_{8,2}X^{-1}_{5,9}//m^{1,2,3,4,5,6,7,8,9,10}_{i}.$
(10)

The most important part of our model of $H$-tangles is the type of
Reidemeister move we allow. We model the hard contact $H^{v}_{u,d}$ as a rigid
square and so the only way it is allowed to interact with other strands is for
strands to pass over or under the vertex. Two vertices are assumed to never be
precisely on top of each other, just like crossings in a usual tangle diagram
are not supposed to coincide. More formally, we say two $H$-tangle diagrams
are equivalent if they can be made equal by repeatedly making local
replacements as shown in Figure 13. We name these the Reidemeister 4 moves.

Since the $H$-contacts are assumed to be rigid we should remember to always
bring the $H$-contacts in the correct position with one up and one down,
rotating in the counter-clockwise direction. Contacts with other orientations
can be modelled using a $H$-contact by adding crossings as shown in Figure 14.

Figure 13: New Reidemeister 4 moves for $H$-tangle diagrams. The orientation
of the horizontal strand is arbitrary so in fact we have four such moves.
Figure 14: In case we want two strands to run in parallel near a $H$-contact
we augment it with a crossing. We can also flip it over using either of the
two pictures on the right.

It may not be immediately clear that the two pictures on the right of Figure
14 represent the same $H$-tangle. This is demonstrated in the next Figure 15.
The first step uses Reidemeisters 1–3 while the second uses Reidemeister 4 to
pass the horizontal strands under the $H$-contact. The next equality is
obtained by sliding the $H$-contact along the loop and the final step is like
the first.

Figure 15: A proof that two crossings can be passed through an $H$-contact
using Reidemeisters 1–4.

It should be noted that many variations on the notion of $H$-tangle introduced
here are possible by adding more Reidemeister moves for the $H$-contact to
satisfy. Each additional Reidemeister move would make the vertex more
flexible. For example one could assume $H_{ij}=H_{ji}$ which would give a
theory equivalent to that of singular tangles. Indeed, this assumption would
correspond to the fifth move for singular knots, see $\Omega 5a$ on p. 3 of
[Bataineh et al.(2018)Bataineh, Elhamdadi, Hajij, and Youmans]. Another
natural symmetry one could require is that the left-hand diagram of Figure 14
should be equivalent to that where the crossing is opposite. Since it is
conceptually simplest we prefer to not assume any Reidemeister moves beyond
the 4-th. In principle it is not difficult to adjust the theory by including
an additional Reidemeister move but we leave this for future work.

##  6 The Double Gamma Invariant of $H$-Tangles

In this section, we aim to extend $\Gamma$-calculus to $H$-tangles. We first
try the most direct approach and show its limitations. Next we propose a more
powerful version of $\Gamma$ called  $\mathbb{L}$ that is more suitable for
working with $H$-tangles.

The key idea of $\Gamma$ calculus is to assign a pair $(\omega,A)$ to any
tangle where $A$ has as many $r$-variables as the number strands in the
tangle. Since $\Gamma(H_{ij})$ involves two strands, we assign to it a value
of the form:

$\Gamma(H^{a}_{ij})=(p_{0},p_{1}r_{i}c_{i}+p_{2}r_{i}c_{j}+p_{3}r_{j}c_{i}+p_{4}r_{j}c_{j}),$
(11)

for some functions $p_{0},p_{1},p_{2},p_{3},p_{4}$ of $t$ and $a$ that are yet
to be determined.

To ensure that $\Gamma$ takes the same value on equivalent $H$-tangle
diagrams, it suffices to choose the value $\Gamma(H^{a}_{ij})$ so that the
Reidemeister 4 equations are satisfied. The Reidemeister 4 moves appeared in
Figure 13, here we wrote them algebraically and applied $\Gamma$ to both
sides:

$\displaystyle\Gamma(H^{a}_{12}X_{34}X^{-1}_{56}//m^{35}_{3}//m^{14}_{1}//m^{62}_{2})$
$\displaystyle=\Gamma(H^{a}_{12}X_{34})X^{-1}_{56}//m^{35}_{3}//m^{41}_{1}//m^{26}_{2})$
$\displaystyle\Gamma(H^{a}_{12}X^{-1}_{43}X_{65}//m^{35}_{3}//m^{14}_{1}//m^{62}_{2})$
$\displaystyle=\Gamma(H^{a}_{12}X^{-1}_{43}X_{65}//m^{35}_{3}//m^{41}_{1}//m^{26}_{2}).$

Since we can explicitly compute both sides using $\Gamma$ calculus, it is left
to the reader to solve these equations for $p_{0},\dots p_{4}$ and verify (By
computer would be most convenient!) that they are equivalent to
$p_{2}=1-p_{1},p_{3}=1-p_{1},p_{4}=p_{1}$.

As a consequence we can compute what a single $H$-vertex connected to itself,
see Figure 16 (Left) evaluates to:
$\Gamma(H^{a}_{1,2}//m^{12}_{i})=(p_{0}p_{1},r_{i}c_{i})$. This is problematic
because it follows that for any diagram $Y_{j}$ $\Gamma$ takes the same value
on the two configurations in the middle and right of Figure 16. In particular
this implies that $\Gamma$ cannot distinguish the two $H$-tangles
$\mathcal{S,P}$ representing two of the basic pair configurations in circuit
topology (see Figure 3): $\mathcal{S}_{1}=H_{1,2}H_{3,4}//m^{1,3,4,2}_{i}$ and
$\mathcal{P}_{1}=H_{1,2}H_{3,4}//m^{1,2,3,4}_{i}$. The same argument shows
that the $\Gamma$ invariant is insensitive to the location in the chain of any
element $\mathcal{B}_{i}$ with $\Gamma(\mathcal{B}_{i})=(\omega,r_{i}c_{i})$

Figure 16: Left: A $H$-vertex on a single strand cannot be detected by the
naive extension of $\Gamma$ calculus. It takes the same value on both pictures
on the Right.

We conclude that introducing the above $\Gamma$ invariant for $H$-tangles
misses too many fundamental features of $H$-tangles such as the basic pair
interactions. This brings us to propose a more sophisticated version of
$\Gamma$ called  $\mathbb{L}$ (double $\Gamma$) that distinguishes a lot more
about $H$-tangles, while retaining a close relation to $\Gamma$.

[  $\mathbb{L}$ calculus] For a $H$-tangle diagram $D$ whose strands are
labelled by a set $L$ of positive integers and $H$-vertices indexed by set $V$
define $\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D)$ by the
following rules.
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D)=(\omega,A)$ where
$A=\sum_{\pm i,\pm j\in L}A_{ij}r_{i}c_{j}$ and $\omega,A_{ij}$ rational
functions of $t,h_{v},v\in V$.

  1. 1.

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(1_{i})=(1,r_{i}c_{i}+r_{-i}c_{-i})$.

  2. 2.

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(X_{i,j})=\big{(}1,r_{i}c_{i}+r_{-i}c_{-i}+t^{2}(r_{j}c_{j}+r_{-j}c_{-j})+(1-t)(tr_{i}+r_{-i})(c_{j}+c_{-j})\big{)}$.

  3. 3.

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(X^{-1}_{i,j})=\big{(}1,r_{i}c_{i}+r_{-i}c_{-i}+t^{-2}(r_{j}c_{j}+r_{-j}c_{-j})+(1-t^{-1})(r_{i}+t^{-1}r_{-i})(c_{j}+c_{-j}))\big{)}$.

  4. 4.

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(H^{B}_{i,j})=\big{(}1,t(r_{-i}c_{-i}+r_{j}c_{-j})+(1-t)(r_{-j}c_{-i}+r_{-i}c_{-j})+(1-h_{B})(tr_{i}c_{-j}-tr_{-j}c_{-j}+r_{-j}c_{j})+r_{-j}c_{i}+h_{B}r_{i}c_{j}\big{)}$.

  5. 5.

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D//m^{i,j}_{k})=(\omega,A)//\mu^{i,j}_{k}//\mu^{-i,-j}_{-k}$,
where $\mu$ was defined in Definition 4.

  6. 6.

If
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D^{\prime})=(\omega^{\prime},A^{\prime})$
then
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D,D^{\prime})=(\omega\omega^{\prime},A+A^{\prime})$.

As the rules for  $\mathbb{L}$ are very similar to those of $\Gamma$-calculus,
we do not work out any examples step by step but instead refer the reader to
the computer code in the Appendix A. For example for the $H$-tangle $E$ from
Figure 12 the value of  $\mathbb{L}$ is:

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(E_{1})=(-h_{A}h_{B}+th_{A}+h_{B}-t,(1-t)r_{-1}c_{-1}+r_{-1}c_{1}+tr_{1}c_{-1})$

.

The problematic $H$-tangle $\mathcal{B}$ discussed at the start of this
section that $\Gamma$ fails to handle properly also has a non-trivial
$\mathbb{L}$ value:

$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(\mathcal{B}_{1})=(t(1-h_{1}),(1-t)r_{-1}c_{-1}+r_{-1}c_{1}+tr_{1}c_{-1}).$

The second part of
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D)=(\omega,A)$ is
often quite lengthly so we often just list the $\omega$-part and use the
notation
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(D)=\omega$. In
fact this notation was already used in the introduction where
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ of the
$H$-tangle from Figure 1 was listed. As expected
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ also
distinguishes the three basic pair interactions $\mathcal{S,P,X}$ of circuit
topology. Labelling the $H$-contacts in order of appearance as one walks the
chain we find

$\displaystyle\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(\mathcal{S})$
$\displaystyle=(1-h_{1})(1-h_{2})$
$\displaystyle\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(\mathcal{P})$
$\displaystyle=\left(h_{2}-1\right)\left(-h_{1}t+t-1\right)$
$\displaystyle\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(\mathcal{X})$
$\displaystyle=-h_{1}\left(h_{2}-1\right)\left(t^{3}-t^{2}+1\right)t^{2}$
$\displaystyle+h_{2}\left(t^{4}-t^{3}+2t-1\right)t-t^{5}+2t^{4}-3t^{3}+3t^{2}-3t+2.$

As with the usual $\Gamma$ calculus, there is a minor ambiguity in
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ due to fact
that it is not invariant under Reidemeister 1.
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ should always
be taken up to multiplication by a factor $\pm t^{k}$. Actually the situation
with Reidemeister 1 is a bit more serious this time and to make  $\mathbb{L}$
an invariant we should always use 0-framed $H$-tangle diagrams. We say an
$H$-tangle diagram is $0$-framed if after cutting out the $H$-vertices, all
the remaining pieces of strand have writhe $0$. Recall that the writhe of a
strand is the sum of the signs of all the crossings of the strand with itself.
This is not a serious restriction as any diagram can be modified to have
writhe $0$ by inserting curls where necessary using the Reidemeister 1 move.

[  $\mathbb{L}$ is an invariant of $H$-tangles] If two $0$-framed $H$-tangle
diagrams $D,E$ are equivalent then
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(D)=\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(E)$
(up to a factor $\pm t^{k}$ in the
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ parts). If $K$
is a knot diagram, then
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}(K)=\Delta_{K}(t^{2})$,
that is, the Alexander polynomial of the knot at $t^{2}$.

###### Proof.

Like ordinary $\Gamma$ calculus, the proof of invariance under the
Reidemeister moves can be carried out by computer. One simply has to verify
that both sides of all Reidemeister moves yield the same result after applying
$\mathbb{L}$ . We already sketched above that in the case of Reidemeister 1,
one needs to correct for the framing and an ambiguity of a factor $\pm t^{k}$
will arise in the $\omega$ part.

When no $H$-contacts are present it can be verified that  $\mathbb{L}$ is just
$\Gamma$ applied to the double of a knot. By the double, we mean that a
parallel strand is added next to the existing strand. It is well-known that
the Alexander polynomial of the doubled knot is the Alexander polynomial of
the original taken at $t^{2}$, see for example Chapter 8 of [Burde and
Zieschang(2003)]. ∎

The invariant  $\mathbb{L}$ is new but as we said it is closely related to
$\Gamma$-calculus, not just for knots. To obtain the rules for  $\mathbb{L}$
calculus we applied $\Gamma$ calculus to the $H$-tangle in which each strand
was doubled. The value for the $H$-contact is however not of this type but
rather found by the method of undetermined coefficients, imposing all the
necessary Reidemeister moves. Imposing the Reidemeister moves does not
determine all coefficients so the remaining ones were chosen in a way to at
least distinguish the basic examples listed in the tables but not involve too
many additional variables.

##  7 Generalized Ciruit Topology

So far, we have have modelled folded chains using $H$-tangles that have a
single strand and we developed a topological invariant  $\mathbb{L}$ of these
objects. In circuit topology a chain is described by its contacts. The hard
contacts are obvious in a $H$-tangle but the concept of soft contacts is less
clear. In [Golovnev and Mashaghi(2020)] four types of soft contacts were
proposed as the four simplest ways a single strand can knot itself, see Figure
17. These coincide with all the possible knots one can tie in which the
projection has only four crossings.

Figure 17: The four soft contacts, reproduced from Figure 8 of [Golovnev and
Mashaghi(2020)].

In this work, we approach the notion of soft contact from a tangle point of
view. This means that we temporarily allow ourselves to think of the chain as
being composed of several strands. This simplifies matters because two strands
can interlock non-trivially already with only two crossings. The simplest way
two strands can interlock is the clasp, shown in Figure 18. Like the crossing
there is a positive and a negative version of the clasp. In what follows we
will refer to either of them as a (soft) clasp contact.

Figure 18: The circuit topology contacts: Left: the positive and negative
clasp, Right: the hard contact.

The clasp contacts are universal in the sense that attaching a clasp next to a
crossing causes the crossing to change sign, see Figure 19. This means any
other soft contact or really any tangle must be expressible using clasps only.
For example the four model configurations of soft contacts from [Golovnev and
Mashaghi(2020)], Figure 17, correspond precisely to the non-trivial
combinations of two clasps. In Figure 20, we sketch more precisely how the
first of the four soft contacts may explicitly be converted to the standard
clasp diagram form of Defintion 20 given below. The three others can be done
in precisely the same way.

Figure 19: Inserting a soft contact (clasp) has the effect of changing the
sign of a crossing. Figure 20: Changing crossings by inserting clasps,
straightening the picture keeping the clasps factors the soft contact
$A^{+e}A$ into a gCT diagram with two clasps.

Even when restricting soft contacts to just be clasps the number of ways to
combine clasps to get the same tangle is enormous making it hard to discern
any clear structure from them directly. To talk about circuit topology more
concretely we prefer a more rigid type of diagrams where the contacts interact
in a very simple way. We call such diagrams generalized circuit topology
diagrams (gCT diagrams). In the case of tangles, such diagrams appeared in
[Mostovoy and Polyak(2019)] under the name descending clasp diagrams. We
basically write the chain as a horizontal line that is interrupted by arcs
representing contacts.

[Generalized circuit topology (gCT) diagrams] A gCT diagram is encoded by a
partition

$\\{\\{a_{1},b_{1}\\}_{\sigma_{1}},\\{a_{2},b_{2}\\}_{\sigma_{2}},\dots,\\{a_{n},b_{n}\\}_{\sigma_{n}}\\}$

where $a_{i},b_{i}$ are integers such that
$\bigcup_{i=1}^{n}\\{a_{i},b_{i}\\}=\\{1,2,\dots 2n\\}$ and
$\sigma_{i}\in\\{-1,0,1\\}$. Given this data define a one-strand $H$-tangle
diagram as follows. Starting with the $x$-axis oriented left to right, for
each pair $\\{a,b\\}_{\sigma}$ connect the points $a$ and $b$ with a contact
of type $\sigma$ as shown in Figure 21. Whenever $a<c<b<d$ the lines
corresponding to pairs $\\{a,b\\}$ and $\\{c,d\\}$ will cross the $\\{a,b\\}$
contact is assumed to be in front. No other crossings appear.

Figure 21: The notation $\\{a,b\\}_{\sigma}$ means a contact of type $\sigma$
connecting $a,b$ below the $x$-axis. Two such contacts can only cross as
shown.

Examples of gCT diagrams already appeared in Figures 1 and 2. The notation for
these diagrams is

$\\{\\{1,4\\}_{1},\\{2,7\\}_{-1},\\{3,5\\}_{0},\\{6,9\\}_{0},\\{8,10\\}_{-1}\\}$
$\\{\\{1,4\\}_{0},\\{2,7\\}_{0},\\{3,5\\}_{0},\\{6,9\\}_{0},\\{8,10\\}_{0}\\}$

respectively. Also the soft contacts from Figure 17 were factored into two
claps using the reasoning in Figure 20 and in the gCT notation we thus find:

$\displaystyle A^{+e}A$ $\displaystyle=\\{\\{1,3\\}_{1},\\{2,4\\}_{1}\\}$
$\displaystyle A^{+o}A$ $\displaystyle=\\{\\{1,3\\}_{1},\\{2,4\\}_{-1}\\}$
$\displaystyle A^{-o}A$ $\displaystyle=\\{\\{1,3\\}_{-1},\\{2,4\\}_{1}\\}$
$\displaystyle A^{-e}A$ $\displaystyle=\\{\\{1,3\\}_{-1},\\{2,4\\}_{-1}\\}.$

To get a sense of what gCT diagrams look like in general we show a randomly
chosen gCT diagrams with 20 contacts in Figure 22. In gCT notation it is

$\left\\{\\{1,14\\}_{1},\\{2,11\\}_{0},\\{3,13\\}_{0},\\{4,12\\}_{1},\\{5,8\\}_{-1},\\{6,9\\}_{1},\\{7,19\\}_{-1},\\{10,18\\}_{0},\\{15,20\\}_{-1},\\{16,17\\}_{0}\right\\}.$

2

Figure 22: A random gCT diagram with twenty contacts.

As a justification for restricting to gCT diagrams, we claim that planar
$H$-tangle diagrams can be represented this way. By planar we mean that viewed
as a decorated graph whose vertices are crossings and $H$-vertices it can be
drawn in the plane. All $H$-tangle diagrams that are actual projections of
three-dimensional chains are planar. Planar in the sense of the underlying
graph, it can of course have crossings. Usually the planarity condition is
part of the definition of a tangle diagram, however since it is not needed for
$\mathbb{L}$ we preferred to work a little more generally.

[[Mostovoy and Polyak(2019)]] [-15]Any planar $H$-tangle diagram with a single
strand is equivalent to a gCT diagram. This theorem is due to [Mostovoy and
Polyak(2019)] although their argument is given for usual tangles they make it
clear the same technique works for singular knots and also $H$-tangles. A
first hint of why it works is that inserting a clasp contact near a crossing
changes the sign of that crossing as shown in Figure 19. This implies that we
can freely change the crossings of our $H$-tangle diagram at the cost of
inserting clasp contacts. This way all the complicated knots and tangles can
be ’factored’ into simple clasp contacts. The hard contacts will remain and
the clasp contacts may interlace in complicated ways but by changing some more
crossings we can simplify further until a gCT diagram remains. An illustration
of this argument in a very simple case already appeared in Figure 20. A
precise argument using the pure braid group would take us too far afield and
for that and more we refer to [Mostovoy and Polyak(2019)].

While convenient and interesting to study, the reader is warned that such
diagrams are not necessarily unique. In fact in [Mostovoy and Polyak(2019)] a
complete set of moves relating equivalent gCT diagrams is suggested in the
knot case and we suspect this too holds in the $H$-tangle setting. Like the
Reidemeister moves themselves these moves are sufficiently complicated that it
is still essential to compute knot invariants such as  $\mathbb{L}$ for
distinguishing gCT diagrams.

Many natural operations in both circuit topology and tangle theory can easily
be handled in terms of gCT diagrams. To give one simple example, the $H$-$H$
interactions studied in [Ceniceros et al.(2021)Ceniceros, Elhamdadi, and
Mashaghi] can be described precisely as follows. Suppose we have two hard
contacts that are interacting by a soft (clasp) contact as Figure 23 Left.
Assuming the two contacts are in series we can just slide the soft (clasp)
contact off the arcs and turn it into an additional arc in the gCT diagram,
shown on the right of the figure. This means inserting soft (clasp) contacts
like this can be done entirely on the level of gCT diagrams.

Figure 23: Implementing a $H$-$H$ interaction by adding an additional soft
(clasp) contact.

We end this section by listing all 27 gCT diagrams with at most two contacts,
see Figure 24. For each diagram we also list its gCT notation. The hard
contacts of a gCT diagram are always labelled in order of first appearance as
one travels along the strand. For the $i$-th hard contact we used the variable
$h_{i}$ in our expression for
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$. Except for the
pair of mirror image trefoil knots $\\{\\{1,3\\}_{1},\\{2,4\\}_{1}\\}$ and
$\\{\\{1,3\\}_{-1},\\{2,4\\}_{-1}\\}$ diagrams with the same value of
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ are in fact
equivalent. It is well known that the Alexander polynomial does not
distinguish mirror images of knots. The ambiguity in multiplication by a power
of $t$ has been used to make sure the lowest power of $t$ is always $t^{0}$.

The first eight diagrams in the table are all gCT diagrams on which
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ takes the value
$1$. Using Reidemeister moves or perhaps with the help of a rope the reader
should be able to verify that each of them is in fact equivalent to the
standard unknot diagram: a straight line with no contacts. Of the diagrams
that come after we now know for sure that they are not equivalent to the
unknot as $\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ takes
a value different from $1$.

In the Appendix A a similar table for all gCT diagrams with three contacts is
given. These tables show that
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$ is able to
distinguish a lot of relevant cases. A table for all gCT diagrams with four
contacts is also feasible but more and more equivalent gCT diagrams will
appear and additional restrictions should be enforced to obtain a useful
table.

Figure 24: All 27 generalized circuit topology diagrams with two contacts,
with their notation and value of
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$.

##  8 Conclusions and Further Directions

Generalized circuit topology studies folded linear molecular chains, taking
into account both the way the chain bonds to itself (to form hard contacts)
and the way it tangles in space (to form soft contacts or clasps). In this
study, we provided a theoretical basis for the language of circuit topology.
For such chains and their corresponding topological circuits, we furthermore
introduced a new topological invariant  $\mathbb{L}$ that summarizes much of
the essential topological features into a single polynomial. A simple computer
implementation of the computation of the invariant was given in the Appendix A
as well as a complete table of the values of the invariant on all generalized
circuit topology diagrams with at most three contacts.

In future work it may be of interest to consider additional topological
features such as rigid beads (vertices of valency two) and vertices of higher
valency. The alpha helices and beta sheets found in proteins would be an
example of an application. Any such object may be modelled by some version of
$\Gamma$-calculus. The general technique is always to first specify the
appropriate Reidemeister moves, to describe how our objects are supposed to
move and interact. Next we propose some general form with undetermined
coefficients for our new objects. Applying $\Gamma$ to both sides of the
Reidemeister moves yields algebraic equations for the coefficients that may be
used to determine them.

$\Gamma$ calculus is by no means the only knot invariant that could be
extended to molecular chains. The techniques used here are appropriate for a
wide class of knot invariants known as universal knot invariants
[Habiro(2006)]. Roughly speaking there is an invariant for any ribbon algebra
and the basic idea is to place special algebra elements on the strands of the
knot near each crossing and then multiply. The $\Gamma$-calculus is an example
of such invariants for a particular algebra [Bar-Natan and van der Veen(2019),
Vo(2018)] but depending on the task at hand other algebras may be more
convenient. For example if one would need a higher resolution than the one
given by  $\mathbb{L}$ one good option is to choose the polynomial time knot
polynomials of [Bar-Natan and van der Veen(2019)]. So far these invariants
have been introduced for tangles only but following the ideas of the present
paper an extension to $H$-tangles seems possible. Unlike the more well known
strong invariants, such as the Jones polynomial, Khovanov and Floer homology,
the polynomial time knot polynomials are still relatively efficient to
compute, allowing for real world applications. In future work we plan
investigate such invariants for molecular chains.

We have established a good way of distinguishing and manipulating folded
linear molecular chains but many exciting challenges remain. Especially
relevant for applications seems to be the design of a workable notion of
relative topological distance between two given chains. This allows for
mapping the topological evolution of biomolecular folds and the development of
useful reaction coordinates for monitoring molecular folding reactions.

Conceptualization, A.M. and R.v.d.V.; methodology, A.M. and R.v.d.V.;
software, R.v.d.V.; investigation, A.M. and R.v.d.V.; resources, A.M.;
writing—original draft preparation, A.M. and R.v.d.V.; writing—review and
editing, A.M. and R.v.d.V. All authors have read and agreed to the published
version of the manuscript.

Data is contained within the article or supplementary material.

The authors declare no conflict of interest.

Abbreviations The following abbreviations are used in this manuscript:

CT |  Circuit topology
---|---
gCT |  Generalized circuit topology

yes

##  Appendix A Implementation and Table

In this appendix we present implementations of $\Gamma$ and  $\mathbb{L}$ in
Mathematica [Wol()]. We also give a few examples of how the code is used in
practice. Finally a further table of gCT diagrams is provided. All the code
can be downloaded from the second author’s website : www.rolandvdv.nl/gCT
(accessed September 2021)

###  A.1 Implementation for $\Gamma$

$\Gamma$ calculus is simple enough for the complete implementation to fit five
lines of Mathematica code as shown below. The code remains quite close to the
mathematics. A minor deviation is that pairs $(\omega,A)$ are represented as
$\Gamma[\omega,A]$. The first line of code implements disjoint union. The
second line makes sure $\omega$ is always in its simplest form and $A$ should
be written as a quadratic in $r,c$ with coefficients that are functions of
$t$. The remaining lines determine the merging and the value on the crossings.
For simplicity we write $X_{ij}$ instead of $\Gamma(X_{ij})$ and $X^{-1}$ is
written more compactly as $\bar{X}$. The effect of merging strands $i,j$ is
written as $\mu_{i,j\to k}$.

Figure 25: A complete implementation of $\Gamma$-calculus. Figure 26: Two
examples of how to run the $\Gamma$ program.

###  A.2 Mathematica Implementation for  $\mathbb{L}$

The implementation of  $\mathbb{L}$ is very similar to that of $\Gamma$ and
builds on top of it. The key difference is that this time we require all
strand labels to be positive in the sense that for each label $i$ the label
$-i$ is supposed to be distinct and not yet occupied.

Pairs $(\omega,A)$ are still represented as $\Gamma[\omega,A]$. For
$\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}(X_{i,j})$ we write
$XX_{ij}$ and $\overline{XX}_{ij}$ for the value of the negative crossing. The
value of the $H$-vertex $H^{h}_{ij}$ with label $h$ is denoted $HH_{ij}[h]$
and the merging is denoted $\mu\mu_{i,j\to k}$ Finally we add a small routine
for comparing pairs $(\omega,A)$.

Figure 27: A complete implementation of

$\mathbb{L}$

-calculus.

To illustrate that the proof of invariance of  $\mathbb{L}$ is indeed
automatic we provide the code to check an instance of Reidemeister 3 and 4.
The other equations can (and should) be checked similarly by computer.

Figure 28: Checking instances of Reidemeister 3 and 4.

###  A.3 Tabulation of All gCT Diagrams with Three Contacts

Finally we list all gCT diagrams with three contacts, grouped by their value
of $\reflectbox{\rotatebox[origin={c}]{180.0}{$\mathbb{L}$}}_{1}$. Trivial
cases that contain a loose clasp $\\{i,i+1\\}_{\pm 1}$ have been ommitted as
they are clearly equivalent to a gCT diagram with two contacts. Although not
perfect we see that  $\mathbb{L}$ is able to distinguish many types of chains.
For example, the gCT diagrams with three hard contacts and no soft contacts
all get a different value of  $\mathbb{L}$ . With a few exceptions such as the
mirror image trefoils already seen in the two-contact table most of the
diagrams that have the same value of  $\mathbb{L}$ in this table are in fact
equivalent.

References

## References

  * [Scalvini et al.(2020)Scalvini, Sheikhhassani, Woodard, Aupic, Dame, Jerala, and Mashaghi] Scalvini, B.; Sheikhhassani, V.; Woodard, J.; Aupic, J.; Dame, R.; Jerala, R.; Mashaghi, A.  Topology of Folded Molecular Chains: From Single Biomolecules to Engineered Origami.  Trends Chem. 2020, 2, 609–622.
  * [Jamroz et al.(2015)Jamroz, Niemyska, Rawdon, Stasiak, Millett, Sulkowski, and Sulkowska] Jamroz, M.; Niemyska, W.; Rawdon, E.; Stasiak, A.; Millett, K.; Sulkowski, P.; Sulkowska, J.  KnotProt: A database of proteins with knots and slipknots.  Nucleic Acids Res. 2015, 43, D306–D314.
  * [Flapan et al.(2019)Flapan, He, and Wong] Flapan, E.; He, A.; Wong, H.  Topological descriptions of protein folding.  Proc. Natl. Acad. Sci. USA 2019, 116, 9360–9369.
  * [Kyoda et al.(2019)Kyoda, Yamamoto, and Tezuka] Kyoda, K.; Yamamoto, T.; Tezuka, Y.  Programmed Polymer Folding with Periodically Positioned Tetrafunctional Telechelic Precursors by Cyclic Ammonium Salt Units as Nodal Points.  J. Am. Chem. Soc. 2019, 141, 7526–7536.
  * [Mashaghi(2021)] Mashaghi, A.  Circuit topology of folded chains.  Not. Am. Math. Soc. 2021, 68, 420–423.
  * [Mashaghi et al.(2014)Mashaghi, van Wijk, and Tans] Mashaghi, A.; van Wijk, R.; Tans, S.  Circuit topology of proteins and nucleic acids.  Structure 2014, 22, 1227–1237.
  * [Heidari et al.(2020)Heidari, Schiessel, and Mashaghi] Heidari, M.; Schiessel, H.; Mashaghi, A.  Circuit Topology Analysis of Polymer Folding Reactions.  ACS Cent. Sci. 2020, 6, 839–847.
  * [Mugler et al.(2014)Mugler, Tans, and Mashaghi] Mugler, A.; Tans, S.; Mashaghi, A.  Circuit topology of self-interacting chains: Implications for folding and unfolding dynamics.  Phys. Chem. Chem. Phys. 2014, 16, 22537–22544.
  * [Conway(1970)] Conway, J.H.  An Enumeration of Knots and Links, and Some of Their Algebraic Properties. In Computational Problems in Abstract Algebra; Leech, J., Ed.; Pergamon Press: Oxford, UK, 1970; pp. 329–358.
  * [Ceniceros et al.(2021)Ceniceros, Elhamdadi, and Mashaghi] Ceniceros, J.; Elhamdadi, M.; Mashaghi, A.  Coloring Invariant for Topological Circuits in Folded Linear Chains.  Symmetry 2021, 13, 101492.
  * [Golovnev and Mashaghi(2020)] Golovnev, A.; Mashaghi, A.  Generalized Circuit Topology of Folded Linear Chains.  iScience 2020, 23, 101492.
  * [Mostovoy and Polyak(2019)] Mostovoy, J.; Polyak, M.  Encoding knots by clasp diagrams.  _arXiv_ 2019, arXiv:1911.02791.
  * [Chmutov et al.(2012)Chmutov, Duzhin, and Mostovoy] Chmutov, S.; Duzhin, S.; Mostovoy, J.  Introduction to Vassiliev Knot Invariants; Cambridge University Press: Cambridge, UK, 2012.
  * [Burde and Zieschang(2003)] Burde, G.; Zieschang, H.  Knots; Birkhauser: Basel, Switzerland, 2003.
  * [Polyak(2010)] Polyak, M.  Minimal generating sets of Reidemeister moves.  Quantum Topol. 2010, 1, 399–411.
  * [Bar-Natan and Selmani(2013)] Bar-Natan, D.; Selmani, S.  Meta-Monoids, Meta-Bicrossed Products, and the Alexander Polynomial.  J. Knot Theory Ramif. 2013, 22, 1350058.
  * [Vo(2018)] Vo, H.  Alexander Invariants of Tangles via Expansions.  Ph.D. Thesis, University of Toronto, Toronto, ON, Canada, 2018.
  * [Bar-Natan(2015)] Bar-Natan, D.  Balloons and Hoops and their Universal Finite Type Invariant, BF Theory, and an Ultimate Alexander Invariant.  Acta Math. Vietnam. 2015, 40, 271–329.
  * [Bataineh et al.(2018)Bataineh, Elhamdadi, Hajij, and Youmans] Bataineh, K.; Elhamdadi, M.; Hajij, M.; Youmans, W.  Generating sets of Reidemeister moves of oriented singular links and quandles.  J. Knot Theory Ramif. 2018, 27, 1850064.
  * [Habiro(2006)] Habiro, K.  Bottom tangles and universal invariants.  Algebr. Geom. Topol. 2006, 6, 1113–1214, doi:10.2140/agt.2006.6.1113.
  * [Bar-Natan and van der Veen(2019)] Bar-Natan, D.; van der Veen, R.  A polynomial time knot polynomial.  Proc. Am. Math. Soc. 2019, 147, 377–397.
  * [Wol()] _Mathematica, Version 12_ ; Wolfram Research Inc.: Champaign, IL, USA, 2019.


11institutetext: 1Tsinghua-Berkeley Shenzhen Institute and Tsinghua Shenzhen
International Graduate School,
2Zhejiang Future Technology Institute (Jiaxing), 3University of Rochester,
4Beihang University

# Cross-Camera Deep Colorization

Yaping Zhao 1122 Haitian Zheng 33 Mengqi Ji 44 Ruqi Huang Ruqi Huang is the
corresponding author (ruqihuang@sz.tsinghua.edu.cn). This work is supported in
part by the Shenzhen Science and Technology Research and Development Funds
(JCYJ20180507183706645), in part by the Provincial Key R&D Program of Zhejiang
(Serial No. 2021C01016), and the Shenzhen Key Laboratory of next generation
interactive media innovative technolog (Funding No. ZDSYS20210623092001004).
The lab website is: http://www.luvision.net.11

###### Abstract

In this paper, we consider the color-plus-mono dual-camera system and propose
an end-to-end convolutional neural network to align and fuse images from it in
an efficient and cost-effective way. Our method takes cross-domain and cross-
scale images as input, and consequently synthesizes HR colorization results to
facilitate the trade-off between spatial-temporal resolution and color depth
in the single-camera imaging system. In contrast to the previous colorization
methods, ours can adapt to color and monochrome cameras with distinctive
spatial-temporal resolutions, rendering the flexibility and robustness in
practical applications. The key ingredient of our method is a cross-camera
alignment module that generates multi-scale correspondences for cross-domain
image alignment. Through extensive experiments on various datasets and
multiple settings, we validate the flexibility and effectiveness of our
approach. Remarkably, our method consistently achieves substantial
improvements, _i.e._ , around 10dB PSNR gain, upon the state-of-the-art
methods. Code is at: github.com/IndigoPurple/CCDC.

###### Keywords:

image colorization image fusion computational imaging.

(a)

(b)

Figure 1: Aiming at pursuing high-resolution and high color depth
simultaneously, we proposed a general binocular imaging framework that
performs cross-camera colorization.

##  1 Introduction

Nowadays, it has become a common practice of leveraging fusion of multi-
sensory data from camera arrays to improve imaging quality [41, 10, 1, 33, 4,
24, 46, 55, 26, 59]. In this paper, we consider the color-plus-mono dual-
camera system and propose a novel learning-based framework for data fusion.
Such a setting enjoys the advantages of photosensibility and high resolution
of the monochrome camera, and the color sensibility of the color camera
simultaneously. More specifically, our pipeline takes as inputs a high-
resolution (HR) grayscale image and a low-resolution (LR) color image, which
are taken of the same scene by the respective cameras from similar viewpoints,
as shown on the left of Fig. 1. After fusion, we obtain a HR color image
illustrated on the right of Fig. 1, which to some extent facilitates the
trade-off between spatial-temporal resolution and color depth in the single
camera due to the space-bandwidth-product (SBP) [31].

In fact, the color-plus-mono dual camera has attracted an increasing amount of
attention [23, 11, 29, 34, 38] recently. Most of the existing methods focus on
addressing the cross-domain image color transfer. For example, traditional
methods [49, 35, 20, 40, 14, 9, 30, 3] employ global color statistics or low-
level feature correspondences while emerging learning-based approaches [27,
17] find high-level feature correspondences between images. Nevertheless, they
commonly assume similar spatial resolution between data, and therefore can
only deal with minor resolution gap, which is typically less than $2\times$.

On the other hand, it is naturally desirable to retain both high spatial-
temporal resolution and high color depth. Thus methods assuming low-resolution
gap have to either sacrifice resolution or require high-resolution input from
_both_ cameras, which can be costly and computationally heavy. Such
discrepancy limits their practical applicability, for example, in using
cameras with huge-resolution gaps to capture gigapixel video [54, 56], or in
reducing the budget of a camera system [47].

In contrast, our method enables the imaging system to flexibly employ various
cameras with different resolution gaps. The key ingredient of our method is a
cross-camera alignment module that generates multi-scale correspondence for
cross-domain image alignment. Without resorting to hand-crafted design on
image registration or fusion [23, 11], we propose a novel neural network that
leverages joint image alignment and fusion for cross-camera colorization. To
improve the correspondence, we design visibility maps computation that
explicitly computes and compensates warping errors. Finally, we utilize a
warping regularization [42] to further improve the alignment quality.

Extensive experiments are performed to evaluate the proposed method under
various settings, i.e., combinations of different resolution gaps, viewpoints,
temporal steps, dynamic/static scenes. We test our method on various datasets,
including the video dataset Vimeo90k [51], the light field dataset Flower
[39], and the light field video dataset LFVideo [45]. Both the quantitative
and qualitative experiments show the substantial improvements of our method
over the state-of-the-art methods – remarkably, we achieve around 10dB gain in
terms of PSNR in most of the test cases upon the best baseline.

Our main contributions are summarized as follows:

  * •

A flexible and cost-effective imaging framework that is applicable to various
color-plus-mono camera settings with multiple resolution gaps. In particular,
our method can adapt to both spatial and temporal resolution gap more than
$8\times$.

  * •

A novel network design for cross-camera colorization: the cross-camera
alignment generates dense correspondence for multi-scale feature alignment;
the fusion module compensates alignment error via visibility map computation
and performs synthesis; the warping regularization further improves the
alignment.

  * •

Extensive evaluation on a wide range of settings, _i.e._ different resolution
gaps, combinations of viewpoints and temporal steps show the substantial
improvements of our method, _i.e._ , around 10dB PSNR gain over the state-of-
the-art ones.

##  2 Related Work

###  2.1 Automatic Image Colorization

Most traditional methods perform colorization by optimization, e.g.,
Regression Tree Fields (RTFs) [22] and graph cuts [7]. With deep learning,
some approaches [8, 58] leverage large-scale color image data [63, 37] to
colorize grayscale images automatically. Yoo _et al._ [53] propose a
colorization network augmented by external neural memory networks. Another
line of works uses GANs [13, 21, 5, 44] to colorize grayscale images by
learning probability distributions.

However, automatic colorization is ill-conditioned since many potential colors
can be assigned to the gray pixels of an input image. As a result, these
methods tend to generate unnatural colorized images.

###  2.2 Reference-based Image Colorization

According to what is used as a reference, reference-based methods can be
divided into strokes/palette-based ones and example-based ones.

Strokes/Palette-based Image Colorization. Strokes/palette-based image
colorization methods [25, 18, 52, 32, 6] seek to reconstruct color images from
the users-provided sparse stroke. However, these methods require intensive
manual works. More importantly, scribbles and palettes provide insufficient
color information for colorization, leading to unsatisfying results.

Example-based Image Colorization. To avoid the manual labor of scribbling or
palette selection while facilitating controllable colorization, example-based
methods are proposed to transfer the color of a reference image to the target
image. The initial works [35, 49] attempt to match color statistics globally.
Targeting at more accurate color transfer, later works [20, 40, 14, 9, 30, 3]
utilize hand-crafted feature correspondences, e.g., SIFT, Gabor wavelet to
enforce local color consistency. Recently, He _et al._ [16] and Liao _et al._
[27] perform dense patch matching for color transfer. However, the patch-based
correspondence [16, 27] are inherently inefficient to compute. Furthermore,
the inter-patch misalignment may hurt the image synthesis. Xiao _et al._ [50]
propose a self-supervised approach that reconstructs a color image from its
grayscale version and global color distribution coding. Although obtaining
impressive results, the proposed approach does not utilize the spatial
correspondence between two images for local colorization.

###  2.3 Flow-based or non-rigid correspondences

The cross-camera image colorization is also related to flow-based and non-
rigid correspondence estimation. Specifically, back-warping reference images
using estimated optical flow fields [28, 19, 60, 61] can serve to align images
from two viewpoints. In addition, HaCohen _et al._ [15] showcase the usage of
estimating non-rigid dense correspondence for image enhancement. However, due
to the domain and resolution gap between the two cameras, the above approaches
fail in finding correct matching and result in poor performance under our
cross-domain and cross-resolution setting.

##  3 Approach

Assume two images from different cameras capturing the same scene at similar
viewpoints are given – an LR color image as the reference and an HR gray image
as the target. We denote the target single-channel image as
$I_{1}\in{\mathbb{R}}^{sH\times sW\times 1}$ and the three-channel reference
as $I_{2}\in{\mathbb{R}}^{H\times W\times 3}$, where $s\geq 1$ is the scale
factor representing the spatial-resolution gap in the dual camera, $H$ and $W$
are the horizontal and vertical spatial-resolution, respectively. Similarly,
the ground-truth color image is denoted as $I_{g}\in{\mathbb{R}}^{sH\times
sW\times 3}$, where $I_{g}$ shares the same viewpoint with the target image
$I_{1}$. Our goal is to generate an HR color image
$I_{c}\in{\mathbb{R}}^{sH\times sW\times 3}$ as similar as possible to
$I_{g}$.

To achieve this goal, we propose an end-to-end and fully convolutional deep
neural network (see illustration in Fig. 2). Our network contains a cross-
camera alignment module and a hierarchical fusion module: the former (Sec.
3.1) consists of a color encoder ${Net}_{E}^{c}$, a luminance encoder
${Net}_{E}^{l}$, and a flow estimator $Net_{flow}$; the latter (Sec. 3.2)
consists of a decoder $Net_{D}$. On top of that, our network trains jointly
the warping flow estimator and colorization neural networks by combining
warping loss $\mathcal{L}_{w}$ and colorization loss $\mathcal{L}_{c}$ (Sec.
3.3).

Figure 2: Our network contains two modules: 1) the alignment module, which
performs non-rigid transformation on the cross-camera image inputs and
extracted features; 2) the fusion module, which performs features fusion and
colorization synthesis.

###  3.1 Cross-camera Alignment Module

This module is designed to perform the temporal and spatial alignment. As
shown on the left of Fig. 2, we first extract feature maps of input images and
then utilize a flow estimator to generate the cross-camera correspondence at
multiple scales. After flow estimation, we use cross-camera warping to perform
the non-rigid transformation. In the following, we provide details of each
part.

#### Pyramidal Feature Extraction.

Considering the different resolutions of the input images, we first upsample
the reference $I_{2}$ to the same resolution as $I_{1}$ via bicubic
upsampling, denoted by $I_{2}^{\uparrow}$. $I_{1}$ and $I_{2}^{\uparrow}$ are
of the same resolution but belong to different image modalities, thus we
design two encoders to extract their pyramidal features respectively:

$\displaystyle\begin{aligned}
&\\{F_{i}^{l}\\}={Net}_{E}^{l}(I_{1}),&\\{F_{i}^{c}\\}={Net}_{E}^{c}(I_{2}^{\uparrow}),\
\ \ i=1,2,3,4,\end{aligned}$ (1)

where $F_{i}^{l}$ (resp. $F_{i}^{c}$) is the feature map of target image
$I_{1}$ (resp. $I_{2}$) at scale $i$.

To measure the image alignment across different modalities, we convert the
upsampled reference image $I_{2}^{\uparrow}$ from RGB color space to YUV [43]
color space. Thus, $I_{2}^{\uparrow}$ can be separated into a luminance
component and two chrominance components. Discarding chrominance components,
we retain the luminance components denoted as $I^{Y}_{2}$, which is a
grayscale counterpart of the upsampled reference image $I_{2}^{\uparrow}$.

Then the luminance encoder is also used to extract multi-scale feature maps of
$I^{Y}_{2}$, which is utilized latter in Sec. 3.2 to calculate warping errors
on feature domain:

$\displaystyle\begin{aligned} \\{F_{i}^{Y}\\}={Net}_{E}^{l}(I^{Y}_{2}),\ \ \
i=1,2,3,4.\end{aligned}$ (2)

#### Cross-camera Flow Estimation.

For image alignment, we adopt the widely used FlowNetS [12] as our flow
estimator, ${Net}_{flow}$, to generate the dense cross-camera correspondence
at multiple scales. Tailored for our setting, we change the input channel
number of the first convolutional layer of FlowNetS from $6$ to $4$, and
obtain the following flow fields:

$\displaystyle\begin{aligned}
\\{f_{i}\\}={{Net}_{flow}}(I_{1},I_{2}^{\uparrow}),\ \ \
i=0,1,2,3,4,\end{aligned}$ (3)

where $f_{i}$ is the estimated flow field at scale $i$.

#### Image and Features Warping.

To perform the temporal and spatial alignment with the estimated flow fields,
we utilize a warping operation similar to [62]. More specifically, our warping
operation considers the cross-camera flow field $f$:

$\displaystyle\begin{aligned} \widetilde{I}=\mathcal{W}(I,f),\end{aligned}$
(4)

where $\mathcal{W}(I,f)$ denotes the result of warping the input $I$ using the
flow field $f$.

After flow estimation, we perform the warping operation on the reference image
features $F_{i}^{c}$ and corresponding luminance features $F_{i}^{Y}$. Using
the multi-scale flow $f_{i}$ in Eq. 3, we generate the temporally and
spatially aligned features $\\{\widetilde{F}_{i}^{c}\\}$ and
$\\{\widetilde{F}_{i}^{Y}\\}$:

$\displaystyle\begin{aligned}
&\widetilde{F}_{i}^{c}=\mathcal{W}(F_{i}^{c},f_{i}),&\widetilde{F}_{i}^{Y}=\mathcal{W}(F_{i}^{Y},f_{i}),\
\ \ i=1,2,3,4.\end{aligned}$ (5)

To measure image alignment latter in Sec. 3.2, we also perform the warping
operation on the image domain. According to Eq. 4, we have:

$\displaystyle\begin{aligned}
\widetilde{I}^{Y}_{2}=\mathcal{W}(I^{Y}_{2},f_{0}),\end{aligned}$ (6)

where $f_{0}$ is the estimated flow field at scale $0$ in Eq. 3,
$\widetilde{I}^{Y}_{2}$ is the warping result utilized latter in Sec. 3.2 to
calculate warping errors on the image domain.

###  3.2 Hierarchical Fusion Module

#### Visibility Maps Computation.

On the image domain, the warping error indicates the different light intensity
between the target and the reference, _i.e._ , optical visibility. While on
the feature domain, warping error represents the different activation value of
the feature maps, _i.e._ , feature recognition. We combine the errors from
both perspectives and define the multi-scale visibility maps $\\{V_{i}\\}$ as
warping errors on both domains:

$\displaystyle\begin{aligned}
&V_{0}=\widetilde{I}^{Y}_{2}-I_{1},&V_{i}=\widetilde{F}_{i}^{Y}-{F_{i}^{l}},\
\ \ i=1,2,3,4,\end{aligned}$ (7)

where $V_{i}$ is the warping error at scale $i$.

Figure 3: Visualizing the visibility map on image domain. Red: invisible
regions in the reference image; green: invisible regions in the target image.
Zoom in to see details.

To give an intuition of the visibility maps, we visualize it on the image
domain in Fig. 3. There are lost details caused by the low-quality reference
(_e.g._ , blurry words shown on the top of Fig. 3), motion blur caused by
fast-moving objects (_e.g._ , waving hand shown on the bottom of Fig. 3),
local occlusion caused by motion or parallax (_e.g._ , the garment occluded by
the hand shown on the bottom of Fig. 3). The pixel-wise positive and negative
values of visibility maps represent the invisible regions in the reference and
the target image, respectively.

#### Feature Fusion.

In the end, we design a U-Net [36] like decoder to fuse feature maps and
visibility maps, and synthesize the colorization result. As shown on the right
of Fig. 2, the target features $\\{F_{i}^{l}\\}$, warped reference features
$\\{\widetilde{F}_{i}^{c}\\}$ and visibility maps $\\{V_{i}\\}$ are
concatenated as the input of fusion decoder. Finally we obtain the result
$I_{c}$ by:

$\displaystyle\begin{aligned}
I_{c}={Net}_{D}(\\{F_{i}^{l}\\},\\{\widetilde{F}_{i}^{c}\\},\\{V_{j}\\}),\ \ \
i=1,2,3,4,j=0,1,2,3,4.\end{aligned}$ (8)

###  3.3 Loss Function

We use two loss functions: warping loss and colorization loss. The former
encourages the flow estimator to generate precise cross-camera correspondence
for image alignment. The latter is responsible for the final synthesized
image.

#### Warping Loss.

Since the ground-truth flow is unavailable, it is difficult to train the flow
estimator in an unsupervised fashion. To solve this problem, we adopt the
warping loss from [42]. Specifically, since the input images capture the same
scene, it is reasonable to require the warped-upsampled reference image owning
a intensity distribution similar to the ground truth ${I_{g}}$ as much as
possible. Thus our warping loss is defined as:

$\displaystyle\begin{aligned}
&\mathcal{L}_{w}=\frac{1}{2N}\sum_{i=1}^{N}\sum_{j}{||{\widetilde{I}^{\uparrow}_{2}}{}^{(i)}-{I}_{g}^{(i)}||}_{2}^{2},&\widetilde{I}^{\uparrow}_{2}=\mathcal{W}(I_{2}^{\uparrow},f_{0}),\end{aligned}$
(9)

where $N$ is the sample number, $i$ iterates over training samples, $f_{0}$ is
the estimated flow field at scale $0$ in Eq. 3.

#### Colorization Loss.

Given the network prediction $\widetilde{I_{c}}$ and the ground truth $I_{c}$,
the colorization loss is defined as:

$\displaystyle\begin{aligned}
\mathcal{L}_{c}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j}\rho(I_{c}^{(i)}-{I}_{g}^{(i)}),\end{aligned}$
(10)

where $\rho(x)=\sqrt{x^{2}+{0.001}^{2}}$ is the Charbonnier penalty function
[2], $\widetilde{I}_{c}$ is obtained from Eq. 8, $N$ is the sample number, $i$
iterates over training samples.

Figure 4: Colorization comparisons on Vimeo90K dataset under cross-scale
$8\times$ settings.  Figure 5: Comparison under cross-scale $8\times$ settings
on Flower (the 1st, 2nd, 3rd rows) and LFVideo (the last row) dataset,
respectively.

##  4 Experiments

###  4.1 Dataset

#### Video Dataset.

The Vimeo90K [51] dataset consists of videos that cover a large variety of
scenes and actions. Following [51], we selected $66,178$ video sequences, each
of which contains $7$ frames with resolution of $448\times 256$. To construct
the training and testing datasets, we randomly divide it into $60,000$
sequences for training and $6,178$ sequences for testing. For training, we
downsample the first frame at each video sequence as a reference image and
randomly select the $t$th$(1<t<8)$ frame converted to the grayscale image as a
target. For testing, reference images are sampled from the first video frame,
while the target images are from the second and the last frame. See the
supplementary material for training details.

#### Light-field Dataset.

The Flower dataset [39] contains flowers and plants light-field images with
the $376\times 541$ spatial resolution, and $14\times 14$ angular samples.
Following [39], we extract the central $8\times 8$ grid of angular sample, and
randomly selected 343 samples for evaluations. The images at viewpoints
$(1,1)$ and $(7,7)$ are converted to grayscale as target, and images at
viewpoint $(0,0)$ are downsampled as reference.

#### Light-field Video Dataset.

The LFVideo dataset [45] contains real-scene light-field videos with the
spatial resolution as $376\times 541$, while the angular samples are $8\times
8$. For evaluations, we randomly selected $270$ video frames. The images of
the $t$th$(t=2,9)$ frame at viewpoints $(i,i),i=1,7$ are converted to
monochrome as target, and images of the first frame at viewpoint $(0,0)$ are
downsampled as reference.

###  4.2 Comparison to State-of-the-Art Methods

Table 1: Quantitative evaluations of the state-of-the-art automatic and
example-based algorithms on different datasets, in terms of
NRMSE/PSNR/SSIM/LPIPS for different scale factors, frame gaps and parallax
settings respectively.

Dataset | Target Position | Scale | Methods | Quantitative evaluations
---|---|---|---|---
Frame | View | NRMSE | PSNR | SSIM | LPIPS | Runtime
Vimeo | 2 | - | N/A | MemoPainter-RGB [53] | 0.3463 | 19.9920 | 0.7989 | 0.4206 | 0.5124
2 | - | N/A | MemoPainter-dist [53] | 0.2352 | 22.5768 | 0.8751 | 0.3015 | 0.5620
2 | - | N/A | ChromaGAN [44] | 0.2097 | 23.5276 | 0.8697 | 0.2917 | 2.6915
2 | - | $4\times$ | Deep Image Analogy [27] | 0.1544 | 25.6775 | 0.7741 | 0.3184 | 219.5779
2 | - | $4\times$ | DEPN [50] | 0.1313 | 27.5916 | 0.9313 | 0.1585 | 0.3425
2 | - | $4\times$ | Ours | 0.0227 | 43.2263 | 0.9884 | 0.0157 | 0.0838
2 | - | $8\times$ | DEPN [50] | 0.1316 | 27.5724 | 0.9310 | 0.1588 | 0.3525
2 | - | $8\times$ | Deep Image Analogy [27] | 0.1951 | 23.5943 | 0.6954 | 0.4142 | 203.4619
2 | - | $8\times$ | Ours | 0.0275 | 41.6039 | 0.9845 | 0.0241 | 0.0847
7 | - | N/A | MemoPainter-RGB [53] | 0.3150 | 20.0014 | 0.7995 | 0.4201 | 0.5301
7 | - | N/A | MemoPainter-dist [53] | 0.2326 | 22.4321 | 0.8738 | 0.3028 | 0.5483
7 | - | N/A | ChromaGAN [44] | 0.2064 | 23.5428 | 0.8696 | 0.2909 | 2.7384
7 | - | $4\times$ | Deep Image Analogy [27] | 0.2063 | 23.0956 | 0.7227 | 0.3529 | 214.3759
7 | - | $4\times$ | DEPN [50] | 0.1306 | 27.5174 | 0.9313 | 0.1593 | 0.3548
7 | - | $4\times$ | Ours | 0.0380 | 39.5223 | 0.9823 | 0.0321 | 0.0846
7 | - | $8\times$ | Deep Image Analogy [27] | 0.2356 | 21.8671 | 0.66771 | 0.4260 | 220.8749
7 | - | $8\times$ | DEPN [50] | 0.1309 | 27.4989 | 0.9311 | 0.1596 | 0.3478
7 | - | $8\times$ | Ours | 0.0408 | 38.6736 | 0.9796 | 0.0382 | 0.0891
Flower | - | (1,1) | N/A | MemoPainter-RGB [53] | 0.4172 | 22.0304 | 0.7508 | 0.3389 | 0.5623
- | (1,1) | N/A | MemoPainter-dist [53] | 0.3237 | 24.2114 | 0.8822 | 0.2668 | 0.6184
- | (1,1) | N/A | ChromaGAN [44] | 0.3046 | 24.4863 | 0.8407 | 0.3081 | 1.1562
- | (1,1) | $4\times$ | Deep Image Analogy [27] | 0.1874 | 28.6252 | 0.8065 | 0.2581 | 411.6000
- | (1,1) | $4\times$ | DEPN [50] | 0.1797 | 29.0692 | 0.9286 | 0.1711 | 0.4899
- | (1,1) | $4\times$ | Ours | 0.0205 | 45.9354 | 0.9938 | 0.0041 | 0.1173
- | (1,1) | $8\times$ | Deep Image Analogy [27] | 0.2620 | 25.6807 | 0.6976 | 0.4029 | 404.4650
- | (1,1) | $8\times$ | DEPN [50] | 0.1797 | 29.0730 | 0.9285 | 0.1710 | 0.4729
- | (1,1) | $8\times$ | Ours | 0.0255 | 43.8623 | 0.9923 | 0.0077 | 0.1171
- | (7,7) | N/A | MemoPainter-RGB [53] | 0.4216 | 21.9233 | 0.7525 | 0.3424 | 0.5912
- | (7,7) | N/A | MemoPainter-dist [53] | 0.3158 | 24.4801 | 0.8858 | 0.2617 | 0.6034
- | (7,7) | N/A | ChromaGAN [44] | 0.3065 | 24.4616 | 0.8404 | 0.3081 | 1.5492
- | (7,7) | $4\times$ | Deep Image Analogy [27] | 0.2532 | 25.9906 | 0.7100 | 0.2954 | 409.2839
- | (7,7) | $4\times$ | DEPN [50] | 0.1775 | 29.2156 | 0.9300 | 0.1663 | 0.4725
- | (7,7) | $4\times$ | Ours | 0.0237 | 44.8138 | 0.9931 | 0.0060 | 0.1170
- | (7,7) | $8\times$ | Deep Image Analogy [27] | 0.2903 | 24.7787 | 0.6703 | 0.4006 | 405.3829
- | (7,7) | $8\times$ | DEPN [50] | 0.1774 | 29.2186 | 0.9299 | 0.1663 | 0.4793
- | (7,7) | $8\times$ | Ours | 0.0276 | 43.2567 | 0.9920 | 0.0092 | 0.1175
LFVideo | 2 | (1,1) | N/A | MemoPainter-RGB [53] | 0.4190 | 21.4680 | 0.7227 | 0.3470 | 0.6633
2 | (1,1) | N/A | MemoPainter-dist [53] | 0.3317 | 23.7711 | 0.8565 | 0.2181 | 0.6927
2 | (1,1) | N/A | ChromaGAN [44] | 0.1982 | 27.6557 | 0.8592 | 0.2120 | 0.5773
2 | (1,1) | $4\times$ | Deep Image Analogy [27] | 0.3304 | 23.2553 | 0.6374 | 0.4040 | 755.4143
2 | (1,1) | $4\times$ | DEPN [50] | 0.1129 | 32.7250 | 0.9668 | 0.0933 | 0.4824
2 | (1,1) | $4\times$ | Ours | 0.0459 | 41.0111 | 0.9857 | 0.0288 | 0.1173
2 | (1,1) | $8\times$ | Deep Image Analogy [27] | 0.3695 | 22.3074 | 0.5970 | 0.4944 | 748.9276
2 | (1,1) | $8\times$ | DEPN [50] | 0.1129 | 32.7220 | 0.9669 | 0.0933 | 0.4793
2 | (1,1) | $8\times$ | Ours | 0.0492 | 40.3612 | 0.9849 | 0.0312 | 0.1167
2 | (7,7) | N/A | MemoPainter-RGB [53] | 0.4261 | 21.3552 | 0.7125 | 0.3527 | 0.6325
2 | (7,7) | N/A | MemoPainter-dist [53] | 0.3058 | 24.4653 | 0.8662 | 0.2037 | 0.6429
2 | (7,7) | N/A | ChromaGAN [44] | 0.1959 | 27.8429 | 0.8609 | 0.2097 | 0.5826
2 | (7,7) | $4\times$ | Deep Image Analogy [27] | 0.3394 | 22.9054 | 0.6315 | 0.4041 | 751.4360
2 | (7,7) | $4\times$ | DEPN [50] | 0.1104 | 32.9745 | 0.9679 | 0.0880 | 0.4529
2 | (7,7) | $4\times$ | Ours | 0.0448 | 41.0335 | 0.9859 | 0.0274 | 0.1185
2 | (7,7) | $8\times$ | Deep Image Analogy [27] | 0.3718 | 22.0710 | 0.5977 | 0.4920 | 753.5917
2 | (7,7) | $8\times$ | DEPN [50] | 0.11070 | 32.9560 | 0.9680 | 0.0883 | 0.4826
2 | (7,7) | $8\times$ | Ours | 0.0480 | 40.4257 | 0.9853 | 0.0301 | 0.1179
9 | (1,1) | N/A | MemoPainter-RGB [53] | 0.4501 | 21.1598 | 0.6991 | 0.3608 | 0.6530
9 | (1,1) | N/A | MemoPainter-dist [53] | 0.3290 | 23.7217 | 0.8582 | 0.2204 | 0.6498
9 | (1,1) | N/A | ChromaGAN [44] | 0.2117 | 27.5192 | 0.8512 | 0.2244 | 0.5728
9 | (1,1) | $4\times$ | Deep Image Analogy [27] | 0.3270 | 23.5479 | 0.6404 | 0.4102 | 749.7418
9 | (1,1) | $4\times$ | DEPN [50] | 0.1165 | 32.5500 | 0.9670 | 0.0917 | 0.4692
9 | (1,1) | $4\times$ | Ours | 0.0508 | 40.7632 | 0.9853 | 0.0323 | 0.1169
9 | (1,1) | $8\times$ | Deep Image Analogy [27] | 0.3709 | 22.4716 | 0.5979 | 0.4954 | 747.7591
9 | (1,1) | $8\times$ | DEPN [50] | 0.1166 | 32.5409 | 0.9671 | 0.0919 | 0.4792
9 | (1,1) | $8\times$ | Ours | 0.0538 | 40.1106 | 0.9845 | 0.0349 | 0.1167
9 | (7,7) | N/A | MemoPainter-RGB [53] | 0.4104 | 21.9786 | 0.7211 | 0.3464 | 0.6009
9 | (7,7) | N/A | MemoPainter-dist [53] | 0.3271 | 23.9009 | 0.8628 | 0.2170 | 0.6947
9 | (7,7) | N/A | ChromaGAN [44] | 0.2112 | 27.5598 | 0.8542 | 0.2248 | 0.5927
9 | (7,7) | $4\times$ | Deep Image Analogy [27] | 0.3221 | 23.7166 | 0.6447 | 0.4058 | 747.9275
9 | (7,7) | $4\times$ | DEPN [50] | 0.1130 | 32.7364 | 0.9676 | 0.0882 | 0.4865
9 | (7,7) | $4\times$ | Ours | 0.0492 | 40.8788 | 0.9858 | 0.0304 | 0.1420
9 | (7,7) | $8\times$ | Deep Image Analogy [27] | 0.3564 | 22.8959 | 0.6035 | 0.4921 | 750.8465
9 | (7,7) | $8\times$ | DEPN [50] | 0.1132 | 32.7194 | 0.9678 | 0.0881 | 0.4539
9 | (7,7) | $8\times$ | Ours | 0.0518 | 40.3270 | 0.9851 | 0.0328 | 0.1453

Our method is compared against the state-of-the-art example-based colorization
methods, namely Deep Image Analogy [27] and DEPN [50], and the recent
automatic colorization approaches, including MemoPainter [53] and ChromaGAN
[44]. Following [53], we evaluated MemoPainter using RGB information and color
distribution for color features, respectively. Both the quantitative
evaluation in Table 1 and the qualitative results in Figs. 4 and 5 suggest
that our performance is far beyond that of the baselines.

Our quantitative evaluation involves four image quality metrics: NRMSE, PSNR,
SSIM [48], and LPIPS [57]. Table 1 shows quantitative comparisons, and it is
evident that our method outperforms the baselines by a large margin in most of
the settings, including combinations of different scale factors, frame gaps,
and parallax. In general, our method achieves approximately 10dB gain of PSNR
upon the baselines.

We provide qualitative comparisons in Figs. 4 and 5 respectively on the
Vimeo90k, Flower, LFVideo datasets under the challenging scale $8\times$,
largest parallax and largest frame gap setting. Firstly, benefiting from the
reference image, the example-based approaches show better results than the
automatic colorization approaches. Moreover, among the example-based
approaches, our method generates the most coherent images with less color
bleeding effects, showing the clear advantage of our algorithm.

Figure 6: The PSNR performance of the different variants with some of the
components disabled during training on Vimeo90K dataset.

###  4.3 Ablation Study

This section investigates the role of proposed visibility maps and warping
loss by two variants of our pipeline, which respectively turn off one of the
components while remaining others. In particular, we run our test on the
Vimeo90k dataset, with resolution gaps being $4\times$ and $8\times$.
According to Fig. 6, disabling either component leads to a performance drop,
suggesting the necessities of both.

#### Visibility Maps

When visibility maps are disabled, slight degradation in PSNR is observed in
Fig. 6. Also, as shown in Fig. 3, artifacts are more evident without
visibility maps, suggesting that visibility maps can reduce the wrong
colorization due to inconsistent visibility between input images.

#### Warping Loss

From Fig. 6, the performance of our network drops if warping loss is remove.
We speculate that it is because the colorization loss is for image synthesis
and does not explicitly define terms for flow estimation. In contrast, the
warping loss is necessary to regularize the flow estimator training and enable
better convergence.

##  5 Conclusion

We propose a novel convolutional neural network to facilitate the trade-off
between spatial-temporal resolution and color depth in the imaging system. Our
method fuses data to obtain an HR color image given cross-domain and cross-
scale input pairs captured by the color-plus-mono dual camera. In contrast to
previous works, our method can adapt to color and monochrome cameras with the
various spatial-temporal resolution; thus it enables more flexible and
generalized imaging systems. The key ingredient of our method is a cross-
camera alignment module that generates multi-scale corre-spondence for cross-
domain image alignment. Experiments on several datasets demonstrate the
superior performance of our method (around 10dB in PSNR) compared to state-of-
the-art methods.

## References

  * [1] Brady, D.J., et al.: Multiscale gigapixel photography. Nature 486(7403), 386–389 (2012)
  * [2] Bruhn, A., Weickert, J., Schnörr, C.: Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. International journal of computer vision 61(3), 211–231 (2005)
  * [3] Bugeau, A., et al.: Variational exemplar-based image colorization. IEEE TIP (2013)
  * [4] Cao, X., Tong, X., Dai, Q., Lin, S.: High resolution multispectral video capture with a hybrid camera system. In: CVPR 2011. pp. 297–304. IEEE (2011)
  * [5] Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: ECML PKDD. pp. 151–166. Springer (2017)
  * [6] Chang, H., et al.: Palette-based photo recoloring. ACM Trans. Graph. (2015)
  * [7] Charpiat, G., et al.: Automatic image colorization via multimodal predictions. In: ECCV. Springer (2008)
  * [8] Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: ICCV. pp. 415–423 (2015)
  * [9] Chia, A.Y.S., et al.: Semantic colorization with internet images. ACM TOG (2011)
  * [10] Cossairt, O.S., et al.: Gigapixel computational imaging. In: ICCP. IEEE (2011)
  * [11] Dong, X., Li, W.: Shoot high-quality color images using dual-lens system with monochrome and color cameras. Neurocomputing 352, 22–32 (2019)
  * [12] Dosovitskiy, A., et al.: Flownet: Learning optical flow with conv networks. In: ICCV (2015)
  * [13] Goodfellow, I., et al.: Generative adversarial nets. In: NeuIPS. pp. 2672–2680 (2014)
  * [14] Gupta, R.K., et al.: Image colorization using similar images. In: ACM MM (2012)
  * [15] HaCohen, Y., Shechtman, E., Goldman, D.B., Lischinski, D.: Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30(4), 1–10 (2011)
  * [16] He, M., Liao, J., Yuan, L., Sander, P.V.: Neural color transfer between images. arXiv (2017)
  * [17] He, M., et al.: Deep exemplar-based colorization. ACM TOG (2018)
  * [18] Huang, Y.C., et al.: An adaptive edge detection based colorization algorithm and its applications. In: ACM MM (2005)
  * [19] Ilg, E., et al.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: CVPR. pp. 2462–2470 (2017)
  * [20] Ironi, R., et al.: Colorization by example. In: Rendering Techniques. Citeseer (2005)
  * [21] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR. pp. 1125–1134 (2017)
  * [22] Jancsary, J., Nowozin, S., Sharp, T., Rother, C.: Regression tree fields—an efficient, non-parametric approach to image labeling problems. In: CVPR. pp. 2376–2383. IEEE (2012)
  * [23] Jeon, H.G., et al.: Stereo matching with color and monochrome cameras in low-light conditions. In: CVPR (2016)
  * [24] Jin, D., et al.: All-in-depth via cross-baseline light field camera. In: ACM MM (2020)
  * [25] Levin, A., et al.: Colorization using optimization. In: ACM SIGGRAPH 2004 Papers (2004)
  * [26] Li, G., et al.: Zoom in to the details of human-centric videos. In: ICIP. IEEE (2020)
  * [27] Liao, J., et al.: Visual attribute transfer through deep image analogy. arXiv (2017)
  * [28] Liu, C., et al.: Sift flow: Dense correspondence across scenes and its applications. IEEE TPAMI (2010)
  * [29] Liu, C., Shan, J., Liu, G.: High resolution array camera (Apr 19 2016), uS Patent 9,319,585
  * [30] Liu, X., et al.: Intrinsic colorization. In: ACM SIGGRAPH Asia 2008 papers, pp. 1–9 (2008)
  * [31] Lohmann, et al.: Space–bandwidth product of optical signals and systems. JOSA A (1996)
  * [32] Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Eurographics conference on Rendering Techniques. pp. 309–320 (2007)
  * [33] Ma, C., Cao, X., Tong, X., Dai, Q., Lin, S.: Acquisition of high spatial and spectral resolution video with a hybrid camera system. IJCV 110(2), 141–155 (2014)
  * [34] Mantzel, W., et al.: Shift-and-match fusion of color and mono images (2017), uS Patent
  * [35] Reinhard, E., et al.: Color transfer between images. IEEE Computer graphics and applications 21(5), 34–41 (2001)
  * [36] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
  * [37] Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. IJCV (2015)
  * [38] Sharif, S., Jung, Y.J.: Deep color reconstruction for a sparse color sensor. Optics express 27(17), 23661–23681 (2019)
  * [39] Srinivasan, P.P., et al.: Learning to synthesize a 4d rgbd light field from a single image. In: ICCV. pp. 2243–2251 (2017)
  * [40] Tai, Y.W., Jia, J., Tang, C.K.: Local color transfer via probabilistic segmentation by expectation-maximization. In: CVPR. vol. 1, pp. 747–754. IEEE (2005)
  * [41] Tai, Y.W., et al.: Image/video deblurring using a hybrid camera. In: CVPR. IEEE (2008)
  * [42] Tan, Y., Zheng, H., Zhu, Y., Yuan, X., Lin, X., Brady, D., Fang, L.: Crossnet++: Cross-scale large-parallax warping for reference-based super-resolution. IEEE TPAMI (2020)
  * [43] Union, I.T.: Encoding parameters of digital television for studios. CCIR Recommend. (1992)
  * [44] Vitoria, P., Raad, L., Ballester, C.: Chromagan: An adversarial approach for picture colorization. arXiv preprint arXiv:1907.09837 (2019)
  * [45] Wang, T.C., et al.: Light field video capture using a learning-based hybrid imaging system. ACM TOG (2017)
  * [46] Wang, X., et al.: Panda: A gigapixel-level human-centric video dataset. In: CVPR (2020)
  * [47] Wang, Y., Liu, Y., Heidrich, W., Dai, Q.: The light field attachment: Turning a dslr into a light field camera using a low budget camera ring. IEEE TVCG 23(10), 2357–2364 (2016)
  * [48] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)
  * [49] Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. In: annual conference on Computer graphics and interactive techniques. pp. 277–280 (2002)
  * [50] Xiao, C., Han, C., Zhang, Z., others, G., He, S.: Example-based colourization via dense encoding pyramids. In: Computer Graphics Forum. Wiley Online Library (2020)
  * [51] Xue, T., et al.: Video enhancement with task-oriented flow. IJCV (2019)
  * [52] Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE transactions on image processing 15(5), 1120–1129 (2006)
  * [53] Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: Few-shot colorization via memory augmented networks. In: CVPR. pp. 11283–11292 (2019)
  * [54] Yuan, X., Fang, L., Dai, Q., Brady, D.J., Liu, Y.: Multiscale gigapixel video: A cross resolution image matching and warping approach. In: ICCP. pp. 1–9. IEEE (2017)
  * [55] Yuan, X., et al.: A modular hierarchical array camera. Light: Science & Applications (2021)
  * [56] Zhang, J., Zhu, T., Zhang, A., et al.: Multiscale-vr: Multiscale gigapixel 3d panoramic videography for virtual reality. In: ICCP. pp. 1–12. IEEE (2020)
  * [57] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR. pp. 586–595 (2018)
  * [58] Zhang, R., et al.: Colorful image colorization. In: ECCV. pp. 649–666. Springer (2016)
  * [59] Zhao, Y., Li, G., Wang, Z., Lam, E.Y.: Cross-camera human motion transfer by time series analysis. arXiv preprint arXiv:2109.14174 (2021)
  * [60] Zhao, Y., et al.: Efenet: Reference-based video super-resolution with enhanced flow estimation. In: CICAI. pp. 371–383. Springer (2021)
  * [61] Zhao, Y., et al.: Manet: Improving video denoising with a multi-alignment network. arXiv preprint arXiv:2202.09704 (2022)
  * [62] Zheng, H., Ji, M., Wang, H., Liu, Y., Fang, L.: Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In: ECCV. pp. 88–104 (2018)
  * [63] Zhou, B., et al.: Learning deep features for scene recognition using places database. In: NeuIPS. pp. 487–495 (2014)