# Learning Visuo-Haptic Skewering Strategies
for Robot-Assisted Feeding

Priya Sundaresan
Stanford University
<EMAIL_ADDRESS>Suneel Belkhale
Stanford University
<EMAIL_ADDRESS>Dorsa Sadigh
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Acquiring food items with a fork poses an immense challenge to a robot-
assisted feeding system, due to the wide range of material properties and
visual appearances present across food groups. Deformable foods necessitate
different skewering strategies than firm ones, but inferring such
characteristics for several previously unseen items on a plate remains
nontrivial. Our key insight is to leverage visual and haptic observations
during interaction with an item to rapidly and reactively plan skewering
motions. We learn a generalizable, multimodal representation for a food item
from raw sensory inputs which informs the optimal skewering strategy. Given
this representation, we propose a zero-shot framework to sense visuo-haptic
properties of a previously unseen item and reactively skewer it, all within a
single interaction. Real-robot experiments with foods of varying levels of
visual and textural diversity demonstrate that our multimodal policy
outperforms baselines which do not exploit both visual and haptic cues or do
not reactively plan. Across 6 plates of different food items, our proposed
framework achieves 71% success over 69 skewering attempts total. Supplementary
material, datasets, code, and videos can be found on our website.

> Keywords: Assistive Feeding, Deformable Manipulation, Multisensory Learning

##  1 Introduction

Realizing the full capabilities of assistive robots in the home, hospitals, or
elderly care facilities remains challenging due to the dexterity required to
complete many day-to-day tasks. Eating free-form meals is one such example
with many nuances in perception and manipulation that can be easy to overlook.
However, automating the task of feeding, one of six essential activites of
daily life (ADL) [1], has the potential to improve quality of life for over
one million people in the U.S. who are unable to feed themselves due to upper-
extremity mobility impairment, their caregivers, families with young children
and elders, and anyone impacted by the substantial time and effort required in
meal preparation and feeding [2, 3, 4].

In recent years, there have been significant efforts to tackle the challenging
problem of robot-assisted feeding. Solutions on the market have limited
traction as they rely heavily on pre-programmed trajectories, pre-specified
foods, have limited autonomy, or require manual utensil interchange [5, 6].
Meanwhile, academic research on assistive feeding largely centers around data-
driven methods but has yet to show widespread generalization across food
groups [7, 8, 9, 10]. As a necessary first step towards feeding, we focus on
the problem of _bite acquisition_ — acquiring bite-sized items from a plate or
bowl — using a robot with a fork-equipped end-effector. Developing a bite
acquisition strategy sensitive to differences in _geometry_ and _deformation_
both _across and within_ food classes is a challenging problem: Skewering
position and orientation matters for items with irregular shape, such as a
broccoli floret where skewering at the stem is preferable to the head for
stability of the acquisition. The fragility of food also affects the optimal
skewering strategy, as delicate items such as thin banana slices are more
likely to slip off a fork oriented vertically and instead benefit from an
angled fork insertion and scooping strategy [11]. On the other hand, hard baby
carrots require a vertical insertion angle for effective and stable
acquisition [11]. In addition, instances within the same class of foods can
also exhibit visual similarity but textural contrast (raw vs. boiled carrots,
silken vs. extra firm tofu, cheddar vs. mozzarella cheese); choosing the
optimal skewering strategy therefore depends on more than just vision.

Prior works show that food classification objectives can lead to visual
features that can be used for downstream policy learning [7, 8, 9]. They also
introduce an action taxonomy for skewering to discretize the complex space of
possible acquisition trajectories [7, 8, 9, 12, 13]. Although these visual-
only skewering strategies are able to classify food with different _geometry_
, they lack critical information about _deformation_ and may fail to
differentiate between foods _within_ the same food class that appear similar
but have drastically different properties, such as boiled and raw carrots.

Preliminary experiments suggest that a boiled carrot requires the fork to
skewer the item at an angle to avoid breakage or dropping during acquisition,
whereas a rigid carrot requires a more forceful vertical approach to pierce
the item. Gordon et al. [10] try to address this issue by leveraging post-hoc
haptic feedback to update a visual policy after skewering. This work requires
multiple trials of interaction per unseen food item to reason about item
_deformation_ through haptic feedback. However, repeated skewering attempts
can easily damage fragile items (e.g. overcooked carrots or thin slices of
banana) and potentially change the properties of the food, leading to breakage
or squishing over multiple robot interactions. These repeated interaction
strategies do not scale to unseen food classes either, for the same reasons.
On the other hand, open-loop strategies that do not adapt skewering plans mid-
motion are also limited in their ability to handle unseen items with unknown
properties.

Therefore, bite acquisition methods should be able to zero-shot generalize to
new foods both within and across food classes, just like how humans skewer
bites of food without the need for multiple interactions with the food.

Our key insight is to jointly fuse haptic and visual information during a
single skewering interaction to learn a more robust and generalizable food
item representation. We develop a bite acquisition system along with a visuo-
haptic skewering policy that leverages this learned representation. The
proposed representation informs the skewering policy of both _geometry_ and
_deformation_ food properties on-the-fly, thus enabling a reactive policy
which zero-shot generalizes to unseen plates of food _within and across_ food
classes. Our experiments with a wide range of food items with varying geometry
and deformability demonstrate that our method outperforms those that (1) do
not jointly use haptic and visual cues and (2) do not reactively plan upon
contact, achieving $71\%$ skewering success across $21$ items total. Our
contributions include:

  * •

A skewering system that employs coarse-to-fine visual servoing to approach a
food item, sense multimodal properties upon contact, and reactively plan
skewering in the same continous interaction

  * •

A zero-shot skewering policy that captures geometry and deformation by fusing
visual and haptic information with demonstrated generalization to unseen food
items

  * •

Experimental validation on diverse seen and unseen food items, with varying
degrees of visual likeness and deformation

  * •

An open-source dataset for multimodal food perception and custom end-effector
mount designs, which we hope expands the scope of assistive feeding research

Figure 1: _Left:_ Our method learns zero-shot skewering of food items with a
Franka Panda robot. Given an overhead plate observation, we localize food
items and _probe_ them to reveal haptic and visual data. Using the multimodal
data as input, HapticVisualNet infers the optimal skewering trajectory on the
fly – _angled skewering_ for soft items like banana slices, or _vertical
skewering_ for firm textures such as a broccoli stem.

##  2 Related Work

We build on prior works studying multisensory robot learning both within and
beyond the food domain. In this section, we will discuss related work in
robot-assisted feeding, food manipulation, and more generally multimodal
robotic perception and manipulation.

##### Robot-Assisted Feeding

Feeding can be split into two stages: bite acquisition and bite transfer.
Previous work in bite transfer — transferring an acquired bite to a user’s
mouth for consumption — suggests that transfer is largely contingent upon
acquisition [14, 7, 13]. To enable reliable bite acquisition and thus
transfer, recent acquisition frameworks combine image-based perception [7, 8,
9, 10, 13] — bounding boxes and food pose estimates — with an action space
consisting of parameterized primitives that modulate fork roll/pitch relative
to item geometry. SPANet (Skewering Position Action Network) [8] is one such
forward model mapping food image observations to actions, which has been shown
to reasonably clear plates containing 16 types of fruits and vegetables.
SPANet is trained on 2.5K fork interactions (81 hours of supervision [9, 8])
which does not readily scale to new foods. Follow-up works aim to rapidly
adapt SPANet to unseen food items using a contextual bandit to learn the
optimal primitive selection strategy from real interactions. Approaches
include updating SPANet predictions online by observing the binary outcome of
acquisition attempts on unseen items [9], and additionally haptic time-series
readings recorded post-hoc [10]. A key assumption in [10] is that the visual
context observed _pre-skewer_ and the haptic context observed _post-hoc_ are
equivalent alternate representations for the underlying food state, which does
not always hold (e.g., firm and soft tofu appear to be almost identical but
yield different haptic readings). In our work, we do not restrict ourselves to
this assumption and consider the more general setting where visually similar
items may have different physical properties. In addition, we do not assume
access to repeated interaction trials with the food, and consider a zero-shot
planning setting. Bhattacharjee et al. [12], Song et al. [15] explore
classifying food compliance or skewering outcomes from haptic data from a
single interaction, but delegate reactive planning given these representations
to future work. To address these gaps, we learn a multisensory policy that
learns zero-shot skewering from _pre-skewer_ and _post-contact_ paired images
and haptic readings.

##### Food Manipulation

Recent simulated benchmarks for household food manipulation explore food
preparation, lunch packing, and food storage [16]; pouring water [17]; pile
manipulation for chopped food [18]; and drinking/feeding [19]. While these
works largely abstract away the state space of food, recent work in real
robotic food slicing explores multimodal food representation learning from
interaction using datasets consisting of paired visual, tactile, and audio
information [20, 21, 22, 23, 24]. In particular, Gemici and Saxena [20]
propose to infer haptic properties (‘brittleness’, ‘tensile strength’,
‘plasticity’, etc.) from probing actions that can inform slicing actions, but
does not consider visual properties. Likewise, Zhang et al. [25] use
vibrations from probing interactions to adapt slicing motions. The learned
haptic representations from these work are not directly transferable to
acquisition due to hardware and task differences, but we adopt the notion of
using probing actions to inform skewering in our work.

##### Integrating Vision and Haptics in Robotics

Vision-only manipulation has proven effective in robotic domains such as
semantic grasping [26, 27] and deformable manipulation [28], but contact-rich
tasks such as peg insertion [29] or robotic Jenga [30] have been shown to
benefit from combining vision, force, and proprioception as inputs to a
learned policy [31, 32]. In light of these works, we propose to learn
reactive, visuo-haptic policies for bite acquisition which remains largely
unexplored.

##  3 Method

Our goal is to learn a multisensory manipulation policy which outputs
skewering actions to clear a plate of bite-sized food without any previous
skewering attempts for this food item. We consider foods of varying degrees of
geometric, visual, and textural similarity, all on the same plate. We first
formalize the bite acquisition setting in Section 3.1 and introduce our action
space to tackle the problem in Section 3.2. Next, we discuss our interaction
protocol for sensing visuo-haptic properties of food (Section 3.3), enabling
us to learn a multimodal skewering policy (Sections 3.4-3.5).

###  3.1 Problem Formulation

Figure 2: Skewering action space: fork pitch $\beta$, roll $\gamma$, and
position $(x,y,z).$

At each timestep $t\in 1,\ldots,T$, we assume access to a the current RGB-D
image observation $I_{t}\in\mathbb{R}^{W\times H\times C}$ of a plate of food
and an $N$-length history of haptic readings $H_{t}\in\mathbb{R}^{N\times 6}$,
denoting 6-axis readings from an F/T sensor on a fork-mounted end-effector.
Similar to prior work, we consider an action space parameterized by fork
$(x,y,z)$ position, roll $\gamma$, and pitch $\beta$. We define an action
$a_{t}\in\mathcal{A}$, visualized in Fig. 1, as follows [9, 10]:

$\displaystyle a_{t}=(x,y,z,\Delta z,\gamma,\Delta\beta)$ (1)

At time $t$, the fork starts in position $(x,y,z)$ with pitch
$\beta=0^{\circ}$ and roll $\gamma$ and moves downward $\Delta z<0$ while
optionally tilting $\Delta\beta\geq 0^{\circ}$ to skewer an item (Figure 2).

We use $l_{t}[a_{t}]\in\\{0,1\\}$ to denote the binary loss of executing
action $a_{t}$, where $l_{t}[a_{t}]=1$ denotes failure, for example food
failing to be picked up, slipping off the fork after skewering, or breakage or
damage as a result of skewering. We opt for a binary loss objective as it is
difficult to quantify more nuanced notions of success such as degree of damage
to an item or stability of a skewer.

We aim to learn a policy $\pi_{\theta}(a_{t}\mid I_{t},H_{t})$ that minimizes
$\sum_{t}^{T}l_{t}[a_{t}]$ given no previous interactions. Our policy learns
to map visual and haptic information to a discrete set of skewering primitives
using a small but diverse labelled dataset that can be collected offline. By
conditioning the policy on haptic readings during a short initial contact
period with the the food item, our method HapticVisualNet can extract food
properties that are inaccessible to vision using only a single skewering
interaction. First we outline how we represent the discrete set of primitives
for skewering.

###  3.2 Skewering Action Primitive Parameterization

To successfully skewer a food item, the fork position and roll must align with
the location and orientation of a food item. The fork pitch must also adapt to
the compliance of a food item (i.e. a soft banana slice favors an _angled_
fork approach to prevent slip, while a raw carrot favors a _vertical_ approach
for piercing). Thus, our action space employs two primitives, vertical skewer
or angled skewer, to account for rigid or compliant items, respectively. We
implement vertical skewer with $\Delta\beta=0^{\circ}$, denoting no tilt
during skewering, and angled skewer with $\Delta\beta>0^{\circ}$, where the
fork gradually tilts from vertical to an angled approach during insertion into
a fragile food item (Figure 2). Prior work includes both angled and vertical
skewering strategies amongst an even larger set of primitives, but we
empirically find that our simplified taxonomy reduces redundancy in this
larger action space and can handle an equivalently broad range of food items,
evaluated in Section 4 [8, 10]. In order to decide between these strategies
on-the-fly, we condition our policy not only on visual information but also on
haptic information at the point of contact with the food item, discussed in
the next section.

###  3.3 Sensing Multimodal Data Via Probing

We introduce a _probing_ motion to obtain visuo-haptic information about a
food item by bringing the fork in contact with the item surface but without
actually skewering it. The multisensory information collected from probing
serves as input to $\pi_{\theta}$ which rapidly decides the skewering
primitive to execute. Between the probing and skewering phases, the fork
remains stationary and in constant contact with the food item, thus enabling a
fluid transition between phases.

Our _probe-then-skewer_ approach requires localizing an item and approaching
with precision so as to not accidentally shift or topple it while making
contact. To accomplish this, we first detect items from a plate image using a
pre-trained RetinaNet food bounding box detector from Gallenberger et al. [7].
Similar to prior work, we also train a network (SkeweringPoseNet) which
refines the estimated item location by predicting a keypoint for the item
center within the local bounding box, and a fork roll angle $\hat{\gamma}$
with which to approach [8]. We can obtain the 3D predicted item location
$(\hat{x},\hat{y},\hat{z})$ in robot frame by using depth information. Next,
we continuously servo to the item using a learned model (ServoNet) which
detects the fork-item offset as keypoints from streamed RGB images. Using this
framework, we probe a food item and record a post-contact image observation
$I_{t}$ and the short window of force magnitude readings $H_{t}$. In Section
3.4, we discuss how these multisensory readings inform the optimal skewering
strategy.

###  3.4 Multimodal Representation and Policy Learning

To learn $\pi_{\theta}(a_{t}\mid I_{t},H_{t})$, we propose HapticVisualNet, a
network which takes observations captured by the probing motion and reactively
outputs the appropriate skewering action. HapticVisualNet takes as input a
post-contact image of a food item $I_{t}\in\mathbb{R}^{W\times H\times 3}$ and
$H_{t}$, the force magnitude readings recorded from the F/T sensor during the
first $N$ milliseconds of contact in the initial probing period.

HapticVisualNet maps $(I_{t},H_{t})$ to an $|\mathcal{A}|$-d vector denoting
the likelihood of success for each action primitive, in our case vertical
skewer and angled skewer. The model first encodes visual information and
haptic information separately, and then concatenates these features to produce
a joint visuo-haptic representation. The policy then predicts action success
likelihood from this representation. We implement HapticVisualNet as a multi-
headed network with a ResNet-18 backbone for the visual encoder and a LSTM for
the haptic encoder. We pass the concatenated visual and haptic encodings to a
linear layer followed by a softmax to obtain primitive successes, and choose
the maximum likelihood predicted primitive as the skewering action.

###  3.5 Training and Data Collection

We train HapticVisualNet on a small but diverse dataset of 300 paired post-
contact images and haptic readings, augmented 8x using image affine and
colorspace transforms as well as temporal scaling and shifting of the haptic
readings. The dataset consists of _hard_ items labeled vertical skewer (raw
carrots / broccoli / zucchini / butternut squash, grapes, cheddar cheese, and
celery) and _soft_ items labeled angled skewer (banana, kiwi, ripe mango,
boiled carrots / broccoli / zucchini / butternut squash, avocado, and
mozzarella cheese).

In practice, we use an $N=$ 26 ms. contact window of haptic readings, from the
initial probing period, which we find adequately captures force-surface
interactions. Using a 20-30 ms. window of contact is also a common choice of
haptic representation in other reactive, contact-rich manipulation settings
[29, 10]. For each paired example, we manually assign the optimal primitive
label — vertical or angled skewer — based on whether the annotator considers
the item hard or soft. This process requires 3 hours of data collection and
labeling time total (a 27x reduction from the 81 hours reported in SPANet
[10]), and _without_ the need for actual skewering attempts during data
collection. We intend for the inclusion of haptic data to prevent overfitting
to visual features, enabling a more generalizable, food representation trained
with greater sample efficiency. Both ServoNet and SkeweringPoseNet are
implemented using a ResNet-18 backbone, each trained on 200 images of the same
items (2 hours of supervision) and augmented to a dataset of 3,500 paired
images and annotations. Additional training and implementation details are in
Appendix C.

In summary, our method leverages offline datasets to learn visuo-haptic
features to more robustly predict between a set of discrete skewering actions.
The probing motion used to obtain haptic information is connected seamlessly
to the chosen skewering motion, leading to one continuous and adaptive zero-
shot skewering policy aware of haptics and vision.

##  4 Evaluation

Figure 3: 6 Plates for Evaluation, covering a wide range of foods. From left
to right: 1: Raw banana, broccoli, zucchini, carrot, grapes, cucumber. 2
(Unseen): Raw pineapple, mango, dragonfruit, canteloupe, honeydew, pear. 3:
Raw butternut squash, boiled butternut squash. 4: Raw broccoli florets. 5
(Unseen): Pasta, dumpling, boiled yam/sweet potato, raw yam/sweet potato. 6
(Unseen): Ice cream mochi, snow peas, canned peaches. Figure 4: Skewering
Success and Failure Modes: We visualize the number of items acquired over
total acquisition attempts for all methods across Plates 1-3. Failure modes
include exceeding the maximum number of consecutive attempts (3+ tries per
item), dropping after skewering, the item being unstable on the fork after
skewering (affecting transfer), damage or breakage to fragile items, or
failure to detect an item due to bounding box anomalies. HapticVisNet (ours)
performs best on each plate, while causing the least failures.

In this section, we seek to evaluate (1) the benefits of combining both vision
and haptics for bite acquisition as opposed to only using a subset of
modalities, (2) the effectiveness of reactive skewering compared to open-loop
strategies, and (3) the generalization capabilities of our system to
previously unseen foods. We first perform classification ablations of
HapticVisualNet in (Section 4.1) and then deploy our system in the real world
for trials on both seen and unseen foods (Section 4.2-4.4).

###  4.1 Ablative Studies

We evaluate the contributions of both visual and haptic data towards primitive
classification by training and evaluating HapticVisualNet against two variants
which observe exclusively the post-contact haptic readings or post-contact
image after probing.

Figure 5: Confusion matrices for classification accuracy for the skewering
primitive (angled skewer or vertical skewer), on a held-out test dataset of 60
images, for each model. Darker off-diagonals and lighter on-diagonals indicate
more accurate models.

Figure 5 shows the confusion matrices for classification accuracy.
HapticVisualNet achieves the highest overall and per-class classification
accuracy, and omitting either modality hurts accuracy. We hypothesize that the
reduced performance of the vision-only model stems from visually similar but
texturally dissimilar foods in our dataset, for which inferring the primitive
is challenging without haptic context. On the other hand, the haptic-only
model learns a naïve solution of mapping high magnitude contact events to
vertical skewering and low readings to angled skewer. This is brittle for
anomalous foods like broccoli which benefit from a vertical skewer, yet may
yield low contact readings if the fork comes in contact at the head instead of
the stem. Similarly, the fork can easily penetrate a thin banana slice and
touch the plate during probing, yielding high contact readings when an angled
skewer is still preferable. In the subsequent sections, we evaluate
HapticVisualNet (HapticVis) against the haptic only (Haptic), vision only
(Vis), and SPANet baselines on real food acquisition trials. See Appendix D
for additional ablations of the learned multimodal representation and
HapticVisualNet sample efficiency.

###  4.2 Hardware Setup

Our setup consists of a 7DoF Franka Emika Panda robot with the default
gripper. We outfit the gripper with a custom 3D-printed mount comprised of a
standard fork, a Mini45 ATI F/T sensor, and a D435i RealSense camera. We
perform all acquisition trials on a plastic dinner plate on an anti-slip
surface. We instantiate each primitive, parameterized according to Equation
(1) as follows, assuming a fixed $\Delta z$ and discretized $\Delta\beta$:

  * •

probe = $(\hat{x},\hat{y},\hat{z}$-APPROACH_HEIGHT, -APPROACH_HEIGHT,
$\hat{\gamma},0^{\circ})$

  * •

vertical skewer = $(\hat{x},\hat{y},\hat{z}$, -DT*0.17m/sec,
$\hat{\gamma},0^{\circ})$

  * •

angled skewer = $(\hat{x},\hat{y},\hat{z}$, -DT*0.08m/sec,
$\hat{\gamma},65^{\circ})$

Here, $(\hat{x},\hat{y},\hat{z})$ denotes a predicted food item location,
obtained by deprojecting a predicted pixel in a depth image from ServoNet to a
3D location. SkeweringPoseNet also predicts the fork roll $\hat{\gamma}$. We
first probe the item starting from an APPROACH_HEIGHT of 0.01cm, observe a
post-contact image $I_{t}$, and record haptic readings
$H_{t}\in\mathbb{R}^{26}$ over the first 26-milliseconds of contact. Given
these inputs, HapticVisualNet infers either vertical skewer or angled skewer
which we execute.

The robot controller runs at 20Hz (DT = $0.05$), and primitives terminate
early if the fork reaches a pre-defined $z$-distance (the plate height) or
force limit. After skewering, the end-effector _scoops_ until the fork is
nearly horizontal with $\beta=80^{\circ}$, emulating the start of a feasible
_transfer_ trajectory.

###  4.3 Baselines

We deploy all methods — HapticVis (ours), Haptic, Vis, and SPANet — on the
real robot setup of Section 4.2 and classification networks trained according
to Section 4.1. The HapticVis, Haptic, and Vis methods all perform _probing-
then-skewering_ , but run inference using both, only haptic, or only visual
sensory information obtained from probing, respectively.

We also implement SPANet given the pre-trained visual models and original
taxonomy of six skewering primitives reported in [8]. SPANet performs zero-
shot primitive inference given an overhead image observation of a food item,
without probing. SPANet still uses our ServoNet to plan, analogous to the
original implementation which similarly accounted for fork-food precision
error.

###  4.4 Real World Bite Acquisition Results

We compare all methods on the challenging task of clearing plates containing
10 bite-sized food items, evaluated according to skewering success and the
distribution of skewering failure modes encountered (Table 1). We define a
skewering success as one in which the fork picks up the item with at least 2
tines inserted and the item remains on the fork for up to 5 seconds after
scooping as in [8]. Failure modes are detailed in Figure (4). Between
successful acquisitions, a human operator removes the acquired item from the
fork. Upon skewering failure, an attempted item remains on the plate and can
be re-attempted up to 3 times before being manually removed and marked as a
failure.

Plates 1-3 – Full System Evaluation: We first perform a full-system evaluation
of HapticVis and all baselines on 3 plates. Plate 1 contains in-distribution
fruits and vegetables that HapticVisualNet was trained on which include both
textural and visual diversity, Plate 2 consists of unseen assorted fruits with
visual diversity but similar textures, and Plate 3 contains in-distribution
boiled and raw butternut squash cubes which appear similar but greatly differ
in softness. When adjusted for perception failures (e.g., bounding box false
negatives) which affect all methods, HapticVis outperforms all methods across
Plates 1-3 (Figure 4). The bulk of HapticVis failures center around near-
misses or perception failures which are less drastic than damaging items or
exceeding skewering attempts (Figure 4). Vis slightly underperforms Haptic
across all plates and achieves lowest performance on Plate 3, where
mispredicted vertical skewer or angled skewer actions can miss/damage soft-
boiled squash, or fail to penetrate hard raw squash. SPANet achieves lowest
performance, mostly due to erroneously executing vertical strategies to pick
up soft items like bananas and ripe dragonfruit in many cases. SPANet’s
underperformance relative to HapticVis, Haptic, and Vis suggests the
effectiveness of reactive strategies compared to open-loop acquisition, and
that our simplified action space is just as expressive as SPANet’s larger
taxonomy. We acknowledge that the performance gap may in part be attributed to
hardware differences (different robot, different fork mount) in the original
SPANet compared to our re-implementation.

Plate 4 – Texturally Misleading Food: While Haptic is the most competitive
baseline on Plates 1-3, we run additional experiments between HapticVis and
Haptic on a plate of only broccoli florets (Plate 4). In cases where the stem
is occluded from view or the fork servos to the leafy region to make contact,
Haptic tends towards misclassifying the low readings as angled skewer, leading
to frequent failures to pierce the item which HapticVis is better equipped to
recognize and avoid.

Plates 5-6 – Generalization to Out-of-Distribution Foods: Finally, we stress-
test the generalization capabilities of HapticVis on two plates of unseen
foods, Plate 5 and Plate 6, containing the items listed in Table 1. Across
Plates 5-6, HapticVis achieves 58% success, a 19% improvement over Vis,
indicating the promise of multimodal representations for zero-shot food
skewering. HapticVis performs best on soft canned pears and boiled/raw root
vegetables which are most comparable to the items in the training
distribution, closely followed by pasta and ice cream mochi. The majority of
failures occur due to the fragility of dumplings and thinness of snow peas
which are difficult to pierce. Still, by fusing haptic and visual information,
HapticVis is better equipped to generalize to visually and texturally diverse
foods. See Appendix E for additional stress-tests of HapticVisualNet.

Plate Type | # Items Acquired / Total Attempts
---|---
_Plate_ | _Items_ | _Visuals_ | _Haptics_ | _Category_ | _HapticVis_ | _Haptic_ | _Vis_ | _SPANet_
1 | Assorted fruits and vegetables | Diverse | Diverse | Seen | 9/10 | 9/12 | 7/10 | 8/20
2 | Assorted tropical fruits | Diverse | Similar | Unseen | 10/11 | 9/11 | 8/13 | 10/14
3 | Boiled/raw butternut squash | Similar | Diverse | Seen | 8/11 | 9/14 | 10/18 | 7/12
4 | Broccoli florets | Similar | Similar | Seen | 8/13 | 7/17 | – | –
5 | Pasta, dumplings, boiled/raw root veggies | Diverse | Diverse | Unseen | 7/11 | – | 9/13 | –
6 | Mochi, snow peas, canned pear | Diverse | Diverse | Unseen | 7/13 | – | 5/23 | –
| 71% | 63% | 51% | 54%

Table 1: Physical Results: We evaluate the ability of all methods to clear 6
plates (Figures 4), each initially containing 10 items. The number of items
acquired refers to items successfully skewered ($\leq 10$ in all cases due to
dropped items or early termination with bounding box false negatives). The
total attempts refers to all attempted acquisitions until termination or
clearance ($\geq 10$ in all cases due to failed items remaining on the plate
for up to 3 consecutive re-attempts). HapticVis outperforms baselines in 5/6
plates. The symbol – denotes experiments that are not useful in the specific
testing scenario.

Overall, HapticVisualNet benefits from the use of both vision and haptics from
just a single interaction, and outperforms single-modality baselines for a
variety of challenging food plates. We show that skewering strategies that
reactively update the strategy (HapticVis, Haptic, Vis) are more robust to
texturally and visually diverse food items than open loop strategies (SPANet).
With its multimodal representation, HapticVisualNet can also zero-shot
generalize to challenging unseen food classes.

##  5 Discussion

Summary In this work, we present a framework for zero-shot food acquisition of
a diverse range of bite-sized items using multimodal representation learning.
Our approach uses interactive probing to sense complex, multisensory food
properties upon contact in order to reactively plan a skewering strategy. We
deploy the learned policy along with a learned visual servoing controller on a
robot for zero-shot skewering of unseen food items. Our experiments span a
wide range of food appearances and textures, and validate the need for
multimodal reasoning and reactive planning to clear plates.

Limitations and Future Work One limitation in our approach is the use of a
small action space for acquisition. While our set of primitives can generalize
to a wide range of foods, our current strategies are not equipped to handle
thin, flat items like finely sliced produce or leafy salad greens, which may
require new techniques like positioning the fork under an item to scoop,
gathering multiple items together before skewering, or using plate walls for
stabilization. Other food groups like noodles would benefit twirling, and
foods with irregular geometries might require toppling into a stable pose
before skewering. Another limitation in this work is the supervision used to
currently train our multimodal policy. However, since our policy learns from
sparse labels, we are excited by the possibility of automatically detecting
skewering outcomes to self-supervise the training procedure for
HapticVisualNet. Finally, in future work we hope to tackle challenging food
groups such as filled dumplings which easily break and extremely thin snow
peas which are hard to pierce.

#### Acknowledgments

This work is supported by NSF Awards 2132847, 2006388, and 1941722 and the
Office of Naval Research (ONR). Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the sponsors. Priya Sundaresan is
supported by an NSF GRFP. We thank our colleagues for the helpful discussions
and feedback, especially Tapomayukh Bhattacharjee, Jennifer Grannen, Lorenzo
Shaikewitz, and Yilin Wu.

## References

  * Katz [1983] S. Katz.  Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living.  _Journal of the American Geriatrics Society_ , 1983.
  * Brault et al. [2012] M. W. Brault et al.  _Americans with disabilities: 2010_.  US Department of Commerce, Economics and Statistics Administration, US …, 2012.
  * Brose et al. [2010] S. W. Brose, D. J. Weber, B. A. Salatin, G. G. Grindle, H. Wang, J. J. Vazquez, and R. A. Cooper.  The role of assistive robotics in the lives of persons with disability.  _American Journal of Physical Medicine & Rehabilitation_, 89(6):509–521, 2010.
  * Maheu et al. [2011] V. Maheu, P. S. Archambault, J. Frappier, and F. Routhier.  Evaluation of the jaco robotic arm: Clinico-economic study for powered wheelchair users with upper-extremity disabilities.  In _2011 IEEE International Conference on Rehabilitation Robotics_ , pages 1–5. IEEE, 2011.
  * [5] Meet Obi.  https://meetobi.com/.  [Online; accessed 6-June-2022].
  * [6] Meal-Mate - Made2Aid.  https://www.made2aid.co.uk/productprofile?productId=8&company=RBF%20Healthcare&product=Meal-Mate.  [Online; accessed 6-June-2022].
  * Gallenberger et al. [2019] D. Gallenberger, T. Bhattacharjee, Y. Kim, and S. S. Srinivasa.  Transfer depends on acquisition: Analyzing manipulation strategies for robotic feeding.  In _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pages 267–276. IEEE, 2019.
  * Feng et al. [2019] R. Feng, Y. Kim, G. Lee, E. K. Gordon, M. Schmittle, S. Kumar, T. Bhattacharjee, and S. S. Srinivasa.  Robot-assisted feeding: Generalizing skewering strategies across food items on a plate.  In _The International Symposium of Robotics Research_ , pages 427–442. Springer, 2019.
  * Gordon et al. [2020] E. K. Gordon, X. Meng, T. Bhattacharjee, M. Barnes, and S. S. Srinivasa.  Adaptive robot-assisted feeding: An online learning framework for acquiring previously unseen food items.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 9659–9666. IEEE, 2020.
  * Gordon et al. [2021] E. K. Gordon, S. Roychowdhury, T. Bhattacharjee, K. Jamieson, and S. S. Srinivasa.  Leveraging post hoc context for faster learning in bandit settings with applications in robot-assisted feeding.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10528–10535. IEEE, 2021.
  * Bhattacharjee et al. [2019a] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa.  Towards robotic feeding: Role of haptics in fork-based food manipulation.  _IEEE Robotics and Automation Letters_ , 4(2):1485–1492, 2019a.  doi:10.1109/LRA.2019.2894592.
  * Bhattacharjee et al. [2019b] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa.  Towards robotic feeding: Role of haptics in fork-based food manipulation.  _IEEE Robotics and Automation Letters_ , 4(2):1485–1492, 2019b.
  * Bhattacharjee et al. [2020] T. Bhattacharjee, E. K. Gordon, R. Scalise, M. E. Cabrera, A. Caspi, M. Cakmak, and S. S. Srinivasa.  Is more autonomy always better? exploring preferences of users with mobility impairments in robot-assisted feeding.  In _2020 15th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pages 181–190. IEEE, 2020.
  * Belkhale et al. [2022] S. Belkhale, E. K. Gordon, Y. Chen, S. Srinivasa, T. Bhattacharjee, and D. Sadigh.  Balancing efficiency and comfort in robot-assisted bite transfer.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 4757–4763. IEEE, 2022.
  * Song et al. [2019] H. Song, T. Bhattacharjee, and S. S. Srinivasa.  Sensing shear forces during food manipulation: resolving the trade-off between range and sensitivity.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8367–8373. IEEE, 2019.
  * Srivastava et al. [2022] S. Srivastava, C. Li, M. Lingelbach, R. Martín-Martín, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al.  Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments.  In _Conference on Robot Learning_ , pages 477–490. PMLR, 2022.
  * Lin et al. [2020] X. Lin, Y. Wang, J. Olkin, and D. Held.  Softgym: Benchmarking deep reinforcement learning for deformable object manipulation.  2020\.
  * Suh and Tedrake [2020] H. Suh and R. Tedrake.  The surprising effectiveness of linear models for visual foresight in object pile manipulation.  In _International Workshop on the Algorithmic Foundations of Robotics_ , pages 347–363. Springer, 2020.
  * Erickson et al. [2020] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp.  Assistive gym: A physics simulation framework for assistive robotics.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10169–10176. IEEE, 2020.
  * Gemici and Saxena [2014] M. C. Gemici and A. Saxena.  Learning haptic representation for manipulating deformable food objects.  In _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 638–645. IEEE, 2014.
  * Zhang et al. [2019] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer.  Leveraging multimodal haptic sensory data for robust cutting.  In _2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids)_ , pages 409–416. IEEE, 2019.
  * Sawhney et al. [2020] A. Sawhney, S. Lee, K. Zhang, M. Veloso, and O. Kroemer.  Playing with food: Learning food item representations through interactive exploration.  In _International Symposium on Experimental Robotics_ , pages 309–322. Springer, 2020.
  * Sharma et al. [2019] M. Sharma, K. Zhang, and O. Kroemer.  Learning semantic embedding spaces for slicing vegetables.  _arXiv preprint arXiv:1904.00303_ , 2019.
  * Heiden et al. [2022] E. Heiden, M. Macklin, Y. Narang, D. Fox, A. Garg, and F. Ramos.  Disect: A differentiable simulator for parameter inference and control in robotic cutting.  _arXiv preprint arXiv:2203.10263_ , 2022.
  * Zhang et al. [2019] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer.  Leveraging multimodal haptic sensory data for robust cutting.  _CoRR_ , abs/1909.12460, 2019.  URL http://arxiv.org/abs/1909.12460.
  * Mahler et al. [2017] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg.  Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.  2017\.
  * Florence et al. [2018] P. R. Florence, L. Manuelli, and R. Tedrake.  Dense object nets: Learning dense visual object descriptors by and for robotic manipulation.  2018\.
  * Ha and Song [2022] H. Ha and S. Song.  Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding.  In _Conference on Robot Learning_ , pages 24–33. PMLR, 2022.
  * Lee et al. [2019] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg.  Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8943–8950. IEEE, 2019.
  * Fazeli et al. [2019] N. Fazeli, M. Oller, J. Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez.  See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion.  _Science Robotics_ , 4(26):eaav3123, 2019.
  * Li et al. [2019] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba.  Connecting touch and vision via cross-modal prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10609–10618, 2019.
  * Lee et al. [2021] M. A. Lee, M. Tan, Y. Zhu, and J. Bohg.  Detect, reject, correct: Crossmodal compensation of corrupted sensors.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 909–916. IEEE, 2021.

Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding
Supplementary Material

Datasets, code, and videos can be found on our website.

##  Appendix A Experimental Hardware

Figure 6: End-Effector Custom Hardware: A) CAD designs for custom end-effector
attachment with inserts for Franka Panda gripper tips and mounting holes for
Intel RealSense D435i, Mini45 ATI F/T sensor, and fork with screw attachment.
B) The 3/D printed full mount, assembled using mounting screws and super glue.

We collect all data and run all experiments using the Franka Panda 7DoF robot
with the custom 3D-printed end-effector mount shown in Figure 6. While the
fork is attached via a screw-in slot, the force of consecutive skewering
attempts in the course of clearing a plate can cause the fork position to
shift slightly within the slot. To address this, we train ServoNet to estimate
the fork-food item offset and continuously servo until the midpoint of the
tines is aligned with the predicted item center (Figure 7).

##  Appendix B Additional Qualitative Results

In this section, we provide additional qualitative results of the full bite
acquisition pipeline: RetinaNet bounding box detection, SkeweringPoseNet for
item pose estimation, ServoNet for pre-probing, and HapticVisualNet for
primitive planning over consecutive actions.

Figure 7: ServoNet: From left to right, we plot the last 8 frames of visual
servoing using ServoNet until the fork tines midpoint (white) is centered with
the predicted item location (orange), across 4 items (zucchini, broccoli,
banana slice, snow pea). Figure 8: HapticVis Action Predictions: We visualize
5 consecutive HapticVis action predictions (left to right) for Plates 1 and
Plates 3. The SkeweringPoseNet predictions per column first illustrate the
predicted skewering points (which are intended to refine bounding box centers
depicted in row 1 to item locations) and secondly the skewering orientations.
In Plate 1, HapticVis correctly infers a sequence of actions to pick up
zucchini, a raw carrot, a broccoli floret, and two slices of banana. In Plate
3, HapticVis skewers several pieces of butternut squash ranging from raw
(1st-3rd, 5th columns) to overcooked (4th column). These intrinstic properties
mostly align with the predicted action predictions, except for one mis-
predicted angled skewer primitive for the first action, which succeeds but
results in an unstable skewer. Plate 3 column 2 is a failed action in which
ServoNet erroneously guides the fork to the wrong location, likely due to the
close proximity of two butternut squash pieces. This causes a drop failure,
but the item is re-attempted and successfully skewered in the next action.

##  Appendix C ServoNet Training Details

We employ ServoNet, a network which continuously estimates the fork-food
offset from images, and precisely guides the fork to a desired item via visual
servoing in order to probe. We implement ServoNet with a fully-convolutional
ResNet-18 backbone. ServoNet takes as input a $200\times 200\times 3$ image
from the end-effector mounted RealSense camera, and outputs a $200\times
200\times 2$ heatmap. The two channels represent a 2-d Gaussian heatmap
centered around the predicted fork pixel and nearest food pixel, respectively.
We take the predicted fork and food pixels to be the argmaxes of these
heatmaps.

To train ServoNet, we annotate 200 images with the corresponding fork and
nearest food item pixels and augment this dataset to 3,500 examples (Figure
9). We train the network using binary cross-entropy loss over the predicted
and ground truth heatmaps. Figure 10 shows some visualizations of the
predicted fork-food heatmaps on unseen images.

Figure 9: ServoNet Dataset Generation: Top row: we annotate 200 images with
the fork tines center (white) and nearest food item center (orange). Bottom
row: we augment this dataset with various colorspace and affine
transformations to yield a dataset of 3,500 examples (shown for the first
strawberry example). Figure 10: ServoNet Predictions: For 8 unseen images, we
visualize the ServoNet Gaussian heatmap predictions. The estimated fork pixels
appear in blue, and the estimated food item pixels are shown in yellow.

##  Appendix D Additional Ablations

###  D.1 Sample Efficiency

The original dataset collected to train HapticVisualNet consists of 300
examples of paired post-contact images, force readings, and manually assigned
primitive labels, augmented 8x artificially (Section 3.5). To understand the
sample efficiency of HapticVisualNet, we ablate for the accuracies of the
network when training on varied amounts of data.

Training Dataset Size (% of orig.) | Overall Acc. | Vertical Skewer Acc. | Angled Skewer Acc.
---|---|---|---
$100\%$ (OURS) | $\mathbf{94.1\%}$ | $\mathbf{95.2\%}$ | $93.1\%$
$75\%$ | $91.8\%$ | $88.3\%$ | $95.7\%$
$50\%$ | $92.1\%$ | $87.6\%$ | $\mathbf{97.8\%}$
$25\%$ | $89.4\%$ | $83.4\%$ | $96.9\%$
$10\%$ | $84.2\%$ | $82.7\%$ | $86.1\%$
Table 2:  HapticVisualNet Accuracy and Sample Efficiency: Training with all
300 of the data points, augmented, yields the highest overall accuracy. We
note a general trend towards lower accuracies as the dataset size decreases,
taking into account that with smaller dataset sizes comes lower state coverage
and higher variance in the accuracies, resulting in the 50% network achieving
high angled skewer accuracy but low vertical skewer accuracy.

###  D.2 Interpretability of HapticVisualNet

Given a paired food image observation and haptic readings, HapticVisualNet
separately encodes visual features and haptic features. The concatenated
features yield a 640-d multimodal embedding. This embedding serves as input to
a final linear layer followed by a softmax activation which yields predicted
primitive likelihoods. To improve interpretability of HapticVisualNet’s
learned food item representation, we visualize the multimodal embeddings for
26 items in the training dataset using 2-d t-SNE projections below.

Figure 11: 2D t-SNE Projection of HapticVisualNet Embeddings: We find that
instances of the same item (i.e. ripe mango) and items with similar textural
properties (i.e. boiled carrot and boiled zucchini, avocado and banana) tend
to cluster. Additionally, HapticVisualNet learns to separate items that favor
vertical skewer (bottom right) from those that favor angled skewer (upper
left) by pushing them apart in latent space (i.e. mozzarella vs. cheddar, soft
tofu vs. firm tofu).

##  Appendix E Additional Physical Experiments

In these set of experiments, we stress-test HapticVisualNet’s capabilities to
generalize to challenging textural and visual food properties. We assess
HapticVisualNet on the tasks of clearing a plate of frozen fruits and a plate
with sauteed vegetables and tofu in sauce.

Items | Skewering Success Rate | Slip/Miss | BBox FP | 3+ Tries
---|---|---|---|---
Frozen mango, pineapple, strawberry | 20/25 [video] | 5 | 0 | 1
Sautéed veggies and tofu | 20/23 [video] | 3 | 3 | 0
Sautéed veggies, tofu, soy sauce | 25/34 [video] | 9 | 6 | 1
Table 3:  HapticVisualNet Stress-Tests: HapticVisualNet successfully skewers
frozen fruit in 20/25 and stir-fried tofu with vegetables in 45/57 total
attempts. Videos of both experiments are available on the website. The
failures observed in the fruit case are slips during probing due to the highly
rigid texture of frozen fruits (and in the case of one strawberry,
consecutively slipping three times on the same item). In this experiment, we
also note that HapticVisualNet predominantly infers vertical skewering for
these items as expected, but occasionally predicts angled tines. This is
possibly due to the effects of thawing and softening over time. In the sautéed
vegetables experiment, HapticVisualNet has more near misses and slips due to
the degree of clutter and oiliness of the surface. As these items are very out
of distribution, this also incurs more bounding box failures leading to early
termination. Still, HapticVisualNet demonstrates generalization to
charred/oiled/sauce-coated/seasoned items of varying levels of doneness.


# Light Field Image Super-Resolution with Transformers

Zhengyu Liang, Yingqian Wang, Longguang Wang, Jungang Yang, Shilin Zhou  This
work was supported in part by the National Natural Science Foundation of China
under Grants 61921001, 61972435, and 61401474. Z. Liang, Y. Wang, L. Wang, J.
Yang, and S. Zhou are with the College of Electronic Science and Technology,
National University of Defense Technology, P. R. China. Emails: {zyliang,
wangyingqian16<EMAIL_ADDRESS>Z. Liang and Y. Wang contribute
equally to this work and are co-first authors. Corresponding author: Jungang
Yang.

###### Abstract

Light field (LF) image super-resolution (SR) aims at reconstructing high-
resolution LF images from their low-resolution counterparts. Although CNN-
based methods have achieved remarkable performance in LF image SR, these
methods cannot fully model the non-local properties of the 4D LF data. In this
paper, we propose a simple but effective Transformer-based method for LF image
SR. In our method, an angular Transformer is designed to incorporate
complementary information among different views, and a spatial Transformer is
developed to capture both local and long-range dependencies within each sub-
aperture image. With the proposed angular and spatial Transformers, the
beneficial information in an LF can be fully exploited and the SR performance
is boosted. We validate the effectiveness of our angular and spatial
Transformers through extensive ablation studies, and compare our method to
recent state-of-the-art methods on five public LF datasets. Our method
achieves superior SR performance with a small model size and low computational
cost. Code is available at https://github.com/ZhengyuLiang24/LFT.

###### Index Terms:

Light field, image super-resolution, transformer

##  I Introduction

Light field (LF) cameras record both intensity and directions of light rays,
and enable many applications such as post-capture refocusing[1, 2], depth
sensing [3, 4], saliency detection[5] and de-occlusion [6, 7]. Since high-
resolution (HR) images are required in various applications, it is necessary
to use the complementary information among different views (i.e., angular
information) to achieve LF image super-resolution (SR).

In the past few years, convolutional neural networks (CNNs) have been the
dominant architectures of LF image SR. Yoon et al. [8] proposed the first CNN-
based method called LFCNN to improve the resolution of LF images. Then, some
works[9, 10, 11, 12, 13] combined CNN-based method with the epipoloar geometry
of LF images to improve the SR performance. Several works [14, 15, 16, 17]
organized LF images into the sub-aperture and macro-pixel patterns to achieve
LF image SR. Meng et al.[18] designed a high-dimensional dense residual
network to learn the geometry information encoded in multiple adjacent views.

Although the SR performance has been continuously improved via delicate
network designs, most existing CNN-based LF image SR methods have the
following two limitations. First, these methods either use part of views to
reduce the complexity of the 4D LF structure [8, 9, 10, 11], or integrate
angular information without considering view position and image content [15,
16, 13]. The under-use of the rich angular information results in performance
degradation especially on complex scenes (e.g., occlusions and non-Lambertain
surfaces). Second, existing CNN-based methods extract spatial features by
applying (cascaded) convolutions on SAIs. The local receptive field of
convolutions hinders these methods to capture long-range spatial dependencies
from input images. In summary, existing CNN-based LF image SR methods cannot
fully exploit both angular and spatial information, and thus face a bottleneck
for further performance improvement.

Recently, Transformers have been demonstrated effective in modeling positional
and long-range correlations, and were applied to various computer vision tasks
such as image classification [19], object detection [20, 21], semantic
segmentation [22], depth estimation [23] and super-resolution[24, 25]. More
research and details about Transformers can be referred to Section II.
Inspired by the recent advances of Transformers, in this paper, we propose a
Transformer-based network (i.e., LFT) to address the aforementioned
limitations of CNN-based methods. Specifically, we design an angular
Transformer to model the relationship among different views, and design a
spatial Transformer to capture both local and non-local context information
within each SAI. Compared to CNN-based methods, our LFT can discriminately
incorporate the information from all angular views, and capture long-range
spatial dependencies in each SAI.

The contributions of this paper can be summarized as: 1) We adapt Transformers
to LF image processing and propose a simple but strong baseline for LF image
SR. 2) We propose a novel paradigm (i.e., angular and spatial Transformers) to
incorporate angular and spatial information in an LF. The effectiveness of our
paradigm is validated through extensive ablation studies. 3) With a small
model size and low computational cost, our LFT achieves superior SR
performance than other state-of-the-art methods.

Figure 1: An overview of our network.

##  II Related Work

###  II-A LF Image SR

Since the CNNs are first introduced to the area of LF image SR [8], CNNs have
been widely used and the performance of reconstruction has been continuously
developed [9, 10, 11, 12, 13, 14, 15, 16, 17, 18]. Yuan et al. [9] applied
EDSR [26] to super-resolve each sub-aperture image (SAI) independently, and
developed an EPI-enhancement network to refine the super-resolved images.
Yeung et al.[14] proposed LFSSR to alternately shuffle LF features between
sub-aperture and macro-pixel patterns for convolution. Zhang et al. [11]
proposed a multi-branch residual network to incorporate the multi-directional
epipolar geometry prior for LF image SR. More recently, Zhang et al. [12]
divided LF images into several image stacks according to multiple epipolar
geometry, and fed them into different branches to effectively learn sub-pixel
information. Since both view-wise angular information and image-wise spatial
information contribute to the SR performance, state-of-the-art CNN-based
methods [13, 15, 16, 17, 18] designed different network structures to leverage
both angular and spatial information for LF image SR. In [15], the spatial and
angular features can be separately extracted from LF images, and then
repetitively interacted to progressively incorporate spatial and angular
information. Jin et al. [13] proposed an all-to-one framework to super-resolve
each view using all the remaining views, and designed a structure-aware loss
to preserve the parallax structure of LF images. Meng et al. [18] designed a
high-dimensional dense residual network (i.e., HDDRNet), and applied 4D
convolutions to learn the geometry information encoded in multiple adjacent
views.

###  II-B Vision Transformers

Different from the convolutions that process an image locally, vision
Transformers regards an image as a sequence of tokens, and build the
relationship between the pairs of all tokens by self-attention mechanism.
Thanks to the capability of modeling long-range dependencies (i.e., the
relationship of all tokens), Transformers have achieved competitive
performance on many vision tasks, e.g., object detection [21], classification
[20] and semantic segmentation [22]. Among them, Carion et al. [21] first
applied Transformers to object detection to model the relations of objects and
global image contexts, and directly generate the set of predictions in
detection. Dosovitskiy et al. [19] designed a pure Transformer model (i.e.,
ViT) for classification tasks, and ViT achieves competitive results as
compared to CNN-based methods. Zheng et al. [22] treated semantic segmentation
as a sequence-to-sequence prediction task, and designed a pure Transformer
encoder and a CNN-based decoder to model global context and pixel-level
segmentation, respectively. In the area of low-level vision, Chen et al. [27]
developed an image processing Transformer with multi-heads and multi-tails.
Their method achieves state-of-the-art performance on image denoising,
deraining, and SR. Wang et al.[24] proposed a hierarchical U-shaped
Transformer to capture both local and non-local context information for image
restoration. Cao et al. [25] proposed a Transformer-based network to exploit
correlations among different frames for video SR.

##  III Method

We formulate an LF as a 4D tensor $\mathcal{L}\in\mathbb{R}^{U\times V\times
H\times W}$, where $U$ and $V$ represent angular dimensions, $H$ and $W$
represent spatial dimensions. Following [11, 14, 18, 13, 15], we achieve LF
image SR using SAIs distributed in a square array (i.e., $U$=$V$=$A$).  As
shown in Fig. 1(a), our network consists of three stages including initial
feature extraction, Transformer-based feature incorporation, and up-sampling.

###  III-A Angular Transformer

The input LF images are first processed by cascaded 3$\times$3 convolutions to
generate initial features $F\in\mathbb{R}^{U\times V\times H\times W\times
C}$. The extracted features are then fed to the angular Transformer to model
the angular dependencies. Our angular Transformer is designed to correlate
highly relevant features in the angular dimension and can fully exploit the
complementary information among all the input views.

Specifically, feature $F$ is first reshaped into a sequence of angular tokens
$T_{Ang}\in\mathbb{R}^{HW\times N_{A}\times d_{A}}$, where $HW$ represents the
batch dimension, $N_{A}$$=$$UV$ is the length of the sequence and $d_{A}$
denotes the embedding dimension of each angular token. Then, we perform
angular positional encoding to model the positional correlation of different
views [28], i.e.,

${\color[rgb]{0,0,0}P_{A}(p,2i)={\rm sin}(p/\alpha^{2i/d_{A}}),}$ (1)
${\color[rgb]{0,0,0}P_{A}(p,2i\text{+}1)={\rm cos}(p/\alpha^{2i/d_{A}}),}$ (2)

where $p$$=$$\\{1,2,...,A^{2}\\}$ represents the angular position and $i$
denotes the channel index in embedding dimension. It is worth mentioning that
each dimension of the positional encoding vector is regarded as a sinusoid,
and $\alpha$ represents the wavelength of sinusoids to form a geometric
progression from 2$\pi$ to $\alpha\cdot$2$\pi$ for the angular positional
encoding. Following [28], we set $\alpha$ to 10000 in our experiments.

As shown in Fig. 1(b) and Fig. 1(d), the angular position codes $P_{A}$ are
directly added to $T_{Ang}$, and passed through a layer normalization (LN) to
generate query ${Q}_{A}$ and key ${K}_{A}$, i.e.,
${Q}_{A}$$=$${K}_{A}$$=$${\rm LN}(T_{Ang}+P_{A})$. Value ${V}_{A}$ is directly
assigned as $T_{Ang}$, i.e., ${V}_{A}$$=$$T_{Ang}$. Afterwards, we apply the
multi-head self-attention (MHSA) to learn the relationship among different
angular tokens. Similar to other MHSA approaches [28, 27, 19], the embedding
dimension of ${Q}_{A}$, ${K}_{A}$ and ${V}_{A}$ is split into $N_{H}$ groups,
where $N_{H}$ is the number of heads.  For each attention head, the
calculation can be formulated as:

$\begin{split}{{H}_{h}}={\rm
Softmax(\frac{\it{Q}_{\it{A,h}}{\it{W}_{{{Q},h}}}(\it{K}_{\it{A,h}}{\it{W}_{{{K},h}}})^{\it{T}}}{\sqrt{\it{\it{d_{A}}/\it{N_{H}}}}})}\it{V}_{\it{A}}{\it{W}_{{\it{V},h}}},\end{split}$
(3)

where $h$$=$$\\{1,2,...,N_{H}\\}$ denotes the index of head groups.
${\it{W}_{{{Q},h}}}$, ${\it{W}_{{{K},h}}}$ and
${\it{W}_{{{V},h}}}\in\mathcal{R}^{(\it{d_{A}}/\it{N_{H}})\times(\it{d_{A}}/\it{N_{H}})}$
are the linear projection matrices. In summary, the MHSA can be formulated as:

$\begin{split}{\rm
MHSA}({Q}_{A},{K}_{A},{V}_{A})=[{{H}_{1}},...,{{H}_{N_{H}}}]W_{{O}},\end{split}$
(4)

where $W_{{O}}\in\mathcal{R}^{d_{A}\times d_{A}}$ is output projection matrix,
$[\cdot]$ denotes the concatenation operation.

As shown in Fig. 1(b), to further incorporate the correlations built by MHSA,
the tokens are further fed to a feed forward network (FFN), which consists of
a LN and a multi-layer perception (MLP) layer. In summary, the calculation
process of our angular Transformer can be formulated as:

$T^{\prime}_{Ang}={\rm MHSA}({Q}_{A},{K}_{A},{V}_{A})+T_{Ang},$ (5)
$\hat{T}_{Ang}={\rm MLP}({\rm LN}(T^{\prime}_{Ang}))+T^{\prime}_{Ang}.$ (6)

Finally, $\hat{T}_{Ang}$ is reshaped into ${F}\in\mathbb{R}^{U\times V\times
H\times W\times C}$ and fed to the subsequent spatial Transformer to
incorporate spatial context information.

###  III-B Spatial Transformer

The goal of our spatial Transformer is to leverage both local context
information and long-range spatial dependencies within each SAI. Specifically,
the input feature ${F}$ is first unfolded in each 3$\times$3 neighbor region
[29], and then fed to an MLP to achieve local feature embedding. That is,

$\begin{split}{F}^{\prime}(x,y)={\rm MLP}(\mathop{\rm
concat}\limits_{x_{r}=\\{-1,0,1\\}\atop
y_{r}=\\{-1,0,1\\}}{F}(x-x_{r},y-y_{r})),\end{split}$ (7)

where $(x,y)$ denotes an arbitrary spatial coordinate on feature ${F}$. The
local assembled feature ${F}^{\prime}$ is then cropped into overlapping
spatial tokens $T_{Spa}\in\mathbb{R}^{UV\times N_{S}\times d_{S}}$, where $UV$
denotes the batch dimension, $N_{S}$ represents the length of the sequence,
and $d_{S}$ represents the embedding dimension of the spatial tokens.

By performing feature unfolding and overlapped cropping, the local context
information can be fully integrated into the generated spatial tokens, which
enables our spatial Transformer to model both local and non-local
dependencies. To further model the spatial position information, we perform 2D
positional encoding on spatial tokens:

${\color[rgb]{0,0,0}P_{\textit{S}}(p_{x},p_{y},2j)={\rm
sin}({p_{x}}/{\alpha^{2j/d_{S}}})+{\rm sin}({p_{y}}/{\alpha^{2j/d_{S}}}),}$
(8) ${\color[rgb]{0,0,0}P_{\textit{S}}(p_{x},p_{y},2j\text{+}1)={\rm
cos}({p_{x}}/{\alpha^{2j/d_{S}}})+{\rm cos}({p_{y}}/{\alpha^{2j/d_{S}}}),}$
(9)

where $(p_{x},p_{y})$$=$$\\{(1,1),...,(H,W)\\}$ denotes the spatial position,
$j$ denotes the index in the embedding dimension and parameter $\alpha$ is set
as 10000. Then, ${Q}_{S}$, ${K}_{S}$ and ${V}_{S}$ can be calculated according
to:

${Q}_{S}={K}_{S}={\rm LN}(T_{Spa}+P_{S}),$ (10) ${V}_{S}=T_{Spa}.$ (11)

As shown in Fig. 1(c), similar to the proposed angular Transformer, we use the
MHSA and FFN to build our spatial Transformer. That is,

$T^{\prime}_{Spa}={\rm MHSA}({Q}_{S},{K}_{S},{V}_{S})+T_{Spa},$ (12)
$\hat{T}_{Spa}={\rm MLP}({\rm LN}(T^{\prime}_{Spa}))+T^{\prime}_{Spa}.$ (13)

Then, $\hat{T}_{Spa}$ is reshaped into $F\in\mathbb{R}^{U\times V\times
H\times W\times C}$ and fed to the next angular Transformer. After passing
through all the angular and spatial Transformers, both angular and spatial
information in an LF can be fully incorporated. Finally, we apply pixel
shuffling [30] to achieve feature up-sampling, and obtain the super-resolved
LF image $\mathcal{L}_{out}\in\mathbb{R}^{U\times V\times\alpha H\times\alpha
W}$.

##  IV Experiments

TABLE I: PSNR/SSIM values achieved by different methods for 2$\times$ and 4$\times$SR. The best results are in bold faces.  Methods | $2\times$ | $4\times$
---|---|---
EPFL | HCInew | HCIold | INRIA | STFgantry | EPFL | HCInew | HCIold | INRIA | STFgantry
Bicubic | 29.74/0.941 | 31.89/0.939 | 37.69/0.979 | 31.33/0.959 | 31.06/0.954 | 25.14/0.833 | 27.61/0.853 | 32.42/0.931 | 26.82/0.886 | 25.93/0.847
VDSR [31] | 32.50/0.960 | 34.37/0.956 | 40.61/0.987 | 34.43/0,974 | 35.54/0.979 | 27.25/0.878 | 29.31/0.883 | 34.81/0.952 | 29.19/0.921 | 28.51/0.901
EDSR [26] | 33.09/0.963 | 34.83/0.959 | 41.01/0.988 | 34.97/0.977 | 36.29/0.982 | 27.84/0.886 | 29.60/0.887 | 35.18/0.954 | 29.66/0.926 | 28.70/0.908
RCAN [32] | 33.16/0.964 | 34.98/0.960 | 41.05/0.988 | 35.01/0.977 | 36.33/0.983 | 27.88/0.886 | 29.63/0.888 | 35.20/0.954 | 29.76/0.927 | 28.90/0.911
resLF[11] | 33.62/0.971 | 36.69/0.974 | 43.42/0.993 | 35.39/0.981 | 38.36/0.990 | 28.27/0.904 | 30.73/0.911 | 36.71/0.968 | 30.34/0.941 | 30.19/0.937
LFSSR [14] | 33.68/0.974 | 36.81/0.975 | 43.81/0.994 | 35.28/0.983 | 37.95/0.990 | 28.27/0.908 | 30.72/0.912 | 36.70/0.969 | 30.31/0.945 | 30.15/0.939
LF-ATO [13] |  34.27/0.976 |  37.24/0.977 |  44.20/0.994 |  36.15/0.984 |  39.64/0.993 | 28.52/0.912 | 30.88/0.914 | 37.00/0.970 | 30.71/0.949 | 30.61/0.943
LF-InterNet [15] |  34.14/0.972 |  37.28/0.977 |  44.45/0.995 |  35.80/0.985 |  38.72/0.992 |  28.67/0.914 |  30.98/0.917 |  37.11/0.972 |  30.64/0.949 |  30.53/0.943
LF-DFnet [16] |  34.44/0.977 |  37.44/0.979 |  44.23/0.994 |  36.36/0.984 |  39.61/0.993 | 28.77/0.917 | 31.23/0.920 | 37.32/0.972 | 30.83/0.950 | 31.15/0.949
MEG-Net [12] |  34.30/0.977 | 37.42/0.978 | 44.08/0.994 |  36.09/0.985 | 38.77/0.991 | 28.74/0.916 | 31.10/0.918 | 37.28/0.972 | 30.66/0.949 | 30.77/0.945
LF-IINet [17] |  34.68/0.977 |  37.74/0.979 |  44.84/0.995 |  36.57/0.985 |  39.86/0.994 |  29.11/0.920 |  31.36/0.921 |  37.62/0.974 |  31.08/0.952 |  31.21/0.950
DPT [33] | 34.48/0.976 | 37.35/0.977 | 44.31/0.994 | 36.40/0.984 | 39.52/0.993 | 28.93/0.917 | 31.19/0.919 | 37.39/0.972 | 30.96/0.950 | 31.14/0.949
LFT(ours) |  34.80/0.978 |  37.84/0.979 |  44.52/0.995 |  36.59/0.986 |  40.51/0.994 |  29.25/0.921 |  31.46/0.922 |  37.63/0.974 |  31.20/0.952 |  31.86/0.955

###  IV-A Implementation Details

Following [16], we used 5 public LF datasets [34, 35, 36, 37, 38] to validate
our method. All LF images in the training and test set have an angular
resolution of 5$\times$5\. In the training stage, we cropped LF images into
patches of 64$\times$64$/$128$\times$128 for 2$\times/$4$\times$ SR, and used
the bicubic downsampling approach to generate LR patches of size 32$\times$32.

We used peak signal-to-noise ratio (PSNR) and structural similarity (SSIM)
[39] as quantitative metrics for performance evaluation. To obtain the metric
score for a dataset with $M$ scenes, we calculated the metrics on the
$A$$\times$$A$ SAIs of each scene separately, and obtained the score for this
dataset by averaging all the $M$$\times$$A^{2}$ scores.

All experiments were implemented in Pytorch on a PC with four Nvidia GTX
1080Ti GPUs. The weights of our network were initialized using the Xavier
method [40], and optimized using the Adam method [41].  The batch size was set
to 4$/$8 for 2$\times/$4$\times$ SR. The learning rate was set to 2$\times
10^{-4}$ and halved for every 15 epochs. The training was stopped after 80
epochs.

###  IV-B Comparison to state-of-the-art methods

We compare our LFT to several state-of-the-art methods, including 3 single
image SR methods [31, 26, 32] and 8 LF image SR methods [11, 14, 13, 15, 16,
12, 17, 33]. We retrained all these methods on the same training datasets as
our LFT.

Figure 2: Visual results achieved by different methods for 2$\times$SR. Figure
3: Visual results achieved by different methods for 4$\times$SR.

####  IV-B1 Quantitative Results

Table I shows the quantitative results achieved by our method and other state-
of-the-art SR methods. Our LFT achieves the competitive PSNR and SSIM results
on all the 5 datasets for both 2$\times$ and 4$\times$ SR.  Note that, the
superiority of our LFT is very significant on the STFgantry dataset [38]
(i.e., 0.65 dB higher than the second top-performing method [17] for both
2$\times$ and 4$\times$ SR). That is because, LF images in the STFgantry
dataset have more complex structures and larger disparity variations. By using
our angular and spatial Transformers, our method can well handle these complex
scenes while maintaining state-of-the-art performance on other datasets.

####  IV-B2 Qualitative Results

Figures 2 and 3 show the qualitative results achieved by different methods.
Our LFT can well preserve the textures and details in the SR images and
achieves competitive visual performance. Moreover, we provide a demo
video111https://wyqdatabase.s3.us-west-1.amazon aws.com/LFT_video.mp4 for a
visual comparison of the angular consistency.

####  IV-B3 Efficiency

We compare our LFT to several competitive methods [11, 14, 13, 15, 16, 12, 17,
33] in terms of the number of parameters and FLOPs. Specifically, DPT [33] is
a recently proposed Transformer-based LF image SR method. As shown in Table
II, compared to other methods, our LFT achieves higher accuracy with smaller
model size and lower computational cost, which demonstrates the high
efficiency of our method.

TABLE II: The parameters (#Param.), FLOPs and average PSNR/SSIM scores achieved by state-of-the-art methods for 2$\times$ and 4$\times$ SR. Note that, FLOPs is computed with an input LF of size 5$\times$5$\times$32$\times$32\. The best results are in bold faces.  Methods | $2\times$ | $4\times$
---|---|---
#Param. | FLOPs | PSNR/SSIM | #Param. | FLOPs | PSNR/SSIM
resLF | 7.98M | 79.63G | 37.50/0.982 | 8.65M | 85.47G | 31.25/0.932
LFSSR | 0.89M | 91.06G | 37.51/0.983 | 1.77M | 455.04G | 31.56/0.937
LF-ATO | 1.22M | 1815.36G | 38.30/0.985 | 1.36M | 1898.91G | 31.54/0.938
LF-InterNet | 4.91M | 38.97G | 38.08/0.985 | 4.96M | 40.25G | 31.59/0.939
LF-DFnet | 3.94M | 57.22G | 38.42/0.985 | 3.99M | 58.49G | 31.86/0.942
MEG-Net | 1.69M | 48.40G |  38.12/0.985 | 1.77M | 102.2G | 31.72/0.940
LF-IINet | 4.84M | 56.16G |  38.74/0.986 | 4.89M | 57.42G |  32.08/0.944
DPT | 3.73M | 57.44G |  38.40/0.985 | 3.78M | 58.64G |  31.92/0.941
LFT(ours) | 1.11M | 28.00G |  38.85/0.986 | 1.16M | 29.45G |  32.28/0.945
TABLE III: PSNR results achieved on the EPFL [34], HCIold [36] and INRIA [37] datasets by several variants of LFT for $4\times$SR. Note that, AngTr and SpaTr represent models using angular Transformer and spatial Transformer, respectively. AngPos and SpaPos denote models using positional encoding in AngTr and SpaTr, respectively. #Param. represents the number of parameters of different variants.  | AngTr | AngPos | SpaTr | SpaPos | #Param. | EPFL | HCIold | INRIA
---|---|---|---|---|---|---|---|---
1 |  |  |  |  | 1.49M | 28.63 | 37.00 | 30.66
2 | ✓ |  |  |  | 1.42M | 28.85 | 37.29 | 30.93
3 | ✓ | ✓ |  |  | 1.42M | 28.98 | 37.38 | 30.93
4 |  |  | ✓ |  | 1.28M | 28.93 | 37.30 | 30.97
5 |  |  | ✓ | ✓ | 1.28M | 28.95 | 37.41 | 30.98
6 | ✓ | ✓ | ✓ | ✓ | 1.16M | 29.25 | 37.46 | 31.20
Figure 4:  Visualization of attention maps (i.e., local angular similarities)
generated by our angular Transformer and LF-InterNet[15]. (a) Center view and
corresponding epipolar plane images of scene Cards[38]. (b) and (c) are local
angular similarity maps of the red patch in (a) by our LFT and LF-InterNet,
respectively. Similarly, (d) and (e) are of the green patch by our LFT and LF-
InterNet, respectively. Note that, each tile in the map illustrates the
similarities between the current view (at the same position as the tile) and
all the views (different pixels within the tile).

###  IV-C Ablation Study

We introduce several variants with different architectures to validate the
effectiveness of our method. As shown in Table III, we first introduce a
baseline model (i.e., model-1) without using angular and spatial Transformers,
then separately added angular Transformer (i.e., model-2) and spatial
Transformer (i.e., model-4) to the baseline model. Moreover, we introduce
model-3 and model-5 to validate the effectiveness of angular and spatial
positional encoding.

####  IV-C1 Angular Transformer

We compare the performance of model-2 to model-1 and model-6 to model-5 to
validate the effectiveness of the angular Transformer. As shown in Table III,
by using the angular Transformer, model-2 achieves a 0.2$\sim$0.3 dB PSNR
improvements over model-1. When the angular positional encoding is introduced,
model-3 can further achieve a 0.1dB improvement over model-2 on the EPFL [34]
and HCIold [36] datasets. By comparing the performance of model-5 and model-6,
we can see that removing the angular Transformer (and angular positional
encoding) from our LFT will cause a notable PSNR drop (around 0.2 dB). The
above experiments demonstrate that our angular Transformer and angular
positional encoding are beneficial to the SR performance.

Moreover, we investigate the spatial-aware modeling capability of our angular
Transformer by visualizing the local angular attention maps. Specifically, we
selected two patches from scene Cards [38], and obtained the attention maps (a
25$\times$25 matrix for a spatial location in a 5$\times$5 LF) produced by the
MHSA in the first angular Transformer at each spatial location in the patches.
Note that, larger values in the attention maps represent higher similarities
between a pair of angular tokens. We then define “local angular attention” by
calculating the ratios of similar tokens (with attention scores larger than
0.025) in the selected patches.  Finally, we visualize the local angular
attention map in Fig. 4 by assembling the calculated attention values
according to their angular coordinates. It can be observed in Fig. 4(b) that
the attention values in the occlusion area (red patch) are distributed
unevenly, where the non-occluded pixels share larger values. It demonstrates
that our angular Transformer can adapt to different image contents and achieve
spatial-aware angular modeling.

Similarly, we calculate the self-correlation of the feature generated by the
LF-InterNet[15] (CNN-based method) in angular dimensions, and the similarity
maps of red patch and green patch are shown in Fig. 4(c) and Fig. 4(d),
respectively. It can be observed that the view-wise correlation of LF-InterNet
is not obvious, i.e., each view has higher similarities with its adjacent
views only.

####  IV-C2 Spatial Transformer

We demonstrate the effectiveness of the spatial Transformer by comparing the
performance of model-4 to model-1 and model-6 to model-3. As shown in Table
III, model-4 achieves a 0.3 dB improvements in PSNR over model-1. Moreover,
when the spatial Transformer is removed from our LFT, model-3 suffers a
notable performance degradation (0.08$\sim$0.27 dB in PSNR). That is because,
compared to cascaded convolutions, the proposed spatial Transformer can better
exploit long-range context information with a global receptive field, and can
capture more beneficial spatial information for image SR.

##  V Conclusion

In this paper, we propose a Transformer-based network (i.e., LFT) for LF image
SR. By using our proposed angular and spatial Transformers, the complementary
angular information among all the views and the long-range spatial
dependencies within each SAI can be effectively incorporated. Experimental
results have demonstrated the superior performance of our LFT over state-of-
the-art CNN-based SR methods.

## References

  * [1] Y. Wang, J. Yang, Y. Guo, C. Xiao, and W. An, “Selective light field refocusing for camera arrays using bokeh rendering and superresolution,” _IEEE Signal Processing Letters_ , vol. 26, no. 1, pp. 204–208, 2018.
  * [2] S. Jayaweera, C. Edussooriya, C. Wijenayake, P. Agathoklis, and L. Bruton, “Multi-volumetric refocusing of light fields,” _IEEE Signal Processing Letters_ , vol. 28, pp. 31–35, 2020.
  * [3] W. Wang, Y. Lin, and S. Zhang, “Enhanced spinning parallelogram operator combining color constraint and histogram integration for robust light field depth estimation,” _IEEE Signal Processing Letters_ , vol. 28, pp. 1080–1084, 2021.
  * [4] J. Lee and R. Park, “Reduction of aliasing artifacts by sign function approximation in light field depth estimation based on foreground–background separation,” _IEEE Signal Processing Letters_ , vol. 25, no. 11, pp. 1750–1754, 2018.
  * [5] A. Wang, “Three-stream cross-modal feature aggregation network for light field salient object detection,” _IEEE Signal Processing Letters_ , vol. 28, pp. 46–50, 2020.
  * [6] Y. Wang, T. Wu, J. Yang, L. Wang, W. An, and Y. Guo, “Deoccnet: Learning to see through foreground occlusions in light fields,” in _Winter Conference on Applications of Computer Vision (WACV)_ , 2020, pp. 118–127.
  * [7] S. Zhang, Z. Shen, and Y. Lin, “Removing foreground occlusions in light field using micro-lens dynamic filter,” in _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , 2021, pp. 1302–1308.
  * [8] Y. Yoon, H. Jeon, D. Yoo, J. Lee, and I. Kweon, “Light-field image super-resolution using convolutional neural network,” _IEEE Signal Processing Letters_ , vol. 24, no. 6, pp. 848–852, 2017.
  * [9] Y. Yuan, Z. Cao, and L. Su, “Light-field image superresolution using a combined deep cnn based on epi,” _IEEE Signal Processing Letters_ , vol. 25, no. 9, pp. 1359–1363, 2018.
  * [10] Y. Wang, F. Liu, K. Zhang, G. Hou, Z. Sun, and T. Tan, “Lfnet: A novel bidirectional recurrent convolutional neural network for light-field image super-resolution,” _IEEE Transactions on Image Processing_ , vol. 27, no. 9, pp. 4274–4286, 2018.
  * [11] S. Zhang, Y. Lin, and H. Sheng, “Residual networks for light field image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 11 046–11 055.
  * [12] S. Zhang, S. Chang, and Y. Lin, “End-to-end light field spatial super-resolution network using multiple epipolar geometry,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 5956–5968, 2021.
  * [13] J. Jin, J. Hou, J. Chen, and S. Kwong, “Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 2260–2269.
  * [14] H. Yeung, J. Hou, X. Chen, J. Chen, Z. Chen, and Y. Chung, “Light field spatial super-resolution using deep efficient spatial-angular separable convolution,” _IEEE Transactions on Image Processing_ , vol. 28, no. 5, pp. 2319–2330, 2018.
  * [15] Y. Wang, L. Wang, J. Yang, W. An, J. Yu, and Y. Guo, “Spatial-angular interaction for light field image super-resolution,” in _European Conference on Computer Vision (ECCV)_. Springer, 2020, pp. 290–308.
  * [16] Y. Wang, J. Yang, L. Wang, X. Ying, T. Wu, W. An, and Y. Guo, “Light field image super-resolution using deformable convolution,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1057–1071, 2020.
  * [17] G. Liu, H. Yue, J. Wu, and J. Yang, “Intra-inter view interaction network for light field image super-resolution,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2021.
  * [18] N. Meng, H. So, X. Sun, and E. Lam, “High-dimensional dense residual convolutional neural network for light field reconstruction,” _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * [19] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Proceedings of the International Conference on Learning and Representation (ICLR)_ , 2021.
  * [20] M. Zheng, P. Gao, X. Wang, H. Li, and H. Dong, “End-to-end object detection with adaptive clustering transformer,” _arXiv preprint arXiv:2011.09315_ , 2020.
  * [21] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European Conference on Computer Vision (ECCV)_. Springer, 2020, pp. 213–229.
  * [22] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. Torr _et al._ , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 6881–6890.
  * [23] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 12 179–12 188.
  * [24] Z. Wang, X. Cun, J. Bao, and J. Liu, “Uformer: A general u-shaped transformer for image restoration,” _arXiv preprint arXiv:2106.03106_ , 2021.
  * [25] J. Cao, Y. Li, K. Zhang, and L. Van Gool, “Video super-resolution transformer,” _arXiv preprint arXiv:2106.06847_ , 2021.
  * [26] B. Lim, S. Son, H. Kim, S. Nah, and K. Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2017, pp. 136–144.
  * [27] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 12 299–12 310.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [29] Y. Chen, S. Liu, and X. Wang, “Learning continuous image representation with local implicit image function,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8628–8638.
  * [30] W. Shi, J. Caballero, F. Huszár, J. Totz, A. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1874–1883.
  * [31] J. Kim, J. Lee, and K. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1646–1654.
  * [32] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 286–301.
  * [33] S. Wang, T. Zhou, Y. Lu, and H. Di, “Detail preserving transformer for light field image super-resolution,” in _Proceedings of the AAAI Conference on Artificial Intelligence,_ , 2022.
  * [34] M. Rerabek and T. Ebrahimi, “New light field image dataset,” in _International Conference on Quality of Multimedia Experience (QoMEX)_ , 2016\.
  * [35] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A dataset and evaluation methodology for depth estimation on 4d light fields,” in _Asian Conference on Computer Vision (ACCV)_. Springer, 2016, pp. 19–34.
  * [36] S. Wanner, S. Meister, and B. Goldluecke, “Datasets and benchmarks for densely sampled 4d light fields.” in _Vision, Modelling and Visualization (VMV)_ , vol. 13. Citeseer, 2013, pp. 225–226.
  * [37] M. Pendu, X. Jiang, and C. Guillemot, “Light field inpainting propagation via low rank matrix completion,” _IEEE Transactions on Image Processing_ , vol. 27, no. 4, pp. 1981–1993, 2018.
  * [38] V. Vaish and A. Adams, “The (new) stanford light field archive,” _Computer Graphics Laboratory, Stanford University_ , vol. 6, no. 7, 2008\.
  * [39] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [40] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2010, pp. 249–256.
  * [41] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _Proceedings of the International Conference on Learning and Representation (ICLR)_ , 2015.