# Pyramidal Blur Aware X-Corner Chessboard Detector

Peter Abeles1 1Peter Abeles is with NINOX 360, Redwood City, CA USA
<EMAIL_ADDRESS>
###### Abstract

With camera resolution ever increasing and the need to rapidly recalibrate
robotic platforms in less than ideal environments, there is a need for faster
and more robust chessboard fiducial marker detectors. A new chessboard
detector is proposed that is specifically designed for: high resolution
images, focus/motion blur, harsh lighting conditions, and background clutter.
This is accomplished using a new x-corner detector, where for the first time
blur is estimated and used in a novel way to enhance corner localization, edge
validation, and connectivity. Performance is measured and compared against
other libraries using a diverse set of images created by combining multiple
third party datasets and including new specially crafted scenarios designed to
stress the state-of-the-art. The proposed detector has the best F1-Score of
0.97, runs 1.9x faster than next fastest, and is a top performer for corner
accuracy, while being the only detector to have consistent good performance in
all scenarios.

##  I Introduction

Camera calibration is an essential part in determining the mapping between an
image’s 2D pixel coordinate and the 3D world. By observing calibration
markers, it is possible to extrapolate a camera’s intrinsic characteristics
(e.g. focal length and lens distortion) to a high level of precision [1].

Chessboard (or checkerboard) patterns are arguably the most popular pattern
for camera calibration with good reason [2], they are easy to detect and
robust to blur due to their symmetry. They are used by themselves or with
self-identifying patterns [3], as done in CALTag [4] and ChArUco [5].

In this work, we focus on fully automatic detection of chessboard patterns in
diverse environments. Outdoors calibration and in factories is increasingly
important and challenging due to poor inconsistent lighting, motion blur, poor
focus, harsh shadows, and complex backgrounds. In addition, typical camera
resolutions have increased dramatically since much of the past work was
proposed and new techniques are needed to handle, the now common, 12+
megapixels images.

A novel chessboard detector is proposed that combines several innovations; a
new x-corner detector, blur estimator, robust blur aware edge validation, and
overall multi-scale approach. The last three innovations are significant
departures from past work and is the first known fiducial detector to estimate
the optimal scale to localize a corner, then uses the scale to dynamically
sample edges and determine corner connectivity. Enabling heavily blurred and
fisheyed images to be processed. The final output can be configured to return
all found patterns or use prior information and find a single known target.
Results include metadata that is useful for camera diagnostics, autofocus, and
estimating corner precision.

To validate performance, a study is performed on what might be the most
diverse set of chessboard scenes to date from multiple authors and new ones
specifically designed to stress and break detectors in real-world situations.
Accuracy is measured using hand labeled corners and synthetic images with
known corners. In this study, the proposed detector demonstrated its
robustness, accuracy, and speed across all scenarios. Making it the only
tested library to produce consistently good results. Both source code and
datasets are freely available online111Detector and source code has been
available since 2019 in BoofCV, with smaller improvements in v0.39.
https://boofcv.org.

##  II Related Work

Planar targets are widely used in camera calibration [1] for their ease of use
and there are a multitude of toolboxes for them, such as OpenCV [6], Bouguet’s
Matlab [7], CamOdoCal [8], and BoofCV [9]. The earliest approaches required
human intervention to find corners, while more recent work finds most of the
corners automatically, to varying degrees of accuracy and reliability.
Deltille patterns have been proposed [10] as an alternative to chessboards for
improved handling of blur and distortion.

One approach to chessboard detection involves using contours around squares.
In OpenCV’s findChessboardCorners [11] the image is adaptively thresholded,
eroded to separate the individual squares/quadrangles, then squares are found
using contours. The chessboard grid is formed by connecting quadrangles
corners. Image borders are challenging since they crop and/or heavily distort
squares. With improvements to this general approach being proposed in [12, 8].
Recently, deep learning has been explored for corner detection [13].

Instead of using square contours, x-corners (named after the intersection of
two squares) can be detected directly, avoiding image border issues. Harris
[14] corner detector is the basis for many x-corner detectors [15, 16, 17].
While Harris is excellent at detecting x-corners, it detects other corner-like
shapes too.

Many papers have proposed custom x-corner intensity functions [18, 17, 10,
19]. In [20] x-corners are found by looking at binary centerlines [21].
Filtering corners after detection is standard. In [16, 18] filtering is done
using local gradient histogram statistics and based on local intensity
patterns in [17].

After candidate corners are found at pixel level precision, they are refined
to sub-pixel accuracy [22, 23]. Specific approaches include: Fitting a
quadratic function [24] or saddle point [15, 20, 10] to x-corner intensity
image. Maximize gradient orthogonality [18]. Iterative perspective undistort
and redetection [25].

The first step in constructing chessboard graphs from x-corners is finding
high confidence pairs. Combinatorics can be limited by only considering
N-nearest-neighbor corners [16, 10, 19]. Require two corners have compatible
orientations/polarities [18, 10]. Reject pairs using image gradient or sample
points along edges [16, 26, 10].

After pairs are known, the graph is then constructed. This is a challenging
process which often drives the final accuracy. In [18, 10, 19] multiple seeds
are used then expanded into a graph. In [8] a spline is used to reject rows if
the error is too large.

##  III Proposed Approach and Contributions

The proposed approach is summarized in Algorithms 1 and 2, with a more
detailed explanation in the following subsections. ”Level” always refers to
level in an image pyramid and is synonymous with scale.

X-corner detector contributions:

  1. 1.

Corner localization at optimal scale: Select lowest level with a strong
response to avoid instability.

  2. 2.

Explicit handling of x-corner symmetry: Applying 2x2 box filter for unique
local maximum, stabilizing non-maximum suppression, see Figure 2.

  3. 3.

Sub-pixel accuracy using mean-shift: Alternative to curve fitting that is less
sensitive to noise.

Graph connectivity and structure contributions:

  1. 1.

Robust edge validation: Score using $n$-best points. Reduces sensitive to
outliers from localized lighting.

  2. 2.

Blur aware edge validation: Avoid sampling poorly defined edges near blurred
x-corners centers.

  3. 3.

Scale Compatibility: Only connect a corner to others at same or higher pyramid
level. Reduces influence of false positives at lower levels.

###  III-A X-Corner Detector

1Let $L$ be the set of images in a pyramid

2 for  _ $l_{i}\in L$ _ do

3 Compute x-corner intensity

4 2x2 box filter for unique maximums

5 Non-maximum suppression

6 Prune using a cascade of filters

7 Mean-shift sub-pixel localization

8 Compute corner orientation and revised intensity

9

10 end for

11Connect corners across pyramid levels

12 Select level for corner location and orientation

Algorithm 1 Pyramidal X-Corner Detection

Figure 1: Left: Circle sampling pattern stored in array ’v’. Right: Workspace
variables formulas. Bottom: Visualization of kernels.

The proposed x-corner intensity function is designed to be accurate, fast, and
have affine lighting invariance. It will be excited only when encountering
x-corners. Other corners and lines cause a negative response. The ”likelihood”
function in [18] is an inspiration, but their formulation has many more
logical branches making it expensive to compute.

(a) input

(b) x-corner

(c) filter 2x2

Figure 2: a) Chessboard corner without perspective distortion. b) Original
x-corner intensity. Red and green indicate positive and negative x-corner
response, respectively. c) Local maximum after applying 2x2 block filter,
which improves corner localization in general.

_Corner Intensity:_ 1) Convert image to grayscale and scale pixel values to be
0 to 1. 2) Apply 3x3 Gaussian blur to image. 3) For each pixel, sample a
circle as shown in Figure 1. 4) Compute x-corner intensity $I\in\Re$

$I=\mbox{max}\left(\mbox{xscore}(a,b,c,d),\mbox{xscore}(e,f,g,h)\right)$ (1)

where $\\{a,b,c,d,e,f,g,h\\}$ are sampled in Step 3.

⬇

fun xscore( $v_{1}$ , $v_{2}$ , $v_{3}$ , $v_{4}$)

$\mu$ = ($v_{1}$+$v_{2}$+$v_{3}$+$v_{4}$)/4

return ($v_{1}$-$\mu$)*($v_{3}$-$\mu$) + ($v_{2}$-$\mu$)*($v_{4}$-$\mu$)

Gaussian blur in Step 2 acts as a computationally efficient template to
construct kernels in Figure 1. The _xscore_ function is at a maximum when the
kernel response $v_{1}$ and $v_{3}$ are both above or below the mean, then the
same is done for $v_{2}$ and $v_{4}$. Affine lighting invariance is achieved
by subtracting the mean then multiplying the difference. Partial rotation
invariance is achieved in Eq. 1 by using two templates offset by 45 degrees.

In the absence of noise and perspective distortion, Eq. 1 will not have unique
local maximums. This is resolved by convolving a 2x2 box kernel to the
intensity image (Figure 2) and apply (0.5, 0.5) pixels offset to undo box
filter induced shift. In typical situations, this helps sub-pixel estimation
converge by providing a better initial estimate. For corners at a skewed angle
and heavy fisheye distortion the box filter still help break the symmetry.

A cascade of filters is applied to reduce candidate corners in order of least
to most computationally expensive. 1) Filter corners based on intensity
relative to maximum intensity in top pyramid level. 2) Filter if too many
positive x-corner intensity values. Actual x-corners tend to have negative
neighbors. 3) Check for expected up-down gray scale pattern. 4) Apply Shi-
Tomasi [27] corner Eigenvalue test.

Figure 3: a) Orientation and a rotationally invariant x-corner intensity are
found using the line integral along 32 spokes centered on the corner. Red
indicates selected orientation. b) Cartoon showing raw spoke orientation
score. c) Unique maximum after smoothing.

Remaining corners are refined to sub-pixel accuracy using mean-shift in the
intensity image. Mean-shift implicitly applies a low pass filter, making it
less sensitive to noise than curve fitting approaches, which attempt to
exactly fit all values. Orientation and an improved x-corner intensity are
estimated using spokes, Figure 3. These spokes are conceptually similar to the
approximated Radeon transform in [19]. Corner half-circle orientation is
estimated by computing the difference between each spoke and one offset by 90
degrees. Resulting array is smoothed with a Gaussian kernel to create a
maximum in the middle, Figure 3. The selected angle is further refined by
fitting a quadratic. The new x-corner intensity is set to the orientation’s
best score and exhibits better rotational invariance than Eq 1. Corner
contrast is computed as the difference in magnitude of spokes for the best
orientation and is used later on.

###  III-B Pyramidal Processing

(a) Level 0

(b) Level 1

(c) Level 2

(d) Level 3

(e) Level 4

Figure 4: Corners detected across pyramid levels. Higher levels have lower
resolution. Yellow circles are found x-corners.

Unlike blob features [28], e.g. Hessian or Laplacian, corners do not have a
unique maximum in intensity across scale-space [29] but have a constant
intensity. This is similar to the aperture problem [30] but across scale-
space. In fact, if there is no noise or blur, a single corner would be
observable in all pyramid levels. In practice, the first level is determined
by the amount of blur and sensor noise, while the last level is based on ratio
of marker size and distance.

Corners are poorly defined under heavy blur and localization is dominated by
noise. Naively assuming higher resolution are better for localization causes
instability. To overcome this issue, we proposed that the selected level for
localization maximizes resolution and intensity:

$\max_{level}\quad\mbox{intensity}(c,level)/(level+1)$ (2)

where $\mbox{intensity}(c,level)$ is a corner’s intensity at $level$.

Metadata from pyramidal processing is useful for end consumers. High
confidence corners will typically be viewed across multiple levels. The first
observed level indicates the amount of local blur/noise. For example, this
could be used to autofocus a camera on chessboard patterns, estimate corner
accuracy, or detect faults during calibration.

###  III-C Corner Connectivity

1Let $L$ be the set of all pyramid levels

2 for  _ $l_{i}\in L$ _ do

3 Let $C$ be the set of all detected corners in $l_{i}$

4 for  _ $c_{i}\in C$ _ do

5 Let $C_{n}$ be the set of nearest neighbors of $c_{i}$ visible in the same
level or greater

6 for  _ $c_{j}\in C_{n}$ _ do

7 Discard $c_{j}$ if incompatible orientation

8 Sample grid of points connecting $c_{i}$ and $c_{j}$

9 Score samples using binary intensities

10 Connection if sufficient score

11

12 end for

13

14 end for

15

16 end for

17for  _ $c_{i}\in C$ _ do

18 Vote on connected neighbors

19 Select neighbors based on perspective geometry and graph constraints

20

21 end for

22Discard ambiguous corners based on votes

23 Sort and enforce graph constraints

Algorithm 2 Chessboard Graph Formation Figure 5: Visualization of corner
orientation (cyan) and corner connections (orange) under fisheye distortion

Connected corner pairs are required to construct a topological graph that
represents the chessboard. These connections have known topological properties
and orientations. Procedure: 1) For each corner, find its $k$-nearest-
neighbors (KNN) in the same pyramid level or above as candidates for
connecting. 2) Reject if the corner orientations are not perpendicular. 3)
Reject if edge validation fails.

Figure 6: Diagram illustrating components of edge validation. Sample locations
are dynamic based on corner level to avoid blur.

Edge validation is done by sampling points in a grid on either side of the
line, Figure 6. $\\{c_{1},c_{2}\\}$ are corners with different amounts of
local blur. Spacing $s_{i}$ is dependent on the pyramid level of $c_{i}$. The
proposed novel dynamic spacing based on level avoids sampling poorly defined
edges near blurry corners. Perpendicular offset $s_{3}$ is determined by the
distance between $c_{1}$ and $c_{2}$. Perpendicular error
$E^{\bot}=I_{i}-I_{j}$ score is found by sampling adjacent points across the
edge and is maximized by high contrast. Longitudinal error
$E^{\parallel}=|I_{i}-I_{j}|$ measures similarity. These scores are sorted and
the worst values removed to prevent outliers caused by noise and hard shadows
from dominating. The edge intensity score is shown below and edges with values
below a threshold are pruned:

$L_{ij}=\frac{\sum_{k}E^{\bot}_{k}-E^{\parallel}_{k}}{\mbox{contrast}(c_{i})+\mbox{contrast}(c_{j})}$
(3)

Corner $\mbox{contrast}(c)$ is defined in Section III-A and is used as a
divisor for lighting invariance.

###  III-D Grid Graph Construction

A correctly formed chessboard topological graph will have the following
properties: 1) All corners must be mutually connected. 2) Every corner must
have 2, 3, or 4 neighbors. 3) There must be one and only one other common
corner between two neighbors that are adjacent. Any connection or corner not
meeting these conditions is pruned. If there are too many connections and
ambiguous corners have been identified, then a vote is taken from the local
neighborhood where each corner decides independently which connection is the
best fit based on edge intensity and expected geometry. The losing corner(s)
are removed.

###  III-E Grid Graph to Chessboard Graph

Grid graphs are reformatted into valid chessboard graphs by putting corners
into a canonical counter-clock-wise graph order using corner orientation
information and ensuring corner (0,0) is connected to a corner square.
Optionally, a single complete grid can be enforced by removing stray
connections: Outer rows and columns are removed if there are missing corners.
This is repeated as required. If the chessboard shape is known in advance then
only graphs which match are returned. If only a single chessboard is expected
then the pattern with the largest apparent image size is returned. In summary,
output can be multiple arbitrary chessboards or a single known chessboard
where false positives are pruned with stronger assumptions.

Please consult source code for specifics as algorithm details in this section
have been simplified for conciseness.

##  IV Performance Study

(a) Border

(b) Caltech Edges

(c) Distant

(d) Gaussian

(e) Large Shadow

(f) OCam Fisheye190

(g) Perfect

(h) Rotation Vertical

Figure 7: Select sample images from 8 of 24 the scenarios. A wide range of
environments, lighting conditions, and sensors have been tested.

A diverse performance study is conducted by combining several datasets from
multiple authors, as well as adding new scenarios specifically designed to
stress detectors, Figure 7. Scenarios vary widely in camera type, image size
(0.3 MP to 12 MP), lighting conditions, noise level, motion blur, focus, and
background clutter. Corner accuracy is measured using hand labeled corners as
well as synthetic images with known corner locations. Every image will have
one and only chessboard pattern with all corners visible, as is typical for
single camera calibration. See technical report [31] for a more complete
description of all scenarios.

Performance is measured for chessboard detection, corner localization
accuracy, and runtime performance. Detection accuracy is measured using
F1-Score statistics. A _true positive_ TP is defined as a detected chessboard
where every corner is within $t_{c}$ pixels of ground truth. A _false
positive_ FP is defined as a detection with the expected grid shape and one or
more corners outside of $t_{c}$. Detections with the wrong shape or missing
corners are ignored. To keep it simple, because chessboard orientation can be
ambiguous, corner error is computed using the closest match in ground truth.
If possible, the known target’s shape is given to a library.

Corner labels are used instead of the common post calibration reprojection
error because reprojection error does not measure the detector’s accuracy
directly. Instead reprojection error measures the calibration system as a
whole and can hide systematic bias and report optimistic results due to camera
model over parameterization. As an example, if a chessboard detector is biased
by 5 pixels the calibration system would compensate by translating the camera
and report perfect results.

Hand labeled x-corners are estimated to have an accuracy of around 0.2 pixels
when images are in focus. Labeling heavily blurred images is difficult to
impossible. For this reason, the _Gaussian_ scenario is simulated with exact
ground truth and _Gaussian Fisheye_ has Gaussian blur added to a real fisheye
image. By default, $t_{c}$ is set to a generous 5 pixels and for _focus_large_
$t_{c}=20$ due to labeling accuracy. The relative rankings of each library is
insensitive to the choice of $t_{c}$, Figure 8. Libraries which define the
pixel at $(1,1)$ to be within $x,y\in(0.5,1.5]$ are offset by (0.5, 0.5)
pixels.

Runtime is measured using system clock in code while excluding IO. Libraries
written in C/C++ are compiled in release mode and built with GCC 9.3.0. Java
libraries use JVM 15.0.3 and are given a warm start. Matlab libraries run in
headless mode using version R2021a.

Libraries and Versions: _Proposed:_ [9] BoofCV v0.39, _DelChe:_ [10] SHA
eed7e86, _Geiger:_ [18] libomnical and libcbdetect from website 2021-Aug-15,
_OCamCalib:_ [32] v3.0, _Binary:_ OpenCV 4.5.3 findChessboardCorners and
cornerSubPix, _ChessSB:_ [19] OpenCV 4.5.3 findChessboardCornersSB. To be
clear, _DelChe_ is the chessboard detector which was adapted from a deltille
pattern detector.

External Datasets: _caltech_edges_ [7], _ocam_ [32], _stefano_2012_ [33],
_challenge_ [34], _kaist fisheye_ [10].

##  V Results

Corner localization accuracy is reported using 50% error $E50$ and max error
statistics $E100$. The same is true for runtime performance, i.e. $R50$ and
$R100$. Median or 50% statistics is intended to represent ”expected” or
nominal performance with outliers removed. Max statistics is the worst case
performance and tends to be volatile. $N$ is the number of images in a
specific scenario. See Tech Report [31] for a more complete discussion of
these results.

Libraries which crashed, froze, or were excessively slow presented a challenge
for summary results. Due to the lack of any better options, summary
performance is computed by omitting scenarios a library could not process.
This gives some libraries a significant advantage by removing the most
difficult scenarios. Here is a list of libraries and the number of omitted
scenarios; DelChe 2, and OCamCalib 1.

Figure 8: F1-Score by combining all scenarios with different thresholds Figure
9: F1-Score in select challenging scenarios

Detection performance as a function of true positive thresholds across all
scenarios combined is shown in Figure 8. The proposed approach has the best
performance with an F1-score of 0.97 using a 5 pixel threshold. The next best
library is Geiger with an F1-score of 0.92. The proposed and Geiger have
comparable performance in nominal situations, but the proposed approach
outperforms Geiger in degraded scenarios. Figure 9 shows performance in select
scenarios with blur or adverse environmental conditions. In these individual
scenarios the proposed is the top performer, often by a considerable margin,
i.e. 3x in ”focus large”. The biggest performance difference tends to be in
scenarios with larger images ($\geq 10$ megapixels, see Table II), validating
the proposed blur aware pyramidal approach.

Figure 10: Overall Corner Location Accuracy

Figure 10 shows nominal corner localization accuracy using 50% error from all
true positive detections across all scenarios. Using 50% error avoids putting
libraries with more detections at a disadvantage by excluding outliers. The
proposed detector is effectively tied for best nominal accuracy. All libraries
exhibit reasonable nominal accuracy.

Library | Sloppy | ThetaS | Motion | Stef2012
---|---|---|---|---
| TP | E50 | E100 | TP | E50 | E100 | TP | E50 | E100 | TP | E50 | E100
Proposed | 13 | 0.42 | 2.17 | 11 | 0.25 | 0.89 | 37 | 0.67 | 4.45 | 30 | 0.18 | 0.83
DelChe | 13 | 0.47 | 2.12 | 11 | 0.26 | 0.80 | 10 | 0.43 | 2.18 | 23 | 0.24 | 1.15
Geiger | 13 | 0.42 | 2.97 | 11 | 0.24 | 0.94 | 14 | 0.60 | 3.23 | 30 | 0.19 | 0.81
OCamCalib | 10 | 0.60 | 3.28 | 6 | 0.44 | 1.44 | 32 | 0.84 | 4.78 | 26 | 0.36 | 4.28
Binary | 11 | 0.44 | 2.87 | 9 | 0.24 | 4.86 | 24 | 0.64 | 4.89 | 14 | 0.20 | 2.94
ChessSB | 13 | 0.53 | 2.47 | 11 | 0.31 | 4.22 | 33 | 0.67 | 4.25 | 26 | 0.25 | 3.15
TABLE I: Comparison of Corner Accuracy in Select Scenarios

Because overall nominal performance paints an unrealistically optimistic
picture, selected specific scenarios are now considered. Table I shows true
positives, 50% corner error, and max corner error in scenarios where most
detectors exhibited ”good” detection rates. Sloppy is a printed on wavy paper,
Motion has motion blur, Stef2012 regular camera, and ThetaS is an 185∘
fisheye. The reason max error tops out around 5 pixels is because detentions
are labeled as false positives at that point. Even if a detector has very low
50% error accuracy, calibration results will be poor if the max and false
positive rates are high.

Figure 11: Accuracy plots with increasing Gaussian blur. Left: Rendered
chessboard without perspective distortion. Right: Fisheye with hand labeled
corners.

Figure 11 shows accuracy degrading as a function of Gaussian blur in two
scenarios. Ideally a detector would slowly degrade as blur increases and
spikes could indicate fundamental instability in the approach. Oddly enough,
the rendered scenario with no lens or perspective distortion is very
challenging for some detectors, while the fisheye scenario exhibits stable
performance. This could be caused by x-corner symmetry discussed previously.
In both scenarios, the proposed approach has the most stable response and best
accuracy.

Figure 12: Average mean relative runtime across all scenarios. A score of 1
would mean it is the top performer in every category.

The proposed library is 1.9 times faster on average than the second fastest
and over 90 times faster than the second most accurate library, Figure 12.
Proposed has a stable runtime across scenarios Table II. Average runtime
metrics hide erratic performance (i.e. very fast or slow on similar images)
exhibited by some libraries, making them undesirable for real-time
applications. One explanation for erratic performance is poor handling of
excessive false positives, especially in higher resolution images. See Figure
4 for an example of how images can be covered in corners.

Dataset | MP | N | FP | FN | E50 | E100 | R50 | R100
---|---|---|---|---|---|---|---|---
|  |  |  |  | (px) | (px) | (ms) | (ms)
border | 12.2 | 16 | 0 | 0 | 0.29 | 1.71 | 103 | 151
caltech edges | 0.3 | 20 | 0 | 1 | 0.26 | 0.89 | 11.9 | 15.7
challenge | 1.4 | 85 | 1 | 20 | 0.23 | 4.51 | 45.3 | 66.9
close | 0.6 | 12 | 0 | 0 | 0.42 | 1.40 | 6.7 | 8.7
kaist fisheye | 1.9 | 58 | 0 | 3 | 0.21 | 1.08 | 26.4 | 41.6
distance angle | 0.5 | 6 | 0 | 0 | 0.21 | 0.59 | 10.6 | 30.8
distance straight | 0.5 | 8 | 0 | 0 | 0.18 | 0.67 | 10.0 | 16.6
distant | 0.5 | 7 | 0 | 0 | 0.20 | 0.63 | 16.4 | 19.7
focus | 0.5 | 36 | 0 | 0 | 0.44 | 2.89 | 7.2 | 17.2
focus large | 12.2 | 16 | 0 | 0 | 1.89 | 13.87 | 118 | 216
gaus fisheye | 0.5 | 12 | 1 | 3 | 0.18 | 1.61 | 5.5 | 7.3
gaus perfect | 0.1 | 12 | 0 | 0 | 0.02 | 0.10 | 2.3 | 3.2
large | 10.0 | 6 | 0 | 0 | 0.31 | 1.25 | 180 | 232
large shadow | 12.2 | 23 | 0 | 0 | 0.42 | 3.77 | 127 | 605
motion blur | 0.3 | 38 | 1 | 0 | 0.67 | 4.45 | 5.4 | 7.5
ocam fisheye190 | 0.4 | 8 | 0 | 1 | 0.22 | 1.16 | 8.3 | 13.0
ocam kaidan omni | 0.3 | 17 | 0 | 1 | 0.26 | 1.03 | 10.3 | 15.2
ocam ladybug | 0.8 | 13 | 0 | 0 | 0.26 | 0.84 | 10.8 | 16.1
ocam mini omni | 0.4 | 15 | 0 | 0 | 0.22 | 3.25 | 17.9 | 27.7
ocam omni | 1.2 | 14 | 0 | 0 | 0.29 | 1.11 | 27.0 | 36.1
perfect | 0.3 | 8 | 0 | 0 | 0.00 | 0.01 | 7.5 | 11.0
theta 5 | 0.5 | 11 | 0 | 0 | 0.25 | 0.89 | 13.0 | 16.0
theta V | 3.7 | 46 | 0 | 0 | 0.32 | 2.59 | 88.2 | 134
rotation flat | 0.5 | 12 | 0 | 0 | 0.28 | 0.97 | 17.1 | 20.5
rotation vertical | 0.5 | 7 | 0 | 0 | 0.24 | 0.66 | 9.8 | 15.8
shadow | 0.5 | 15 | 0 | 0 | 0.22 | 0.63 | 16.7 | 28.0
sloppy13x10 | 2.1 | 13 | 0 | 0 | 0.42 | 2.17 | 34.1 | 43.6
DSC-HX5V | 0.3 | 13 | 0 | 0 | 0.31 | 0.85 | 4.3 | 5.3
stefano 2012 | 0.3 | 30 | 0 | 0 | 0.18 | 0.83 | 10.2 | 13.8
TABLE II: Summary of Proposed Performance across all Scenarios

Proposed approach’s detection, accuracy, and runtime performance in each
scenario is shown in Table II. MP stands for image megapixels. When compared
against other libraries in individual scenarios [31], the proposed is always a
top performer. Making it the most consistent library.

##  VI Conclusion

A new chessboard detector is proposed that is designed to handle larger images
while being fast, accurate, and robust. This is accomplished using a new
x-corner detector, where for the first time, blur is detected and used to
select the best level in scale-space for corner location, dynamically adjust
sample points for edge validation, and determine corner connectivity. Several
incremental improvements are also proposed, such as correct handling of
x-corner symmetry and n-best edge score. Performance is validated across a
large set of scenarios using hand labeled and simulated data in what might be
the most challenging chessboard dataset yet. Only the proposed detector is a
top performer in both overall and all individual scenarios (particularly in
larger degraded images), while being the fastest library by a large margin.

## References

  * [1] Z. Zhang, “A flexible new technique for camera calibration,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 22, no. 11, pp. 1330–1334, 2000.
  * [2] J. Mallon and P. F. Whelan, “Which pattern? biasing aspects of planar calibration patterns and detection methods,” _Pattern recognition letters_ , vol. 28, no. 8, pp. 921–930, 2007.
  * [3] M. Fiala and C. Shu, “Self-identifying patterns for plane-based camera calibration,” _Machine Vision and Applications_ , vol. 19, no. 4, pp. 209–216, 2008.
  * [4] B. Atcheson, F. Heide, and W. Heidrich, “Caltag: High precision fiducial markers for camera calibration.” in _VMV_ , vol. 10, 2010, pp. 41–48.
  * [5] S. Garrido, “Charuco: Chessboard + aruco,” https://opencv.org/, 2015.
  * [6] Various, “Opencv library,” https://opencv.org, 2000.
  * [7] J.-Y. Bouguet, “Camera calibration toolbox for matlab,” 2001.
  * [8] L. Heng, B. Li, and M. Pollefeys, “Camodocal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 1793–1800.
  * [9] P. Abeles, “Boofcv’s camera calibration tools,” http://boofcv.org/, 2011-2021.
  * [10] H. Ha, M. Perdoch, H. Alismail, I. So Kweon, and Y. Sheikh, “Deltille grids for geometric camera calibration,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5344–5352.
  * [11] V. Vezhnevets and P. Gruebele, “Part of the opencv image processing library: findchessboardcorners(),” https://opencv.org/, 2010.
  * [12] M. Rufli, D. Scaramuzza, and R. Siegwart, “Automatic detection of checkerboards on blurred and distorted images,” in _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2008, pp. 3121–3126.
  * [13] S. Donné, J. De Vylder, B. Goossens, and W. Philips, “Mate: Machine learning for adaptive calibration template detection,” _Sensors_ , vol. 16, no. 11, p. 1858, 2016.
  * [14] C. G. Harris and M. Stephens, “A combined corner and edge detector.” in _Alvey vision conference_ , vol. 15, no. 50, 1988.
  * [15] L. Lucchese and S. K. Mitra, “Using saddle points for subpixel feature detection in camera calibration targets,” in _Asia-Pacific Conference on Circuits and Systems_ , vol. 2. IEEE, 2002, pp. 191–195.
  * [16] A. Kassir and T. Peynot, “Reliable automatic camera-laser calibration,” in _Proceedings of the 2010 Australasian Conference on Robotics & Automation_. ARAA, 2010.
  * [17] Y. Liu, S. Liu, Y. Cao, and Z. Wang, “Automatic chessboard corner detection method,” _IET Image Processing_ , vol. 10, no. 1, pp. 16–23, 2016.
  * [18] A. Geiger, F. Moosmann, Ö. Car, and B. Schuster, “Automatic camera and range sensor calibration using a single shot,” in _2012 IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 3936–3943.
  * [19] A. Duda and U. Frese, “Accurate detection and localization of checkerboard corners for calibration.” in _BMVC_ , 2018, p. 126.
  * [20] S. Placht, P. Fürsattel, E. A. Mengue, H. Hofmann, C. Schaller, M. Balda, and E. Angelopoulou, “Rochade: Robust checkerboard advanced detection for camera calibration,” in _European Conference on Computer Vision_. Springer, 2014, pp. 766–779.
  * [21] C. W. Niblack, P. B. Gibbons, and D. W. Capson, “Generating skeletons and centerlines from the distance transform,” _CVGIP: Graphical Models and image processing_ , vol. 54, no. 5, pp. 420–437, 1992.
  * [22] J.-M. Lavest, M. Viala, and M. Dhome, “Do we really need an accurate calibration pattern to achieve a reliable camera calibration?” in _European Conference on Computer Vision_. Springer, 1998, pp. 158–174.
  * [23] W. Sun and J. R. Cooperstock, “Requirements for camera calibration: Must accuracy come with a high price?” in _2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05)-Volume 1_ , vol. 1. IEEE, 2005, pp. 356–361.
  * [24] W. Förstner and E. Gülch, “A fast operator for detection and precise location of distinct points, corners and centres of circular features,” in _Proc. ISPRS intercommission conference on fast processing of photogrammetric data_ , 1987, pp. 281–305.
  * [25] A. Datta, J.-S. Kim, and T. Kanade, “Accurate camera calibration using iterative refinement of control points,” in _2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops_. IEEE, 2009, pp. 1201–1208.
  * [26] V. N. Dao and M. Sugimoto, “A robust recognition technique for dense checkerboard patterns,” in _2010 20th International Conference on Pattern Recognition_. IEEE, 2010, pp. 3081–3084.
  * [27] J. Shi and C. Tomasi, “Good features to track,” in _1994 Proceedings of IEEE conference on computer vision and pattern recognition_. IEEE, 1994, pp. 593–600.
  * [28] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the seventh IEEE international conference on computer vision_ , vol. 2. IEEE, 1999, pp. 1150–1157.
  * [29] A. P. Witkin, “Scale-space filtering,” in _Readings in Computer Vision_. Elsevier, 1987, pp. 329–332.
  * [30] J. Shi and C. Tomasi, “Good features to track,” in _Proceedings of IEEE conference on computer vision and pattern recognition_ , 1994, pp. 593–600.
  * [31] P. Abeles, “Yet to be published detailed technical report,” in _TBD_ , 2021\.
  * [32] D. Scaramuzza, A. Martinelli, and R. Siegwart, “A toolbox for easily calibrating omnidirectional cameras,” in _2006 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2006, pp. 5695–5701.
  * [33] S. Mattoccia, “Stereo images from videredesign monochrome megad,” http://vision.deis.unibo.it/ smatt/stereo/Calibration.html, 2006.
  * [34] T. Peynot, S. Scheding, and S. Terho, “The marulan data sets: Multi-sensor perception in a natural environment with challenging conditions,” _The International Journal of Robotics Research_ , vol. 29, no. 13, pp. 1602–1607, 2010\.


# UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation
for Multilingual Visual Word Sense Disambiguation

Michael Ogezi, Bradley Hauer, Talgat Omarov, Ning Shi, Grzegorz Kondrak
Alberta Machine Intelligence Institute
Department of Computing Science
University of Alberta, Edmonton, Canada
<EMAIL_ADDRESS>

###### Abstract

We describe the systems of the University of Alberta team for the SemEval-2023
Visual Word Sense Disambiguation (V-WSD) Task. We present a novel algorithm
that leverages glosses retrieved from BabelNet, in combination with text and
image encoders. Furthermore, we compare language-specific encoders against the
application of English encoders to translated texts. As the contexts given in
the task datasets are extremely short, we also experiment with augmenting
these contexts with descriptions generated by a language model. This yields
substantial improvements in accuracy. We describe and evaluate additional
V-WSD methods which use image generation and text-conditioned image
segmentation. Overall, the results of our official submission rank us 18 out
of 56 teams. Some of our unofficial results are even better than the official
ones. Our code is publicly available at https://github.com/UAlberta-NLP/v-wsd.

##  1 Introduction

This paper addresses our work on SemEval-2023 Task 1: Visual Word Sense
Disambiguation111https://raganato.github.io/vwsd/ Raganato et al. (2023). The
V-WSD task is closely related to WSD, and similarly involves understanding and
classifying the meaning of a polysemous word in context. The distinction is in
how classes are defined: In WSD, a system has access to a sense inventory that
enumerates the possible senses of each word, and the task is to classify the
focus word according to the sense that best corresponds to its intended
meaning. In V-WSD, a system is given a set of candidate images, and the task
is to select the image which depicts the intended meaning of the focus word.

Figure 1:  The task is to select the image that best represents the meaning of
the focus word (e.g., bat) in the context (e.g., “baseball bat.”)

The multi-modal nature of V-WSD introduces challenges not encountered in WSD.
First, image processing is generally more computationally intensive than text
processing. Second, a V-WSD system must represent the meanings of both images
and text, and must have mechanisms to compare these multi-modal semantic
representations. Last, since the candidate images in V-WSD are not restricted
to a sense inventory, they may exhibit highly variable levels of sense
granularity.

The V-WSD task is motivated by cases where textual context alone is
insufficient to disambiguate a word. In such cases, visual context may be
available to facilitate disambiguation. For example, the word _play_ is
ambiguous in the context “that was a good play,” as it may refer to a
theatrical performance or an action in a sport. However, an associated image
of a stage or a sports field would enable a V-WSD system to disambiguate
_play_.

We propose a novel V-WSD algorithm that ranks candidate images by embedding
images and words-in-context in a shared semantic space, while also taking
advantage of lexical knowledge bases commonly used in WSD. In particular, our
method uses sense glosses of the focus word to create representations of the
possible meanings that word may have. Our algorithm is flexible, It includes
several optional modules, as well as hyper-parameters that facilitate
customization, optimization, and detailed analysis.

We test various configurations of our method and analyze their performance.
Our three principal conclusions are as follows: First and foremost, the
augmention of the original textual context plays a crucial role in improving
performance. Second, there is a considerable gap between English and non-
English performance, indicating that bias towards English models extends to
the multi-modal setting. Third, we observe a major distribution shift between
the train and test sets, which is confirmed by our ablation study.

##  2 Related Work

Recent work on WSD can be divided into supervised and knowledge-based systems.
Supervised WSD methods depend on large training corpora in which some or all
of the content words have been tagged with their correct senses Blevins and
Zettlemoyer (2020); Barba et al. (2021). Knowledge-based methods depend on
other sources of linguistic knowledge Wang and Wang (2020). In general,
knowledge-based methods are outperformed by contemporary supervised methods
Pasini et al. (2021). Today, state-of-the-art WSD systems approach accuracy
limits imposed by inter-annotator agreement Maru et al. (2022).

Research on the incorporation of visual information for WSD is relatively
sparse. Barnard et al. (2003) propose a statistical model that associates
image regions and words to predict word senses. Loeff et al. (2006) apply
spectral clustering to group similar images corresponding to the same senses.
Saenko and Darrell (2008) employ an unsupervised approach to assign senses to
images using surrounding texts and dictionary definitions, and then train a
visual SVM classifier to disambiguate unseen images. Gella et al. (2019)
introduce the task of visual verb sense disambiguation, in which one image is
selected based on a given context. Vascon et al. (2021) propose a graph-based
semi-supervised transductive learning method for visual verb sense
disambiguation.

Multi-modal foundation models Bommasani et al. (2021) such as CLIP Radford et
al. (2021) and ALIGN Jia et al. (2021) can represent both text and images in a
shared embedding space. Recent work222https://github.com/moein-
shariatnia/OpenAI-CLIP Bianchi et al. (2021); Sajjad Ayoubi (2022) improves
the text encoder by using a pre-trained text-only encoder such as BERT Devlin
et al. (2018).

##  3 Task & Dataset

#### Task Definition:

Given a focus word $w$ in a short context $c$, and a set of candidate images
$I$, the task is to select the image $i^{*}\in I$ which best represents the
meaning of $w$ in $c$. For example, given the context “baseball bat” with bat
as the focus word, a V-WSD system should choose the image that depicts the bat
used in baseball (Figure 1).

#### Dataset:

The training data provided for this shared task consists of a silver dataset
with 12,869 V-WSD instances. Each sample is a 4-tuple $\langle f,c,I,i^{*}\in
I\rangle$ where $|I|=10$. The contexts are generally very short, often just a
single word in addition to the focus word. We randomly select 10% of the
training data for use as a development set. The test dataset consists of 968
instances, of which 463 are English, 200 are Farsi, and 305 are Italian. We
observe that many of the incorrect candidate images in the training data have
nothing to do with any sense of the focus word. However, in the test data, we
observe that this is less often the case, making the test set considerably
more difficult.

#### Evaluation Metrics:

The primary metric is the hit rate, which is equivalent to top-1 accuracy, or
simply accuracy. This is the proportion of instances for which the system
selects the correct image. We also compute the mean reciprocal rank Voorhees
and Tice (2000) which represents how highly V-WSD systems rank the ground-
truth image, on average.

Language | % | Example
---|---|---
English | 82 |  waxflower wildflower
Latin | 15 |  shorea genus
German | 2 |  truppenübungsplatz workplace
French | 1 |  brumaire month
Table 1: The language distribution of 100 instances from the training set. The
focus word is underlined.

#### Language Distribution:

We observed some instances where the context contained non-English words. To
estimate the prevalence of this phenomenon, we randomly selected 100 instances
from the training set and manually identified the language of each. For
example, the focus word _shorea_ in “shorea genus” comes from new Latin, and
refers to a genus of mainly rainforest trees. Table 1 shows the frequency of
each language in our sample.

##  4 Method

In this section, we describe the key components of our systems, including an
algorithm that combines text and image similarity measures.

###  4.1 Algorithm

We propose an algorithm to select a single image from a set of candidates that
best matches the context. To reiterate the problem, we are given a context $c$
containing a focus word $w$ and a set $I$ of candidate images. We assume that
we also have a non-empty set $G$ containing possible glosses of $w$; in
practice, we obtain $G$ from BabelNet using the freely available
API.333https://babelnet.org/guide

Our algorithm makes calls to two similarity functions: The first is $sim^{L}$,
a _written language_ similarity function, which takes as input two text
strings and returns a value indicating the semantic similarity between them.
The second is $sim^{VL}$, a _vision-to-written language_ similarity function,
which takes as input an image and a text string and returns a value indicating
the similarity between what the image depicts and what the text describes.

With these functions, for each candidate image $i\in I$, and for each gloss
$g\in G$ of the focus word $w$, we compute the pairwise similarity between:

  1. 1.

The image and context: $s_{ic}=sim^{VL}(i,c)$

  2. 2.

The image and gloss: $s_{ig}=sim^{VL}(i,g)$

  3. 3.

The context and gloss: $s_{cg}=sim^{L}(c,g)$

This allows us to identify the pair of a candidate image $i^{*}$ and gloss
$g^{*}$ that maximizes a weighted average of these three similarity scores.
Algorithm 1 shows the pseudocode for this algorithm.

Algorithm 1 Candidate Image Scoring

1:$c\leftarrow$ the context of the focus word

2:$G\leftarrow$ list of glosses for the focus word

3:$I\leftarrow$ list of candidate images

4:for $i$ in $I$ do

5: $s_{g}\leftarrow 0$

6: for $g$ in $G$ do

7: $s_{ig}\leftarrow w_{ig}\cdot sim^{VL}({i,g})$

8: $s_{cg}\leftarrow w_{cg}\cdot sim^{L}({c,g})$

9: $s_{g}\leftarrow\max(s_{g},s_{ig}+s_{cg})$

10: $scores[i]\leftarrow s_{g}+w_{ic}\cdot sim^{VL}({i,c})$

11:return $scores$

#### Hyperparameters:

Our algorithm depends on three weight hyperparameters: $w_{ic}$, $w_{ig}$, and
$w_{cg}$. They represent the weights for image-context, image-gloss, and
context-gloss similarity, respectively. Table 2 shows the results of the
hyperparameter binarized grid search performed on a 500-sample of the training
set. Based on our development experiment results, we decided to set all
hyperparameter weights to $1$ for simplicity, except where otherwise noted. We
discuss the hyperparameters further in Section 4.

$w_{ic}$ | $w_{ig}$ | $w_{cg}$ | Accuracy (%)
---|---|---|---
1 | 1 | 1 | 79.2
1 | 1 | 0 | 79.2
1 | 0 | 1 | 72.2
1 | 0 | 0 | 72.2
0 | 1 | 1 | 68.4
0 | 1 | 0 | 68.6
0 | 0 | 1 | 11.0
Table 2: Binarized grid search results for weight hyperparameters.

#### Context Augmentation:

For each instance, we prompt InstructGPT Brown et al. (2020); Ouyang et al.
(2022) to generate a definition for the context phrase. We use the following
prompt template: “For each line, define the phrase:” followed by the contexts,
one per line. For example, the context “baseball bat” is augmented to become
“baseball bat: a bat used to hit a baseball during the game of baseball.” The
use of this additional context is described in Section 5.3

#### Supplementary Training Data:

We speculate that the size of the training dataset may be a limiting factor in
the accuracy of our method. We, therefore, experiment with augmenting the
training data with additional data derived from BabelPic Calabrese et al.
(2020), a multi-modal resource which maps a subset of BabelNet synsets to sets
of one or more images. For each pair of a synset and an image, we enumerate a
lemma from the base synset and a lemma from a related synset. The two lemmas
are concatenated, starting with the lemma from the related base synset, to
form a two-word context. We then select nine other random images from
BabelPic, forming an instance comparable to those in the training set: a two-
word context with a single focus word, with ten images, one depicting the
correct sense of the focus. We create 54,968 instances this way and experiment
with adding this dataset to the training data at training time.

#### Glosses:

For each instance, we enumerate the BabelNet Navigli and Ponzetto (2012)
glosses corresponding to each sense of the focus word. If there are multiple
glosses for a single sense, we pick the first and add it to the set $G$. This
prevents senses from being over-represented due to the number of glosses in
BabelNet.

##  5 Systems

In this section, we describe our systems for the V-WSD task, Our official
system submissions are based on our primary systems: Tr and LangSpec. We also
describe two alternative systems, which do not use Algorithm 1. Both perform
worse than the primary systems, but their results are nevertheless valuable
for the purpose of analysis. We also present a supplementary method, which can
be optionally used in combination with our other systems. Non-English
instances are translated using DeepL444https://www.deepl.com/translator for
Italian and ChatGPT555https://openai.com/blog/chatgpt/ for Farsi.

###  5.1 Primary Systems

####  Tr: Image Scoring with Translations

If the input instance is not English, we translate it into English. Then we
apply Algorithm 1. We compute $sim^{VL}$ using embeddings from CLIP Radford et
al. (2021), an English-only model which encodes text and images in a shared
embedding space. We compute $sim^{L}$ using BERT Devlin et al. (2018) as an
English-only text encoder. We set the weight parameters: $w_{ic}$, $w_{ig}$,
and $w_{cg}$ to 1 in this specific case.

####  LangSpec: Image Scoring with Language-Specific Models

This system is similar to Tr, except that non-English instances are not
translated into English. This is our only system which directly operates in
other languages. Given a non-English instance, we replace CLIP and BERT with
language-specific models to compute $sim^{VL}$ and $sim^{L}$. For English
instances, this method is the same as Tr. For Italian, we use CLIP-Italian
Bianchi et al. (2021) to compute $sim^{VL}$ and Italian
BERT666https://huggingface.co/dbmdz/bert-base-italian-xxl-uncased to compute
$sim^{L}$. For Farsi, we use CLIPfa Sajjad Ayoubi (2022) to compute $sim^{VL}$
and ParsBERT Farahani et al. (2021) to compute $sim^{L}$.

###  5.2 Alternative Systems

####  Gen: Generative Image Model

This method takes a different approach compared to Tr and LangSpec; it does
not use Algorithm 1. Instead, we provide the context (translated into English,
if needed, as outlined above) as input to Stable Diffusion Rombach et al.
(2022), a generative model which takes a text prompt as input and produces
candidate images to depict what the text describes. For each context, we
generate $15$ images using $20$ diffusion steps each. We set the guidance
scale hyperparameter to $7.5$. For each candidate image, we compute its cosine
similarity with each generated image based on embeddings produced by CLIP. The
candidate with the highest similarity to the generated images is chosen as the
output.

####  Seg: Text-Conditioned Image Segmentation

As with Gen, this method does not use Algorithm 1. Instead, we use a zero-shot
image segmentation system Lüddecke and Ecker (2022) to segment images based on
the provided context. This system produces a _mean mask value_ , which we use
as a measure of similarity between the context and the segmented image; we
return the image with the highest mean mask value, given the context.

###  5.3 Supplementary Method

####  Def: Generating Additional Context

Tr, Gen, and Seg make use of the input context, translated to English as
needed. However, the contexts provided in the official dataset for this task
are extremely short. With Def, we generate additional context by using the
original context to prompt InstructGPT for a more extensive description, as
described in Section 4.1. We then concatenate the generated text to the
context and pass this augmented context to Tr, Gen, or Seg. We refer to the
methods using this supplementary method as Tr+Def, Gen+Def, and Seg+Def,
respectively. We do not combine Def with LangSpec, as we observe that
InstructGPT is less robust to short non-English contexts.

For Tr+Def, we set $w_{ig}$ and $w_{cg}$ to 0, as the improved context
obviates the need for their corresponding terms in Algorithm 1. Gen+Def and
Seg+Def, being based on Gen and Seg, do not depend on Algorithm 1.

##  6 Experiments

In this section, we present, discuss, and analyze our results.

| EN | IT | FA | Avg
---|---|---|---|---
Baseline | 60.5 | 22.6 | 28.5 | 37.2
Tr* | 61.1 | 59.3 | 43.0 | 54.5
Tr+Def | 69.1 | 63.3 | 40.0 | 57.5
LangSpec* | 56.8 | 37.7 | 14.5 | 36.3
Gen | 51.6 | 45.9 | 39.0 | 45.5
Gen+Def | 58.1 | 48.5 | 34.5 | 47.0
Seg | 31.5 | 29.8 | 20.5 | 27.3
Seg+Def | 34.1 | 36.7 | 20.0 | 30.3
Table 3:  Accuracy for English, Italian, and Farsi, along with the macro
average for all languages. We indicate our official system submissions with *.

###  6.1 Results

Table 3 shows our performance on the test set. We find that accuracy has a
99.46% Pearson correlation with mean reciprocal rank, and so for conciseness,
we report accuracy alone. The translation-based systems, Tr and Tr+Def, yield
the best results. One explanation for this outcome is the disproportionate
amount of English training data available to the models we build upon: CLIP
and BERT. The higher performance of these models on English appears to
compensate for the noise introduced by the translation process. We discuss
this further in Section 6.4.

An interesting trend is the benefit of context augmentation, (Section 5.3).
Between Tr and Tr+Def, we observe a 3% average improvement in accuracy. We
observe a similar trend in Gen versus Gen+Def and Seg versus Seg+Def.

We further observe that accuracy on English instances is highest, accuracy on
Farsi instances is lowest, while accuracy on Italian instances is in between
both. This corresponds to the quality and quantity of resources available for
each language. We undertake more thorough analyses in the next section.

###  6.2 Distribution Shift

As shown in Table 4, we observe a clear disparity in polysemy, and the
proportion of focus words which are nouns, between the training and test sets.
This difference is especially notable when considering the performance gap
between the sets.

| Train | Test
---|---|---
| EN | EN | IT | FA
Polysemy | 6.8 | 23.1 | 13.6 | 10.7
Nouns (%) | 74.7 | 88.1 | 91.5 | 92.5
Table 4: Distribution shifts between the training and test sets. Polysemy
indicates the average number of senses each focus word in the set has.

#### Zero-shot vs. Fine-tuning:

We observe that fine-tuning on the training set leads to a drop in performance
on the test set (Figure 2). This may be due to the divergence between the
training and test datasets outlined above.

$0$$1$$2$$3$$4$$5$$30$$30$$40$$50$$60$$70$$80$$90$Fine-tuning EpochsHit rate
(%)Dev PerformanceTest Performance Figure 2:  As we fine-tune on the training
set for more epochs, we see an increase in dev set performance, but a drop in
test set performance. Epoch 0 refers to using the model zero-shot, with no
fine-tuning.

###  6.3 Traditional WSD

Although both the V-WSD and WSD tasks have some similarities, we found that
some ideas drawn from WSD prove ineffective for V-WSD.

#### Using Glosses:

We observe empirically worse performance when using glosses in our algorithm.
Specifically, with Tr on the English test set, we obtain a hit rate of 61.1%
when we do not use glosses and 56.8% when we do. Such a steep drop (4.3%) is
surprising, especially since most state-of-the-art WSD systems explicitly use
glosses in their methods.

We posit that sense disambiguation in V-WSD is more focused on homonymy than
polysemy and, as a result, can be less nuanced than in WSD. For example, apple
could refer to a fruit or a tree. In an image depicting both, the focus may be
unclear. In WSD, this distinction is critical since tree and fruit are
distinct senses. In V-WSD, however, we can make a correct prediction without
deciding between both senses. As a result of this lower granularity, glosses
become less important.

#### Performance of WSD Systems on Context:

We manually disambiguated the sense of the focus word in a randomly-selected
set of 16 instances from the training set. We then applied a state-of-the-art
WSD system, ConSec Barba et al. (2021), to these instances. We observe that
ConSec sense predictions were accurate 50% of the time, falling considerably
below its reported accuracy of 82%.

###  6.4 English Hegemony

Natural language processing research often focuses on the English language, at
the expense of other languages Magueresse et al. (2020). The relative
performance of Tr and LangSpec reflects this phenomenon: Translating non-
English text to English, in order to apply an English encoder, can be expected
to introduce some noise due to translation errors and information loss.
However, we observe that this pipeline approach produces better results than
using an Italian or Farsi encoder directly. This suggests that the field’s
focus on English has yielded English encoders which are much better than those
available for Italian or Farsi. We speculate that advancing the state-of-the-
art for non-English encoders may yield even better performance, by avoiding
the need to translate to English.

###  6.5 Image Generation

As shown in Figure 3, when applying our image generation system (Gen), we
observe an increase in performance as we generate more images. Although the
performance jump when transitioning from 1 to 5 images is most pronounced, we
see benefits from scaling until a certain point, 10 images, where the trend
becomes unreliable.

$1$$5$$10$$15$$45$$50$$55$$60$Number of Images GeneratedHit rate (%)GenGen+Def
Figure 3: Hit rate (%) vs. number of images generated for Gen and Gen+Def.

###  6.6 Text-Conditioned Image Segmentation

With Seg, we can sometimes robustly segment images and predict masks
indicating the correct image, conditioning on the full context. However, this
method sometimes forms incorrect semantic representations. Appendix A details
more examples of Seg’s usage. In addition to Figure 4, we present more
extensive examples in Appendix A.

Figure 4:  Original images from the dataset depicting andromeda (Japanese
plant), andromeda (galaxy) and their two masks conditioned on “andromeda
tree.”

###  6.7 Algorithm Hyperparameters

Our algorithm uses three weights hyperparameters to balance pairwise
similarities. We set all weights to 1 based on Table 2. Comparison of results
with $w_{cg}$ set to 0 or 1 suggests that $sim^{L}(c,g)$ does not improve
performance. Two reasons support this finding. Firstly, images encode richer
representations, producing more precise $sim^{VL}(i,g)$ and $sim^{VL}(i,c)$,
while both context and glosses are discrete textual features, introducing
uncertainty to $sim^{L}(c,g)$. Secondly, we use CLIP and BERT to calculate
$sim^{VL}$ and $sim^{L}$, respectively. CLIP’s multi-modal pre-training may
offer better similarity scores, fitting this task better. Understanding these
findings more deeply is an interesting avenue for future research.

##  7 Conclusion

In this paper, we outlined our work on the recently proposed task of V-WSD. We
found that many ideas from traditional WSD are difficult to adapt to V-WSD,
and, moreover, WSD systems are generally not useful for V-WSD. We were
particularly surprised to find that, unlike in WSD, glosses appear to be
unhelpful for V-WSD. Contrariwise, our innovation of augmenting the context
did yield substantial gains in accuracy.

Further research will be needed to establish the connection between V-WSD and
the broader field of lexical semantics. We speculate that developing systems
for joint WSD and V-WSD may yield improvements in one or both tasks. Our work
here serves as a proof-of-concept establishing the utility of language models
and lexico-semantic resources in the developing task of V-WSD.

## Acknowledgements

This research was supported by the Natural Sciences and Engineering Research
Council of Canada (NSERC), and the Alberta Machine Intelligence Institute
(Amii). We also thank Behzad Shayegh for providing advice on the Farsi test
set.

## References

  * Barba et al. (2021) Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2021.  ConSeC: Word sense disambiguation as continuous sense comprehension.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1492–1503, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Barnard et al. (2003) Kobus Barnard, Matthew Johnson, and David Forsyth. 2003.  Word sense disambiguation with pictures.  In _Proceedings of the HLT-NAACL 2003 Workshop on Learning Word Meaning from Non-Linguistic Data_ , pages 1–5.
  * Bianchi et al. (2021) Federico Bianchi, Giuseppe Attanasio, Raphael Pisoni, Silvia Terragni, Gabriele Sarti, and Sri Lakshmi. 2021.  Contrastive language-image pre-training for the italian language.  _arXiv preprint arXiv:2108.08688_.
  * Blevins and Zettlemoyer (2020) Terra Blevins and Luke Zettlemoyer. 2020.  Moving down the long tail of word sense disambiguation with gloss informed bi-encoders.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1006–1017, Online. Association for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Calabrese et al. (2020) Agostina Calabrese, Michele Bevilacqua, and Roberto Navigli. 2020.  Fatality killed the cat or: BabelPic, a multimodal dataset for non-concrete concepts.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4680–4686, Online. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Farahani et al. (2021) Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2021.  Parsbert: Transformer-based model for persian language understanding.  _Neural Processing Letters_.
  * Gella et al. (2019) Spandana Gella, Frank Keller, and Mirella Lapata. 2019.  Disambiguating visual verbs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(2):311–322.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR.
  * Loeff et al. (2006) Nicolas Loeff, Cecilia Ovesdotter Alm, and David A. Forsyth. 2006.  Discriminating image senses by clustering with multimodal features.  In _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_ , pages 547–554, Sydney, Australia. Association for Computational Linguistics.
  * Lüddecke and Ecker (2022) Timo Lüddecke and Alexander Ecker. 2022.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7086–7096.
  * Magueresse et al. (2020) Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020.  Low-resource languages: A review of past work and future challenges.  _arXiv preprint arXiv:2006.07264_.
  * Maru et al. (2022) Marco Maru, Simone Conia, Michele Bevilacqua, and Roberto Navigli. 2022.  Nibbling at the hard core of Word Sense Disambiguation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4724–4737, Dublin, Ireland. Association for Computational Linguistics.
  * Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012.  Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network.  _Artificial intelligence_ , 193:217–250.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * Pasini et al. (2021) Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021.  XL-WSD: An extra-large and cross-lingual evaluation framework for word sense disambiguation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13648–13656.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.
  * Raganato et al. (2023) Alessandro Raganato, Iacer Calixto, Asahi Ushio, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2023.  SemEval-2023 Task 1: Visual Word Sense Disambiguation.  In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , Toronto, Canada. Association for Computational Linguistics.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695.
  * Saenko and Darrell (2008) Kate Saenko and Trevor Darrell. 2008.  Unsupervised learning of visual sense models for polysemous words.  In _Advances in Neural Information Processing Systems_ , volume 21. Curran Associates, Inc.
  * Sajjad Ayoubi (2022) Amir Ahmadi Sajjad Ayoubi, Navid Kanaani. 2022.  Clipfa: Connecting farsi text and images.  https://github.com/SajjjadAyobi/CLIPfa.
  * Vascon et al. (2021) Sebastiano Vascon, Sinem Aslan, Gianluca Bigaglia, Lorenzo Giudice, and Marcello Pelillo. 2021.  Transductive visual verb sense disambiguation.  In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 3049–3058.
  * Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000.  The TREC-8 question answering track.  In _Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00)_ , Athens, Greece. European Language Resources Association (ELRA).
  * Wang and Wang (2020) Ming Wang and Yinglin Wang. 2020.  A synset relation-enhanced framework with a try-again mechanism for word sense disambiguation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6229–6240, Online. Association for Computational Linguistics.

##  Appendix A Text-Conditioned Image Segmentation

###  A.1 Success Mode

In the successful case of this system, we see that we are able to segment the
object based on the text provided properly. See the figures below for details.

Figure 5: Original images from the dataset depicting on the left: andromeda
and on the right: andromeda.

Figure 6: Conditioned on the full “andromeda tree”

Figure 7: Conditioned on “andromeda”

Figure 8: Conditioned on “tree”

###  A.2 Failure Mode

In the failure case of this system, we see that we are unable to confidently
segment the object based on the text provided. See the figures below for
details.

Figure 9: Original images from the dataset depicting on the left: bank
(finance) and on the right: bank (river).

Figure 10: Conditioned on the full “bank erosion”

Figure 11: Conditioned on “bank”

Figure 12: Conditioned on “erosion”