# On Utilizing Relationships for Transferable Few-Shot Fine-Grained Object
Detection

Ambar Pal
Johns Hopkins University
<EMAIL_ADDRESS>Arnau Ramisa, Amit Kumar K C, René Vidal
Amazon Visual Search & AR
{aramisay, amitkrkc<EMAIL_ADDRESS>
###### Abstract

State-of-the-art object detectors are fast and accurate, but they require a
large amount of well annotated training data to obtain good performance.
However, obtaining a large amount of training annotations specific to a
particular task _i.e_., fine-grained annotations, is costly in practice. In
contrast, obtaining common-sense relationships from text, _e.g_. “a table-lamp
is a lamp that sits on top of a table”, is much easier. Additionally, common-
sense relationships like “on-top-of” are easy to annotate in a task-agnostic
fashion. In this paper, we propose a probabilistic model that uses such
relational knowledge to transform an off-the-shelf detector of coarse object
categories (_e.g_. “table”, “lamp”) into a detector of fine-grained categories
(_e.g_. “table-lamp”). We demonstrate that our method, RelDetect achieves
performance competitive to finetuning based state-of-the-art object detector
baselines when an extremely low amount of fine-grained annotations is
available ($0.2\%$ of entire dataset). We also demonstrate that RelDetect is
able to utilize the inherent transferability of relationship information to
obtain a better performance ($+5$ mAP points) than the above baselines on an
unseen dataset (zero-shot transfer). In summary, we demonstrate the power of
using relationships for object detection on datasets where fine-grained object
categories can be linked to coarse-grained categories via suitable
relationships.

##  1 Introduction

Figure 1: Overview of the proposed method. Given an image, we firstly use an
object detector trained on base classes to produce detections for lamp, table,
chair and their corresponding class probabilities. A relationship predictor
then takes in these detections as input and produces possible relations
between various objects in the image. Finally our model looks into a
relational knowledge base (details in Sec. 4.3) to deduce the fine-grained
categories table-lamp and coffee-table.

††This work was done while AP was a research intern at Amazon. AP is with the
Department of Computer Science, and the Mathematical Institute for Data
Science at Johns Hopkins University.

Object detection is an essential component of many computer vision
applications. Accordingly, steady progress over the past decade has led to
modern object detectors that are both fast and accurate _e.g_. [7, 6].
However, these state-of-the-art object detectors require large amounts of
training data in order to obtain good detection performance. For instance,
while off-the-shelf object detectors pre-trained on large public datasets like
MS-COCO [4] can detect coarse-grained classes like “table”, “lamp”, and
“chair” very well, they struggle at detecting and differentiating among
derivative fine-grained classes like “coffee-table”, “floor-lamp” and “table-
lamp”. Obtaining training data for such fine-grained classes is very expensive
and when obtained, training state-of-the-art detectors on such large amounts
of task-specific data requires significant computational resources. All these
factors create an important practical bottleneck in the applicability of
state-of-the-art detectors to specialized tasks.

On the other hand, common sense _relational knowledge_ that links coarse-
grained to fine-grained classes is readily available from text, _e.g_. “a
table-lamp is a lamp that sits on top of a table”, “a coffee-table is a table
in front of a sofa or chair arrangement”. Moreover, detecting and labelling
relationships between objects in an image, such as “on top of” and “in front
of”, has been studied widely under the name of scene-graph generation [14, 15,
10, 9]. One might immediately wonder whether annotating relationships requires
an even higher annotation effort. While the number of general spatial
relationships between objects in an image can be large (quadratic in the
number of object instances), general relationships are often not very
discriminative for the task of fine-grained classification. Discriminative
relationships often occur between a few pairs of objects, and are much easier
to label. Additionally, such discriminative relationships can often be derived
from fine-grained labels using automatic rules (as we shall see in Sec. 5.1),
eliminating the need for an expert annotator. Thus annotation of
discriminative relationships is a cheaper and much less time-consuming task
than fine-grained object bounding box annotation.

Such advances in object detection and relationship prediction motivate us
towards the central question that we aim to investigate in this work: _Can we
combine off-the-shelf coarse-grained object detection networks and
relationship prediction networks to obtain a fine-grained object detector that
achieves a performance comparable to that of the state of the art while
requiring much fewer training labels?_

In this work, we propose a principled approach to tackle the above question
which builds upon observations of humans solving the fine-grained annotation
task: Firstly, they identify the coarse-grained labels in the image, say
“lamp” and “table”. Secondly, they identify the relationships between the
coarse-grained objects, say “on top of”. Finally, they use the relationship
information available to label the fine-grained object, say “table-lamp”. We
propose a probabilistic model that factors into products corresponding to the
steps of the human reasoning process: identify the coarse-grained objects
_i.e_.,

$\displaystyle p({\rm ObjectBoundingBoxes,ObjectCoarseLabels|Image}),$

identify relationships given coarse-grained objects _i.e_.,

$\displaystyle p({\rm RelationshipLabels|Objects}),$

and finally identify fine-grained objects given relationships and coarse-
grained objects _i.e_.,

$\displaystyle p({\rm ObjectFineGrainedLabels|Relationships,Objects}).$

We evaluate the effectiveness of our method for two tasks: Few-Shot Learning
for a source domain and Zero-Shot transfer learning from a source domain to a
target domain. We find that exploiting relationships significantly reduces the
need for fine-grained levels and leads to more transferable representations.

In summary, we make the following contributions:

  1. 1.

We propose a probabilistic model we call RelDetect for detection of fine-
grained object classes. The model has three main components. First, an object
detection model for coarse-grained classes. Second, a detector of spatial
relationships among detected objects in the scene. Third, a relational
knowledge model linking the coarse classes to the fine-grained classes via the
detected spatial relationships to obtain a fine-grained class for each
detected object.

  2. 2.

We demonstrate the effectiveness of our approach for few-shot object detection
on the DeepRooms dataset [1] suited for furniture detection on images of
indoor home scenes. We evaluate our method quantitatively using the COCO
mAP@50 metric [4] commonly used in object detection. We demonstrate that our
method RelDetect is able to obtain few-shot performance competitive with
state-of-the-art baselines when training with $0.2\%$ of the available fine-
grained labels. We further evaluate qualitatively with examples of success and
failure cases.

  3. 3.

We demonstrate the effectiveness of our approach for zero-shot domain transfer
from DeepRooms to LivingRoom, a dataset having the same classes as DeepRooms
but a different data distribution. Our experiments show that learning using
relationships leads to transferrable representations that perform better than
fine-tuning baselines.

##  2 Related Work

Object Detection has improved significantly since the advent of deep learning,
and nowadays it is often used as a first step in many computer vision
processing pipelines. Object detectors are typically divided into one-stage
[6] and two-stage [7], with the former usually regarded as being faster at
inference time, and the later being easier to train and more accurate.
Although Convolutional Networks have dominated the state-of-the-art for object
detection for a long time, new transformer-based models are starting to
challenge their dominance [12].

Much of the object detection literature is built around the MSCOCO [4]
dataset, with comparatively few works tackling related problems like object
detection under domain shift [5] or fine-grained object detection. In Yang et
al. [13], the authors use existing image-level annotations for fine-grained
categories to augment the training of a detector on a coarse-level bounding
box dataset via visual correlations, producing a fine-grained detector. In
contrast, in our work we do not assume a pre-existing fine-grained
classification dataset, but only common-sense knowledge on the spatial
relationships between the coarse-level and fine-grained object categories, as
well as a generic relationship classifier.

Few-shot object detection can also potentially reduce the high cost of
labelling many fine-grained categories, by learning to detect new categories
given only a few examples. Wang et al. [11] propose to fine-tune only the last
layer of an existing detector on rare clases, to drastically reduce the amount
of data needed to fit the parameters, while leveraging the intermediate
representations learned in the baseline training. Sohn et al. [8] uses pseudo-
labeled data and strong data augmentations to obtain improved performance in
low labelled data regimes.

Scene Graph Generation (SGG) focuses on determining relationships between
objects and their attributes in an image as a basis for higher level
understanding of the visual scene. Yang et al. [14] first determine relevant
relations between object proposals using a Relation Proposal Network, and then
they update the resulting graph using an attention Graph Convolutional Network
to exploit correlations between the object and relationships to improve
prediction accuracy. Zellers et al. [15] analyze motifs (regularly appearing
substructures in scene graphs) and determine a frequency prior that results on
improved performance. Tang et al. [10] use causal and counterfactual reasoning
to remove bias induced by the suboptimal annotation of most SGG dataset, where
certain simple relationship labels are heavily overused with respect to more
informative ones. Suhail et al. [9] propose an energy-based loss for scene
graph generation that takes into account the structure of the scene, instead
of treating the objects and relationships as independent entities. Recently,
Han et al. [2] surveyed the existing literature and published a benchmark to
compare SGG methods.

##  3 Probabilistic Model for Relationship-Based Fine-Grained Detection

Problem Statement Assume we are given an off-the-shelf object detector trained
to detect $\tilde{K}$ coarse classes and a relational knowledge base $B$ that
links coarse classes to $K$ fine-grained classes via spatial relationships of
$L$ different types. For example, the knowledge base entry “a table-lamp is a
lamp that sits on top of a table” links the coarse classes “lamp” and “table”
to the fine-grained class “table-lamp” via the relationship “on top of”.
Assume further we are given a small dataset of images of coarse-grained class
objects annotated with a few relevant spatial relations. Our task is to build
a detector to accurately localize and classify objects belonging to fine-
grained derived classes with little to no training examples for the fine-
grained classes.

Notation Let $I$ denote the input image and let $\\{S_{i}\\}_{i=1}^{N}$ denote
the spatial bounding box descriptions for $N$ detected bounding boxes, _e.g_.
their location and size. Let $\tilde{C_{i}}\in\\{1,\ldots\tilde{K}\\}$ and
$C_{i}\in\\{1,\ldots K\\}$ denote, respectively, the coarse and fine-grained
classes for bounding box $i$, and let $R_{i,j}\in\\{1,2,\ldots,L+1\\}$ denote
the relation between bounding boxes $i$ and $j$. The relation $L+1$ denotes
_no-relation_. Finally, given a bounding box $i$, the set ${\rm NB}_{i}$
denotes the spatial neighbors of $i$.

Probabilistic Model Given image $I$, our joint probabilistic model
$p(C,R,\tilde{C},S|I)$ can be factored as

$p(C|R,\tilde{C},S,I)p(R|\tilde{C},S,I)p(\tilde{C}|S,I)p(S|I).$ (1)

In what follows, we define each term in Eq. 1, and make some simplifying
assumptions to make the corresponding part of our model tractable.

Region Proposal The term $p(S|I)$ can be seen as a region proposal network
RPN, which takes an image as input and produces a collection of bounding boxes
$S=\\{S_{i}\\}_{i=1}^{N}$. The term $p(\tilde{C}|S,I)$ can be seen as an
object classification head OBJ which predicts the coarse-grained base class of
the objects given the image and the bounding boxes. Hence, the factor
$p(S|I)p(\tilde{C}|S,I)$ represents an off-the-shelf object detector for
coarse classes. Sec. 4.1 details the specific implementation for RPN and OBJ
we use.

Relationship Prediction The term $p(R|\tilde{C},S,I)$ predicts all
relationships given all the coarse object classes and their corresponding
bounding boxes. In order to enable make the inference more tractable, we make
the following simplifying assumption:

  1. A1.

_The relationships are conditionally independent given the coarse classes, the
image and the spatial coordinates of the bounding boxes_. That is,
$\\{R_{i,j}\\}_{i,j=1}^{N}$ are conditionally independent given $I,S$.
Intuitively, this assumption says that the relationships do not affect each
other once we know the bounding boxes, their image features, and their coarse
classes.

We simplify $p(R|\tilde{C},S,I))$ using assumption A1 as

$\displaystyle p(R|\tilde{C},S,I)=\prod_{i,j}p(R_{i,j}|\tilde{C},S,I).$ (2)

Each of the terms $p(R_{i,j}|\tilde{C},S,I)$ can be seen as a relationship
prediction network REL that takes in object bounding boxes, their image
features and the coarse class predictions as input, and produces the
relationship between boxes $i$ and $j$ as output. We utilize a state-of-the-
art network architecture from the Scene Graph Generation literature for REL
and detail the resultant implementation in Sec. 4.2.

#### Knowledge Base Query

The term $p(C|R,\tilde{C},S,I)$ finally predicts all the fine-grained classes
given all the relationships, coarse object classes and their bounding boxes.
Again, we can make inference more tractable by making the following
simplifying assumption:

  1. A2.

_The fine-grained class of a bounding box is conditionally independent of (a)
the coarse classes of bounding boxes not in its neighborhood, (b) its
relationships with boxes not in its neighborhood, and (c) the relationships
among other boxes, given the coarse classes and relationships in its
neighborhood 111 This definition is generic, but for our experiments we take
the neighborhood ${\rm NB}_{i}$ to be the set of bounding boxes $j$ whose
centers are at most a fixed distance $\tau$ from the center of box $i$. ._
That is, $C_{i}$ is conditionally independent of $\tilde{C}_{j}$ and $R_{ij}$
given $I,S$ when $j\not\in{\rm NB}_{i}$, and $C_{i}$ is conditionally
independent of $R_{j,k}$ given $I,S$ when $j\neq i$ and $k\neq i$.
Intuitively, this assumption says that objects very far away in the scene do
not affect each other once we know the objects in the neighborhood and the
relationships among them.

Using assumption A2, we obtain the simplification

$p(C|R,\tilde{C},S,I)=\prod_{i}p(C_{i}|R_{i,{\rm NB}_{i}},\tilde{C}_{{\rm
NB}_{i}}),$ (3)

where each of the factors $p(C_{i}|R_{i,{\rm NB}_{i}},\tilde{C}_{{\rm
NB}_{i}})$ can be seen as querying the relational knowledge base $B$. As an
example, let $C_{1}$ be the bounding box for a “lamp” and its neighborhood
${\rm NB}_{1}=\\{1,2\\}$ be the objects spatially close to this lamp in the
scene. If the knowledge base contains “table-lamp is lamp on top of table”,
then the conditional probability $p(C_{1}=\textsc{table-
lamp}|C_{1}=\textsc{lamp},R_{1,2}=\textsc{on top of},C_{2}=\textsc{table})$
should be $1$. This can be implemented by a rule-based model detailed in Sec.
4.3.

Figure 2: Different components of our proposed model. Given an image $I$, the
region proposal model is used to produce spatial coordinates $S$ of the object
bounding boxes. This is then used in the coarse classification model to
produce the coarse classes $\tilde{C}$. Then, the relationship prediction
model takes the coarse classes and bounding boxes as inputs to produce the
relationships $R$. Finally, the fine-grained classes $C$ are obtained by
utilizing the knowledge base $B$.

Finally, we put together Eqs. 2, 3 and 1 to obtain the full factorization:

$\displaystyle p(C,R,\tilde{C},S|I)$
$\displaystyle=p(S|I)p(\tilde{C}|S,I)\prod_{i,j}p(R_{i,j}|\tilde{C},S,I)$
$\displaystyle\quad\prod_{i}p(C_{i}|R_{i,{\rm NB}_{i}},\tilde{C}_{{\rm
NB}_{i}})$ (4)

The model described above is summarized in Fig. 2. We now proceed to describe
how each component is implemented in Sec. 4.

##  4 RelDetect \- Model Implementation

In this section, we describe our implementation of the three components of our
model: coarse detector, the relationship prediction network and the fine-
grained classifier.

###  4.1 Region Proposal Network and Coarse Classifier

Our Region Proposal Network RPN is implemented as a neural network that
predicts bounding boxes $S$ given the image $I$. We use the region proposal
network $f_{\rm RPN}$ from an off-the-shelf object detector, Faster RCNN [7].
Each spatial description $S_{i}\in\mathbb{R}^{4}$ is a 4-tuple containing the
height, width and the center $x,y$ coordinates of the bounding box. Note that
there is no uncertainty involved in the spatial description, hence the
probability model $p(S|I)$ can be seen as a dirac delta defined as:

$p(S|I)=\delta_{f_{\rm RPN}(I)}$ (5)

The coarse classifier is then implemented as an object classification head
$f_{\rm coarse}$ on top of image features extracted from each predicted
$S_{i}$. The feature extraction is performed using RoI Align [3] on top of
features extracted by a deep network from the image $I$. The probability model
is given by:

$p(\tilde{C}_{i}|S_{i},I)=f_{\rm coarse}(\textsc{Roi-Align}(S_{i},I))$ (6)

###  4.2 Relationship Prediction Network

Our Relationship Prediction Network REL is implemented as a network that takes
as input the predicted bounding boxes $S_{1},\ldots,S_{N}$ and the predicted
classes $C_{1},\ldots,C_{N}$ and outputs the relationship probabilities for
all $N^{2}$ pairs of objects. Note that since the number of relationships to
be predicted is quadratic in $N$, a non-maximum suppression step is applied to
the bounding boxes as a preprocessing step to make computation feasible. We
follow the architecture design in [15, 10] and set up REL as a bidirectional
LSTM followed by a MLP.

An object representation $\mathbf{o}_{i}$ is computed for each object $i$ by
concatenating an embedding of the predicted coarse class, the RoI-Align
feature, and an spatial embedding $f_{\rm SP}$ of the predicted bounding box
$S_{i}$:

$\mathbf{o}_{i}=[\mathbf{W}_{1}\mathbf{\tilde{c}}_{i};\textsc{Roi-
Align}(S_{i},I);f_{\rm SP}(S_{i})]$ (7)

In the above, $\mathbf{W}_{1}$ is a learnt embedding and
$\mathbf{\tilde{c}}_{i}\in\\{0,1\\}^{\tilde{K}}$ is a one-hot vector
representing the predicted coarse class.

A global contextualized embedding $\mathbf{e}_{i}$ is then computed for each
object using a bidirectional LSTM on the object embedding sequence
$\\{\mathbf{o}_{i}\\}$:

$\mathbf{e}_{i}={\rm
biLSTM}([\mathbf{o}_{1},\mathbf{o}_{2},\ldots,\mathbf{o}_{N}])$ (8)

Finally, for every pair $\mathbf{e}_{i},\mathbf{e}_{j}$, the relationship
probability model is given by a network $f_{\rm REL}$ that additionally takes
as input the RoI-Align feature computed from the union box corresponding to
$S_{i},S_{j}$ denoted as $S_{i}\cup S_{j}$:

$p(R_{i,j}|\tilde{C},S,I)=f_{\rm
REL}(\mathbf{e}_{i},\mathbf{e}_{j},\textsc{Roi-Align}(S_{i}\cup S_{j},I))$ (9)

###  4.3 Knowledge Base Model

Our knowledge base $B$ consists of relationship tuples linking the base
classes to the fine-grained classes via relationships. These tuples are
extracted from common-sense knowledge sources, _e.g_. Wikipedia, and take the
form “an object $i$ belonging to coarse-class $\tilde{c}_{2}$ should be
classified as fine-grained-class $c$ when there is a relation relationship $r$
between object $i$ and object $j$ which belongs to coarse-class
$\tilde{c}_{2}$”. The collection of such tuples forms our knowledge base $B$:

$B=\\{(c_{i},\tilde{c}_{i,1},\tilde{c}_{i,2},r_{i})\\}$ (10)

Now given $B$ and a predicted bounding box $i$, we search for possible
knowledge-base matches among the neighboring boxes of $i$ _i.e_. $N_{i}$. This
gives the final probability model for the fine-grained class prediction:

$\displaystyle p(C_{i}|R_{i,{\rm NB}_{i}},\tilde{C}_{{\rm
NB}_{i}})=\operatorname*{\scalerel*{+}{\sum}}_{j\in{\rm
NB}_{i}}\mathbbm{1}[(C_{i},\tilde{C}_{i},\tilde{C}_{j},R_{ij})\in B]$ (11)

In the above the $\operatorname*{\scalerel*{+}{\sum}}$ operator evaluates to
$1$ if any of the operands is $1$, and evaluates to $0$ otherwise.

During inference, given an image, we first of all use the neural network RPN
for predicting a set of bounding boxes in the image, which is then refined by
applying non-maximum suppression. The obtained boxes are then classified into
coarse categories by the object classification head $f_{\rm coarse}$. The
relationship prediction network REL is then used to predict all relationships
between the obtained bounding boxes. Finally, the knowledge base model Eq. 11
is applied using the predicted coarse classes and relationships to obtain the
fine-grained predictions. This completes the description of our model
RelDetect. We now proceed to describe the experimental protocols and results
in Sec. 5.

Method |  DeepRooms Labels Used |  Relationship Annotations Used | mAP DeepRooms | mAP LivingRoom
---|---|---|---|---
|  |  |  Coarse |  Fine-Grained |  Coarse |  Fine-Grained
RelDetect |  Coarse |  AUTO ($0.2\%$) |  95.59 |  75.46 |  71.64 |  52.70
RelDetect |  Coarse |  AUTO ($2\%$) |  95.97 |  82.17 |  72.18 |  55.33
RelDetect |  Coarse |  AUTO ($4\%$) |  96.00 |  79.78 |  72.26 |  55.36
RelDetect |  Coarse |  AUTO ($10\%$) |  97.55 |  79.36 |  75.16 |  56.45
RelDetect |  Coarse |  AUTO ($40\%$) |  96.26 |  75.21 |  74.30 |  53.82
RelDetect |  Coarse |  AUTO ($80\%$) |  95.90 |  70.16 |  73.46 |  50.89
RelDetect |  Coarse |  AUTO ($100\%$) |  95.53 |  70.48 |  73.30 |  49.94
Fine-Tuning |  Fine-Grained ($0.2\%$) |  $-$ |  78.56 |  78.28 |  49.86 |  46.86
Fine-Tuning |  Fine-Grained ($100\%$) |  $-$ |  89.85 |  94.89 |  75.29 |  68.97
Table 1: Experimental Results: Comparing Few-Shot Capabilities. The metric
reported is mAP@50 over the Coarse or Fine-Grained classes. Detailed per-class
AP scores can be found in the Appendix. No labels from the LivingRoom dataset
were used for training. We use Faster RCNN as the detector for coarse classes.
For any row, a dashed ($-$) column indicates an invalid setting _e.g_., the
fine-tuning baselines do not need any relationship annotations for training.
See Sec. 5.1 for details and experimental protocol.

##  5 Experiments

In this section, we describe our experimental studies. We first describe the
datasets that we use for evaluation, and then detail our experimental
protocols and results.

Datasets We show experimental results on two datasets curated for the task of
furniture detection: (1) DeepRooms [1], having $48,474$ images with $169,877$
bounding box annotations and (2) LivingRoom, having $1506$ images with $8,193$
bounding box annotations. Each dataset consists of images of living rooms with
furniture items (objects) labelled with the 8 fine-grained classes {chair,
sofacouch, coffee-table, end-table, console-table, table-lamp, floor-lamp,
rug} corresponding to the 5 coarse classes {chair, sofacouch, table, lamp,
rug}. There are no relationship labels available in the datasets. For the
purposes of our experiments, we will outline our Auto protocol to obtain the
relationship labels.

We clarify two important aspects about our experiments on these datasets:

  * •

The training split of the DeepRooms dataset is used for learning our detector
and we use the testing split for demonstrating our first main result in Sec.
5.1: Detection using relationships provides comparable extremely few-shot
performance than fine-tuning based methods [11].

  * •

The LivingRoom dataset contains more in-the-wild images than DeepRooms. We use
it only for testing transfer performance, and demonstrating our second main
result in Sec. 5.2: Detection using relationships provides stronger zero-shot
domain transfer performance than fine-tuning based methods when very few fine-
grained labels are available.

Fine-Tuning Baselines We compare RelDetect to fine-tuning baselines in Tabs. 1
and 2. The detectors used are FasterRCNN [7] and YOLOv3 [6]. In all the fine-
tuning baselines, the coarse detector was first of all pre-trained on a class
converted version of DeepRooms, where all the fine-grained labels were
converted to their corresponding coarse labels. The the detectors were then
fine-tuned on fine-grained labels (hence the name Fine-Tuning baseline). In
each table, the dataset and the size of these fine-grained labels is mentioned
on each fine-tuning row.

###  5.1 Few-Shot Detection using Relationships

In order to demonstrate the few-shot detection capabilities of RelDetect in
Tab. 1, we follow an experimental protocol of using a fraction of the fine-
grained labels available in our training dataset, DeepRooms.

Method |  DeepRooms Labels Used |  Relationship Annotations Used |  LivingRoom Labels Used |  Coarse Detector | mAP LivingRoom
---|---|---|---|---|---
|  |  |  |  |  Coarse |  Fine-Grained
RelDetect |  Coarse |  AUTO ($0.2\%$) |  None |  Faster RCNN |  71.64 |  52.70
Fine-Tuning |  Fine-Grained ($0.2\%$) |  $-$ |  None |  Faster RCNN |  49.86 |  46.86
Fine-Tuning |  Fine-Grained ($100\%$) |  $-$ |  None |  Faster RCNN |  89.85 |  68.97
Fine-Tuning |  Coarse (100%) |  $-$ |  Coarse (100%) |  YOLOv3 |  80.03 |  $-$
Fine-Tuning |  Coarse (100%) |  $-$ |  Fine-Grained ($11\%$) |  YOLOv3 |  67.56 |  60.34
Fine-Tuning |  Fine-Grained ($100\%$) |  $-$ |  None |  YOLOv3 |  70.77 |  62.96
Fine-Tuning |  Fine-Grained ($100\%$) |  $-$ |  Fine-Grained ($100\%$) |  YOLOv3 |  79.62 |  72.56
Table 2: Experimental Results: Comparing Zero-Shot Domain Transfer
Capabilities. The metric reported is mAP@50 over the Coarse or Fine-Grained
classes. Detailed per-class AP scores can be found in the Appendix. For any
row, a dashed ($-$) column indicates an invalid setting _e.g_., a fine-tuning
baseline trained with only coarse labels cannot predict fine-grained
categories. See Sec. 5.2 for details.

Experimental Protocol More specifically, the experimental protocol consists of
two steps.

  1. 1.

To mimic a coarse-grained object detector trained on a large dataset, we train
the methods on coarse-grained labels {chair, sofacouch, table, lamp, rug} on
the entire dataset.

  2. 2.

Then, to mimic availibility of a small amount of fine-grained task specific
labels, {chair, sofacouch, coffee-table, end-table, console-table, table-lamp,
floor-lamp, rug}, we only allow the methods to access $x\%$ of the images in
the entire dataset, annotated with fine-grained object labels. $x$ is varied
from $0.2$ to $100$.

Auto Relationship Annotation Since RelDetect needs relationship annotations
between object boxes labelled with the coarse classes, we propose a method for
automatic relationship annotation (AUTO) using the only $x\%$ fine-grained
labels (as described above). In Auto, we find object pairs in the image that
satisfy the knowledge base, _i.e_. for a tuple
$\\{(c,\tilde{c}_{1},\tilde{c}_{2},r)\\}$, we find all boxes $i,j$ such that
the coarse class of object $i$ is $\tilde{c}_{1}$ and that of object $j$ is
$\tilde{c}_{2}$, and the fine-grained class of object $i$ is $\tilde{c}$.
Among such pairs $(i,j)$, we retain only those pairs where the centers of the
predicted bounding boxes are at most a fixed distance $\tau$ from each other
222We find that this fixed-distance based filtering scheme works reasonably
for our datasets. Dynamic neighborhood generation schemes might perform better
for datasets having a large variation in object sizes.. These retained pairs
are labelled with the corresponding relationship $r$. These annotated
relationships are used to train the relationship prediction network REL from
Sec. 4.2, and this process is labelled AUTO $x\%$ in Tab. 1.

With the above protocol, we compare RelDetect to a strong baseline which is
trained with access to all ($100\%$) of the fine-grained and coarse-grained
labels in DeepRooms. The results are presented in Tab. 1. In order to keep the
comparison fair across datasets of widely varying sizes, we have fixed the
total number of iterations that each method is trained with to $10,000$
iterations. This in turn also ensures that each row is trained for the same
amount of GPU time.

Result Analysis Firstly, we observe that training using relationships inferred
from fine-grained annotations on $0.2\%$ of DeepRooms images leads to a
performance which is comparable to training an object detector with the same
data, _i.e_. $75.46$ mAP vs $78.28$ mAP. This demonstrates that learning with
relationships approximately retains performance in low-data regimes. As the
fraction of available fine-grained annotations ($x$) is increased, we see that
the performance on DeepRooms gradually increases, peaking at $x=2$, where the
performance is within $12$ mAP points of the baseline trained with $100\%$ of
the data ($82.17$ mAP vs $94.89$ mAP). While this gap might seem significant,
note that RelDetect is trained using no fine-grained bounding boxes, while the
Fine-Tuning baseline is trained with fine-grained bounding boxes on $50\times$
more image data. This demonstrates that learning with relationships provides a
good trade-off between obtaining fine-grained annotations and performance. We
observe that increasing $x$ further deteriorates the performance. We believe
this might be due to the inaccuracies in the AUTO annotation rule which might
negate the benefits of learning with relationships. On the other hand, we note
that the fine-tuning baseline trained on the full dataset is able to obtain
the best Fine-Grained performance on DeepRooms, _i.e_. $94.89$ mAP, reassuring
us that training a standard detector for a long time with task-specific labels
does provide good performance on the task.

Figure 3: Qualitative Results from our method RelDetect. Considering the top-
left image: The bounding boxes show the fine-grained labels output from
RelDetect. We draw an arrow with the label $r_{i,j}$ between the centers of
two bounding boxes $i,j$ when the probability of corresponding relationship
$r_{i,j}$ is above a threshold. In this example, the base categories “lamp”
(yellow) and “table” (green) were predicted, and the relationship “on top of”
was predicted, leading to a fine-grained label of “table-lamp”. Best viewed in
color.

###  5.2 Zero-Shot Domain Transfer using Relationships

In this section we compare the first RelDetect row in Tab. 1 to fine-tuning
baselines in order to understand the benefits of learning with relationships
when we have no access to labels from a target domain.

Experimental Protocol For our method RelDetect, the protocol remains the same
as Sec. 5.1. For the finetuning baselines, we allow access to $y\%$
($y\in\\{0,11,100\\})$333$11\%$ corresponds to $1000$ bounding box annotations
in LivingRoom of fine-grained labels from the target domain, which is taken to
be the LivingRoom dataset. We then report the mAP on the target domain in Tab.
2.

Result Analysis We firstly observe (from Tab. 1) that the source domain
(DeepRooms) and target domain (LivingRoom) have a large distribution shift, as
all methods drop around $20$ to $30$ mAP points from DeepRooms to LivingRoom.
This also suggests that detection on LivingRoom is a harder task. Secondly, we
observe that RelDetect is able to utilize relationships in a low-data regime
($0.2\%$ fine-grained labels) to improve performance on LivingRoom by $+5.68$
mAP considering a baseline trained with the same number fine-grained
annotations, _i.e_. $52.70$ mAP vs $47.02$ mAP. This demonstrates that
relationships are good for learning transferrable representations, which is
the second main experimental insight from this section. Finally, we observe
from the last few rows of Tab. 2 that training on even a small fraction
($11\%$) of fine-grained labels from the target domain leads to a large
increase in performance, reassuring us that training with good task-specific
labels provides the best performance. Having analysed RelDetect
quantitatively, we now show a qualitative evaluation in Fig. 3.

##  6 Conclusion, Shortcomings and Future Work

In this work, we proposed RelDetect, a relationship based probabilisitic model
for fine-grained object detection. RelDetect utilizes relationships between
coarse-grained object classes and an external knowledge base to predict fine-
grained classes. We demonstrated via experiments that such learning from
relationships enables us to obtain reasonable performance in limited data
regimes where we have access to a small amount of fine-grained training data
(few-shot), and superior performance where we have access to no training data
from a target domain (zero-shot transfer).

In this work, we limit ourselves to datasets where there is a tight coupling
between the coarse-grained classes and the fine-grained classes via suitable
relationships. Our method is intended for settings where we can obtain a set
of common sense rules (_i.e_., the knowledge-base $B$ in our case) where one
can infer a fine-grained object category by looking at relationships between
objects belonging to coarse categories in the scene. While this hierarchy is
very natural in many fine-grained object detection tasks, there are tasks
where such a hierarchy is not readily available, _e.g_., detecting and
categorizing fine-grained breeds of dogs. However, recent works in zero-shot
learning (ZSL) have developed methods to identify a suitable hierarchy from
large text and image corpora. More specifically, we have the hierarchy coarse-
grained $\to$ relationships $\to$ fine-grained, whereas the ZSL literature
works with base classes $\to$ class attributes $\to$ novel-classes. Such
attributes would lead to a natural generalization of our relationship-based
detection framework. Our relationship prediction network would then generalize
to an attribute prediction network, which takes as input one or more coarse-
objects, and predicts an attribute.

While we show improvements with our noisy AUTO annotation pipeline, it can
often lead to inaccurate relationship labels depending on the distance
threshold $\tau$. A better method for relationship annotation, _e.g_.
selecting $\tau$ dynamically per image, could lead to an improvement in
RelDetect performance.

## References

  * [1] Raghudeep Gadde, Qianli Feng, and Aleix M Martinez.  Detail me more: Improving gan’s photo-realism of complex scenes.  In ICCV, pages 13950–13959, 2021.
  * [2] Xiaotian Han, Jianwei Yang, Houdong Hu, Lei Zhang, Jianfeng Gao, and Pengchuan Zhang.  Image scene graph generation (sgg) benchmark, 2021.
  * [3] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [4] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [5] Poojan Oza, Vishwanath A. Sindagi, Vibashan VS, and Vishal M. Patel.  Unsupervised domain adaptation of object detectors: A survey, 2021.
  * [6] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [7] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: towards real-time object detection with region proposal networks.  PAMI, 39(6):1137–1149, 2016.
  * [8] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.  A simple semi-supervised learning framework for object detection, 2020\.
  * [9] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal.  Energy-based learning for scene graph generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936–13945, 2021.
  * [10] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3716–3725, 2020.
  * [11] Xin Wang, Thomas Huang, Joseph Gonzalez, Trevor Darrell, and Fisher Yu.  Frustratingly simple few-shot object detection.  In International Conference on Machine Learning, pages 9919–9928. PMLR, 2020.
  * [12] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu.  End-to-end semi-supervised object detection with soft teacher.  Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [13] Hao Yang, Hao Wu, and Hao Chen.  Detecting 11k classes: Large scale object detection without fine-grained bounding boxes, 2019.
  * [14] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh.  Graph r-cnn for scene graph generation.  In Proceedings of the European conference on computer vision (ECCV), pages 670–685, 2018.
  * [15] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi.  Neural motifs: Scene graph parsing with global context.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5831–5840, 2018.


Present address: ]Uni Bayreuth, Univertitätstr. 30, 95447, Bayreuth

# Hole-initiated melting process of thin films

Chenyu Jin  [<EMAIL_ADDRESS>Hans Riegler<EMAIL_ADDRESS>Max Planck Institute of Colloids and Interfaces, Science Park Golm, 14476
Potsdam, Germany

###### Abstract

It is generally believed that melting of a thin film starts from a small
defect, e.g., a hole on the film that acts as the nucleation site. We show in
this study that when the contribution of the capillary surface (liquid/air
interface) is taken into consideration, the melting process is non-trivial: 1)
There may be multiple melting pathways for one system, and hence different “de
facto” melting points. 2) Melting of a thin film may start from the long,
straight edge rather than the small holes if the edge facet wets better than
the substrate. This is verified by experimental data from melting alkane
layers between silica and air. Our findings provides a general approach for
solving the non-trivial melting scenarios in systems with capillary surfaces.

###### pacs:

36.40.Ei, 64.70.dj, 68.08.Bc

##  I Introduction

Thermostability of thin films is of great interest in theory and in practice
Huber (2015). For example, it is very important to know when and how an anti-
corrosion coating, a filter membrane, or a semi-conductor layer, would
collapse under high temperature. Generally, we are to look into the melting
process of the thin film: in such a confined system, the “de facto” melting
point is different from the melting point of the bulk material, due to the
ineligible contribution from the interfacial free energies.

The classical description of melting under confinement is based on a two-phase
system: a small solid particle covered by its melt, with one solid-liquid
interface. The area of this interface varies during melting, hence the
interfacial energy varies and contributes to the total free energy change.
Melting point shift calculated with this model is the Gibbs-Thomson equation
widely used by experimentalists and engineers Pawlow (1909); Tammann (1920);
Meissner (1920); Rie (1923). For the thin films, a variation of this model is
used, but same as the original model, only the solid-liquid interface is
considered.

However, in most real cases, there will be at least one capillary surface,
i.e., the liquid-air interface. The prerequisite of the Gibbs-Thomson equation
is that the liquid melt completely covers the solid particle floating in the
air, so that the capillary surface is a spherical shell that doesn’t change in
shape or area. In practice, this means all the facets of the floating particle
must be “premelting” Frenken and Van der Veen (1985); Dash (1989), i.e., a
thin layer of liquid melt covers the solid facets BEFORE the temperature
reaches the bulk melting point. A bad news is that “premelting” is only
confirmed on certain facets of ice and lead (Pb). Experimental study with lead
particles shows that the Gibbs-Thomson equation breaks down once a non-
premelting facet is involved, i.e., the liquid melt wets the facet with a non-
zero contact angle Métois and Heyraud (1989). This will also happen when the
particle is not floating but in contact with a substrate. Liquid could be
pinned at the grain boundaries, or wet the substrate with a non-zero contact
angle. The shape of the capillary surface will become non-trivial, so does the
interfacial energy. For long it has been known that the effects of the
capillary surface on melting is complex Maeda and Christenson (1999);
Christenson (1995); Fretwell et al. (1996); Glicksman et al. (2006); Moerz et
al. (2012). But still, this problem is rarely systematically analysed.

Previously, we have observed an unexpected melting scenario of molecularly
thin terrace of long-chain alkanes: liquid drops appear at the edge and “eat”
into the terrace Lazar and Riegler (2005). We have revealed that the emergence
of these drops is due to the Plateau-Rayleigh instability of the capillary
surface Kusumaatmaja et al. (2012). Later we have analysed the melting
behaviour of cylindrical aggregates Jin and Riegler (2016) in gaseous
environment based on the same experimental system, and have calculated the
deviation of the “de facto” melting point from Gibbs-Thomson equation due to
the capillary surface.

In this work, we study the melting behaviour of a thin, planar solid film from
a defect in the shape of a cylindrical hole. We show that the melting scenario
may include a morphology transition of the capillary surface, and the
morphology of the capillary surface will decide the melting path as well as
the “de facto” melting point of the system. Although the simulation is based
on our experimental system of long-chain alkane film between air and silica
substrate, the model is easily generalised to other systems.

##  II Theory and methods

###  II.1 Thermodynamics

We analyze the melting process in a continuum approach. 222B. Henrich et al.
showed that continuum models are adequate for $\geq 2$ adjacent fluid mono-
layers using computer simulation Henrich et al. (2008).. Contributions from
the line tension Tolman (1949); Berg et al. (2010), the disjoining pressure
Derjaguin and Churaev (1978), and the gravity are neglected. When alkane solid
melts, the volume increase typically by $\approx 10\%$ Seyer et al. (1944).
This change in volume can be neglected without changing our conclusion. We
hence use the volume of the liquid melt $V_{\text{l}}$ to represent the amount
of the material that undergoes melting.

In this study, we focus on the slow, equilibrated process: the capillary
surface is always fully relaxed to the minimum of the interfacial energy
(area), and the energy fluctuation is generally not taken into consideration
hence metastable states are allowed. The system status is determined by the
total free energy, which is the sum of the contributions from the bulk energy
$G_{\text{B}}$ and the interfacial energies $G_{\text{I}}$. If we heat up the
system to an arbitrary temperature $T$, the system will adjust to the new
thermodynamic equilibrium. For example, some solid may melt ($V_{\text{l}}$
increases) until the total free energy is minimized. At equilibrium we always
have

$\frac{dG_{\text{total}}}{dV_{\text{l}}}=\Delta\mu+\frac{dG_{\text{I}}}{dV_{\text{l}}}=0$
(1)

where $\Delta\mu$ is the bulk chemical potential difference per volume between
liquid and solid under a constant pressure (in this case, 1 bar).

Gibbs free energy is the indicator for the direction of a process: if
$G_{\text{total}}$ decreases ($dG_{\text{total}}/dV_{\text{l}}<0$), then the
process proceed forward; if increases, then backward. In Eq. (1), $\Delta\mu$
only depends on the temperature $T$. But $dG_{\text{I}}/dV_{\text{l}}$ does
not change with $T$, rather it is a function of $V_{\text{l}}$. If we know
these two functions, we can predict the melting scenario of the system from
any arbitrary $T$ and $V_{\text{l}}$, given that $V_{\text{l}}$ naturally
increases when melting proceeds. For example, if the system starts with a
$(T,V_{\text{l}})$ where $dG_{\text{total}}/dV_{\text{l}}<0$, the system will
start to melt and $V_{\text{l}}$ increases, then:

  * •

if $dG_{\text{I}}/dV_{\text{l}}$ decreases, $dG_{\text{total}}/dV_{\text{l}}$
is always negative, melting continues;

  * •

if $dG_{\text{I}}/dV_{\text{l}}$ increase, $dG_{\text{total}}/dV_{\text{l}}$
will approach $0$. Melting will stop at another $V_{\text{l}}$ that makes
$dG_{\text{total}}/dV_{\text{l}}=0$. Note that in this case, the solid will
coexist with some amount of liquid melt.

Following we will have a look at the exact function for $\Delta\mu$ and
$dG_{\text{I}}/dV_{\text{l}}$.

As the relevant temperature range is often small, a constant melting entropy
$\Delta S_{\text{fus}}$ can be assumed. $\Delta\mu$ at an arbitrary
temperature $T$ is calculated as Adkins (1983); Dash et al. (2006)

$\Delta\mu\simeq-\Delta S_{\text{fus}}\cdot(T-T_{\text{0}})$ (2)

$T_{\text{0}}$ is the bulk melting point. $\Delta\mu$ decreases linearly with
the temperature $T$: the higher $T$ is, the more negative $dG_{\text{B}}$ is.

In a simple system described by the Gibbs-Thomson model, i.e., a small
spherical solid particle (radius $r_{0}$) covered with liquid melt (solid-
liquid interfacial tension $\gamma_{\text{sl}}$).
$dG_{\text{I}}/dV_{\text{l}}=-2\gamma_{\text{sl}}/(r_{0}^{3}-3V_{\text{l}}/(4\pi))^{1/3}$
is a strictly decreasing function over the change of $V_{\text{l}}$. Hence
there will not be co-existence of the solid and liquid: once the temperature
is above a certain $T_{\text{m}}$, so that $dG_{\text{total}}/dV_{\text{l}}<0$
at $V_{\text{l}}=0$, melting will complete instantly.

However, because of the non-trivial shape of the capillary surface, the
function for $dG_{\text{I}}/dV_{\text{l}}$ of $V_{\text{l}}$ is also non-
trivial. 333Please note that in our case, $dG_{\text{I}}/dV_{\text{l}}$ is not
simply equal to the Laplace pressure, as in other systems Glicksman et al.
(2006); Glicksman (2011). For example, in Fig. 3(b),
$dG_{\text{I}}/dV_{\text{l}}$ first rises with $V_{\text{l}}$. As a result
there will be a small temperature range inside which the liquid melt co-exists
with the solid. This co-existence can be stable or metastable, and was
reported in our previous studies Kusumaatmaja et al. (2012); Jin and Riegler
(2016).

With Eq. (1) and Eq. (2), we can deduce that the “equilibrium” temperature
$T_{\text{eq}}$ is related to the “equilibrium” volume
$V_{\text{l}}^{\text{eq}}$ as:

$T_{\text{eq}}-T_{\text{0}}=\frac{1}{\Delta
S_{\text{fus}}}\cdot(\frac{dG_{\text{I}}}{dV_{\text{l}}}\bigg{|}_{V_{\text{l}}=V_{\text{l}}^{\text{eq}}})$
(3)

Naturally, the upper limit of $T_{\text{eq}}$ corresponds to the maximum of
${dG_{\text{I}}}/{dV_{\text{l}}}$. This is the temperature under which the
system will melt completely without any energy barrier. We call it the “de
facto” melting temperature $T_{\text{m}}$ of the system. 444Sometimes called
the “true” or the “critical” melting temperature. As shown in Fig. 3(b), in
our system ${dG_{\text{I}}}/{dV_{\text{l}}}$ increases and then decreases with
$V_{\text{l}}$, hence the liquid volume $V_{\text{eq}}$ at $T_{\text{m}}$ is
also the maximum. It characterize the maximum coexistence liquid volume beyond
which the system melt completely. We write it as $V_{\text{m}}$.

$\Delta T_{\text{m}}=T_{\text{m}}-T_{\text{0}}=\frac{1}{\Delta
S_{\text{fus}}}\cdot(\frac{dG_{\text{I}}}{dV_{\text{l}}})_{\text{max}}$ (4)

Therefore, in order to solve the melting scenario of a system, one approach
is: First look into various intermediate states during melting, relax the
capillary surfaces and calculate the free energies. Then get the function for
$dG_{\text{I}}/dV_{\text{l}}$ of $V_{\text{l}}$ by connecting these states
into melting pathways. The whole melting scenario could then be constructed as
shown in the following sections.

###  II.2 Simulation details

Figure 1: Melting scenario from a hole of finite size in an infinite film of
uniform thickness. The solid walls are assumed cylindrical and perpendicular
during melting. The melt completely wets the side walls. The substrate is
wetted only partially with contact angle $\theta_{ls}$. Different amounts of
liquid melt (corresponding to different temperatures) will lead to different
liquid morphologies such as and open channel (I) or a closed concave film
(II).

We focus on the melting of a film between two different media: a substrate and
air. Melting of a film between two identical media or substrates is either
straight-forward to analyze, or closed related to what we analyze here. A
cylinder hole (radius $r_{0}$) penetrating the film until the substrate is
where the melting would start. For calculation we assume the followings:

  * •

The hole maintains a smooth, right cylinder geometry. The substrate is planar
and the film is of uniform thickness, hence the hole also maintains a constant
height $h$ during melting;

  * •

The total volume of solid and liquid phase together remains constant as the
density difference between solid and liquid is neglected;

  * •

The liquid wets completely the side walls of the cylindrical hole, but only
partially the film surfaces and the substrates, with a non-zero contact angle;

  * •

The liquid is “pinned” at the solid/air/liquid contact line of the hole edge,
meaning that there is no fixed contact angle at this line;

  * •

Young’s equation holds for contact angles at the liquid/substrate/air contact
lines.

The non-trivial shape of the capillary surface during the melting is solved by
relaxing the surface in Surface Evolver Brakke (1992) with each arbitrary
$V_{\text{l}}$ and corresponding boundary conditions. For the simulations we
use real data of long-chain alkane systems Dirand et al. (2002); Yi and
Rutledge (2011); Riegler and Köhler (2007): $\gamma_{\text{lv}}=25\times
10^{-3}$N/m, $\gamma_{\text{lw}}\text{(hole side wall)}=10\times 10^{-3}$N/m,
$\gamma_{\text{ls}}\text{(film surface and substrate)}=4\times 10^{-3}$N/m,
and $\Delta S_{\text{fus}}=5\times 10^{5}$ J/Km-3. The contact angle is
$\theta_{\text{ls}}=15^{\circ}$ on both the film surface and the substrate.
The model can be easily tuned to simulate different systems. For example, if
use $\theta_{\text{ls}}=90^{\circ}$, the model represents not only a film
coated on a substrate, but also half of a self-supporting film in the air.

Fig. 1 illustrates the different liquid morphologies that may emerge during
melting at a round hole in a planar film. The holes have a radius $r_{0}$ in
the all-solid case (at $T\ll T_{0}$) and a depth $h$ (same with the film
thickness). When some solid melt into the liquid state, the radius of the hole
is $r$, the liquid volume $V_{\text{l}}=\pi h(r^{2}-r_{0}^{2})$. Upon
increasing temperature liquid melt will first appear as a uniform channel at
edge of the hole. To be exact, the capillary surface will be a rotational-
symmetric constant-mean-curvature surface (Delaunay surface, or rouloid)
Delaunay (1841); Eells (1987). As the liquid volume increases, two
distinctively different morphologies are possible: a ring-like channel of the
liquid with a opening in the center exposing part of the bare substrate
(“open” shape, (I)), and a spherical concave covering the entire substrate
(“closed” shape (II)). In (I) the liquid meet the substrate at contact angle
$\theta_{\text{ls}}$.

There is a geometric limit for the “closed” shape: It will only appear when a
certain volume of the solid has melted. This lower limit can be calculated
analytically as the top surface of the liquid is a concave spherical cap whose
edge pinned at the upper rim. Calculation results in a minimum radius
$r=r_{\text{min}}$ of a hole that can be filled with a liquid concave from
melting the solid film:

$\frac{r_{\text{min}}}{h}=\sqrt{2(\frac{r_{0}}{h})^{2}-\frac{1}{3}}$ (5)

Note that this limit is different for very small/deep holes with $r_{0}/h<1$,
where the liquid front on the substrate closes before the liquid front on the
wall can be pinned at the hole edge.

In some cases, typically when a large amount of liquid accumulate at a long
edge, there are two possible “open” morphologies (Fig. 2), which are (a)
uniform channels and (b) bulged morphologies in isobaric (isocurvature)
coexistence with channel sections Kusumaatmaja et al. (2012); Jin and Riegler
(2016). In this work, mostly the bulged morphologies are only possible at such
large $V_{\text{l}}$ (i.e. comparatively high temperatures) that typically the
systems already gets absolutely unstable, as will be shown in later analysis.
Hence only the morphology transition between the “open” channel and the
“closed” concave affects the melting.

Figure 2: Two types of “open”morphologies: (a) rouloid and (b) bulged channel
.

In the following, we shall use normalized variables:
$\overline{G}=G/(\gamma_{\text{lv}}r_{0}h)$, $\overline{V}=V/(\pi
r_{0}^{2}h)$, $\overline{r}=r/h$, $\overline{\Delta T}=\Delta T\cdot h$, and
the in-plane curvature $\overline{\kappa}=\kappa\cdot h$.

###  II.3 Experimental methods

We use silicon wafer covered with a $300$ nm thick oxide layer as substrate.
The oxide layer has a close refractive index to the long-chain alkane. Such a
thickness is chosen because it provides interference enhancement of contrast
from thin alkane layers on the substrate Köhler et al. (2006). We coat the
piranha-cleaned wafer with long-chain alkane (triacontane C30H62 or
hexatriacontane C36H74) solution in toluene. After a heat-cool cycle, a
“surface frozen” alkane monolayer Merkl et al. (1997) will be formed on the
wafer. The sample is then annealed at one of two degree below $T_{0}$ for a
few minutes. Films and islands of various size and height will form. The
suitable ones are chosen for the melting experiments. The height of the alkane
films are measured with atomic force microscope (AFM).

The substrate in such a system is actually the surface frozen alkane
monolayer, covering the silica surface. This layer, same with the top of
alkane film, can only be partially wetted by liquid alkane melt with a contact
angle around $15^{o}$. This contact angle is directly calculated from
microscope observation or alkane droplets: The light (peak wavelength $550$
nm) reflected from liquid-air interface and the silica-silicon interface in
the wafer will interfere. From the Newton’s rings we get the droplet surface
profile, and from fitting a circle to the profile we can get the contact
angle.

##  III Discussions

###  III.1 Transitions between different morphologies during melting

Figure 3: Melting process of a film from a hole of $\overline{r_{0}}=5$ on a
substrate with $\theta_{\text{ls}}=15^{o}$. (a) Interfacial energy
$\overline{G_{\text{I}}}$ as a function of liquid volume
$\overline{V_{\text{l}}}$. The solid black lines indicate “open” shape, while
the dashed lines the closed concave. For small volumes (left of A) only
uniform channels are possible, for intermediate volumes (between C and D) the
hole can only be filled with a closed concave film. Between A and C, also
above D, both morphologies are possible but with a difference in energy. B and
E are the two iso-energy points. (b) The corresponding
$\overline{dG_{\text{I}}/dV_{\text{l}}}$ as a function of
$\overline{V_{\text{l}}}$.

Melting process is characterized by the increase in the volume of liquid melt
$\overline{V_{\text{l}}}$. Fig. 3 shows the change of the interfacial free
energy $\overline{G_{\text{I}}}$ and $\overline{dG_{\text{I}}/dV_{\text{l}}}$
as the liquid volume increases. The data represent the melting process of a
film from a hole of $\overline{r_{0}}=5$ on a substrate with
$\theta_{\text{ls}}=15^{o}$. This corresponds to the real cases of holes of
about $500$ nm in radius in an alkane film of with $h=100$ nm.

At the bulk melting temperature $T_{0}$, $G_{\text{total}}=G_{\text{I}}$.
$\overline{G_{\text{I}}}$ first decrease with $\overline{V_{\text{l}}}$, this
means liquid melt already appears and co-exists with the solid at $T_{0}$.
Following our previous analysis, the same conclusion can be read from Fig.
3(b): Starting from $(T_{0},0)$, $\Delta\mu=0$ so that
${dG_{\text{total}}}/{dV_{\text{l}}}<0$, the system will melt. Then, because
$\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}$ increases with
$\overline{V_{\text{l}}}$, melting will stop at an equilibrium volume
$\overline{V_{\text{l}}^{\text{eq}}}$, in exact, the volume when the
$\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}=0$

Geometrically, the minimum volume for the “closed” shape
$\overline{V_{\text{min}}}\approx 1$ according to Eq. (5), indicated by A on
Fig. 3. B and E indicate the two iso-energy points: After B the “closed” shape
becomes more favorable in energy; above E, the “open” shape becomes more
favorable in energy again. Between C and D, the “open” shape is forbidden.

Combining the energy and geometry analysis, we have such a general melting
scenario: If the melting proceeds always with the lowest free energy, it will
start from a uniform channel, then the liquid will close into a concave at B.
The closed liquid concave will rapture into “open” channel at E. However,
there is an energy barrier between the two morphologies, hence the first
morphology transition may happen anywhere between A and C. In addition, once
$\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}$ reaches the maximum and starts to
decrease, the system will melt completely, hence the morphology change around
D and E will be barely noticeable.

###  III.2 Melting scenarios and blurred melting points

Following we show how to read the melting scenarios from the
$\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}$ to $\overline{V_{\text{l}}}$
plots, and analyze the $T_{\text{eq}}$, $T_{\text{m}}$ and $V_{\text{m}}$. In
Fig. 4 we plot $\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}$ to
$\overline{V_{\text{l}}}$, for different $\overline{r_{0}}$ and
$\theta=15^{o}$. Different $\overline{r_{0}}$ means the film melt from a hole
of different size. A to E have the same meanings as in Fig. 3. Following Eq.
(3) and Eq. (4), in the coexisting temperature range, under equilibrium,
$\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}$ is proportional to
$T_{\text{eq}}-T_{0}$. $\overline{{dG_{\text{I}}}/{dV_{\text{l}}}}=0$
corresponds to the bulk melting temperature $T_{0}$, and maximize at
$(T_{\text{m}},V_{\text{m}})$. We could then see the curves, especially the
rising part, as system status represented by temperature $T_{\text{eq}}$ and
liquid volume $V_{\text{l}}$, given that the system is under equilibrium.
Following the curves we see directly how melting will proceed as the
temperature rises slowly.

Figure 4: Melting scenarios of film from holes with different
$\overline{r_{0}}$ and $\theta_{\text{ls}}=15^{o}$.
$\overline{dG_{\text{I}}/dV_{\text{l}}}$ is plotted as a function of the
liquid volume scaled to the hole volume, $\overline{V_{\text{l}}}$. Just as in
Fig. 3, the solid black lines indicate “open” uniform channels, the dashed red
lines closed concaves. Point A to E have also the same meaning: when the
liquid volume is below A only uniform channels are possible. Between C and D
the hole can only be filled with a closed concave film. Between A and C, also
above D, both morphologies are possible but with a difference in energy. B and
E are the two iso-energy points. $T_{\text{m}}^{\text{open}}$ and
$T_{\text{m}}^{\text{closed}}$ indicate the de facto melting temperatures of
the corresponding morphology.

When $\overline{r_{0}}=4$, melting start from liquid forming a uniform channel
at the hole edge. Beyond A the close concave becomes possible, but its energy
is still higher than the uniform channel shape until B. Upon the temperature
rises, the liquid volume increase as well. The transition from the “open”
uniform channel to the closed concave shape may happen somewhere between A and
C by overcoming an energy barrier, or at C when the “open” channel shape
becomes absolutely impossible and the liquid front coalesce.

Assuming the transition happens at B, the iso-energy point. From b1 to b2 it
will first be an isochoric transition. However, $dG_{\text{I}}/dV_{\text{l}}$
and thus the $T_{\text{eq}}$ on the “closed” curve at b2 is lower than that on
the “open” curve at b1. If the temperature doesn’t change, liquid volume
$V_{\text{l}}$ will rapidly increase from b2 to b3. The system is still
metastable because the temperature at b3 is lower than maximum on this curve,
$T_{\text{m}}^{\text{closed}}$. Even if the transition does not happen until
C, the system is stable after transition. Hence $T_{\text{m}}^{\text{closed}}$
is the only “de facto” melting point in this system.

In practice, this melting scenario means we could seal some small holes in the
solid film: We rise the temperature slowly until there is enough liquid inside
the hole to form a closed concave, while carefully keep the temperature below
$T_{\text{m}}^{\text{closed}}$. Then we freeze quickly the system and solidify
the closed concave of liquid melt into a glass state, the small hole will be
sealed.

When $\overline{r_{0}}=5$, the scenario becomes different around the first
morphology transition: If the transition occurs already at A, it will start
from a1, goes via a2 to a3. Then the system may remain metastable at a3
because this $T_{\text{eq}}$ is below the maximum on the “closed” curve,
$T_{\text{m}}^{\text{closed}}$. However, if the transition happens later,
e.g., at B the iso-energy point, same process will lead to a complete melting
as the corresponding temperature is already higher than
$T_{\text{m}}^{\text{closed}}$. If the morphology transition does not happen,
the film will melt when the temperature reaches $T_{\text{m}}^{\text{open}}$,
the maximum on the “open” curve. This means, the “de facto” melting point
$T_{\text{m}}$ of such a system is a temperature range rather than a single
temperature point. It can be any temperature between
$T_{\text{m}}^{\text{closed}}$ and $T_{\text{m}}^{\text{open}}$, depending on
when the system jumps to the other morphology.

When $\overline{r_{0}}=6$, once the morphology transition happens, even at the
minimum volume (A), temperature is already higher than
$T_{\text{m}}^{\text{closed}}$, and the whole system will melt immediately.
But if the morphology transition does not happen, the film will then melt at
$T_{\text{m}}^{\text{open}}$. The “de facto” melting point $T_{\text{m}}$ of
this system is also a temperature range, but between
$T_{\text{eq}}^{\textbf{A}}$ and $T_{\text{m}}^{\text{open}}$

In the last case with $\overline{r_{0}}=20$, the liquid in the system will
stay in uniform channel shape up to $V_{\text{m}}^{\text{open}}$, where
complete melting starts. A closed liquid concave is only possible after that,
hence absolutely unstable in the sense of melting.

###  III.3 Phase diagram of the various morphologies

Figure 5: Phase diagram for the two different morphologies (“open” channel,
“closed” concave) in a plot of the liquid volume (scaled to the original hole
volume) as function of the original hole radius (scaled to the film
thickness). Frame (a) shows regions where only “open” channels (i), only
closed concaves (ii), or both morphologies (iii) can exist due to the
geometrical boundary conditions ($\theta_{\text{ls}}=15^{\circ}$, uniform
pressure within the liquid). The bulge boundary is also indicated. At volumes
above this boundary, open channel shape is still possible, but higher in
energy than the bulged channel. Frame (b) includes the curve where the “open”
and “closed” morphologies have the same interfacial energy. A volume-
increasing path is also indicated, with the A to E points corresponds to those
in Fig. 3. Frame (c) shows the critical volumes of both morphologies, i.e.,
the absolute volume limits of thermal stability.

Combining the analysis of the geometry, the energy, and the stability, we plot
in the Fig. 5 a phase diagram of various morphologies. The phase diagram
exhibits the melting scenarios of films with holes of different sizes.

Frame (a) depicts the regions where the two different morphologies are (not)
allowed due to mechanical and geometrical constraints (contact angle, volume,
uniform internal pressure). The whole phase diagram is divided into three
regions where: (i) only open morphologies are possible, (ii) only closed
concaves are possible and (iii) both morphologies are possible. The line
separating the region (i) and region (iii) is derived analytically from Eq.
(5) All the other data result from numerical simulations with Surface Evolver.
The “bulge boundary” divides region (i) and (iii) further into the areas
where: a) only uniform channels are possible (volumes below the boundary) and
b) both uniform channels and bulged morphologies are possible.

In frame (b) we add the iso-energy boundary, the boundary where open
morphologies and closed films have the same interfacial energy
($G_{\text{I}}^{\text{closed}}=G_{\text{I}}^{\text{open}}$). The iso-energy
boundary separate a small region (iv) from region(iii). The interfacial energy
of the closed concave shape is lower than the “open” shape inside region (iv),
but higher in the rest part of region (iii). The vertical line at $r_{0}/h=5$
shows the same scenario as presented in Fig. 3. The points B and E lie on the
iso-energy boundary, while points A, C, and E lie on the boundaries of regions
for certain morphologies.

Finally, in frame (c) we add the absolute thermodynamic stability limits of
two morphologies, i.e. the “critical” volumes $V_{\text{m}}$. In Fig. 4, these
volumes are shown as $V_{\text{m}}^{\text{open}}$ and
$V_{\text{m}}^{\text{closed}}$, the liquid volume at the “de facto” melting
points $T_{\text{m}}^{\text{open}}$ and $T_{\text{m}}^{\text{closed}}$.

It should be emphasized that Fig. 5 presents the phase diagram with respect to
the (scaled) liquid volume and not with respect to the temperature. The volume
increase of the liquid melt after a morphology transition does not show here.

###  III.4 Experimental results of different hole melting scenarios

Figure 6: Top: Melting behavior of solid alkane film ($h\approx 0.1\,\mu$m)
with three different local structures: (1) an almost straight edge
($\overline{r_{0}}\to\infty$); (2) a hole with $\overline{r_{0}}\approx 30$;
(3) a hole with $\overline{r_{0}}\approx 10$. In the same experiment they melt
sequentially. Bottom: Simulated result for alkane film / island melting
behavior. Bulging is not included here. Melting point shifts multiplied by
film thickness are plotted as the function of the scaled initial in-plane
curvature $\overline{\kappa_{0}}$. Therefore the grey region shows film
melting from hole, and the white region shows melting of island. The boundary
between the two region indicates melting of a straight terrace edge. Melting
point shifts for the three local structures shown in top frame are marked
respectively. Dotted rectangular indicates the geometries whose “de facto”
melting points are not single temperature, but within an interval as discussed
before.

Combined with our previous calculation on melting of cylindrical island Jin
and Riegler (2016) and straight-edge terrace Kusumaatmaja et al. (2012), we
have now the full picture of melting point shifts of different geometries. In
Fig. 6 we plot the scaled melting point shifts $\Delta T\cdot h$ in our model
system as the function of the scaled in-plane curvature of the all-solid hole
$\overline{\kappa_{0}}=h/r_{0}$. We use in-plane curvature instead of radius
to distinguish hole and island: hole has negative in-plane curvature
$\overline{\kappa_{0}}=-h/r_{0}$. The region with two separate lines shows the
geometry that can have multiply “de facto” melting points depending on the
melting paths. The prediction is that our thin film will start to melt from
the long, straight edge rather than the tiny defects (holes) inside the film.
This is contradict to the instinct, but is verified by our experiment. Fig. 6
shows the melting process of a alkane film (C36H74) in which different
geometry can be found: The curved edge behave locally like islands with
different radius. Inside the film there are two holes of different size. Top
right panel shows that under the same temperature, most liquid appear at the
edge (1), around the big hole a significant amount of solid melt as well,
while around the smallest hole (3) barely any change can be observed. Such a
melting sequence is consistent with the estimated “de facto” melting points at
these three geometries: (1) with the lowest “de facto” melting point melts the
first, (3) with the highest “de facto” melting point melts the last.

##  IV Conclusions

In this work, we have presented an approach to solve the non-trivial melting
scenarios in systems with capillary surfaces: first calculate the shape and
energy of various intermediate states during melting, connecting them into
possible melting pathways, then plot the function for
$dG_{\text{I}}/dV_{\text{l}}$ on $V_{\text{l}}$ along each pathway. Under
equilibrium, $dG_{\text{I}}/dV_{\text{l}}$ is approximately linear to $T$,
hence the plots show the details how the system will melt when temperature
rises gradually. This approach is especially useful in unveiling the non-
trivial effect of capillary surfaces on the melting of systems.

We analyzed and discussed the melting scenarios of thin films from round holes
of different size. The model is built based on the real system of alkane
layers between silica and air. Liquid melt wets the substrate and the wall of
the hole differently, resulting a capillary surface of various shape: rouloid,
bulged channel, and spherical concave. The possible shapes of the capillary
surface and its energy are calculated by Surface Evolver, then the possible
melting scenarios are constructed. We have found that depending on the system
history (melting path), the system may have multiple “de facto” melting
temperatures.

Quite contradict to the instinct, our analysis has predicted that in this
system (thin films of long chain alkanes between silica and air), a film will
melt from a straight edge rather than the small defects (holes). This
prediction is supported by experiment results.

##  V Acknowledgement

Discussion with H. Möhwald, H. Kusumaatmaja are gratefully acknowledged. C.Jin
was support from IMPRS on Biomimetic Systems.

## References

  * Huber (2015) P. Huber, Journal of Physics: Condensed Matter 27, 103102 (2015).
  * Pawlow (1909) P. Pawlow, Z. Phys. Chem 65, 1 (1909).
  * Tammann (1920) G. Tammann, Z. Anorg. Allg. Chem. 110, 166 (1920).
  * Meissner (1920) F. Meissner, Z. Anorg. Allg. Chem. 110, 169 (1920).
  * Rie (1923) E. Rie, Z. Phys. Chem. 104, 354 (1923).
  * Frenken and Van der Veen (1985) J. W. Frenken and J. Van der Veen, Phys. Rev. Lett. 54, 134 (1985).
  * Dash (1989) J. Dash, Contemp. Phys. 30, 89 (1989).
  * Métois and Heyraud (1989) J. Métois and J. Heyraud, J. Phys. France 50, 3175 (1989).
  * Maeda and Christenson (1999) N. Maeda and H. K. Christenson, Colloids Surf A Physicochem Eng Asp. 159, 135 (1999).
  * Christenson (1995) H. Christenson, Phys. Rev. Lett. 74, 4675 (1995).
  * Fretwell et al. (1996) H. Fretwell, J. Duffy, M. Alam, and R. Evans, J. Radioanal. Nucl. Chem 210, 575 (1996).
  * Glicksman et al. (2006) M. Glicksman, A. Lupulescu, and M. Koss, in _Free Boundary Problems_ (Springer, 2006), pp. 219–230.
  * Moerz et al. (2012) S. T. Moerz, K. Knorr, and P. Huber, Physical Review B 85, 075403 (2012).
  * Lazar and Riegler (2005) P. Lazar and H. Riegler, Phys. Rev. Lett. 95, 136103 (2005).
  * Kusumaatmaja et al. (2012) H. Kusumaatmaja, R. Lipowsky, C. Jin, R. C. Mutihac, and H. Riegler, Phys. Rev. Lett. 108, 126102 (2012).
  * Jin and Riegler (2016) C. Jin and H. Riegler, J. Phys. Chem. (2016), accepted.
  * Note (2) Note2, b. Henrich et al. showed that continuum models are adequate for $\geq 2$ adjacent fluid mono-layers using computer simulation Henrich et al. (2008).
  * Tolman (1949) R. C. Tolman, J. Chem. Phys. 17, 333 (1949).
  * Berg et al. (2010) J. K. Berg, C. M. Weber, and H. Riegler, Phys. Rev. Lett. 105, 076103 (2010).
  * Derjaguin and Churaev (1978) B. Derjaguin and N. Churaev, J. Colloid Interface Sci. 66, 389 (1978).
  * Seyer et al. (1944) W. F. Seyer, R. F. Patterson, and J. L. Keays, J. Am. Chem. Soc. 66, 179 (1944).
  * Adkins (1983) C. J. Adkins, _Equilibrium Thermodynamics_ (Cambridge University Press, 1983).
  * Dash et al. (2006) J. G. Dash, A. W. Rempel, and J. S. Wettlaufer, Rev. Mod. Phys. 78, 695 (2006).
  * Note (3) Note3, please note that in our case, $dG_{\text{I}}/dV_{\text{l}}$ is not simply equal to the Laplace pressure, as in other systems Glicksman et al. (2006); Glicksman (2011).
  * Note (4) Note4, sometimes called the “true” or the “critical” melting temperature.
  * Brakke (1992) K. A. Brakke, Exp. Math. 1, 141 (1992).
  * Dirand et al. (2002) M. Dirand, M. Bouroukba, A.-J. Briard, V. Chevallier, D. Petitjean, and J.-P. Corriou, J. Chem. Thermodyn. 34, 1255 (2002), ISSN 0021-9614.
  * Yi and Rutledge (2011) P. Yi and G. C. Rutledge, J. Chem. Phys. 135, 024903 (2011).
  * Riegler and Köhler (2007) H. Riegler and R. Köhler, Nat. Phys. 3, 890 (2007).
  * Delaunay (1841) C. Delaunay, J. Math. Pures Appl. pp. 309–314 (1841).
  * Eells (1987) J. Eells, Math. Intell. 9, 53 (1987).
  * Köhler et al. (2006) R. Köhler, P. Lazar, and H. Riegler, Appl. Phys. Lett. 89, 241906 (2006).
  * Merkl et al. (1997) C. Merkl, T. Pfohl, and H. Riegler, Phys. Rev. Lett. 79, 4625 (1997).
  * Henrich et al. (2008) B. Henrich, C. Cupelli, M. Santer, and M. Moseler, New J. Phys. 10, 113022 (2008).
  * Glicksman (2011) M. E. Glicksman, _Crystal-Melt Interfaces_ (Springer New York, New York, NY, 2011), pp. 181–212, ISBN 978-1-4419-7344-3, URL https://doi.org/10.1007/978-1-4419-7344-3_8.