# Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection

Ruoshi Liu and Carl Vondrick
Columbia University
thermal.cs.columbia.edu

###### Abstract

The relatively hot temperature of the human body causes people to turn into
long-wave infrared light sources. Since this emitted light has a larger
wavelength than visible light, many surfaces in typical scenes act as infrared
mirrors with strong specular reflections. We exploit the thermal reflections
of a person onto objects in order to locate their position and reconstruct
their pose, even if they are not visible to a normal camera. We propose an
analysis-by-synthesis framework that jointly models the objects, people, and
their thermal reflections, which combines generative models with
differentiable rendering of reflections. Quantitative and qualitative
experiments show our approach works in highly challenging cases, such as with
curved mirrors or when the person is completely unseen by a normal camera.

Figure 1: We introduce a method to reconstruct the 3D position and pose of a
person from their thermal reflections in everyday objects with non-planar
surfaces. Given an RGBD image and a thermal image in the middle of the figure,
our method is able to recover the 3D mesh of the person (blue) as well as the
objects (pink), even though they are not within the field of view of the
camera system. Our system never sees the scene on the left, which is only
shown for visualization purposes.

##  1 Introduction

One of the major goals of the computer vision community is to locate people
and reconstruct their poses in everyday environments. What makes thermal
cameras particularly interesting for this task is the fact that humans are
often the hottest objects in indoor environments, thus becoming infrared light
sources. Humans have a relatively stable body temperature of 37 degrees
Celcius, which according to the Stefan-Boltzmann law, turns people into a
light source with constant brightness under long-wave infrared (LWIR). This
makes LWIR images a robust source of signals of human activities under many
different light and camera conditions.

Since infrared light on the LWIR spectrum has a wavelength that is much longer
than visible light (8$\mu$m-14$\mu$m vs. $0.38\mu$m-$0.7\mu$m), the objects in
typical scenes look qualitatively very different from human vision. Many
surfaces of objects in our daily life – such as a ceramic bowl, a stainless
steel fridge, or a polished wooden table top – have stronger specular
reflections than in the visible light spectrum[58, 7]. Figure 1 shows the
reflection of a person with the surface of salad bowls, which is barely
visible to the naked eye, if at all, but clearly salient in the LWIR spectrum.

In cluttered environments, a visible light camera may not always be able to
capture the person, such as due to a limited field of view or occlusions. In
such scenes, the ideal scene for locating and reconstructing a person would be
an environment full of mirrors. This is what the world looks like under the
LWIR spectrum. Infrared mirrors are abundant in the thermal modality, and
reflections reveal significant non-line-of-sight information about the
surrounding world.

In this paper, we introduce a method that uses the image of a thermal
reflection in order to reconstruct the position and pose of a person in a
scene. We develop an analysis-by-synthesis framework to model objects, people,
and their thermal reflections in order to reconstruct people and objects. Our
approach combines generative models with differentiable rendering to infer the
possible 3D scenes that are compatible with the observations. Given a thermal
image, our approach optimizes for the latent variables of generative models
such that light emitting from the person will reflect off the object and
arrive at the thermal camera plane.

Our approach works in highly challenging cases where the object acts as a
curved mirror. Even when a person is completely unseen by a normal visible
light camera, our approach is able to localize and reconstruct their 3D pose
from just their thermal reflection. Traditionally, the increased specularity
of surfaces has posed a challenge to thermography, making it extremely
difficult to measure the surface temperature of a thermally specular surface,
which brings out a line of active research aiming to remove the specular
reflection for more accurate surface temperature measurement [4, 40, 5, 80].
We instead exploit these “difficulties” of LWIR to tackle the problem of 3D
human reconstruction from a single view of thermal reflection image.

The primary contribution of the paper is a method to use the thermal
reflection of the human body on everyday objects to infer their location in a
scene and its 3D structure. The rest of the paper will analyze this approach
in detail. Section 2 provides a brief overview of related work for 3D
reconstruction and differentiable rendering. Section 3 formulates an
integrated generative model of humans and objects in a scene, then discusses
how to perform differentiable rendering of reflection, which we are able to
invert to reconstruct the 3D scene. Section 4 analyzes the capabilities of
this approach in the real world. We believe thermal cameras are powerful tools
to study human activities in daily environments, extending computer vision
systems’ ability to function more robustly even under extreme light
conditions.

Figure 2: High-level overview of our analysis-by-synthesis framework. We
sample random initializations from the latent space of pretrained generative
models of humans and objects in 3D. Through a differentiable rendering
process, we synthesize a reflection image of a human body on object surfaces.
This synthesized reflection is compared with the observed reflection with an
$L_{1}$ loss. Gradients are backpropagated through differentiable rendering
and generative models to the latent variables.

##  2 Related Work

Differentiable Rendering. Differentiable rendering is a differentiable process
of rendering 2D images given 3D scenes. The gradient obtained from the image
space w.r.t. the scene parameters can be calculated and used to perform
optimization. Recent advances in implicit 3D representations, especially
Neural Radiance Field (NeRF) [52, 2, 3, 54, 69, 60], have made impressive
results on rendering photo-realistic images for the view-synthesis problems.

Another line of work focuses on differentiable rasterization [32, 47, 73, 46,
42, 63]. These works aim to replace the traditional rasterization process in
computer graphics based on 2D projections of primitives such as polygons with
z-buffering, with a differentiable rasterization process.

While differentiable, these methods are limited by the intrinsic difficulty of
modeling single or multiple bounces of light in a scene, which can be modeled
with physics-based differentiable ray tracing [41, 81, 57, 25, 73, 30]. In our
problem, because humans are light sources and we need to perform
differentiable rendering of one-bounce reflection, we extended Soft Rasterizer
[46].

Single-View 3D Reconstruction. From a practical point of view, obtaining 3D
ground truth for supervision is often difficult and expensive [28]. In terms
of the quantity of data available, the unlabeled 3D data is not comparable to
the 2D data on the internet. This spurs a long-standing interest from the
general computer vision community to pursue 3D reconstruction with as little
information as a single-view[31, 46, 19, 75, 43, 77, 45].

In addition to general 3D object reconstruction, another line of research
focus on the 3D reconstruction of human body from single-view images and
videos [44, 35, 36, 53, 64, 61]. Representatively, SMPL-X [61] is an
expressive whole-body model with details around hands and faces, represented
as a triangle mesh with 10,475 vertices. In the same paper, SMPLify-X was
proposed to estimate an SMPL-X model from just a single RGB image. This is
done by first detecting human keypoints from the image with an off-the-shelf
keypoint detector [10, 10, 70, 14, 17, 6, 38, 21]. Then the parameters of an
SMPL-X model is optimized to fit the keypoints which serve as the observation
of human in the 2D image.

3D Generative Model. Our system utilizes generative models for both objects
and humans. For 3D objects, generative models are usually trained with
synthetic datasets composed of CAD models[11]. Different generative
architectures including VAE [20, 9, 76, 20], GAN [62, 62, 26, 74], normalizing
flow [33, 34], and diffusion models [48] were proposed to generate objects in
meshes, point clouds, or voxels. More recently, implicit 3D representation, or
coordinate-based models, become a popular choice of modality to perform
generative tasks [23, 51, 59, 16, 27, 56].

For humans, [61, 71] proposed generative models for 3D humans, represented as
SMPL-X models. In [61], a VAE is trained to generate human poses from 4
datasets including Human3.6M, LSP, CMU Panoptic, and PosePriors [78, 22, 55,
1]. The VAE samples a latent vector from a high-dimensional Gaussian
distribution and generates a human pose vector. This pose vector is applied
with a sparse linear regressor and a linear blend skinning function to
generate a triangle mesh in a fully differentiable manner.

Thermal Computer Vision. Previous work has applied computer vision to thermal
images for various problems [37, 72, 18, 24, 15, 13, 65]. ContactDB [8] used
thermal imaging to obtain accurate human grasps of everyday objects for
robotics applications. [49] studied the problem of thermal non-line-of-sight
imaging. In comparison, this work focuses on the 3D reconstruction of people
from their thermal reflections in non-planar objects. Other work pursues 3D
reconstruction of objects from thermal images [12, 67, 50, 66]. To our
knowledge, we are the first to perform 3D reconstruction of humans from their
thermal reflection.

##  3 Methods

Our system takes an RGBD image and a thermal image of everyday objects with
thermally reflective surfaces and performs a 2-stage optimization to estimate
3D objects and a human not in sight from both cameras’ perspectives. In the
first stage, a 6 DoF pose, scale, and a neural signed distance function [59]
are jointly estimated for each object present in the scene. In the second
stage, the location, orientation, and pose of the human are jointly estimated
to reconstruct the observed thermal reflection.

Section 3.1 formulates the problem we aim to solve. Section 3.2 gives an
overview of the approach. Section 3.3 describes the generative models we used
in our approach in detail. Section 3.4 lays out a differentiable rendering
algorithm of human thermal reflection. Section 3.5 formulates the optimization
process and the objective functions.

###  3.1 Problem Formulation

We decompose a scene into 3 components: a human body, objects with specular
surfaces in LWIR spectrum, and environmental heat sources. We first obtain a
segmentation mask from each object in the scene from the RGBD image. To obtain
the thermal reflection image, we perform ray tracing starting from the camera
sensor to the light source – the human body, under Helmholtz reciprocity.
Assuming a pinhole camera model, let n be the surface normal of the object at
point p, r be the vector from the camera sensor to p and $\textbf{{\bf
r}}^{\prime}$ the reflected ray vector. We model the intensity of each pixel
$I_{\bf x}$ in the thermal camera as a binary value:

$I_{\bf x}=\begin{cases}1,&\textbf{{\bf r}}^{\prime}\textrm{ intersects with
}\mathcal{T}_{\phi,T}(M_{h})\\\ 0,&\textrm{otherwise}\end{cases}$ (1)

where $M_{h}$ represents the human shape in the form of a triangle mesh, and
$\mathcal{T}_{\phi,T}$ represents an SE(3) transformation matrix parameterized
by rotation, translation, and scale. With background subtraction, the noise
coming from environmental heat sources can be mitigated.

As described in figure 1, the calibrated thermal camera and RGBD camera with
known intrinsic matrix and unknown extrinsic matrix capture an RGB image, a
depth map, and a thermal image. Given these images as our observation
containing N objects, we solve for the following 7 variables via optimization:
locations $\\{{\bf T}_{obj}\\}_{i=0}^{N}$, rotations
$\\{\phi_{obj}\\}_{i=0}^{N}$, scales $\\{s_{obj}\\}_{i=0}^{N}$, and the shape
$\\{M_{obj}\\}_{i=0}^{N}$ of the objects, location ${\bf T}_{h}$, rotation
$\phi_{h}$, and the shape $M_{h}$ of the human, all in camera’s perspective.

Figure 3: Differentiable Rendering of Reflection. Ray direction shown reverses
the physical propagation direction of light by Helmholtz reciprocity.

###  3.2 Overview of Approach

The optimization problem we are solving is severely under-constrained, so we
choose to leverage the priors provided by pretrained generative models. As
described in figure 2, we first randomly sample the aforementioned 7 variables
as initial input to the generative models to generate a 3D human and objects
in the scene. Then we perform a differentiable rendering of human thermal
reflection. For every ray from the camera sensor that intersects with an
object, we can analytically calculate the reflected ray vector, given that the
surface normals of the objects are defined by the output of the object
generative model. With these reflected ray vectors, we can render a binary
reflection image based on whether the reflected ray vectors intersect with
humans, whose exact 3D shape and location are defined by the output of the
human generative model. The optimization objective is to maximize the
similarity between the rendered reflection image and the observed image
captured by the thermal camera.

In order for such a pipeline to be differentiable, we need both the generative
models of humans and objects, as well as the rendering algorithm, to be
differentiable. In the following sections, we will describe how we achieve
this.

###  3.3 Generative Models

Object: DeepSDF. We decided to use DeepSDF [59] as our generative models for
objects. SDF, or signed distance function, is a function between a point in
space and its orthogonal distance to the closest surface. In essence, DeepSDF
is an SDF parameterized by a neural network $G_{obj}$ whose input is a 3D
coordinate ${\bf p}$ and output is a signed distance $s$. Following [59], we
condition a DeepSDF model on a latent vector ${\bf z}_{obj}$ from a
probabilistic latent space to make them generative model:

$G_{obj}({\bf p},{\bf z}_{obj})=s:{\bf p}\in\mathbb{R}^{3},s\in\mathbb{R}$ (2)

Figure 4: 3D Object Reconstruction from RGBD

Human: SMPL-X. We adopted SMPL-X [61] as our generative models of 3D humans.
Broadly, SMPL-X is composed of 2 components. The first is a variational
autoencoder (VAE) that projects a latent vector ${\bf z}_{h}$ sampled from a
probabilistic latent space with Gaussin prior to the human pose space, in the
form of rotations of human body joints. The generated human body pose is then
applied with a differentiable sparse linear regressor to generate vertices and
triangle meshes representing the surface skins of a human body. Because both
the VAE and the linear vertex regressor are differentiable, the location of
each vertex is differentiable w.r.t. the latent vector ${\bf z}_{h}$.

Figure 5: 3D Human Reconstruction (visualized from another camera view). From
an RGBD image, we recover the 3D location ${\bf T}_{obj}$, pose $\phi_{obj}$,
and shape ${\bf z}_{obj}$ of each object. A marching cube visualization of the
reconstructed 3D objects is shown in pink. With reconstructed objects, we
recover 3D location ${\bf T}_{h}$, pose $\phi_{h}$, and shape ${\bf z}_{h}$ of
the human from a denoised thermal input showing reflections of the human on
object surfaces, which we visualize in blue. We also include the original
scene and our reconstruction from a calibrated third-camera view for
comparison. This image is not seen by our system during reconstruction. The
black mesh where the objects are located is the depth pointclouds captured by
the RGBD camera.

###  3.4 Differentiable Rendering of Reflection

The information we have from the thermal image of objects is a reflected human
silhouette. Soft rasterizer (SoftRas) [46] is a method of choice to perform
differentiable rendering from 2D silhouette images. However, SoftRas is a
differentiable rasterization algorithm, which does not directly apply to
reflection, especially when the reflective surface is a curved surface defined
by a DeepSDF. To overcome this limitation, we extended SoftRas to ray tracing
under non-planar reflection off the zero-isosurface of a DeepSDF. This process
is visualized in figure 3.

DeepSDF Depth Estimation. The complex geometry of an everyday object prevents
us from projecting all triangles to the 2D image plane as in [46]. Thus, we
need to march rays $\\{{\bf r}_{i}\\}$ from camera sensor ${\bf c}$, through
the reflection point on the surface $\\{{\bf p}_{i}\\}$ with a surface normal
$\\{{\bf n}_{i}\\}$, to the reflected rays $\\{{\bf r}_{i}^{\prime}\\}$. To
obtain the intersection point with the surface $\\{{\bf p}_{i}\\}$ given an
SDF representation of an object, we need a differentiable method to extract
the zero-isosurface and calculate the depth of the surface along the incoming
ray ${\bf r}_{i}$. Previously, [79] proposed to perform surface projection by
first grid-searching for a point close to the zero-isosurface, then projecting
along gradient direction $\frac{\partial G}{\partial{\bf p}}$ with the
predicted distance. However, because the gradient direction is not in the same
direction as the incoming ray, performing such an operation could yield a
point far from the intersection point between the incoming ray and zero-
isosurface, especially when the attack angle is small. To mitigate this error,
we perform finite steps of sphere tracing along the ray to estimate the
intersection point $\\{{\bf p}_{i}\\}$ as shown in figure 3.

DeepSDF Surface Normal. With the estimated intersection point $\\{{\bf
p}_{i}\\}$ between $\\{{\bf r}_{i}\\}$ and the surface of the object, we
calculate the surface normal of the object at $\\{{\bf p}_{i}\\}$:

${\bf n}_{i}=\frac{\partial G_{obj}({\bf p}_{i},{\bf z}_{obj})}{\partial{\bf
p}_{i}}$ (3)

We can then calculate the reflected ray vector as:

${\bf r}_{i}^{\prime}={\bf r}_{i}+2\cdot{\bf r}_{i}\cdot\frac{{\bf
n}_{i}}{\|{\bf n}_{i}\|_{2}}$ (4)

Figure 6:  Real-world 3D human reconstruction from thermal reflections of
cars. A diverse set of human poses can be reconstructed by using the surfaces
of different types of cars as infrared mirrors. RGB input (1st row) and
thermal input (2nd row) captured by a depth camera and a thermal camera are
used as input to our method. Our reconstruction (3rd row) is compared with the
original scene (4th row), both rendered/captured from another camera
viewpoint.

3D Ray-Triangle Distance. We can then calculate the pairwise distance matrix,
denoted as $\mathcal{D}_{i,j}$ between each reflected ray ${\bf
r}_{i}^{\prime}$ and each triangle $t_{j}\in\\{{\bf M}_{h}\\}$, where ${\bf
M}_{h}$ represents human body mesh. Each element in the distance matrix
$d_{i,j}\in\mathcal{D}$ can be expressed as a differentiable function of
vertices of $t_{j}$ and the reflected ray vector ${\bf r}_{i}^{\prime}$. We
can also obtain a ray-triangle intersection matrix $\Lambda$ with the same
dimension as the distance matrix. Since the value of the ray-triangle
intersection is binary, this calculation is not required to be differentiable.

Differentiable Ray Occupancy. Following SoftRas [46], we define the influence
of each triangle $t_{j}$ on each ray $r_{i}^{\prime}$ where the influence is
expressed as a function of distance $d_{i,j}$:

$d^{\prime}_{i,j}=\textrm{sigmoid}\left(\lambda_{i,j}\frac{d^{2}_{i,j}}{\sigma}\right),\;\;\lambda_{i,j}\in\Lambda,\;\;d_{i,j}\in\mathcal{D}$
(5)

where $\lambda_{i,j}=1$ if reflected ray ${\bf r}_{i}^{\prime}$ intersects
with triangle ${\bf t}_{j}$, otherwise $-1$. $d_{i,j}$ denotes the distance
between ray ${\bf r}_{i}^{\prime}$ and triangle ${\bf t}_{j}$, $\sigma$ is a
hyperparameter that controls the “softness” of the influence. We then
aggregate the influence of each triangle for a ray reflected ${\bf
r}_{i}^{\prime}$ to obtain the estimated binary occupancy of the ray by human
body mesh ${\bf M}_{h}$:

$\hat{I_{i}}=\mathcal{A}(\\{\mathcal{D}\\}_{j})=1-\Pi_{j}(1-d_{i,j})$ (6)

The estimated binary occupancy of ray $\hat{I_{i}}$ is a value between 0 and 1
and is compared with the ground truth binary thermal image defined in Eq. 1.

###  3.5 Optimization for Inference

3D Object Reconstruction. We first estimate the 6 DoF pose, scale, and shape
of the objects present in the scene following a similar method as in [29]. We
optimize the locations $\\{{\bf T}_{obj}\\}_{i=0}^{N}$, rotations
$\\{\phi_{obj}\\}_{i=0}^{N}$, scale $\\{{\bf s}_{obj}\\}_{i=0}^{N}$, and the
shape of the objects $\\{{\bf z}_{obj}\\}_{i=0}^{N}$, where $\\{{\bf
z}_{obj}\\}_{i=0}^{N}$ are latent variables sampled from the probabilistic
latent space of DeepSDF ${\bf G}_{obj}$ s.t. ${\bf M}_{obj}=G_{obj}({\bf
z}_{obj})$. For each object, we minimize the objective:

$\mathcal{L}_{obj}=\mathcal{L}_{depth}+\mathcal{L}_{mask}+\mathcal{L}_{prior}$
(7)

where $\mathcal{L}_{depth}$ is the $L_{1}$ loss between the estimated depth
map and the measured depth map, $\mathcal{L}_{mask}$ denotes a pixel-wise
$L_{2}$ loss between the estimated segmentation mask and the observed
segmentation mask obtained from RGB observation, and $\mathcal{L}_{prior}$ is
a shape prior regularization term.

Figure 7: Visualization of reconstruction obtained from ablated variations of
our full model. While all variations can still find the 3D location of humans
relatively accurately, the fine-grained details of human poses are
significantly improved in our full model.

3D Human Reconstruction Given the estimated translations, rotations, scales,
and shape latent vectors from 3D object reconstruction, we optimize
translation ${\bf T}_{h}$, rotation $\phi_{h}$, and shape ${\bf z}_{h}$ of the
human where ${\bf z}_{h}$ is the latent vector sampled from the pose VAE in
SMPL-X s.t. ${\bf M}_{h}=G_{h}({\bf z}_{h})$. Upon obtaining the estimated
reflection image $\hat{{\bf I}}$ and observed thermal silhouette image ${\bf
I}$, we minimize the objective:

$\mathcal{L}_{human}=\mathcal{L}_{silhouette}+\mathcal{L}_{prior}$ (8)

where

$\mathcal{L}_{silhouette}=1-\frac{\|\hat{{\bf I}}\otimes{\bf
I}\|_{1}}{\|\hat{{\bf I}}\oplus{\bf I}-\hat{{\bf I}}\otimes{\bf I}\|_{1}}$ (9)

and $\mathcal{L}_{prior}$ is an $L_{2}$ regularization term on the human
latent vector ${\bf z}_{h}$

##  4 Experiments

The goal of our experiments is to validate our hypothesis that LWIR thermal
reflection on everyday objects provides sufficient information to perform
accurate 3D human reconstruction in the real world. In section 4.1, we first
demonstrate the accurate 3D reconstruction of objects from a single RGBD
image, which serves as a foundation for 3D human reconstruction from
reflection. We showcase our results on 3D human reconstruction with different
poses and object types with everyday objects (section 4.2) and cars (section
4.3). Lastly, in section 4.4 we perform quantitative and qualitative ablation
studies to evaluate the effectiveness of our technical approach.

###  4.1 3D Object Reconstruction

Real-world depth sensors are subject to often significant measurement errors
and are sensitive to lighting conditions (assuming an active stereo sensor).
The surface depth estimated is often noisy, non-smooth, and full of “holes”,
as shown in figure 4. Performing differentiable rendering of reflection using
the direct output of the depth sensor will necessarily introduce an excessive
amount of noise, given the reflected ray direction is calculated from the
surface normal. Therefore, we opted to perform 3D object reconstruction from
RGBD input first, then use the reconstructed surfaces for differentiable ray
tracing.

In figure 4, we visualize the reconstructed objects from the RGBD input.
Because our 3D representation of objects is an implicit function – DeepSDF, we
perform marching cubes to extract the zero-isosurface of each object generated
from the latent vector ${\bf z}_{obj}$. We then applied the SE(3)
transformation matrix which is calculated from the estimated location
$T_{obj}$ and pose $\phi_{obj}$.

Evaluation Method |  Object Type |  w/o Edge Sampling |  w/o Sphere Tracing |  w/o Surface Smoothing |  Full Model |  Random
---|---|---|---|---|---|---
2D Keypoints [17] |  Bowl |  0.231 |  0.224 |  0.145 |  0.116 |  0.346
2D Keypoints [17] |  Mug |  0.101 |  0.209 |  0.109 |  0.094 |  0.371
3D Skeleton [39] |  Bowl |  0.309 |  0.272 |  0.212 |  0.152 |  0.322
3D Skeleton [39] |  Mug |  0.223 |  0.215 |  0.202 |  0.126 |  0.317
Table 1: Quantitative evaluation of our reconstructed 3D human. We used two
evaluation methods by comparing the extracted 2D keypoints and 3D skeleton
with a calibrated 3rd camera view. Object type indicates the type of objects
serving as reflectors. Columns 3-5 are three variations of our full model with
some parts ablated. Random shows the corresponding metric if a random sample
were to be drawn from the HumanEva[68] dataset, which includes diverse poses
in daily human activities. Numbers show the average normalized Euclidean
distance between reconstruction and ground truth.

As shown in figure 4, the location, pose, and shape of 3D objects can be
faithfully reconstructed. Most importantly, we are able to obtain a high-
fidelity, smooth, and accurate object surface without an explicit
regularization on surface smoothness, which sets the foundation for the
differentiable rendering of reflection. The successful reconstruction even
when the depth input is noisy can be largely attributed to the object priors
provided by searching in the latent space of a pretrained generative model.
Generative priors such as a bowl is usually symmetric, the outside surface of
a mug is often smooth, are enforced during the optimization process.

###  4.2 3D Human Reconstruction

Given the reconstructed objects represented as individual DeepSDF models and
their locations, we perform joint optimization of human location ${\bf
T}_{h}$, orientation $\phi_{h}$, and shape ${\bf z}_{h}$. The input to the
differentiable rendering algorithm is a single binary thermal reflection
image, representing a mask of human silhouette on each reflective object, as
shown in figure 5. The binary mask of reflection is obtained from the thermal
camera pointing towards the reflective objects, with simple denoising and
thresholding. In addition to RGBD and thermal cameras, we put a third
calibrated camera in the scene to capture the scene from another angle for
evaluation and visualization. Note that any images from this camera are not
used as input to our system.

We render the reconstruction from the third camera’s perspective for
comparison with the original scene at the exact time input data was captured.
As shown in figure 5, the output of our method very accurately reconstructs
the original scene. Note that the subject in the original scene is wearing
normal clothing and the data is collected in a normal office environment
without special lab environmental control. Besides, the objects used to
reflect human thermal radiation are everyday objects with a variety of
textures and materials that we purchased from supermarkets. This indicates the
robustness of our system and its practical applicability to various settings.

###  4.3 Cars as Infrared Mirrors

Non-line-of-sight information of human activity plays a crucial role in the
safe deployment of autonomous driving systems. Therefore, we showcase an
experiment where we use cars as infrared mirrors to reconstruct the 3D
location, orientation, and shape of a pedestrian that’s not in the line-of-
site of a camera system. In figure 6, we show the results in a similar fashion
as figure 5. 3D reconstruction from thermal imaging could allow new
opportunities for autonomous vehicles to sense and safely avoid occluded
pedestrians.

###  4.4 Ablation Studies

To solve the extremely under-constrained and challenging problem, we made a
lot of design decisions that turned out to be crucial to the quality of
reconstruction. To evaluate the effectiveness of our technical approach, we
perform ablation studies and compare our reconstruction with a baseline. We
have included both quantitative evaluations as well as qualitative
visualizations. Here we described some representative design decisions in
detail.

Edge Sampling. As pointed out by [41], edge sampling plays an important role
in differentiable ray tracing. This is even more significant for human
reflection silhouettes. In addition, unless a person is standing right in
front of the reflector, the reflection silhouette usually occupies a small
region of the thermal image. We therefore perform edge detection on the
reflection image to extract edges of human silhouette and sampling ray with a
probability distribution concentrated at the vicinity of these edges and
increasing the concentration as training progresses as a type of curriculum
training.

Sphere Tracing. As we’ve described in 3.4, direct surface projection from the
vicinity of an SDF will yield a point far from the real intersection between
the incoming ray and the zero-isosurface of the SDF. Therefore, we perform 3
steps of sphere tracing to estimate the intersection point on the object.

Surface Smoothing. From experiments, we discovered that even after we perform
sphere tracing, the reflection surface normals are still noisy, causing the
differentiable rendering algorithm to produce a noisy reflection. This
effectively injects noise into the gradients, making the optimization more
challenging. We discovered that this is caused by the reconstructed DeepSDF
having a locally non-smooth zero iso-surface. In figure 8, we visualize the
surface normals calculated from a small region of zero-isosurface which shows
the non-smoothness. To mitigate this error, we perform surface smoothing
during differentiable rendering by sampling 8 neighboring rays surrounding the
main ray and averaging all estimated surface normals for reflection
calculation.

Figure 8: Visualization of DeepSDF Surface Normals within a 1.6 cm$\times$1.6
cm area. From the visualization, we can clearly see an improvement in surface
smoothness at a small scale, which is beneficial to the differentiable
rendering process. The X-Y plane (horizontal) indicates the location on a
surface with a step size of 0.2mm. Given the unit surface normal vector at a
point on the grid $(x,y)$, we compute its dot product with the unit surface
normal vector at $(0,0)$, and plot this value on the Z-axis. This shows the
curvature of the surface as well as its level of smoothness.

Evaluation. We evaluate our reconstruction as well as the 3 aforementioned
ablated methods by comparing the 2D keypoints and 3D skeleton estimated from
synchronized images captured by a calibrated third camera. We used [17] for 2D
keypoints detection and [39] for 3D skeleton estimation. For comparison, we
compared the reconstruction to 200 randomly sampled 2D human keypoints and 3D
skeletons from the HumanEva dataset [68].

Both the quantitative experiments and qualitative visualizations have shown
the effectiveness of our technical approach as well as the design decisions.
Particularly, we believe our findings regarding differentiable rendering of
reflections on implicit surfaces will provide insights to other computer
vision researchers working with reflections.

##  5 Conclusion

This paper shows that 3D position and pose of a human can be reconstructed
from a single thermal image of everyday objects reflecting human thermal
radiations. We approach this problem by combining the priors learned by
pretrained 3D generative models and differentiable rendering of reflections.
By formulating the problem as an optimization problem, we perform analysis by
synthesis to explain the observations. We believe thermal cameras are powerful
tools to study human activities in daily environments and integrating them
with modern computer vision models will bring out many downstream applications
in robotics, graphics, and 3D perception.

Acknowledgements: This research is based on work partially supported by the
Toyota Research Institute, the NSF NRI Award #1925157, and the NSF CAREER
Award #2046910. We acknowledge Shree Nayar, Shuran Song, Runlin Xu, James
Tompkin, Mark Sheinin, Mia Chiquier, Jeremy Klotz for helpful feedback, and Su
Li, Dylan Chen, Sophia Su for helping with data collection.

## References

  * [1] Ijaz Akhter and Michael J Black.  Pose-conditioned joint angle limits for 3d human pose reconstruction.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1446–1455, 2015.
  * [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [4] Ganbayar Batchuluun, Na Rae Baek, Dat Tien Nguyen, Tuyen Danh Pham, and Kang Ryoung Park.  Region-based removal of thermal reflection using pruned fully convolutional network.  IEEE Access, 8:75741–75760, 2020.
  * [5] Ganbayar Batchuluun, Hyo Sik Yoon, Dat Tien Nguyen, Tuyen Danh Pham, and Kang Ryoung Park.  A study on the elimination of thermal reflections.  IEEE Access, 7:174597–174611, 2019.
  * [6] Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann.  Blazepose: On-device real-time body pose tracking.  arXiv preprint arXiv:2006.10204, 2020.
  * [7] HE Bennett and JO119764 Porteus.  Relation between surface roughness and specular reflectance at normal incidence.  JOSA, 51(2):123–129, 1961.
  * [8] Samarth Brahmbhatt, Cusuh Ham, Charles C Kemp, and James Hays.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8709–8719, 2019.
  * [9] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Generative and discriminative voxel modeling with convolutional neural networks.  arXiv preprint arXiv:1608.04236, 2016.
  * [10] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [11] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012, 2015.
  * [12] Chia-Yen Chen, Chia-Hung Yeh, Bao Rong Chang, and Jun-Ming Pan.  3d reconstruction from ir thermal images and reprojective evaluations.  Mathematical Problems in Engineering, 2015, 2015.
  * [13] I-Chien Chen, Chang-Jen Wang, Chao-Kai Wen, and Shiow-Jyu Tzou.  Multi-person pose estimation using thermal images.  IEEE Access, 8:174964–174971, 2020.
  * [14] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang.  Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5386–5395, 2020.
  * [15] James W Davis and Mark A Keck.  A two-stage template approach to person detection in thermal imagery.  In 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05)-Volume 1, volume 1, pages 364–369. IEEE, 2005.
  * [16] Yu Deng, Jiaolong Yang, and Xin Tong.  Deformed implicit field: Modeling 3d shapes with learned dense correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286–10296, 2021.
  * [17] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  In Proceedings of the IEEE international conference on computer vision, pages 2334–2343, 2017.
  * [18] Lu Gan, Connor Lee, and Soon-Jo Chung.  Unsupervised rgb-to-thermal domain adaptation via multi-domain attention network.  arXiv preprint arXiv:2210.04367, 2022.
  * [19] Shubham Goel, Angjoo Kanazawa, , and Jitendra Malik.  Shape and viewpoints without keypoints.  In ECCV, 2020.
  * [20] Yanran Guan, Tansin Jahan, and Oliver van Kaick.  Generalized autoencoder for volumetric shape generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 268–269, 2020.
  * [21] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297–7306, 2018.
  * [22] Fei Han, Brian Reily, William Hoff, and Hao Zhang.  Space-time representation of people based on 3d skeletal data: A review.  Computer Vision and Image Understanding, 158:85–105, 2017.
  * [23] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely, and Serge Belongie.  Dualsdf: Semantic shape manipulation using a two-level representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7631–7641, 2020.
  * [24] Jingu Heo, Seong G Kong, Besma R Abidi, and Mongi A Abidi.  Fusion of visual and thermal signatures with eyeglass removal for robust face recognition.  In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 122–122. IEEE, 2004.
  * [25] Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand.  Difftaichi: Differentiable programming for physical simulation.  arXiv preprint arXiv:1910.00935, 2019.
  * [26] Le Hui, Rui Xu, Jin Xie, Jianjun Qian, and Jian Yang.  Progressive point cloud deconvolution generation network.  In European Conference on Computer Vision, pages 397–413. Springer, 2020.
  * [27] Moritz Ibing, Isaak Lim, and Leif Kobbelt.  3d shape generation with grid-based implicit functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13559–13568, 2021.
  * [28] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.
  * [29] Muhammad Zubair Irshad, Sergey Zakharov, Rares Ambrus, Thomas Kollar, Zsolt Kira, and Adrien Gaidon.  Shapo: Implicit representations for multi-object shape, appearance, and pose optimization.  arXiv preprint arXiv:2207.13691, 2022.
  * [30] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker.  Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1251–1261, 2020.
  * [31] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In ECCV, 2018.
  * [32] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3d mesh renderer.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907–3916, 2018.
  * [33] Hyeongju Kim, Hyeonseung Lee, Woo Hyun Kang, Joun Yeop Lee, and Nam Soo Kim.  Softflow: Probabilistic framework for normalizing flow on manifolds.  Advances in Neural Information Processing Systems, 33:16388–16397, 2020.
  * [34] Roman Klokov, Edmond Boyer, and Jakob Verbeek.  Discrete point flow networks for efficient point cloud generation.  In European Conference on Computer Vision, pages 694–710. Springer, 2020.
  * [35] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5253–5263, 2020.
  * [36] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2252–2261, 2019.
  * [37] Zülfiye Kütük and Görkem Algan.  Semantic segmentation for thermal images: A comparative survey.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 286–295, 2022.
  * [38] Jia Li, Wen Su, and Zengfu Wang.  Simple pose: Rethinking and improving a bottom-up approach for multi-person pose estimation.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11354–11361, 2020.
  * [39] Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu.  Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3383–3393, 2021.
  * [40] Ning Li, Yongqiang Zhao, Quan Pan, and Seong G Kong.  Removal of reflections in lwir image with polarization characteristics.  Optics express, 26(13):16488–16504, 2018.
  * [41] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen.  Differentiable monte carlo ray tracing through edge sampling.  ACM Transactions on Graphics (TOG), 37(6):1–11, 2018.
  * [42] Tzu-Mao Li, Michal Lukávc, Michaël Gharbi, and Jonathan Ragan-Kelley.  Differentiable vector graphics rasterization for editing and learning.  ACM Transactions on Graphics (TOG), 39(6):1–15, 2020.
  * [43] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  In European Conference on Computer Vision, pages 677–693. Springer, 2020.
  * [44] Kevin Lin, Lijuan Wang, and Zicheng Liu.  End-to-end human pose and mesh reconstruction with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1954–1963, 2021.
  * [45] Ruoshi Liu, Sachit Menon, Chengzhi Mao, Dennis Park, Simon Stent, and Carl Vondrick.  Shadows shed light on 3d objects.  arXiv preprint arXiv:2206.08990, 2022.
  * [46] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708–7717, 2019.
  * [47] Matthew M Loper and Michael J Black.  Opendr: An approximate differentiable renderer.  In European Conference on Computer Vision, pages 154–169. Springer, 2014.
  * [48] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837–2845, 2021.
  * [49] Tomohiro Maeda, Yiqin Wang, Ramesh Raskar, and Achuta Kadambi.  Thermal non-line-of-sight imaging.  In 2019 IEEE International Conference on Computational Photography (ICCP), pages 1–11. IEEE, 2019.
  * [50] E Maset, A Fusiello, F Crosilla, R Toldo, and D Zorzetto.  Photogrammetric 3d building reconstruction from thermal images.  ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 4:25, 2017.
  * [51] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [52] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [53] Gyeongsik Moon and Kyoung Mu Lee.  I2l-meshnet: Image-to-lixel prediction network for accurate 3d human pose and mesh estimation from a single rgb image.  In European Conference on Computer Vision, pages 752–768. Springer, 2020.
  * [54] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv preprint arXiv:2201.05989, 2022.
  * [55] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan.  Single-stage multi-person pose machines.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6951–6960, 2019.
  * [56] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [57] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob.  Mitsuba 2: A retargetable forward and inverse renderer.  ACM Transactions on Graphics (TOG), 38(6):1–17, 2019.
  * [58] Michael Oren and Shree K Nayar.  Generalization of lambert’s reflectance model.  In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 239–246, 1994.
  * [59] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [60] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [61] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975–10985, 2019.
  * [62] Sameera Ramasinghe, Salman Khan, Nick Barnes, and Stephen Gould.  Spectral-gans for high-resolution 3d point-cloud generation.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8169–8176. IEEE, 2020.
  * [63] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [64] Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas.  Humor: 3d human motion model for robust pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11488–11499, 2021.
  * [65] Rafael E Rivadeneira, Angel D Sappa, Boris X Vintimilla, Jin Kim, Dogun Kim, Zhihao Li, Yingchun Jian, Bo Yan, Leilei Cao, Fengliang Qi, et al.  Thermal image super-resolution challenge results-pbvs 2022.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 418–426, 2022.
  * [66] Agata Sage, Daniel Ledwoń, Jan Juszczyk, and Paweł Badura.  3d thermal volume reconstruction from 2d infrared images—a preliminary study.  In International Scientific Conference Advances in Applied Biomechanics, pages 371–379. Springer, 2020.
  * [67] Sebastian Schramm, Phil Osterhold, Robert Schmoll, and Andreas Kroll.  Combining modern 3d reconstruction and thermal imaging: Generation of large-scale 3d thermograms in real-time.  Quantitative InfraRed Thermography Journal, 19(5):295–311, 2022\.
  * [68] Leonid Sigal, Alexandru O Balan, and Michael J Black.  Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion.  International journal of computer vision, 87(1):4–27, 2010.
  * [69] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben @articleli2018differentiable, title=Differentiable monte carlo ray tracing through edge sampling, author=Li, Tzu-Mao and Aittala, Miika and Durand, Frédo and Lehtinen, Jaakko, journal=ACM Transactions on Graphics (TOG), volume=37, number=6, pages=1–11, year=2018, publisher=ACM New York, NY, USA Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495–7504, 2021.
  * [70] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693–5703, 2019.
  * [71] Garvita Tiwari, Dimitrije Antić, Jan Eric Lenssen, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll.  Pose-ndf: Modeling human pose manifolds with neural distance fields.  In European Conference on Computer Vision, pages 572–589. Springer, 2022.
  * [72] Andre Treptow, Grzegorz Cielniak, and Tom Duckett.  Real-time people tracking for mobile robots using thermal vision.  Robotics and Autonomous Systems, 54(9):729–739, 2006.
  * [73] Delio Vicini, Sébastien Speierer, and Wenzel Jakob.  Differentiable signed distance function rendering.  ACM Transactions on Graphics (TOG), 41(4):1–18, 2022.
  * [74] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  Advances in neural information processing systems, 29, 2016.
  * [75] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3d objects from images in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1–10, 2020.
  * [76] Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Sagnet: Structure-aware generative network for 3d-shape modeling.  ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  * [77] Yufei Ye, Shubham Tulsiani, and Abhinav Gupta.  Shelf-supervised mesh prediction in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8843–8852, 2021.
  * [78] Xiang Yu, Feng Zhou, and Manmohan Chandraker.  Deep deformation network for object landmark localization.  In European conference on computer vision, pages 52–70. Springer, 2016.
  * [79] Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon.  Autolabeling 3d objects with differentiable rendering of sdf shape priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12224–12233, 2020.
  * [80] Björn Zeise and Bernardo Wagner.  Temperature correction and reflection removal in thermal images using 3d temperature mapping.  In ICINCO (2), pages 158–165, 2016.
  * [81] Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao.  A differential theory of radiative transfer.  ACM Transactions on Graphics (TOG), 38(6):1–16, 2019.


Anna Zamansky and Eitan Farchilogo_ttlM. Antonia Huertas, João Marcos, María
Manzano, Sophie Pinchinat,
François Schwarzentruber54th International Conference on Tools for Teaching
Logic11273TTL2015

# Teaching Logic to Information Systems Students: Challenges and Opportunities

Anna Zamansky  Information Systems Department, University of Haifa
Haifa, Israel
<EMAIL_ADDRESS>Eitan Farchi  IBM Research
Haifa, Israel
<EMAIL_ADDRESS>
###### Abstract

In contrast to Computer Science, where the fundamental role of Logic is widely
recognized, it plays a practically non-existent role in Information Systems
curricula. In this paper we argue that instead of Logic’s exclusion from the
IS curriculum, a significant adaptation of the contents, as well as teaching
methodologies, is required for an alignment with the needs of IS
practitioners. We present our vision for such adaptation and report on
concrete steps towards its implementation in the design and teaching of a
course for graduate IS students at the University of Haifa. We discuss the
course plan and present some data on the students’ feedback on the course.

###### keywords:

Information Systems, Logic, Education, Formal Methods

##  1 Introduction

The fundamental role of Logic in the Computer Science curriculum is widely
recognized. The ACM CS undergraduate curriculum guidelines ([16]) explicitly
state logic as a mathematical requirement which is directly relevant for the
large majority of all CS undergraduates (together with elements of set theory
and discrete probability). This recommendation is implemented in most of the
standard CS undergraduate study programs by including in the curriculum a
course in discrete structures which includes a significant amount of formal
logic.

The (academic) field of Information Systems (IS) encompasses two broad areas:
(i) acquisition, deployment, and management of information technology
resources and services, and (ii) development and evolution of infrastructure
and systems for use in organization processes. Thus, as opposed to CS, IS’s
primary focus is on an organization’s mission and objectives and the
application of information technology to further these goals. Yet both IS and
CS require a common subset of technical knowledge, reflected also in the
intersection of the respective study programs’ curricula.

Logic, however, does not appear to be in this intersection – almost none of
the IS undergraduate study programs include such course in their curriculum.
The ACM IS curriculum guidelines ([18]), which mention statistics and
probability as required core IS topics and discrete mathematics as an optional
one, but do not refer to logic as relevant: “Even though IS professionals do
not need the same level of mathematical depth as many other computing
professionals, there are, however, some core elements that are very important
for IS professionals. To support in-depth analysis of data, IS professionals
should have a strong background in statistics and probability. For those who
are interested in building a strong skill set in algorithmic thinking,
discrete mathematics is important."

We believe the current state of affairs is suboptimal for several reasons.
First of all, all the reasons for including logic in the CS curricula still
hold in the IS domain: it is widely acknowledged that studying logic directly
contributes to software development skills, confirmed recently also by
empirical studies ([14]). Secondly, the lack of experience with formal
notation forms a major cognitive barrier to the adoption of formal methods in
the industry ([21]). This is further reinforced by the fact that because many
IS study programs tend to be marketed as programs "excluding the hard
math"111Below are some exemplary quotes from webpages of academic institutions
providing both CS and IS degrees on the comparison between the two:  • “As a
rule, computer science requires more mathematics and analytical skill than
information systems. Also, our experience has shown that it is easier to move
from being a CS major to an IS major than the other way around. Therefore, if
you feel reasonably comfortable with the math requirements, then you should
start out as a computer science major.” (Saint Michael’s College) • “A major
in CS will know a considerable amount of mathematics which will help in
technological applications…An IS major needs to be aware of what information
technology can contribute to an organization and how to bring that solution to
fruition.” (University of Missouri-St. Louis). , the students come to see the
lack of mathematical courses as a benefit, and express
disappointment222Quoting one of our graduate students who was assigned to read
a research paper on formal methods: “When I see formal definitions, I just
want to cry.” Notably, she is one of the best students in her class. when any
formal notations are integrated in the IS core courses – thus creating a
vicious circle.

Although we tend to agree that a typical IS major may need a less extensive
mathematical background than a CS major, we believe that rather than excluding
logic from the IS curriculum, a significant adaptation of the contents that
are taught to align with IS objectives. Recently more voices are calling for a
reconsideration of the traditional logic curriculum and its adaptation to the
needs of future pracitioners. Several proposals on what to teach and how to
teach it have been made in the context of CS ([13, 17, 19]).

In this paper we address these questions in the context of the IS curriculum.
We report on our experience in designing and teaching the course "Logic and
Formal Specification" to graduate students at the Information Systems (IS)
department at the University of Haifa, which is one of the few to include a
mandatory course on logic and formal methods in its graduate study program. We
discuss our view of what should be included in the “IS logic toolbox" in order
for the students to be able to carry out activities for checking (by proof,
analysis or testing) that a software system meets specifications and fulfills
its intended purpose. These tools should at least (i) by providing background
on induction and propositional and first-order logic, and (ii) by providing
the ability to read, write and reason about formal specifications. We share
our insights on how the above can be achieved: (i) excluding complex
mathematical intricacies, and (ii) providing simple yet software-related
examples. Concerning the latter, we report on an ongoing work to develop a
tool for measuring “simplicity" of Z specifications. While the above insights
are yet to be empirically validated, some indications of the benefits of our
approach are reflected in our students’ feedback, which we briefly discuss
below.

##  2 The IS Logic Toolbox

The main practical objective in teaching logic to IS practitioners is to give
them the ability to apply formal methods in industry. Application of formal
aspects is particularly important for software quality control, i.e.,
activities for checking (by proof, analysis or testing) that a software system
meets specifications and that it fulfills its intended purpose.

Due to the density of the IS curricula, currently one cannot afford to have
one course on pure formal logic and then another on formal methods (this
problem is also discussed in [20] in the context of CS). Therefore, one must
develop some mixture of a introductory formal logic together with introduction
to formal methods relevant for the IS domain. In what follows we briefly
survey previous reflections on the content of logic and formal methods courses
that practitioners really need and their integration into the curricula, and
propose how to adapt the proposed ideas for the context of IS.

###  2.1 Previous Proposals

Recently there has been an ongoing discussion on whether the traditional logic
syllabus for CS is relevant for practitioners. Since our main goal in this
paper is to extend and adapt this discussion to the context of IS, we start by
briefly outlining some relevant proposals (mostly in the context of CS), the
ideas of which are close in spirit to the vision we present below.

In his paper “From Hilbert to a Logic Toolbox" [13], J. Makowsky questions the
suitability of the standard logic syllabus to the needs of CS practitioners.
He states: “The current syllabus is often justified more by the traditional
narrative than by the practitioner’s needs." He further notes that most
classical logic textbooks follow the narrative of the rise and fall of
Hilbert’s program, emphasizing the following ideas: (i) Logic is needed to
resolve the paradoxes of set theory; (ii) First-order logic (FOL) is The logic
due to its Completeness theorem; (iii) The main theorems of FOL are the
Completeness and Compactness theorems; (iv) The tautologies of FOL are not
recursive; (v) One cannot prove consistency within rich enough systems. This,
according to Makowsky, is not what a CS practitioner needs: “The proof of the
Completeness Theorem is a waste of time at the expense of teaching more the
important skills of understanding the manipulation and meaning of formulas."
What he needs is: (i) understand the meaning and implications of modeling the
environment as precise mathematical objects and relations; (ii) understand and
be able to distinguish intended properties of this modeling and side-effects;
(iii) be able to discern different level of abstraction, and (iv) understand
what it means to prove properties of modeled objects.

In her papers [20, 19], J. Wing stresses the importance of integrating formal
methods into the existing CS curriculum by teaching their common conceptual
elements, including state machines, invariants, abstraction, composition,
induction, specification and verification. She states discrete mathematics and
mathematical logic as crucial prerequisites.

The above proposals on what to teach are extremely relevant for IS
practitioners. On the question of how to teach, the paper “Integrating Formal
Methods into Computer Science Curricula at a University of Applied Science"
([17]) of Tavolato and Vogt offers some useful insights. It discusses teaching
formal methods at universities of applied sciences, where there are usually
limiting factors which are relevant to the IS context as well: (i) students
have very limited theoretical background, and (ii) they are strongly focused
on the direct applicability of what they are taught. In this context the
authors stress the importance of making the practical applicability of the
theory understandable to students, and making use of real industry-inspired
examples.
In what follows, we extend and adapt the above proposals to the context of IS,
and provide our vision on aligning the teaching of logic to the needs of IS
practitioners.

###  2.2 Our Vision: Making Logic Relevant for IS

Logic is a prerequisite for understanding and successfully using formal
methods, which in their turn can significantly contribute to software quality
control. We agree with [20] that the main basic formal conceptual elements
that the students need to be familiar with include state machines,
abstraction, composition, induction, invariants, specification and
verification. While the students encounter the concepts of state machines,
abstraction and composition at other IS courses (such as modeling and design),
aspects related to working with formal specifications are not covered
elsewhere. This leads to the following practical needs of an IS practitioner:
(1) read, write and understand formal specifications, (2) be able to formalize
informal specifications, (3) analyze specifications and detect sources of
incompleteness, inconsistency and complexity, (4) reason about specifications,
and (5) check a system against a specification.

Based on the above, adapting and extending the previous proposals to the
context of IS, we arrive at the following IS logic toolbox: (a) Basic
principles for reasoning about sets; (b) Induction and invariants; (c)
Propositional and first-order logic and their axiomatizations; (d) Formal
specification and verification.

As to how to teach logic to IS students, i.e., designing concrete teaching
methodologies, the following considerations need to be taken into account:

  * •

Examples from software domains are useful. Although it has been believed for
some time that studying logic improves software development skills, this
common belief has recently been empirically validated. In a three-year study
in the framework of the Beseme project ([14]), empirical data on the
achievements of two student populations was collected: those who studied
discrete mathematics (including logic) through examples focused on reasoning
about software, and those who studied the same subject illustrated with more
traditional examples. An analysis of the data revealed significant differences
in the programming effectiveness of these two populations in favor of the
former. As pointed out by [17], software related examples are also useful for
increasing the motivation of students, who can see the applications of the
studied material in the domain of their interest.

  * •

Cognitive difficulties should not be ignored. Empirical studies show that the
use of formal methods poses objective difficulties for practitioners ([5, 9]).
They are also hypothesized to be a major hindering factor for the acceptance
of formal methods in industry ([21]). Although the cognitive processes of
students when studying logic and formal methods are not well understood, they
should not be ignored ([17]). Numerous studies in the education community
addressed the gap between the students’ intuition and formal thinking in
mathematics (see, e.g., [7]). Implementing similar ideas in the domain of
teaching logic and formal methods may help deal with these barriers.

  * •

Intricate complexities are not always needed. Exposing the students to full
intricate complexities of mathematical logic (such as a full proof of the
completeness theorem, or dealing with variables not free for substitution) has
the potential to confuse novices struggling to understand new ideas. However,
most of the practitioners will not encounter them in industry. This is in line
the research agenda of indirect application of formal methods ([11]), calling
for hiding the intricate complexities behind automatic tools with intuitive
user interface. The benefits of hiding logical complexity behind the more
intuitive interface of functional programming are also mentioned in [14].

In view of the above considerations, the basic principles in the design of the
course described below have been (i) use mainly examples from software
domains, (ii) use comprehensible examples, and (iii) introduce the logical
concepts at a basic level. The issue of comprehensibility also led to our
ongoing work of automatically measuring comprehensibility of Z specifications,
which we briefly describe below.

##  3 Teaching Logic at the IS Department of the University of Haifa

In this section we demonstrate how the vision presented in Section 2.2 has
been implemented in our design of the course “Logic and Formal Specification".
The course has been taught at the IS department at the University of Haifa for
several years by both of the authors333Perhaps it is important to mention here
the authors’ relevant background. The first author is an associate professor
at the Information Systems Department at the University of Haifa with active
research interests in applied logic. The second author is the manager of the
Software Performance and Quality research group at the IBM Haifa Research
Laboratory, and a member of the IBM corporate Board of Software Quality. Both
of the authors have several years of experience in teaching logic and formal
methods to various audiences of students. . The course is a mandatory course
for graduate students, and its length is one semester, 4 hours per week.

###  3.1 Course Description

Below we provide a short description of the course’s main topics, which are
divided into two main parts:

Part I: Introduction to Logic

  * •

Informal laws of mathematical reasoning

Our starting point is the place where the students left off in a discrete
mathematics course: with basic set-theoretical concepts. However, our primary
focus is not on understanding the concepts themselves, but on reasoning about
them by applying informal logical laws. Accordingly, the students are asked to
provide proofs of basic claims, explaining which laws were used at each stage.
The presentation of the informal laws and other proof tips is adapted from
[12]. The informal laws become explicit at the object level when classical
propositional and first-order logic are introduced to the students (E.g., the
law for proving general statements can be captured by the rule inferring
$\forall x\psi$ from $\psi(x)$, and the law for proving conditional statements
is captured by the deduction theorem.)

  * •

Induction: mathematical, structural and computational induction.
Induction is in the heart of several formal concepts relevant for verification
and validation of software: fixed point constructions, model checking, program
analysis and many more. Therefore a special emphasis is put on the topic
throughout the course, highlighting its various manifestations, e.g. proving
syntactic properties of logical formulas, proving the deduction theorem,
proving invariants with respect to a Z specification. Software-related
examples are adapted from Chapter 2 of ([3]).

  * •

Classical Propositional and First-Order Logic: syntax and semantics,
satisfiability and validity, Hilbert-style axiomatization, formalization of
natural language sentences.
For this part of the course we mostly adapt parts of the standard presentation
of most mathematical logic textbooks. We make a special emphasis on
formalization of natural language specifications and induction at the expense
of omitting the proofs of the Completeness and Compactness theorems (in line
with the recommendation of [13]).

  * •

Survey of non-classical logics444This part of the course is implemented by
assigning each of the students a short presentation on a non-classical logic
or its applications of his choice. While the importance of temporal logic in
this context is perhaps the most obvious one due to its well-known
applications in verification, also other non-classical logics have IS-relevant
applications. Our goal here is to increase the awareness of the students to
the immense variety of logics outside the realm of classical logics, as well
as engage them more actively in the course. Several students have reported
that exploring new logics on their own was the part they enjoyed the most in
the course. : temporal logic, modal logic, many-valued logic, fuzzy logic,
non-monotonic logic, paraconsistent logic.

Part II: Introduction to Formal Specification

This part of the course builds up on the knowledge obtained at the previous
part. The final aim is for the students to be able to understand and write
formal specifications using the Z notation. For this we have adapted the
material from the textbook [15], covering the basic aspects of Z: types,
schemas and reasoning about Z specifications. However, staying faithful to our
vision oulined above, we have developed our own set of examples, which are (i)
"simple" and (ii) related to software domains. In what follows we shortly
discuss what we mean by "simple" and how "simplicity" can be measured.

#### Measuring comprehensibility of Z specifications

The notion of simplicity (item (i) above) is not well understood.
Comprehensibility (or understandability) of specifications is usually thought
of as the degree to which information contained in a specification is
understandable to the reader, and this is a well-studied topic in software
engineering (see, e.g., [6] for a survey). However, we are aware of only a few
works on Z specifications ([8, 10]), all of which on the structural dimension.
We believe, however, that simplicity is a key to comprehensibility of
specifications, at least at the stage of learning the topic. Our attempt to
quantitatively measure "simplicity has led to our ongoing project of
developing automatic tools for this purpose. Our current hypothesis is that
the nesting of definitions is and shortening notations by introducing
additional symbols decrease the understandability of specifications. We are
currently developing a tool555The tool is based on the open-source Java
framework Community Z tools ([2]). for measuring "simplicity" of
specifications. Using this tool, we plan to empirically check our hypothesis,
as well as to consider other comprehensibility dimensions.

#### Students’ Acceptance

The course has only been taught in its current form for five years, so making
decisive conclusions about its effectiveness is perhaps premature. However, an
important dimension in evaluating such effectiveness is the students’
acceptance and reaction. To gain a better understanding of these factors, the
first author has undertaken a preliminary qualitative study using a
questionnaire filled by twenty three students who took the course in
2013-2014.

It should be noted that the limiting factors typical of our target audience
are in many aspects similar to those described in [17]. The first is lack of
mathematical background: the undergraduate IS study program at the University
of Haifa does not include a course in logic, and the majority of students have
only a background in discrete mathematics, where they are taught very basic
concepts of set theory. The second limiting factor is their lack of
motivation: the majority of the students return to graduate school several
years after receiving their B.A, while working full-time. They typically
expect the topics to be directly relevant to their IS practice, and usually
exhibit difficulty in coping with the dense and abstract material taught in
the course. In light of these factors, we were expecting some of the students
to claim, basically, that the course was too hard without being helpful for
their future as IS practitioners. However, only one student out of 23 felt the
course was not useful for his practice. A full analysis of the data obtained
from students is out of the scope of this paper. However, some aspects
highlighted by the data were that (i) the majority of students felt the course
has improved their analytical thinking abilities666Examples: ”I don’t use
logic or Z on a daily basis in my research or my work, but it improved my
modelling skills”, “I think it can be even more useful as an introduction to
programming, because it teaches you to think systematically”, ”Every graduate
student needs this course as a basis for study and research”.; (ii) the
majority of students felt the part related to formal specification is relevant
for their IS practice777Examples: “I think Z language is the most useful part,
because it is applied in industry”, “If I encounter any other formal notation
in industry, it will take me less time to get familiar with the subject”.; and
(iii) some students felt the course has directly improved their daily IS
practice888Examples: ”I started using truth tables at work to rule out
impossible behaviors”, ”I was surprised to find out how helpful the tools we
obtained are in my daily work”..

##  4 Summary and Future Research

There has recently been a discourse on the relevance of traditional logic
courses to future computer science practitioners. Jeannette Wing writes in
[19]: “…we still face the educational challenge of teaching mathematical
foundations like logic and discrete mathematics to practicing or aspiring
software engineers. We need to go beyond giving the traditional courses and
think about who the target students are." This paper discusses these issues
for the target population of IS students, for whom the lack of direct
relevance of the traditional logic courses seems to have led to their
exclusion from the curriculum. We believe logic is central to IS objectives,
as it is the key to applying formal methods in specification, verification and
validation of information systems. Ideally, we need more empirical evidence in
the spirit of the Beseme project ([1]) that such courses are useful for IS
practitioners. In addidion, there is a need for a wider discussion on what
logical background is needed for Information Systems practitioners and how it
should be taught. In a contribution to such discussion, we have reported on
our insights from teaching the “Logic and Formal Specification" course to
graduate IS students. Like previous authors report in the context of CS, we
have seen that using software-related and comprehensible examples, as well as
simplification of logical intricacies contributes to achieving the courses’
objectives. From a more practical perspective, a future direction is an
empirical investigation of how to make formal specification more
understandable for students. This question is particularly interesting due to
its direct relation to the more general topic of comprehensibility of
specifications. In this context we plan to extend and refine our tool for
automatic analysis of Z specifications and carry out an empirical evaluation.
From the angle of education, strategies for an efficient integration of logic
and formal methods into the IS curricula are required (along the lines of [4,
20]), as well as an investigation of the ways to bridge intuitive and
analytical thinking processes in logic and formal methods (along the lines of
[7]). In this context we would also like to point out that a textbook with an
IS-orientation would be a welcome addition to the large existing variety of
CS-oriented books.

## References

  * [1] Beseme website, http://www.cs.ou.edu/ beseme/.
  * [2] CZT website, http://czt.sourceforge.net/manual.html.
  * [3] Alfred V Aho and Jeffrey D Ullman.  Foundations of computer science, volume 2.  Computer Science Press New York, 1992.
  * [4] Ian Barland, Matthias Felleisen, Kathi Fisler, Phokion Kolaitis, and Moshe Y Vardi.  Integrating logic into the computer science curriculum.  Innovation and Technology in Computer Science Education, 2000.
  * [5] Deirdre Carew, Chris Exton, and Jim Buckley.  An empirical investigation of the comprehensibility of requirements specifications.  In Empirical Software Engineering, 2005. 2005 International Symposium on, pages 10–pp. IEEE, 2005.
  * [6] Nelly Condori-Fernández, Maya Daneva, Klaas Sikkel, and Andrea Herrmann.  Practical relevance of experiments in comprehensibility of requirements specifications.  In Empirical Requirements Engineering (EmpiRE), 2011 First International Workshop on, pages 21–28. IEEE, 2011.
  * [7] Lisser Rye Ejersbo, Uri Leron, and Abraham Arcavi.  Bridging intuitive and analytical thinking: Four looks at the 2-glass puzzle.  For the Learning of Mathematics, 2014.
  * [8] K Finney, N Fenton, and A Fedorec.  Effects of structure on the comprehensibility of formal specifications.  In Software, IEE Proceedings-, volume 146, pages 193–202. IET, 1999\.
  * [9] Kate Finney.  Mathematical notation in formal specification: Too difficult for the masses?  Software Engineering, IEEE Transactions on, 22(2):158–159, 1996\.
  * [10] Kate Finney, Keith Rennolls, and Alex Fedorec.  Measuring the comprehensibility of Z specifications.  Journal of Systems and Software, 42(1):3–15, 1998.
  * [11] Heinrich Hussmann.  Indirect use of formal methods in software engineering.  In ICSE-17 Workshop on Formal Methods Application in Software Engineering Practice, Seattle (WA), USA. Proceedings, pages 126–133. Citeseer, 1995.
  * [12] David Makinson.  Sets, logic and maths for computing.  Springer, 2012.
  * [13] Johann A Makowsky.  From Hilbert’s program to a logic tool box.  Annals of Mathematics and Artificial Intelligence, 53(1-4):225–250, 2008.
  * [14] Rex L Page.  Software is discrete mathematics.  In ACM SIGPLAN Notices, volume 38, pages 79–86. ACM, 2003.
  * [15] Ben Potter, David Till, and Jane Sinclair.  An introduction to formal specification and Z.  Prentice Hall PTR, 1996.
  * [16] Mehran Sahami, Mark Guzdial, Andrew McGettrick, and Steve Roach.  Setting the stage for computing curricula 2013: computer science–report from the acm/ieee-cs joint task force.  In Proceedings of the 42nd ACM technical symposium on Computer science education, pages 161–162. ACM, 2011.
  * [17] Paul Tavolato and Friedrich Vogt.  Integrating formal methods into computer science curricula at a university of applied sciences.  In TLA+ Workshop at the 18th International Symposium on Formal Methods, Paris, Frankreich., 2012.
  * [18] Heikki Topi, Joseph S Valacich, Ryan T Wright, Kate Kaiser, Jay F Nunamaker Jr, Janice C Sipior, and Gert-Jan de Vreede.  Is 2010: Curriculum guidelines for undergraduate degree programs in information systems.  Communications of the Association for Information Systems, 26(1):18, 2010.
  * [19] Jeannette M. Wing.  Teaching mathematics to software engineers.  In Algebraic Methodology and Software Technology, 4th International Conference, AMAST ’95, Montreal, Canada, July 3-7, 1995, Proceedings, pages 18–40, 1995.
  * [20] Jeannette M Wing.  Weaving formal methods into the undergraduate computer science curriculum.  In Algebraic Methodology and Software Technology, pages 2–7. Springer, 2000.
  * [21] Marc K Zimmerman, Kristina Lundqvist, and Nancy Leveson.  Investigating the readability of state-based formal requirements specification languages.  In Proceedings of the 24th International Conference on Software engineering, pages 33–43. ACM, 2002.