0 Research please specify Masaki Takeuchi, Hiroki Kusuyama, Daisuke Iwai, and
Kosuke Sato are with Graduate School of Engineering Science, Osaka University.
E-mail: see https://www.sens.sys.es.osaka-u.ac.jp.

# Projection Mapping under Environmental Lighting
by Replacing Room Lights with Heterogeneous Projectors

Masaki Takeuchi0000-0002-0773-7705  Hiroki Kusuyama0009-0007-9406-2120
Daisuke Iwai0000-0002-3493-5635  and Kosuke Sato0000-0003-1429-9990

###### Abstract

Projection mapping (PM) is a technique that enhances the appearance of real-
world surfaces using projected images, enabling multiple people to view
augmentations simultaneously, thereby facilitating communication and
collaboration. However, PM typically requires a dark environment to achieve
high-quality projections, limiting its practicality. In this paper, we
overcome this limitation by replacing conventional room lighting with
heterogeneous projectors. These projectors replicate environmental lighting by
selectively illuminating the scene, excluding the projection target. Our
contributions include a distributed projector optimization framework designed
to effectively replicate environmental lighting and the incorporation of a
large-aperture projector, in addition to standard projectors, to reduce high-
luminance emitted rays and hard shadows—undesirable factors for collaborative
tasks in PM. We conducted a series of quantitative and qualitative
experiments, including user studies, to validate our approach. Our findings
demonstrate that our projector-based lighting system significantly enhances
the contrast and realism of PM results even under environmental lighting
compared to typical lights. Furthermore, our method facilitates a substantial
shift in the perceived color mode from the undesirable aperture-color mode,
where observers perceive the projected object as self-luminous, to the
surface-color mode in PM.

###### keywords:

Augmented reality, projection mapping, cooperative distributed projector
optimization, large-aperture projector.

The proposed projection mapping system mitigates the reduction in contrast
under environmental lighting. The left image provides an overall view of the
texture projection in a well-lit room using the proposed technique. The crack
pattern texture is projected onto the white statue located at the center of
the desk. The images on the right show enlarged views of the statue without
(left) and with (right) the texture projection. These images were captured in
three different lighting conditions: a dark room, where projection mapping is
typically conducted; under typical room lighting with LED lights; and under
the proposed environmental lighting using projectors. The results demonstrate
that our technique reproduces typical room lighting using multiple projectors
and enhances the contrast of the projected result on the target surface.

Introduction

Projection mapping (PM) is a technology that allows computer-generated images
to be superimposed on a target object in physical space using projectors.
Previous studies have investigated the utility of PM in a range of fields,
including, but not limited to, medicine [33], teleconferencing [37, 18, 39],
museum guides [41, 4], makeup [3, 45], object searches [38, 19, 29, 24, 20],
urban planning [54], and artwork creation [40, 12, 2]. However, there is a
strong and crucial constraint in PM—images need to be projected in dark
environments. Typical environment lighting globally elevates the scene
brightness. Therefore, if PM is performed under it, the dark areas of a
projected result inevitably get brighter, which causes significant contrast
degradation in the projected result. This constraint has become so ingrained
that it is unquestionably accepted as the norm. However, it significantly
limits the application fields of PM because PM in dark environments suffer
from various drawbacks.

First, a dark environment makes it difficult for users to see each other’s
faces. An important advantage of PM over other types of augmented reality (AR)
displays, such as optical see-through and video see-through systems, is that
PM allows multiple co-located users to view augmented objects without the need
for wearable or handheld devices, facilitating face-to-face discussions about
the objects. However, this advantage is significantly diminished by this
constraint. Secondly, while most PM applications, such as product design [28,
9, 30, 36, 50], are developed with the goal of visually altering surface
materials through projected imagery, the augmentations in these applications
appear physically and perceptually unnatural. Objects are only visible when
illuminated by light if they are not self-luminous. Consequently, it is
physically unnatural that only the target object in PM is visible in a dark
environment. In particular, PM results must appear completely dark when the
altered material is specular or transparent because surrounding environments
are reflected and refracted by the specular and transparent objects,
respectively. However, this aspect was not explicitly taken into account in
previous work [44, 34, 35]. In terms of the perceptual unnaturalness, despite
the absence of explicit investigation into this aspect, it is likely that
projected surfaces are perceived in the aperture-color mode (as self-luminous)
rather than in the surface-color mode (as illuminated). Our visual system
estimates the highest luminance achievable through reflected light under a
given illuminant, and an object appears self-luminous when its luminance
exceeds the estimated highest luminance [31]. If there is no environmental
lighting, the highest luminance should be estimated as zero. Consequently, any
projected results are theoretically perceived in the aperture-color mode, even
when the altered material is not inherently self-luminous.

In this study, we propose substituting multiple projectors for typical room
lights to selectively illuminate the scene, while excluding the projection
target, as a means of overcoming the dark environment constraint of PM.
Projector-based environmental lighting allows for local control of the
illuminance and chromaticity incident on scene surfaces, enabling us to avoid
undesirable global illuminance elevation on the target surface. Consequently,
we can mitigate the contrast reduction caused by typical environment lighting.
There are two primary contributions aimed at addressing technical challenges
unique to the proposed framework. First, we introduce a novel distributed
optimization method for multiple projectors that determines the projector
pixel values to accurately reproduce the illuminance and chromaticity incident
on surfaces under typical luminaires. Leveraging the spatially low-frequency
nature of environmental lighting, this technique computes pixel values more
efficiently than a conventional pixel-wise radiometric optimization framework
[13]. Furthermore, this method takes into account the inter-reflection of
projected light between adjacent surfaces. Inter-reflection significantly
increases the illuminance and can lead to inaccurate lighting reproduction
results without proper consideration. Secondly, we propose the use of a custom
projector with a much larger aperture than existing off-the-shelf projectors
to illuminate the environmental surfaces surrounding the projection target. As
a result, our multi-projection system employs heterogeneous projectors.
Typical luminaires function as area light sources, emitting diffuse light,
which results in low-luminance emitted rays and soft shadows when they are
occluded. In contrast, projectors function as point light sources and emit
directional light, resulting in high-luminance rays and hard shadows, which
could potentially disrupt users in completing their tasks using PM. To resolve
this inconsistency, we utilize the large-aperture projector, which can act as
an area light source, to illuminate scene surfaces around the projection
target, assuming that multiple observers frequently occlude the light from
this projector.

We validate the feasibility of our proposal by installing a prototype in a
room and conducting a series of quantitative and qualitative experiments.
First, we demonstrate that our technique can replicate various environmental
lighting effects, similar to typical luminaires, using multiple projectors.
Then, in a PM experiment, we illustrate how the system enhances the contrast
of projected results on a target surface, thus increasing the realism of PM.
Additionally, we show that the surface material of the target object is
naturally transformed into a polished mirror, reflecting the surrounding
environments lit by our PM-based lighting. Finally, we conduct user studies to
confirm that PM results presented by our system are perceived more in surface-
color mode compared to those under a typical dark room condition, and that our
large-aperture projector provides lighting that is more consistent with
typical lighting than a standard projector, thus offering better collaborative
PM environments.

Our primary contributions are as follows:

  * •

Introducing a novel environmental lighting framework that replaces typical
room lights with a combination of heterogeneous projectors, including multiple
standard projectors and a large-aperture projector, to address the dark
environment constraint in PM

  * •

Developing a distributed optimization method for the projectors to accurately
replicate the environmental lighting of typical luminaires

  * •

Employing a large-aperture projector as an area light source to illuminate the
surrounding environmental surfaces of the projection target

  * •

Demonstrating the feasibility of the proposed methods by installing a
prototype in a room

  * •

Confirming, through user studies, that observers perceive that the PM in our
system alters the reflectance properties of the target surface as if under
typical luminaires

##  1 Related Work

Researchers have worked on covering an entire room by PM, while turning off
the room lights. Jones et al., using multiple projectors, transformed an
ordinary room by projecting images onto all of its surfaces, creating a CAVE-
like immersive gaming space [22]. The same research group has also developed a
technology that converts an entire meeting room into an information display
using PM [11]. These studies aim to provide immersive experiences without the
need for dedicated immersive environments by seamlessly blending projected
images from different projectors onto various surfaces within an ordinary
environment. In contrast, our goal is to faithfully reproduce the illuminance
and chromaticity incident on the room surfaces under typical room lighting
using a multi-projection system. Therefore, while blending technology plays a
crucial role in our system, we also need to estimate the illuminance and
chromaticity of room lighting and generate projection images that accurately
replicate it. The PM-based illumination was also explored for occlusion-
capable optical see-through AR, realizing augmented imagery to appear bright
and opaque even in a lit room [6, 27]. They also focused on selectively
darkening a part of a scene, although they did not address the dark
environment constraint in PM. Specifically, accurate room lighting
reproduction was still an unsolved issue.

The estimation of illuminance and chromaticity for a given illuminant has been
extensively researched in the fields of computer vision and virtual production
[1, 25, 42, 15, 26]. Previous techniques employ multiple color patches with
known spectral reflectance properties and analyze these captured patches for
estimation purposes. The estimated results are subsequently employed to ensure
color constancy in various computer vision tasks and to relight physical
objects situated within a virtual production setup, such as a light stage.
While our task shares similarities with the latter scenario, we must
additionally account for inter-reflections among the room surfaces, which have
a more pronounced impact on the illuminance and chromaticity incident upon the
surfaces in our setup compared to a standard virtual production environment.

A well-studied technology related to illuminant reproduction in PM is
radiometric compensation, which is developed to display desired colors on
textured surfaces [5, 7, 14, 13, 17]. When we capture a scene under room
lighting and use the captured image as the target image, radiometric
compensation techniques can compute projector pixel colors that replicate the
scene in the projected result. However, most of the previous techniques do not
take inter-reflections into account. While some solutions address the inter-
reflection issue [55, 43], they are not scalable for systems with many
projectors since they require the computation of the inverse of a large matrix
(the light transport matrix), the dimensions of which increase linearly with
the number of projectors. In contrast, our approach proposes an illuminant
reproduction technique in a distributed manner, where no inversion computation
of a large matrix is required and input pixel values for each projector are
calculated without relying on information about those for other projectors.
Additionally, the radiometric compensation techniques are based on pixel-wise
computation. Because typical environmental lighting does not exhibit rapid
spatial changes in illuminance and chrominance, we compute identical pixel
values for a group of projector pixels—and, in some cases, for the entire set
of pixels in a projector—to enhance computational efficiency.

Shadows must be eliminated in PM. Researchers have extensively explored this
topic using multi-projection systems [32, 47, 21, 51] and specialized optics
[16]. However, achieving complete shadowlessness is not our primary goal, as
we aim to replicate typical luminaires or diffuse lighting, which naturally
results in soft shadows, using projectors. One solution is to implement light
field illumination, which can function as near-diffuse lighting, creating
pseudo soft shadows while allowing spatial control of illuminance and
chromaticity on scene surfaces [58, 49, 56, 48]. Nevertheless, light field
illumination requires time-consuming calibration to determine the position and
orientation of each ray in 3D space. In contrast, we propose a simpler yet
equally effective solution. Specifically, we employ a large-aperture projector
as a diffuse light source to create genuine soft shadows, illuminating the
area surrounding the PM target where observers’ shadows commonly occur.

##  2 Method

We distribute projectors under the ceiling of the room, among which a portion
of the projectors is used for performing PM onto a target object, while the
others are used to replicate the environmental lighting provided by the
original room lights. We refer to the former projectors as the _texture
projectors_ and the latter projectors as _luminaire projectors_. We establish
the correspondence between a point on the scene surfaces and the incident
projector pixel by projecting graycode patterns from each projector and
capturing them with a wide-angle camera. To mitigate the seams caused by
overlapping projected areas from different projectors, we employ a standard
feathering technique [8]. The remainder of this section elaborates on our two
primary contributions.

###  2.1 Distributed optimization for reproducing environmental lighting with
projectors

We compute pixel values for the luminaire projectors to accurately replicate
the illuminance and chromaticity of environmental lighting in our system,
while accounting for inter-reflections among the scene surfaces. We position
the projectors so that the projected areas of different projectors do not
significantly overlap, while also avoiding the occurrence of seams between
them. We position color charts on the scene surfaces, following a guideline of
placing one or two charts within each projector’s frustum. In cases with
significant variations in surface height on the floor or walls, we add
additional color charts as needed. We divide the projected area of each
luminaire projector into multiple segments, with each segment covering
approximately one of the color charts. We treat each luminaire projector as a
collection of multiple projector nodes, where the projected area of each node
corresponds to one of these segments. Each of the projector nodes for all the
luminaire projectors is labeled as $n$.

We input a single pixel value $x_{n}\in[0,1]$ into each projector node,
resulting in the projection of a uniformly colored image from the node. This
approach allows us to sparsely compute optimal pixel values, leveraging the
spatially low-frequency characteristics of environmental lighting. Compared to
conventional pixel-wise optimization frameworks [55, 43], our low-frequency
approach not only reduces computational costs but also mitigates high-
frequency artifacts caused by noise in the capturing process. To ensure
scalability with respect to the number of luminaire projectors, we have
developed a distributed optimization framework.

Figure 1: Obtaining attenuation factor $p_{1}^{i_{m}}$ by projecting a
uniformly white image from projector node 1, while other nodes project
uniformly black image. The projected area of a physical projector, depicted by
dashed lines, is divided into two segments. Projector nodes 1 and 2 represent
virtual projectors, and their projected areas correspond to the respective
segments.

A color chart comprises multiple patches of varying colors and gray levels.
Suppose the reflected RGB color of a patch $i$ in a chart $m$ under the
environmental lighting to be reproduced is denoted as $r^{i_{m},c}\in[0,1]$,
where $c$ represents the color channel (i.e., R, G, or B). Our goal is to
reproduce $r^{i_{m},c}$ using the luminaire projectors. In the following
explanation, we omit $c$ for simplicity since the optimization process is
color channel-independent. After deactivating the environmental lighting, we
measure a black offset $d^{i_{m}}$ at each patch by capturing its color under
projection of a uniformly black image from all the projectors (i.e., $x_{n}=0$
for all $n$). Next, we project a uniformly white image from a projector node
$n$ (i.e., $x_{n}=1$), while projecting a uniformly black image from the other
projector nodes (i.e., $x_{n^{\prime}}=0$ for $n^{\prime}\neq n$). After
capturing the reflected color of each patch $i_{m}$, we subtract the black
offset $d^{i_{m}}$ from the captured value, obtaining $p_{n}^{i_{m}}\in[0,1]$,
which is an attenuation factor representing the proportion of light from a
projector node $n$ that is reflected at a color patch $i_{m}$ (Figure 1).
Therefore, in cases where the projected area of the projector node $n$ does
not cover the color chart $m$, $p_{n}^{i_{m}}$ represents the effect of inter-
reflection. We repeat this process by changing the projector node that
projects a uniformly white image.

The captured color $y^{i_{m}}$ of each patch under arbitrary illumination by
the luminaire projectors can be represented as follows:

$y^{i_{m}}=\sum_{n}p_{n}^{i_{m}}x_{n}+d^{i_{m}}.$ (1)

To accurately reproduce the environmental lighting, we minimize the objective
function

$G=\frac{1}{2}\sum_{m}\sum_{i_{m}}\\{e^{i_{m}}\\}^{2},$ (2)

which is the sum of the squared error

$e^{i_{m}}=r^{i_{m}}-y^{i_{m}}=r^{i_{m}}-\sum_{n}p_{n}^{i_{m}}x_{n}-d^{i_{m}}.$
(3)

The gradient of Equation 2 is given by

$\frac{\partial G}{\partial
x_{n}}=-\sum_{m}\sum_{i_{m}}p_{n}^{i_{m}}e^{i_{m}}.$ (4)

Therefore, from the difference between the target color and the captured color
at a color patch $i_{m}$ (i.e., $e^{i_{m}}$), we can compute the gradient for
each projector node without relying on information about the other projector
nodes. We update the input pixel values for projector nodes as follows:

$x_{n}[t+1]=\mathcal{P}\left[\frac{1}{K}\sum_{i_{m}}\left\\{x_{n}[t]-\epsilon\frac{\partial
G}{\partial x_{n}}[t]\right\\}\right],$ (5)
$\mathcal{P}[\bar{x}]:=\begin{cases}{\bar{x},\quad 0\leq\bar{x}\leq 1,}\\\
{1,\quad\bar{x}>1,\quad\quad\quad\bar{x}\in\mathbb{R}}\\\
{0,\quad\bar{x}<0,}\end{cases}$ (6)

where $t$ represents the frame number (or time), and $K$ is the number of
color patches in each color chart. This algorithm is equivalent to a projected
gradient method used to minimize $G$, where $\epsilon$ functions as a step
size. It can be proven that this update rule, with sufficiently small
$\epsilon$, causes $x_{n}$ to converge to the optimal value in a suitable
sense [51, 46]. As described above, we can compute pixel values independently
for each projector. This is a completely distributed process, and furthermore,
it is theoretically guaranteed that the addition or removal of projector nodes
at runtime does not affect the convergence performance.

###  2.2 Large-aperture projector for spatially controlling diffuse lighting
on scene surfaces

Figure 2: Aperture size of a projector affects the emitted ray intensity and
the appearance of a shadow. (a) A standard projector is a point light source
producing a high-luminance ray and a hard shadow. (b) The proposed large-
aperture projector is an area light source creating a low-luminance ray and a
soft shadow. Figure 3: An issue and its solution with the large-aperture
projector illuminating the surrounding environmental surfaces of the PM
target. (a) Some light rays intended to converge on an environmental surface
point near the target unintentionally hit the target surface. (b) Turning off
these pixels and compensating for the darkened area with the texture
projectors.

Illumination lights incident on the scene surfaces near the PM target tend to
be occluded by observers, thus requiring special attention. When we use a
standard projector to illuminate this area, occlusions result in hard shadows
(Figure 2(a)), whereas shadows under typical room lighting are soft. This
inconsistency arises from the difference between a point light source that
produces directional light (projector) and an area light source that generates
diffuse light (typical room light). To bridge this gap, we propose employing a
large-aperture projector as the luminaire for the surfaces near the PM target.
The large-aperture projector comprises a large-format lens and a flat display
plane (Figure 2(b)). When the lens size is sufficiently large, an observer’s
body, such as a hand or a head, cannot fully occlude all the light from the
lens, resulting in soft shadows. The large-aperture projector offers another
advantage compared to a standard projector. Assuming the illuminance of
incident light on a scene surface is identical for both projectors, the
luminance of each ray emitted from the lens is significantly lower in the
large-aperture projector because the light energy is distributed across the
lens. Therefore, the large-aperture projector provides a more comfortable
environment where users experience much less dazzling sensation when they see
the projector’s lens than under standard projector illumination.

Figure 4: Experimental system. (a) A close-up view of our large-aperture
projector. (b) An overview where standard projectors project uniformly colored
images.

A straightforward method to illuminate the scene surfaces using the large-
aperture projector while avoiding the PM target is to turn off the pixels that
focus on the target. However, some light rays emitted from other pixels may
still hit the target, as illustrated in Figure 3(a). This significantly
degrades the contrast of the projected results on the target. We address this
issue by simply turning off a larger area of pixels on the display plane of
the large-aperture projector, which includes all the pixels emitting light
rays that hit the target. While this solution mitigates contrast degradation,
it may cause some of the scene surfaces around the PM target to darken. To
compensate for the reduction in illuminance, we utilize the texture projectors
to function as luminaire projectors for the darkened area of the scene
surfaces (Figure 3(b)).

##  3 Experiment

We conducted both quantitative and qualitative experiments including user
studies to validate the proposed method.

###  3.1 Experimental system

Figure 4 shows our experimental system. We arranged a sofa, desk and other
furniture in our lab space measuring 2800$\times$2200 mm to recreate a typical
living room. Additionally, we placed small objects on the desk and bookshelf,
including a white bust statue of 200 mm height on the desk to serve a
projection target. We measured the reflectance of the object surface using a
spectroradiometer (Topcon SR-LEDW) and a reflection standard white ceramics
plate (Evers Corporation EVER-WHITE No. 9582), then averaged the measured
reflectance values within the visible spectral range. The resulting average
value was 0.56. Our method is effective for other objects with unknown
reflectance properties, provided they have a diffuse surface. In such cases,
radiometric compensation techniques [13, 17] are applied to reproduce the
desired appearances. For capturing the scene, we used a camera (Canon EOS M6
Mark II) equipped with a fisheye lens (Canon EF8-15 mm f/4L Fisheye USM)
positioned at the edge of the room.

Figure 5: The experimental results of the environmental lighting reproduction.
(a) Projecting a uniformly white image from each projector node (indicated by
the overlaid number) for obtaining $p_{n}^{i_{m}}$. (b) Captured images under
three different types of environmental lighting; typical LED lights (target
appearance), the reproduced lighting with projectors using a conventional
method [5], and that using our method. (c) The appearance of the color chart
under the three lighting conditions.

We employed six projectors as luminaire projectors, comprising five standard
projectors and one custom-made large-aperture projector. Among the standard
projectors, two (Optoma ML1050ST+S1J) were directed at the side walls, one
(Acer H6517ST) was directed at the front wall, and two (RICOH PJ WXC1110) were
directed at the floor. The large-aperture projector, comprising an LCD display
(OSEE T7) and a 500$\times$500 mm fresnel lens (SIGMAKOKI FRLN-500S-250P),
provided illumination for the desk. We designated a standard projector (BenQ
TK850) as the texture projector for projecting textures onto the target
surface. The projectors and the camera were controlled by a PC (CPU: AMD Ryzen
9 3900 X 12-Core Processor, RAM: 64 GB). For typical environmental lighting
sources, we utilized LED-based luminaires (NEEWER NL660 Bi-Color LED Panel
Light) with adjustable luminance and color temperature settings. All of these
projectors and LED lights were mounted on the ceiling. As the step size of our
method, $\epsilon=0.00001$ was used in all the experiments.

###  3.2 Reproduction of environmental lighting

We validated our method of reproducing typical environmental lighting using
projectors, explained in subsection 2.1. We positioned a total of ten color
charts on the front and side walls, as well as the floor. Then, we captured
the reflected RGB color of each patch under the LED lighting and stored it as
$r^{i_{m}}$. Next, after turning off the LED lights, we measured the black
offset $d^{i_{m}}$ at each patch. We then measured the attenuation factor
$p_{n}^{i_{m}}$ by projecting a white image from each projector node and
capturing the reflected color at each color patch as shown in Figure 5(a).

The initial input pixel values in the iteration process of our method (i.e.,
$x_{n}[1]$) were computed using a conventional radiometric compensation
technique [5]. This technique relies on the same linear projector-camera
reflection model as Equation 1, but it does not account for inter-reflection.
Specifically, assuming the projected area of the $n$-th projector node covers
the color chart $m^{\prime}$, the conventional technique reformulates Equation
1 as $y^{i_{m^{\prime}}}=p^{i_{m^{\prime}}}_{n}x_{n}+d^{i_{m^{\prime}}}$.
Then, it solves this equation for $x_{n}$, given the target appearance
$r^{i_{m^{\prime}}}$, yielding
$x_{n}=(r^{i_{m^{\prime}}}-d^{i_{m^{\prime}}})/p^{i_{m^{\prime}}}_{n}$, which
is used as the initial input value in our method.

Figure 6: Time series of data in the proposed distributed optimization
technique over 25 iterations. (a) The averaged error observed in each color
chart. (b) The input pixel values of each color channel for each projector
node (b1: R, b2: G, b3: B).

Figure 5(b) shows the appearance of our room under three different lighting
conditions: the LED lights (representing the target appearance), the luminaire
projectors using the conventional method [5], and the luminaire projectors
using the proposed method after $t=20$ iterations. It took 9 seconds to
compute the projection images, project them, and capture the projected results
in each iteration. Therefore, the entire process of 20 iterations requires
about 180 seconds. When focusing on the computation time alone, it took about
18 seconds for 20 iterations. Figure 5(c) showcases the captured colors of the
color charts under these conditions. These results underscore the ability of
our method to more accurately replicate the target environmental lighting
compared to the conventional method, particularly emphasizing the significant
brightness elevation in the conventional method, which does not account for
inter-reflection. Figure 6 shows the transition of the output of our objective
function (Equation 2) through the iteration. The figure also includes the
transition of the input pixel values for the projector nodes in all three
color channels. These results demonstrates the convergence of our method in a
distributed cooperative manner within around 20 iterations.

Figure 7: Captured scene appearances (left) under four different environmental
lighting conditions with varying light positions and color temperatures, and
(right) those under our projector-based environmental lighting.

We conducted a qualitative evaluation of our technique by preparing different
target environmental lighting conditions. Specifically, we altered the
position and color temperature of the LED light, resulting in four distinct
room appearances. Figure 7 shows these new target appearances, alongside the
replication results achieved using our method with the luminaire projectors.
We can confirm that the target appearances were accurately reproduced at a
level where we can easily discern how the position and color temperature of
the original LED light are altered.

###  3.3 Texture manipulation of the projection target

Figure 8: PM results of various texture patterns onto a white bust statue
under three different lighting conditions; dark room, typical LED lighting,
and our method.

By conducting PM in the prototype, we verified the ability of our method in
alleviating the contrast degradation of PM under environmental lighting.
Projection Mapping under Environmental Lighting by Replacing Room Lights with
Heterogeneous Projectors shows the white bust statue positioned on the desk
before and during PM of a crack pattern under three lighting conditions: dark
room, typical environmental lighting by LED lights, and the proposed
replicated environmental lighting. The images captured before PM, as shown in
Projection Mapping under Environmental Lighting by Replacing Room Lights with
Heterogeneous Projectors(b), demonstrate that our method can selectively
remove environmental light from the target object, while typical lighting
significantly increases the brightness of the target. The PM results shown in
Projection Mapping under Environmental Lighting by Replacing Room Lights with
Heterogeneous Projectors(c) demonstrate that our method can manipulate the
target appearance under environmental lighting while minimizing the intensity
elevation in dark areas of the projected texture compared to the typical
lighting. A similar effect of alleviating the contrast degradation is also
exhibited in the PM results of other textures in Figure 8.

Projection Mapping under Environmental Lighting by Replacing Room Lights with
Heterogeneous Projectors and Figure 8 also include the root mean square (RMS)
contrast values of the PM results. The dark room condition yielded the highest
contrast among all PM results, with the typical environmental lighting
producing the lowest contrast. We quantitatively confirmed that the proposed
method yielded a higher level of contrast compared to typical lighting. We
also measured ANSI contrast by replacing the bust statue with a planar surface
(200$\times$200 mm) on the desk and projecting a 4$\times$4 checker pattern
onto the plane using the texture projector. We measured the luminance values
of each checker area using the spectroradiometer. The ANSI contrast values
were 180:1, 30:1, and 65:1 in the dark room condition, the typical
environmental lighting condition, and the proposed replicated environmental
lighting condition, respectively. We confirmed that the proposed method
produced a contrast ratio more than two times higher than that obtained under
the typical lighting condition.

We altered the surface material of the target object to a mirror-like specular
material using PM under the same three lighting conditions. To simulate the
glossy appearance of the object in the real environment from an observer’s
viewpoint, we generated the projection image using a conventional 2-pass
rendering technique [8] as follows. As preparation, we measured the color and
3D shape of the experimental room using an RGB-D camera (Apple iPad Pro 5th
gen), calibrated the projector’s intrinsic and extrinsic parameters, and
measured the position of the observer’s eyes (or the camera capturing the
experimental result) in the room in advance. First, employing a game engine,
specifically Unity, we reproduced the experimental room in a virtual space and
placed the 3D model of the bust statue in the same location as the physical
setup. Then, we rendered the bust statue with a specular material and captured
it with a virtual camera placed at the observer’s location, constituting the
first rendering pass. The captured image was then mapped onto the target
object from the virtual camera position using the projective texture mapping
technique. In the second rendering pass, the mapped object was captured by
another virtual camera with intrinsics and extrinsics identical to the
projector’s. Finally, the captured image was used as the projection image for
the physical space.

Figure 11 displays the PM results. In the dark room condition, the PM result
exhibits the highest contrast. However perceiving its surface material as
specular is challenging because it is not physically correct for specular
surfaces to appear visible when the surrounding environment is not visible.
The PM result in the typical environmental lighting condition shows the lowest
contrast due to the increased black offset caused by the lighting, which
degrades the realism of the projected material. In contrast, the PM result in
the proposed method exhibits higher contrast than the typical lighting
condition, and the surrounding environment remains visible. As a result, the
projected bust statue can be perceived as having a specular material to the
greatest extent under the proposed condition.

###  3.4 User study 1: Perceived color mode of projected results

We conducted a user study to investigate how much the proposed method affects
the observer’s perceived color mode. In vision science, it has been well known
that there are several perceptual modes of color appearance, with surface-
color and aperture-color modes being the most common in our everyday lives
[23, 53]. These modes are determined by the luminance relationship between an
observed object and its surroundings [52]. The proposed technique replicates
the environmental lighting, resulting in bright surroundings. Therefore,
assuming observers perceive PM results in a dark room in aperture-color mode,
the hypothesis in this study is:

H1

The perceived color mode for PM results is shifted toward the surface-color
mode by the proposed environmental lighting.

We showed projected results to each participant in two lighting conditions,
the dark room condition and the proposed replicated environmental lighting
condition, and asked them to rate the perceived mode of its appearance. We
excluded the typical environmental lighting condition in this study because
the appearance of a PM result is significantly affected by the room lights,
and at times, the details of the projected content become invisible. We used
the bust statue as the projection target and presented the PM results of four
projection images, including the crack pattern (Projection Mapping under
Environmental Lighting by Replacing Room Lights with Heterogeneous
Projectors), as well as the camouflage pattern, radial pattern, and the tiled
pattern as shown in Figure 8. Thus, each participant completed eight trials in
total (2 lighting conditions$\times$4 projection images). In each trial, the
participants sat 1.5 m away from the projected object and adapted to the
surround environment for 5 min. After that, they looked at the projected
object and answered a five-point scale question about the perceived mode (1:
aperture-color mode, $\ldots$ , 5: surface-color mode). There was no time
limit for the participant to look at each stimulus before deciding the
response. The study was approved by the Research Ethics Review Committee of
the institute to which the corresponding author belongs (approval number;
R5-6).

Figure 9: Summary of user study results: (a) the first study described in
subsection 3.4, and (b) the second study described in subsection 3.5.

Twenty-one participants (20 males and 1 female, aged 21 to 27 years old,
university students) volunteered for the study. All participants were naïve to
the purpose of the experiment, had corrected visual acuity, and gave informed
consent. Before the experiment, we instructed participants that the aperture-
color mode and surface-color mode respectively correspond to the qualities of
color appearance for a self-luminous object and an illuminated surface,
respectively. All participants agreed that they understood the concept of
these color modes. The order of experimental conditions was randomized and
balanced across participants. Figure 9(a) summarizes the participants’
responses. The median scores were shifted towards the surface color modes by
the proposed environmental lighting for all the projected patterns. A Wilcoxon
signed rank test showed that there was a significant difference (p$<$0.01)
between scores given for the proposed replicated environmental lighting
compared to the dark room condition when the crack pattern was projected. On
the other hand, the test did not reveal any significant differences for the
other projected patterns (p$\geq$0.05). Therefore, the hypothesis of H1 was
partially supported. The results indicate that our environmental lighting
replication method provides a better PM environment for surface material
editing with reduced self-luminous appearance, making it a more natural choice
than a typical dark room. We will discuss the results in more details in
section 4.

###  3.5 User study 2: Subjective evaluation of large-aperture projector

Figure 10: Comparison of shadows created by a standard projector and our
large-aperture projector. Note that the eyes are intentionally blurred to
protect the individual’s identity. Figure 11: Visually altering the surface
material of the diffuse statue to a mirror-like specular appearance in three
lighting conditions: (left) in a dark room, (center) under typical room
lighting, and (right) under the proposed projector-based environmental
lighting.

In our second user study, we evaluated our large-aperture projector as a
luminaire for illuminating the surrounding environmental surfaces of the
projection target, comparing it to a standard off-the-shelf projector. We
focused on the following two issues: First, the illumination produced by the
standard projector appears to result from directional light, which is
inconsistent with common diffuse environmental lighting. In contrast, the
illumination produced by the proposed large-aperture projector seems to reduce
the perception of directional light. Secondly, the standard projector,
considered a point light source, emits high-luminance rays and creates hard
shadows, while the proposed projector, seen as an area light source, generates
much lower-luminance rays and softer shadows. Hard shadows appear, disappear,
and change shape significantly even with slight movements of an occluder. In
contrast, soft shadows are more temporally stable. Figure 10 compares the
shadows created by a standard projector with those created by the proposed
large-aperture projector. Regarding the emitted rays, observers may experience
a dazzling sensation only when the standard projector is in use and they
possibly see the projector lens. Consequently, the illumination produced by
the standard projector has the potential to disturb users’ tasks in PM due to
the presence of hard shadows and high-luminance emitted rays, in contrast to
the proposed large-aperture projector. To investigate these aspects, this
study specifically set the task as observers discussing the surface material
of a PM target with each other and examined the following hypotheses.

H2

The proposed projector mitigates the impression of directional light compared
to a standard projector.

H3

The proposed projector alleviates the optical disturbance in communication
compared to a standard projector.

While one might think that H2 is evident because the impression of directional
light primarily depends on how much an observer pays attention to the shadows,
we believe H2 addresses an important aspect. That is because even a small bump
causes a hard shadow in a standard projector condition. Given that such small
bumps are common on real-world surfaces, users would unconsciously notice them
without explicitly focusing on the shadows.

Each participant and an experimenter sat by the projection target and engaged
in a conversation about the material impression of the PM results, while the
experimenter changed the projected textures among those introduced in this
paper so far including the mirror-like specular texture according to the
conversation. Note that they did not discuss the environmental lighting. The
conversation lasted for about 5 minutes with the first 2.5 minutes and the
rest conducted under different projector conditions. The participant was
allowed to use their hands in the conversation, and the experimenter
frequently pointed to the projection object to create shadows. Afterward, the
participant was asked two questions and answered on a five-point scale
questionnaire (1: agree, $\ldots$ , 5: disagree). The questions were: (1) Was
the environmental lighting directional? and (2) Did the environmental lighting
and shadows disturb the communication? The study was approved by the Research
Ethics Review Committee of the institute to which the corresponding author
belongs (approval number; R5-6).

The same participants from user study 1 volunteered for this study. All
participants were naïve to the purpose of the experiment. The order of
experimental conditions was randomized and balanced across participants.
Figure 9(b) summarizes the participants’ responses. A Wilcoxon signed-rank
test showed significant differences (p$<$0.01) between the scores given for
the proposed projector condition compared to the standard projector condition
in both questions. Specifically, the environmental lighting provided by the
standard projector was perceived as significantly more directional than that
provided by the proposed large-aperture projector. Additionally, the
participants judged that the illumination by the standard projector disturbed
the communication significantly more than that by the proposed projector.
Therefore, the hypotheses of H2 and H3 were both supported. We confirm that
the proposed large-aperture projector provides more consistent illumination
with typical environmental lighting and offers a better environment for
communication about the projected objects and potentially for other tasks.

##  4 Discussion

Through a series of quantitative and qualitative experiments, we demonstrated
the effectiveness of our method for achieving PM in a well-lit room. The
experiment in subsection 3.2 showed that typical environmental lighting is
reproducible to the extent that we can recognize the locations and color
temperatures of the original light sources. The remaining discrepancies
between the target and reproduced appearances can be mitigated by placing the
color charts more densely in the scene. This adjustment does not require any
modifications to our framework but increases the computational cost. The
accuracy of the reproduction depends on the application, and we believe that
there are not many applications requiring very accurate reproduction because
PM users typically do not focus on the environment.

The experiment in subsection 3.3 demonstrated that our replicated
environmental lighting mitigates contrast degradation in PM results compared
to typical environmental lighting. As shown in Figure 11, our framework
effectively reproduces the reflection of light from a scene surface onto the
PM target in a physically accurate manner, by which the realism of the
specular reflection was significantly improved. We can naturally extend this
framework to replicate bidirectional light field interactions between the PM
target and its surroundings. This study represents a promising first step
toward achieving _perceptual realism_ in the context of PM, defined as the
production of imagery indistinguishable from real-world 3D scenes [57].

The first user study in subsection 3.4 confirmed a tendency for observers to
perceive the PM results in surface-color mode when using the proposed
environmental lighting. However, the analysis did not show this tendency to be
statistically significant in three out of four projected textures. A possible
explanation for this result is as follows: Previous psychophysical experiments
used visual stimuli consisting solely of color patches or spatial patterns
without any contextual information [52, 53, 31]. In contrast, our stimuli
provided ample contextual information for participants to identify the object
as a bust statue and understand the meaning of the projected images.
Additionally, all participants knew that the appearance of the target object
was altered in PM, suggesting that they might have implicitly understood that
the object reflected light from the projector rather than being self-luminous.
These factors may have biased their judgments. On the other hand, it is worth
mentioning that even under such conditions, there was a case where the
proposed method caused a significant shift from aperture-color mode to
surface-color mode. To the best of our knowledge, we are the first to focus on
the perceived color mode in PM. We believe that our research is essential
because we investigated the color mode in real PM scenarios and demonstrated
that environmental lighting influences the perceived color mode, albeit with
the degree of influence varying depending on the projected contents.

Through the second user study (subsection 3.5), we confirmed that selecting a
large-aperture projector as the luminaire for illuminating the surrounding
environmental surfaces of the projection target is crucial to creating a
lighting environment familiar to users and minimizing disruptions during their
tasks. Specifically, the large-aperture projector significantly reduced the
impression of directional light which is uncommon in typical environmental
lighting, and substantially decreased bothersome high-luminance emitted rays
and hard shadows during communication compared to a standard projector.
Additionally, we found that the standard projector produces intense
(potentially dazzling) specular reflections, whereas the proposed projector
generates much weaker specular reflections. Therefore, an important design
guideline that we have learned from this result, when considering the
replacement of room lights with projectors, is the necessity of incorporating
a projection system that, alongside standard projectors, can function as an
area light source to illuminate areas in proximity to the projection target.
Such projection systems include multi-projection systems, light field
projectors, and large-aperture projectors. An interesting avenue for future
work would be to compare the performance of these different types of area
light projectors in this context. On the other hand, we applied a standard
projector to the texture projector in the current setup, resulting in a hard
shadow on the PM target when a user approached it. We consider applying area
light projectors as texture projectors to be another important area for future
work.

Although our evaluated setup mimics a living room, it is primarily designed
for special cases of PM. The camera used for calibration must be stable in the
room and observe example lighting before installing the projectors. Here, we
explain why we chose a living room for PM. We envision a future where more
projectors replace standard luminaires in ordinary rooms, and PM supports our
daily activities. With this vision, we selected a living room as a testbed to
address the typical dark environment constraint, which becomes more
significant when PM is applied in ordinary rooms. While the necessity of a
stably fixed camera for distributed projector optimization is a current
limitation, we could potentially relax this limitation by representing the
entire PM system using a neural network [10]. Once the neural representation
is established by casually capturing the projected scene using a hand-held
camera, we can optimize the projected images to reproduce various lighting
conditions without physically projecting calibration images during the
optimization process. Another limitation is the necessity of capturing example
lighting. We believe that an interesting avenue for future work would involve
using professional illumination simulation software applied in the
illumination design industry, such as DIALux111https://www.dialux.com/, for
synthesizing example lighting rather than capturing it.

##  5 Conclusion

In this paper, we discussed and validated a technical solution for achieving
PM in environmental lighting. As an initial step toward this intriguing goal,
we attempted to replace room lights with projectors to selectively illuminate
the scene, excluding the projection target. We proposed two technical
contributions in this paper: a distributed projector optimization framework
designed to replicate environmental lighting and the application of a large-
aperture projector as an area light source to illuminate areas near the
projection target. Thanks to these contributions, we achieved two significant
outcomes: (1) accurate replication of environmental lighting using projectors
and (2) the mitigation of disturbances to users’ tasks in PM by reducing
undesirable high-luminance emitted rays and hard shadows that typically occur
with standard projectors. A PM experiment demonstrated that our projector-
based luminaire could enhance the contrast in PM results under environmental
lighting compared to typical lights, thereby increasing the realism of
projected results. Additionally, we confirmed that it allows for a shift in
perceived color mode from the undesirable aperture-color mode, where observers
perceive the projected object as self-luminous, to the surface-color mode in
PM. As part of our future work, we plan to extend this system to support
dynamic PM.

###### Acknowledgements.

This work was supported by JSPS KAKENHI grant number JP20H05958, JST PRESTO
Grant Number JPMJPR19J2, and the Future Social Value Co-Creation Project,
Osaka University.

## References

  * [1] A. Abdelhamed, A. Punnappurath, and M. S. Brown.  Leveraging the availability of two cameras for illuminant estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6637–6646, June 2021.
  * [2] D. Bandyopadhyay, R. Raskar, and H. Fuchs.  Dynamic shader lamps : painting on movable objects.  In Proceedings IEEE and ACM International Symposium on Augmented Reality, pp. 207–216, 2001.
  * [3] A. H. Bermano, M. Billeter, D. Iwai, and A. Grundhöfer.  Makeup lamps: Live augmentation of human faces via projection.  Computer Graphics Forum, 36(2):311–323, 2017.
  * [4] O. Bimber, F. Coriand, A. Kleppe, E. Bruns, S. Zollmann, and T. Langlotz.  Superimposing pictorial artwork with projected imagery.  IEEE MultiMedia, 12(1):16–26, 2005.
  * [5] O. Bimber, A. Emmerling, and T. Klemmer.  Embedded entertainment with smart projectors.  Computer, 38(1):48–55, 2005. doi: 10 . 1109/MC . 2005 . 17
  * [6] O. Bimber and B. Frohlich.  Occlusion shadows: using projected light to generate realistic occlusion effects for view-dependent optical see-through displays.  In Proceedings. International Symposium on Mixed and Augmented Reality, pp. 186–319, Oct 2002. doi: 10 . 1109/ISMAR . 2002 . 1115088
  * [7] O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhöfer.  The visual computing of projector-camera systems.  Computer Graphics Forum, 27(8):2219–2245, 2008. doi: 10 . 1111/j . 1467-8659 . 2008 . 01175 . x
  * [8] O. Bimber and R. Raskar.  Spatial Augmented Reality: Merging Real and Virtual Worlds.  A K Peters, Ltd., 2005.
  * [9] G. Cascini, J. O’Hare, E. Dekoninck, N. Becattini, J.-F. Boujut, F. Ben Guefrache, I. Carli, G. Caruso, L. Giunta, and F. Morosi.  Exploring the use of ar technology for co-creative product and packaging design.  Computers in Industry, 123:103308, 2020.
  * [10] Y. Erel, D. Iwai, and A. H. Bermano.  Neural projection mapping using reflectance fields.  IEEE Transactions on Visualization and Computer Graphics, 29(11):4339–4349, Nov 2023. doi: 10 . 1109/TVCG . 2023 . 3320256
  * [11] A. R. Fender, H. Benko, and A. Wilson.  Meetalive: Room-scale omni-directional display system for multi-user content and control sharing.  In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, ISS ’17, p. 106–115. Association for Computing Machinery, New York, NY, USA, 2017. doi: 10 . 1145/3132272 . 3134117
  * [12] M. Flagg and J. M. Rehg.  Projector-guided painting.  In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, p. 235–244, 2006.
  * [13] A. Grundhöfer and D. Iwai.  Robust, error-tolerant photometric projector compensation.  IEEE Transactions on Image Processing, 24(12):5086–5099, 2015. doi: 10 . 1109/TIP . 2015 . 2478388
  * [14] A. Grundhöfer and D. Iwai.  Recent advances in projection mapping algorithms, hardware and applications.  Computer Graphics Forum, 37(2):653–675, 2018. doi: 10 . 1111/cgf . 13387
  * [15] G. C. Guarnera, Y. Gitlina, V. Deschaintre, and A. Ghosh.  Spectral Upsampling Approaches for RGB Illumination.  In A. Ghosh and L.-Y. Wei, eds., Eurographics Symposium on Rendering. The Eurographics Association, 2022. doi: 10 . 2312/sr . 20221150
  * [16] K. Hiratani, D. Iwai, Y. Kageyama, P. Punpongsanon, T. Hiraki, and K. Sato.  Shadowless projection mapping using retrotransmissive optics.  IEEE Transactions on Visualization and Computer Graphics, 29(5):2280–2290, 2023. doi: 10 . 1109/TVCG . 2023 . 3247104
  * [17] B. Huang, T. Sun, and H. Ling.  End-to-end full projector compensation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2953–2967, 2022. doi: 10 . 1109/TPAMI . 2021 . 3050124
  * [18] D. Iwai, R. Matsukage, S. Aoyama, T. Kikukawa, and K. Sato.  Geometrically consistent projection-based tabletop sharing for remote collaboration.  IEEE Access, 6:6293–6302, 2018.
  * [19] D. Iwai and K. Sato.  Limpid desk: See-through access to disorderly desktop in projection-based mixed reality.  In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST ’06, p. 112–115. Association for Computing Machinery, New York, NY, USA, 2006. doi: 10 . 1145/1180495 . 1180519
  * [20] D. Iwai and K. Sato.  Document search support by making physical documents transparent in projection-based mixed reality.  Virtual Reality, 15(2):147–160, Jun 2011. doi: 10 . 1007/s10055-010-0159-5
  * [21] C. Jaynes, S. Webb, and R. Steele.  Camera-based detection and removal of shadows from interactive multiprojector displays.  IEEE Transactions on Visualization and Computer Graphics, 10(3):290–301, 2004.
  * [22] B. Jones, R. Sodhi, M. Murdock, R. Mehra, H. Benko, A. Wilson, E. Ofek, B. MacIntyre, N. Raghuvanshi, and L. Shapira.  Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units.  In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, p. 637–644. Association for Computing Machinery, New York, NY, USA, 2014. doi: 10 . 1145/2642918 . 2647383
  * [23] D. Katz.  The World of Colour.  London: Kegan Paul, Trench, Trubner & Co., Ltd., 1935.
  * [24] Y. Kitajima, D. Iwai, and K. Sato.  Simultaneous projection and positioning of laser projector pixels.  IEEE Transactions on Visualization and Computer Graphics, 23(11):2419–2429, 2017.
  * [25] C. LeGendre, L. Lepicovsky, and P. Debevec.  Jointly optimizing color rendition and in-camera backgrounds in an rgb virtual production stage.  In The Digital Production Symposium, DigiPro ’22. Association for Computing Machinery, New York, NY, USA, 2022. doi: 10 . 1145/3543664 . 3543681
  * [26] C. LeGendre, X. Yu, D. Liu, J. Busch, A. Jones, S. Pattanaik, and P. Debevec.  Practical multispectral lighting reproduction.  ACM Trans. Graph., 35(4), jul 2016. doi: 10 . 1145/2897824 . 2925934
  * [27] A. Maimone, X. Yang, N. Dierk, A. State, M. Dou, and H. Fuchs.  General-purpose telepresence with head-worn optical see-through displays and projector-based lighting.  In 2013 IEEE Virtual Reality (VR), pp. 23–26, March 2013. doi: 10 . 1109/VR . 2013 . 6549352
  * [28] M. R. Marner, R. T. Smith, J. A. Walsh, and B. H. Thomas.  Spatial user interfaces for large-scale projector-based augmented reality.  IEEE Computer Graphics and Applications, 34(6):74–82, 2014.
  * [29] K. Matsushita, D. Iwai, and K. Sato.  Interactive bookshelf surface for in situ book searching and storing support.  In Proceedings of the 2nd Augmented Human International Conference, 2011.
  * [30] C. Menk and R. Koch.  Truthful color reproduction in spatial augmented reality applications.  IEEE Transactions on Visualization and Computer Graphics, 19(2):236–248, 2013. doi: 10 . 1109/TVCG . 2012 . 146
  * [31] T. Morimoto, A. Numata, K. Fukuda, and K. Uchikawa.  Luminosity thresholds of colored surfaces are determined by their upper-limit luminances empirically internalized in the visual system.  Journal of Vision, 21(13):3–3, 12 2021. doi: 10 . 1167/jov . 21 . 13 . 3
  * [32] M. Nagase, D. Iwai, and K. Sato.  Dynamic defocus and occlusion compensation of projected imagery by model-based optimal projector selection in multi-projection environment.  Virtual Reality, 15(2):119–132, Jun 2011.
  * [33] H. Nishino, E. Hatano, S. Seo, T. Nitta, T. Saito, M. Nakamura, K. Hattori, M. Takatani, H. Fuji, K. Taura, and S. Uemoto.  Real-time navigation for liver surgery using projection mapping with indocyanine green fluorescence: Development of the novel medical imaging projection system.  Annals of Surgery, 267(6):1134–1140, 2018.
  * [34] T. Nomoto, R. Koishihara, and Y. Watanabe.  Realistic dynamic projection mapping using real-time ray tracing.  In ACM SIGGRAPH 2020 Emerging Technologies, SIGGRAPH ’20. Association for Computing Machinery, New York, NY, USA, 2020. doi: 10 . 1145/3388534 . 3407297
  * [35] T. Nomoto, W. Li, H.-L. Peng, and Y. Watanabe.  Dynamic multi-projection mapping based on parallel intensity control.  IEEE Transactions on Visualization and Computer Graphics, 28(5):2125–2134, 2022. doi: 10 . 1109/TVCG . 2022 . 3150488
  * [36] M. K. Park, K. J. Lim, M. K. Seo, S. J. Jung, and K. H. Lee.  Spatial augmented reality for product appearance design evaluation.  Journal of Computational Design and Engineering, 2(1):38–46, 2015. doi: 10 . 1016/j . jcde . 2014 . 11 . 004
  * [37] T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. Wilson.  Room2room: Enabling life-size telepresence in a projected augmented reality environment.  In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing, CSCW ’16, p. 1716–1725, 2016.
  * [38] R. Raskar, P. Beardsley, J. van Baar, Y. Wang, P. Dietz, J. Lee, D. Leigh, and T. Willwacher.  Rfig lamps: Interacting with a self-describing world via photosensing wireless tags and projectors.  ACM Trans. Graph., 23(3):406–415, Aug. 2004.
  * [39] R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs.  The office of the future: A unified approach to image-based modeling and spatially immersive displays.  In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’98, p. 179–188, 1998.
  * [40] A. Rivers, A. Adams, and F. Durand.  Sculpting by numbers.  ACM Trans. Graph., 31(6), Nov. 2012.
  * [41] S. Schmidt, G. Bruder, and F. Steinicke.  Effects of virtual agent and object representation on experiencing exhibited artifacts.  Computers & Graphics, 83:1 – 10, 2019.
  * [42] J. Shi, H. Yu, X. Huang, Z. Chen, and Y. Tai.  Illuminant spectrum estimation using a digital color camera and a color chart .  In Q. Dai and T. Shimura, eds., Optoelectronic Imaging and Multimedia Technology III, vol. 9273, p. 927307. International Society for Optics and Photonics, SPIE, 2014. doi: 10 . 1117/12 . 2071915
  * [43] C. Siegl, M. Colaianni, M. Stamminger, and F. Bauer.  Adaptive stray-light compensation in dynamic multi-projection mapping.  Computational Visual Media, 3(3):263–271, Sep 2017. doi: 10 . 1007/s41095-017-0090-8
  * [44] C. Siegl, M. Colaianni, L. Thies, J. Thies, M. Zollhöfer, S. Izadi, M. Stamminger, and F. Bauer.  Real-time pixel luminance optimization for dynamic multi-projection mapping.  ACM Trans. Graph., 34(6), nov 2015. doi: 10 . 1145/2816795 . 2818111
  * [45] C. Siegl, V. Lange, M. Stamminger, F. Bauer, and J. Thies.  Faceforge: Markerless non-rigid face multi-projection mapping.  IEEE Transactions on Visualization and Computer Graphics, 23(11):2440–2446, 2017.
  * [46] Stephen Boyd.  http://www.stanford.edu/class/ee364b/lectures/subgrad_method_notes.pdf, 2015.
  * [47] R. Sukthankar, T.-J. Cham, and G. Sukthankar.  Dynamic shadow elimination for multi-projector displays.  In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 2, pp. II–II, 2001.
  * [48] Y. Takeuchi and K. Nagamine.  Theory and implementation of integral illumination.  IEEE Access, 10:939–950, 2022. doi: 10 . 1109/ACCESS . 2021 . 3139108
  * [49] Y. Takeuchi, S. Suwa, and K. Nagamine.  Anylight: Programmable ambient illumination via computational light fields.  In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces, ISS ’16, p. 39–48. Association for Computing Machinery, New York, NY, USA, 2016. doi: 10 . 1145/2992154 . 2992188
  * [50] T. Takezawa, D. Iwai, K. Sato, T. Hara, Y. Takeda, and K. Murase.  Material surface reproduction and perceptual deformation with projection mapping for car interior design.  In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 251–258, 2019. doi: 10 . 1109/VR . 2019 . 8797923
  * [51] J. Tsukamoto, D. Iwai, and K. Kashima.  Radiometric compensation for cooperative distributed multi-projection system through 2-dof distributed control.  IEEE Transactions on Visualization and Computer Graphics, 21(11):1221–1229, 2015. doi: 10 . 1109/TVCG . 2015 . 2459905
  * [52] H. Uchikawa, K. Uchikawa, and R. M. Boynton.  Influence of achromatic surrounds on categorical perception of surface colors.  Vision Research, 29(7):881–890, 1989. doi: 10 . 1016/0042-6989(89)90099-0
  * [53] K. Uchikawa, K. Koida, T. Meguro, Y. Yamauchi, and I. Kuriki.  Brightness, not luminance, determines transition from the surface-color to the aperture-color mode for colored lights.  J. Opt. Soc. Am. A, 18(4):737–746, Apr 2001. doi: 10 . 1364/JOSAA . 18 . 000737
  * [54] J. Underkoffler and H. Ishii.  Urp: A luminous-tangible workbench for urban planning and design.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 386–393, 1999.
  * [55] G. Wetzstein and O. Bimber.  Radiometric compensation through inverse light transport.  In 15th Pacific Conference on Computer Graphics and Applications (PG’07), pp. 391–399, 2007. doi: 10 . 1109/PG . 2007 . 47
  * [56] M. Yasui, Y. Watanabe, and M. Ishikawa.  Dynamic and occlusion-robust light field illumination.  In SIGGRAPH Asia 2021 Posters, SA ’21 Posters. Association for Computing Machinery, New York, NY, USA, 2021. doi: 10 . 1145/3476124 . 3488624
  * [57] F. Zhong, A. Jindal, A. O. Yöntem, P. Hanji, S. J. Watt, and R. K. Mantiuk.  Reproducing reality with a high-dynamic-range multi-focal stereo display.  ACM Trans. Graph., 40(6), dec 2021. doi: 10 . 1145/3478513 . 3480513
  * [58] Z. Zhou, T. Yu, X. Qiu, R. Yang, and Q. Zhao.  Light field projection for lighting reproduction.  In 2015 IEEE Virtual Reality (VR), pp. 135–142, 2015. doi: 10 . 1109/VR . 2015 . 7223335