# Recursive Camera Painting: A Method for Real-Time Painterly Renderings of 3D
Scenes

Ergun Akleman
Visual Computing & Computational Media,
Texas A&M University, College Station, TX, 77831
<EMAIL_ADDRESS>
& Cassie Mullins
Department of Visualization
Texas A&M University, College Station, TX, 77831
<EMAIL_ADDRESS>
&Christopher Morrison
Department of Visualization
Texas A&M University, College Station, TX, 77831
<EMAIL_ADDRESS>
&David Oh
Department of Visualization
Texas A&M University, College Station, TX, 77831
<EMAIL_ADDRESS>
Joint with Computer Science and Engineering Department.

###### Abstract

In this work, we present the recursive camera-painting approach to obtain
painterly smudging in real-time rendering applications. We have implemented
recursive camera painting as both a GPU-based ray-tracing and in a Virtual
Reality game environment. Using this approach, we can obtain dynamic 3D
Paintings in real-time. In a camera painting, each pixel has a separate
associated camera whose parameters are computed from a corresponding image of
the same size. In recursive camera painting, we use the rendered images to
compute new camera parameters. When we apply this process a few times, it
creates painterly images that can be viewed as real-time 3D dynamic paintings.
These visual results are not surprising since multi-view techniques help to
obtain painterly effects.

##  1 Introduction

Figure 1: Examples of images created in real-time with our approach
implemented as a GPU-based ray tracing system.

During the end of the nineteenth century, traditional perspective conventions
were broken down in art. From Cezanne, artists quickly discovered that the
multi-view perspective is an important tool for obtaining visually appealing
images. Not only traditional cubists, but also a wide variety of artists such
as Pablo Picasso, Willem de Kooning, and David Hockney from a variety of art
schools employed the multi-view perspective for the creation of expressive
depictions Hess and De Kooning (2004); Hockney (1995).

(a) First iteration.

(b) Second iteration.

(c) Third iteration.

(d) Forth iteration.

(e) Fifth iteration.

(f) Sixth iteration.

Figure 2: Recursion process that demonstrates more and more smudging.

Figure 3: More examples of GPU-based ray tracing with color-based
displacements. These are six-iteration images that were captured during a
real-time walk-through.

Figure 4: Examples of unreal engine renderings using color-based
displacements. These are also six-iteration images that are captured in a
real-time walk-through. Note how much this forest resembles the forests in
Disney’s animations.

Figure 5: Examples of unreal engine renderings using color based
displacements. These are also six-iteration images that are captured in a
real-time walk-through.

Multi-view perspective is also an important tool in traditional animation to
create moving cameras Glassner (2000). There exists a significant number of
techniques to obtain multi-view perspective such as creating cameras using
parametric surfaces Glassner (2000); Smith et al. (2004); Yu and McMillan
(2004), using images to control camera parameters Meadows and Akleman (2000);
Morrison and Akleman (2020), and using high-saliency regions to obtain cubist
looking images Collomosse and Hall (2003); Arpa et al. (2012); Wang et al.
(2011).

One of the problems of multi-view rendering is to avoid the shower door
effect, which is a well-known problem in non-photo-realistic animations Meier
(1996). Avoiding the shower-door effect requires dynamically controlling the
parameters of all cameras during multi-view animations. Recently, another
method of camera painting was developed, “Remote Empathetic Viewpoint”, to
avoid the effect of the shower door by decomposing the scene using control
lights Morrison and Akleman (2020). However, this system is not real-time, and
the control lights decompose the scene in a predictive way. Moreover, it is
hard to control the style using “Remote Empathetic Viewpoint”.

In this paper, we present a new approach that can remove shower door effects
by using rendered data to control camera parameters. An additional advantage
of this approach is that the recursive application of this process creates
painterly blurs that are obtained by deliberately mixing charcoal and oil
paint to smudge the brighter colors that resemble the styles of a wide variety
of representational and abstract artists Hess and De Kooning (2004).

##  2 Theoretical Framework

Let $I_{n}(u,v)$ denote the image/data rendered in time step $n$ and $(u,v)$
denote the texture coordinates of the image. Note that we are using $(u,v)$
instead of pixels. In this way, we can compute any number of samples per pixel
using standard jittering to avoid aliasing. $I_{n}(u,v)$ can provide not only
color information but also other data such as z-depth, shading normal, surface
normal, material properties of the shading point, or object ID for the given
coordinate. Our recursive process is straightforward and can be represented as

$I_{n}(u,v)=F(I_{n-1}(u,v),S(n)),$

where $I_{n-1}(u,v)$ denote the image / data rendered previously and $S(n)$ is
a given scene in the time step $n$.

$F$ is a function that represents a camera-painting process that defines a new
camera parameter per pixel. The process $F$ consists of two stages:

  1. 1.

Assigning a camera to each point $u,v$ based on previously computed image/data
$I_{n-1}(u,v)$ and the current scene description $S(n)$;

  2. 2.

For every $u,v$ compute the new image/data using new camera definition and
$S(n)$.

The camera assignment, in general, is simply defining a ray that is usually
provided by an eye point that is given by its position $\mathbf{P_{e}}(u,v)$
and a direction that is given a unit vector $\vec{N}(u,v)$. For simplicity,
the unit vector is computed indirectly from a second position
$\mathbf{P_{p}}(u,v)$ that corresponds to the 3D position of the texture
coordinate $(u,v)$. Then,

$\vec{N}(u,v)=\frac{\mathbf{P_{p}}(u,v)-\mathbf{P_{e}}(u,v)}{|\mathbf{P_{p}}(u,v)-\mathbf{P_{e}}(u,v)|}$

We compute the new 3D positions of the eye(s) and 3D positions of the texture
coordinates from their original positions using two displacement vectors,
$\vec{V_{e}}(u,v)$ and $\vec{V_{p}}(u,v)$ as

$\mathbf{P^{\prime}_{p}}(u,v)=\mathbf{P_{p}}(u,v)+\vec{V_{e}}(u,v)$
$\mathbf{P^{\prime}_{p}}(u,v)=\mathbf{P_{p}}(u,v)+\vec{V_{p}}(u,v)$

Note that the whole problem reduces the development of mappings to compute the
displacement vectors $\vec{V_{e}}(u,v)$ and $\vec{V_{p}}(u,v)$ from
$I_{n-1}(u,v)$. This provides significant flexibility for artists to construct
their mappings. In the next section, we will provide a specific example that
uses only colors in $I_{n-1}(u,v)$.

##  3 Displacement Using Colors

Now, let us demonstrate the process using a simple mapping that only involves
color to define cameras and uses a constant $\mathbf{P_{e}}$, i.e.
$\vec{V_{e}}(u,v)=(0,0,0)$. Now, only the new positions of 3D texture
coordinates, $\mathbf{P^{\prime}_{p}}(u,v)$, need to be produced to create a
new camera. Let

$I_{n-1}(u,v)=(R_{n-1}(u,v),G_{n-1}(u,v),B_{n-1}(u,v))$

denote the three channels of low dynamic range that are previously computed in
time $n-1$. Let us assume that the vector $\vec{V_{p}}(u,v)$ is computed as
follows:

$\vec{V_{p}}(u,v)=\lambda\left(\begin{matrix}2R_{n-1}(u,v)-1\\\
2G_{n-1}(u,v)-1\\\ 2B_{n-1}(u,v)-1\\\ \end{matrix}\right)$

where $\lambda$ is a positive real number and $2I_{n-1}(u,v)-1)\in(-1,1)^{3}$.
Note that $\lambda(2I_{n-1}(u,v)-1))$ is a vector in $(-\lambda,\lambda)^{3}$
in the local coordinate system of the camera.

This process can provide a new camera for any given $(u,v)$. We can then
compute the color of $I_{n}(u,v)$ by shooting rays from $\mathbf{P_{e}}$ to
$\mathbf{P^{\prime}_{p}}(u,v)$ for a given scene at time $n$, $S(n)$. We apply
the process $n$ times starting from $I_{0}(u,v)=(0.5,0.5,0.5)$. Note that this
gives us a traditional ray-traced image in the first iteration (see Figure
2(a)). The image computed in $I_{1}(u,v)$ determines how the next image is
computed. For example, $R=1$ moves the pixel position to the right, and $R=0$
to the left in the local camera coordinate system. Similarly, $G=1$ moves the
position of the pixels to the upward, $G=0$ to the downward in the local
camera coordinate system. Blue does not have too much of an effect.

An important property of this process is that average color values do not move
the rays. Only extreme values move the rays. Moreover, in regions closer to a
boundary, the ray moves up and down, or right and left, based on the values in
each iteration. This process produces smudges by mixing extreme colors and
produces an effect that is similar to painterly smudges.

This particular smudging effect is different from any type of anisotropic blur
that can be obtained by bilateral filters or line integral convolution since
the process is not entirely predictable: it allows sharp regions while
blurring other regions. The iterative process can be shown in 2.

Note that even using colors to create displacements is general enough. We can
choose to display any particular color by simply using more complicated
mappings in the form of

$\vec{V_{p}}(u,v)=W\left(R_{n-1}(u,v),G_{n-1}(u,v),B_{n-1}(u,v)\right)$

Figure 6: Changing painterly focus by controlling displacements with depth.
These are also six-iteration images created using Unreal.

Figure 7: A painterly focus scene with moving sea waves. These are also six-
iteration images created using Unreal.

##  4 Displacements Based on Depth

As discussed in Section 2, the z-depth can also be used to control
displacements. Let us say that we want to create an illusion similar to out of
focus by controlling painterly smudge based on depth. Let $z(u,v)$ denote the
depth information for a given texture coordinate, and let $z_{0}$ denote the
depth that needs to be in focus. We can use $|z-z_{0}|$ as a multiplier as
follows:

$\vec{V_{p}}(u,v)=\lambda|z-z_{0}|\left(\begin{matrix}2R_{n-1}(u,v)-1\\\
2G_{n-1}(u,v)-1\\\ 2B_{n-1}(u,v)-1\\\ \end{matrix}\right)$

Using this approach, we obtain visuals that resemble 2D animation aesthetics
in 3D. This is because in 2D animations the backgrounds are drawn in a
painterly way and characters are drawn sharp.

##  5 Displacement Based on Normals

Another way to control displacements is to use world normals and camera
direction. In this case we would replace the usual color values,
$R_{n-1}(u,v)$, $G_{n-1}(u,v)$, and $B_{n-1}(u,v)$, with the cross product of
the camera direction $\vec{C}(u,v)$ and the world normals $\vec{N}(u,v)$.

$\vec{V_{p}}(u,v)=\lambda(\vec{C}(u,v)\times\vec{N}(u,v))$

The normal of a surface is a vector perpendicular to the surface itself,
pointing outward from the shape. So taking its cross product with camera
direction results in a tangent or a vector parallel to the surface. Small
adjustments to this equation can have many widely varied effects. For example,
looping through the equation created images that looked layered. Adding two
different kinds of randomness, on the other hand, resulted in one version that
looks fractured and another that looks like cross-hatching.

##  6 Implementations

We have first implemented this approach using the Nvidia OptiX Engine Parker
et al. (2010). In the examples in Figures 1, 2, and 3, we used simple scenes
that consist of only a spherical texture mapped onto an infinite sphere. The
process also handles 3D scenes in real-time. We have also created examples
with complex 3D objects in videos captured in real time.

One problem with the ray tracing solution is finding high-quality content,
i.e. very richly designed scenes. Therefore, we turned our attention to game
engines. Now, there exists a significant amount of Virtual Reality content
developed for Game Engines such as Unreal Sanders (2016). The problem with
game engines is that they do not support real-time ray tracing. We then
observed that the $\vec{V_{e}}(u,v)=(0,0,0)$ solution does not require Ray
Tracing. If we keep $\mathbf{P_{e}}$ in the same position, the rendered image
already includes most of the potential directions. Then, our original mapping
reduces into an image processing problem. We can simply pick colors from
another position in the image on the basis of the displacement. This process
can be done in the Unreal Engine in real-time. We can also access other
information such as z-depth in the Unreal Engine.

In conclusion, we have implemented the color- and depth-based displacement
methods in Unreal Engine. We tested with a wide variety of 3D scenes that are
created by Unreal users with real-time walk-throughs. Figures 4, 5, 6, and 7
show frames from these walks-through.

(a) Version 1 - Smudge Direction calculated using World Normals.

(b) Version 2 - Smudging from Version 1 is looped to run multiple times to
create a layered effect.

(c) Version 3 - A small amount of randomness is added using Unreal’s Material
Expression Vector Noise.

(d) Version 4 - Smudging using Normals and Noise from Version 3 is looped to
run multiple times.

Figure 8: Examples using world normals and camera direction to determine
smudge direction.

###  6.1 Handling Edge of Screen Flickering

One issue we originally had with our Unreal implementation was flickering at
the edge of the screen. Because the smudging works by taking the neighboring
pixel colors, when the original pixel is close to the edge of the screen,
often the values being used to get the neighboring pixel were outside of the
screen range. This resulted in unwanted glitching where Unreal attempted to
compensate for the bad input.

To fix this issue, we experimented with methods used in convolution filters
that deal with the same problem. The first pass adjusted how the new uv
coordinates are calculated. Now it uses the red and green channels of the
current pixel as uv, multiplies by the maximum smudge distance, divides by the
size of the screen, and finally adds the original uv to find the new pixel
location. In the second pass, the edge values were clamped so that if the new
uv position is less than 0 it becomes 0, and if it is greater than the screen
size, it is set to the maximum screen size. This fixed the glitching but left
unseemly bars of color at the edge of the screen that are very distracting.
The next version wraps around instead of just clamping. In some cases, this
produced a good result, but often when the top and bottom or left and right
sides of the screen have very different colors, wrapping produces a very
obvious border of color.

Finally, we tried mirroring the values so that the new pixel color would be
still somewhat near its original. The mirroring technique completely fixed the
glitching seen at the edge of the screen and also produced the smoothest,
least distracting edge result, as shown in Figure 8.

##  7 Discussion and Future Work

In this work, we have demonstrated that recursive camera painting can provide
a painting effect. We have scratched the surface in terms of controlling the
results. We think that it will be possible to obtain a wider variety of
effects such as cubist rendering by controlling cameras with the data
collected from scenes. In terms of control, it is much better to use a real-
time ray-tracer; however, the major bottleneck is finding high-quality scenes
available. Currently, we only work with realistic materials and scenes. The
results can be further improved by using nonphotorealistic shaders Akleman et
al. (2016) and nonrealistic scenes Elber (2011). An implication of
implementation with Unreal Engine is that we can use a similar approach to
manipulate real images during Augmented Reality. In other words, we can turn
real scenes into painted environments in real-time using the same type of
image processing operations.

## References

  * Hess and De Kooning [2004] Barbara Hess and Willem De Kooning.  _Willem de Kooning, 1904-1997: Content as a Glimpse_.  Taschen, 2004.
  * Hockney [1995] David Hockney.  _David Hockney_ , volume 1.  Manchester University Press, 1995.
  * Glassner [2000] Andrew S Glassner.  Cubism and cameras: Free-form optics for computer graphics.  _Microsoft Research MSR-TR-2000_ , 5:2, 2000.
  * Smith et al. [2004] Jeffrey Smith, Ergun Akleman, Richard Davison, John Keyser, et al.  Multicam: A system for interactive rendering of abstract digital images.  In _Bridges: Mathematical Connections in Art, Music, and Science_ , pages 265–272, Arizona, 2004. Bridges Conference.
  * Yu and McMillan [2004] Jingyi Yu and Leonard McMillan.  A framework for multiperspective rendering.  _Rendering Techniques_ , 4:61–68, 2004.
  * Meadows and Akleman [2000] Scott Meadows and Ergun Akleman.  Creating abstract digital paintings with painting camera technique.  In _2000 IEEE Conference on Multimedia. An International Conference on Computer Visualization and Graphics_ , pages 250–254, NewYork, NY, 2000. IEEE.
  * Morrison and Akleman [2020] Chris Morrison and Ergun Akleman.  Remote empathetic viewpoint: A novel approach to extending cubism.  In _ACM SIGGRAPH 2020 Posters_ , pages 1–2. ACM SIGGRAPH, 2020.
  * Collomosse and Hall [2003] John P Collomosse and Peter M Hall.  Cubist style rendering from photographs.  _IEEE Transactions on Visualization and Computer Graphics_ , 9(4):443–453, 2003.
  * Arpa et al. [2012] Sami Arpa, Abdullah Bulbul, Tolga Capin, and Bulent Ozguc.  Perceptual 3d rendering based on principles of analytical cubism.  _Computers & Graphics_, 36(8):991–1004, 2012\.
  * Wang et al. [2011] Yu-Shuen Wang, Min-Wen Chao, Chin-Chueng Yi, and Chao-Hung Lin.  Cubist style rendering for 3d polygonal models.  _Journal of information science and engineering_ , 27(6):1885–1899, 2011.
  * Meier [1996] Barbara J Meier.  Painterly rendering for animation.  In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , pages 477–484, 1996.
  * Parker et al. [2010] Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, and David McAllister.  Optix: a general purpose ray tracing engine.  _ACM Transactions on Graphics (tog)_ , 29, no. 4:1–13, 2010\.
  * Sanders [2016] Andrew Sanders.  _An introduction to Unreal engine 4_.  CRC Press, 2016.
  * Akleman et al. [2016] Ergun Akleman, S Liu, and Donald House.  Barycentric shaders: Art directed shading using control images.  In _Proceedings of Expressive Conference_ , pages 39–49, Geneva, Switzerland, 2016. Eurographics Association.
  * Elber [2011] Gershon Elber.  Modeling (seemingly) impossible models.  _Computers and Graphics_ , 35(3):632–638, June 2011.