<p>There are two different approaches that can be used.  Motion/Camera tracking and Motion Control.</p>

<p>In Motion or Camera tracking, multiple non-coplanar fixed points in the scene are tracked and from that it is possible to extrapolate the position of the camera within the scene.  This information can then be fed into a virtual camera used to generate the output from the 3d model such that when the 3d model output is superimposed over the video, the movements line up.</p>

<p>Motion Control is similar but uses a special camera rig to either track the camera movement exactly or to reproduce a movement plotted in the 3d software exactly in real life.  This is the more precise but also much more sophisticated approach as it requires far more elaborate gear to do.  It also allows for more accurate tracking of things like change in zoom verses actually moving the camera in (which are otherwise only evident by fairly subtle shifting of perspective and vanishing points.)</p>

<p>The result of both is effectively similar though.  You end up with data on how the camera moves in the real life scene and it matches the movement done in the 3d scene.  There are other factors such as lighting that also have to be matched for the best looking composition.  Then both outputs are brought in to a composition package like After Effects where they can be overlayed and blended together.  Sometimes various additional effects like depth of field and other virtual camera effects may be applied to help it blend.</p>

<p>Another thing is that if anything in the real scene needs to move in front of the 3d object, then they need something called a mask layer.  This is normally produced by rotoscoping, which is a process of updating the image on every frame to follow the subject.  This mask then tells the compositing software that the real video plate should be placed over the 3d plate on the areas where the mask is painted.  Otherwise the 3d plate is used over the real plate.  Alpha channel blending is used to convey information about the transparency of a 3d shape such that the image behind it can be visible.</p>

<p>If the 3d object has any refractive properties, then the real life plate may be fed into the 3d software as an environment.  Then the 3d render engine can actually sample information from the environment in order to figure out refraction in things like a glass object or a 3d object with a mirror.  Since it is mapping a 2d environment onto a 3d one, a lot of the selection of how the environment is applied would be done manually though.</p>

CC BY-SA 3.0
2013-06-03T14:42:19.597Z
Thanks! I take it there is no consumer-grade solution for this?
Dmitri Nesteruk
CC BY-SA 3.0
2013-06-03T16:50:51.123Z
Using After Effects and the 3d motion tracking it can provide is the closest you are going to get to consumer-grade.  It's still a pretty intense process though.  The most expensive part would be the software for actually making the 3d models.  Typical 3d animation software runs for several thousand dollars a seat unless you do something lower end with Blender (which is free).  After Effects is only a few hundred dollars for a seat or can be obtained for $20 a month through Adobe's Creative Cloud.  It would still be labor intensive to rotoscope though.
AJ Henderson
CC BY-SA 3.0
2013-06-06T15:11:28.197Z
I guess the question is - can the software infer movement and rotation of the camera just by analyzing the image?
Dmitri Nesteruk
CC BY-SA 3.0
2013-06-06T15:24:12.853Z
@DmitriNesteruk - yes, though not as accurately as direct collection.  Any plane in the video can be tracked and the corners of that plane will reveal information about the rotation, position and focal length of the camera in question.
AJ Henderson
CC BY-SA 3.0
2013-06-01T16:01:28.083Z
2013-06-01T16:01:28.083Z
AJ Henderson
<p>One popular feature on TV nowadays is to create 3D overlays on top of the recorded content. Also, as the camera changes direction or position, the 3D content view is subsequently adjusted. How is this implemented?</p>

<video><3d>
How are 3-dimensional overlays synchronized with videos?
Dmitri Nesteruk
# How are 3-dimensional overlays synchronized with videos?

(Asked by: username_0 on 2013-06-01)

One popular feature on TV nowadays is to create 3D overlays on top of the recorded content. Also, as the camera changes direction or position, the 3D content view is subsequently adjusted. How is this implemented?

**Tags:** video, 3d


---

## 1 answers

### Answer by username_1 on 2013-06-01. Score: 4 | Accepted answer

There are two different approaches that can be used. Motion/Camera tracking and Motion Control.

In Motion or Camera tracking, multiple non-coplanar fixed points in the scene are tracked and from that it is possible to extrapolate the position of the camera within the scene. This information can then be fed into a virtual camera used to generate the output from the 3d model such that when the 3d model output is superimposed over the video, the movements line up.

Motion Control is similar but uses a special camera rig to either track the camera movement exactly or to reproduce a movement plotted in the 3d software exactly in real life. This is the more precise but also much more sophisticated approach as it requires far more elaborate gear to do. It also allows for more accurate tracking of things like change in zoom verses actually moving the camera in (which are otherwise only evident by fairly subtle shifting of perspective and vanishing points.)

The result of both is effectively similar though. You end up with data on how the camera moves in the real life scene and it matches the movement done in the 3d scene. There are other factors such as lighting that also have to be matched for the best looking composition. Then both outputs are brought in to a composition package like After Effects where they can be overlayed and blended together. Sometimes various additional effects like depth of field and other virtual camera effects may be applied to help it blend.

Another thing is that if anything in the real scene needs to move in front of the 3d object, then they need something called a mask layer. This is normally produced by rotoscoping, which is a process of updating the image on every frame to follow the subject. This mask then tells the compositing software that the real video plate should be placed over the 3d plate on the areas where the mask is painted. Otherwise the 3d plate is used over the real plate. Alpha channel blending is used to convey information about the transparency of a 3d shape such that the image behind it can be visible.

If the 3d object has any refractive properties, then the real life plate may be fed into the 3d software as an environment. Then the 3d render engine can actually sample information from the environment in order to figure out refraction in things like a glass object or a 3d object with a mirror. Since it is mapping a 2d environment onto a 3d one, a lot of the selection of how the environment is applied would be done manually though.

> **Comments:**
> **username_0**: Thanks! I take it there is no consumer-grade solution for this?  
> **username_1**: Using After Effects and the 3d motion tracking it can provide is the closest you are going to get to consumer-grade.  It's still a pretty intense process though.  The most expensive part would be the software for actually making the 3d models.  Typical 3d animation software runs for several thousand dollars a seat unless you do something lower end with Blender (which is free).  After Effects is only a few hundred dollars for a seat or can be obtained for $20 a month through Adobe's Creative Cloud.  It would still be labor intensive to rotoscope though.  
> **username_0**: I guess the question is - can the software infer movement and rotation of the camera just by analyzing the image?  
> **username_1**: @DmitriNesteruk - yes, though not as accurately as direct collection.  Any plane in the video can be tracked and the corners of that plane will reveal information about the rotation, position and focal length of the camera in question.  

---

