<p>In the end I decided to just learn how Blender's scripting works and wrote a script. I know there would be simpler ways to calculate the transformations but I'm no maths expert and I wanted to spend my time debugging my logic errors rather my maths errors.</p>

<p>Basically what this script does is take 3 input points (in the form of empties) relative to the photogrammetry mesh and 3 input points in the same position relative to the tracking data, then attempts to line them up with each other, bringing the photogrammetry mesh along with it. This actually works pretty well in my testing. It's obviously not the fastest workflow but it does work and that's the important part.</p>

<pre class="lang-python prettyprint-override"><code>import bpy
import bmesh
from mathutils import Matrix

#####################################
# TO USE:
#
# 1) create empties at three easily-identified positions in tracking data, called ea1, ea2 and ea3
# 2) create empties at the same three positions but in photogrammetry data, called eb1, eb2 and eb3
#    NOTE: eb1, eb2 and eb3 must be positioned IN THE SAME ORDER as ea1, ea2 and ea3!
# 3) name the photogrammetry mesh "photogrammetry"
# 4) run the script and be amazed
#
#####################################

# from https://blender.stackexchange.com/a/118783/41192
def createLookRotation(forward, up):
    rot = Matrix.Identity(3)
    rot[0] = up                 # x
    rot[1] = forward.cross(up)  # y
    rot[2] = forward            # z
    rot = rot.transposed()
    return rot.to_quaternion()

def doMeshThing(obj, mesh, empty1, empty2, empty3):
    bm = bmesh.new()
    bm.verts.new(empty1.location)
    bm.verts.new(empty2.location)
    bm.verts.new(empty3.location)
    bm.verts.ensure_lookup_table()
    bm.faces.new([bm.verts[0], bm.verts[1], bm.verts[2]])
    bm.normal_update()
    bm.to_mesh(mesh)
    bpy.ops.object.select_all(action='DESELECT')
    obj.select = True
    bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')
    bm.faces.ensure_lookup_table()
    normal = bm.faces[0].normal
    guide = empty2.location - empty1.location
    quatRot = createLookRotation(normal, guide)
    obj.rotation_mode = 'QUATERNION'
    obj.rotation_quaternion = quatRot.inverted()
    bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
    obj.rotation_quaternion = quatRot
    bm.free()

ea1 = bpy.data.objects['ea1']
ea2 = bpy.data.objects['ea2']
ea3 = bpy.data.objects['ea3']
eb1 = bpy.data.objects['eb1']
eb2 = bpy.data.objects['eb2']
eb3 = bpy.data.objects['eb3']

mesha = bpy.data.meshes.new('mesha')
meshb = bpy.data.meshes.new('meshb')

obja = bpy.data.objects.new('obja', mesha)
objb = bpy.data.objects.new('objb', meshb)

scene = bpy.context.scene
scene.objects.link(obja)
scene.objects.link(objb)

doMeshThing(obja, mesha, ea1, ea2, ea3)
doMeshThing(objb, meshb, eb1, eb2, eb3)

bpy.context.scene.update()

# position and rotate objb (photogrammetry) to match position and rotation of obja (tracking)
photogrammetryMesh = bpy.data.objects['photogrammetry']
photogrammetryMesh.parent = objb
photogrammetryMesh.matrix_parent_inverse = objb.matrix_world.inverted()

objb.location = obja.location
objb.rotation_quaternion = obja.rotation_quaternion

# calculate average dimension difference here and scale
scaleDiffX = obja.dimensions.x / objb.dimensions.x
scaleDiffY = obja.dimensions.y / objb.dimensions.y
scaleDiffZ = obja.dimensions.z / objb.dimensions.z
#averageScaleDiff = (scaleDiffX + scaleDiffY + scaleDiffZ) / 3
averageScaleDiff = (scaleDiffX + scaleDiffY) / 2 # seems to be in local space, so z is useless
objb.scale.x = averageScaleDiff
objb.scale.y = averageScaleDiff
objb.scale.z = averageScaleDiff

bpy.context.scene.update()

parented_wm = photogrammetryMesh.matrix_world.copy()
photogrammetryMesh.parent = None
photogrammetryMesh.matrix_world = parented_wm

bpy.ops.object.select_all(action='DESELECT')
obja.select = True
bpy.ops.object.delete()
bpy.ops.object.select_all(action='DESELECT')
objb.select = True
bpy.ops.object.delete()
</code></pre>

CC BY-SA 4.0
2020-01-13T12:25:14.177Z
2020-01-13T12:25:14.177Z
Clonkex
<p>I have some footage, which I've tracked in After Effects. I've exported that 3D tracked camera to C4D Lite. I then took photos of the filming location and used Meshroom (a photogrammetry tool) to generate a mesh from the photos.</p>

<p>Now I want to line up the photogrammetry mesh in C4D with nulls that I've placed relative to the tracking points in After Effects. What's the best way to do this? Or is there a better workflow to get a photogrammetry mesh lined up with tracked footage?</p>

<p><strong>More info:</strong> Specifically, after watching Corridor's real-life portals video I decided to try to create my own portal effect. My plan is to take a rough photogrammetry mesh of the filming area. I would then track the camera in After Effects, somehow get that tracking data into a 3D modelling program, load the photogrammetry mesh in, line everything up so the tracked camera moves through the mesh as it does in the footage then rotate the camera and its animation 180° around the point where the portal mask is tracked onto the footage in AE. This would create a perfect replica of the camera's original motion, but looking out at the world from a different angle.</p>

<p>As such I can't just use the footage as the photogrammetry input, as I specifically need the photogrammetry mesh to include unseen areas of the world.</p>

CC BY-SA 4.0
2020-01-11T08:19:10.190Z
Ideally you would have used images of the footage in mushroom too then your solutions would converge and the locations of the camera positions would have been solved in one go. (photogramettery is sort of automated tracking)
joojaa
CC BY-SA 4.0
2020-01-11T09:01:03.177Z
@joojaa I'm not really sure what you mean. I'd still just get a mesh with no particular way to line it up with the camera tracking from AE. As in, I would have no way to get the scale, rotation and position of the mesh correct relative to the tracked camera.
Clonkex
CC BY-SA 4.0
2020-01-11T09:08:31.173Z
No you get a mesh and postion of the cameras. See mushroom solves the camera position in the first pass.
joojaa
CC BY-SA 4.0
2020-01-11T10:10:47.123Z
@joojaa Well sure it solves the camera positions, but how would that be useful? I'm just not seeing the practical connection. I can't export the camera positions, and even if I could, I couldn't easily turn those positions into an animated camera in a 3D modelling program or After Effects. Plus the results of photogrammetry from video would be far inferior to those from photos, since sharp images give drastically better results.
Clonkex
CC BY-SA 4.0
2020-01-11T10:11:22.033Z
Positions are animation
joojaa
CC BY-SA 4.0
2020-01-11T10:29:40.783Z
@joojaa You seem very confident of this process, but I still don't see how it could be useful. The positions are _not_ animation on their own. Please explain how you would have 3D modelling software interpret a series of positions as animation unless they were already defined as such.
Clonkex
CC BY-SA 4.0
2020-01-11T10:34:35.327Z
Well i dont use C4D but i have done this in Maya, Blender, and 3DS max several times. Just loop over the objects and keyframe one other camera to their potions.
joojaa
CC BY-SA 4.0
2020-01-11T10:42:49.773Z
@joojaa I have no particular interest in using C4D if I can use Blender, but I assumed this would be something other people do commonly with C4D Lite and AE so I only mentioned C4D. Perhaps you would like to write a more in-depth answer? Because "loop over the objects" doesn't sound like something you "just" do. Unless you are implying I should write a script to do this? But aside from that, that ultimately doesn't solve my problem. I'll update my question to be more specific.
Clonkex
CC BY-SA 4.0
2020-01-11T10:47:17.843Z
Well everything seems overly complex if you dont know anything about it
joojaa
CC BY-SA 4.0
2020-01-11T10:49:21.727Z
Anyway you should be using video production stackexchange. Also meshroom can export camera as animation
joojaa
CC BY-SA 4.0
2020-01-11T10:54:42.707Z
@joojaa It seems overly complex because it _should_ be overly complex. As in, there should be no reason to have to go to that much effort. It essentially boils down to needing a way to make a mesh in Blender (or C4D) line up based on 3 or 4 empties to another mesh with 3 or 4 empties, where the relative positions will never be quite precisely the same. I only now realise that's my actual problem. The difficulty comes in trying to search for such a thing...
Clonkex
CC BY-SA 4.0
2020-01-11T10:56:16.680Z
Its not allways possible to make 3 things work together without a bit of script glue. Its not much harder than writing for items selected take item copy attribute to other item
joojaa
CC BY-SA 4.0
2020-01-11T10:58:12.767Z
@joojaa Any scripting is hard when you have to sift through a reference for every second line. I've never written a Blender script, except 10 years ago when I tried to make games with it. I could learn for sure but this is feeling like way more effort than it's worth.
Clonkex
CC BY-SA 4.0
2020-01-11T10:58:58.710Z
You can allways give up on your idea isnetad. But im sure somebody has wirtten this script
joojaa
CC BY-SA 4.0
2020-01-11T11:26:09.893Z
@joojaa Oh I'm not giving up yet. https://blender.stackexchange.com/q/163459/41192 But it's definitely pretty tiresome when something seems like it should be so simple and easy but no one else even wanted to do the same thing so there's no help on it.
Clonkex
CC BY-SA 4.0
2020-01-11T12:04:06.493Z
Your issue might be simpler than your question says. It does not particularity matter where the warp to object is from users point of view. just use meshroom to track scene 1 and scene 2. Export scene 1 as a camera animation. place yoru other stuff in that scene wherever you want it. make the portal in 3d render that. render scene 2.  do not track anything in AE. just comp the results so that 3d portal is alphamask. AE really does not do you any favors here.
joojaa
<after-effects><cinema4d>
Line up photogrammetry mesh with tracking markers
Clonkex
# Line up photogrammetry mesh with tracking markers

(Asked by: username_0 on 2020-01-11)

I have some footage, which I've tracked in After Effects. I've exported that 3D tracked camera to C4D Lite. I then took photos of the filming location and used Meshroom (a photogrammetry tool) to generate a mesh from the photos.

Now I want to line up the photogrammetry mesh in C4D with nulls that I've placed relative to the tracking points in After Effects. What's the best way to do this? Or is there a better workflow to get a photogrammetry mesh lined up with tracked footage?

**More info:** Specifically, after watching Corridor's real-life portals video I decided to try to create my own portal effect. My plan is to take a rough photogrammetry mesh of the filming area. I would then track the camera in After Effects, somehow get that tracking data into a 3D modelling program, load the photogrammetry mesh in, line everything up so the tracked camera moves through the mesh as it does in the footage then rotate the camera and its animation 180° around the point where the portal mask is tracked onto the footage in AE. This would create a perfect replica of the camera's original motion, but looking out at the world from a different angle.

As such I can't just use the footage as the photogrammetry input, as I specifically need the photogrammetry mesh to include unseen areas of the world.

**Tags:** after-effects, cinema4d

> **Comments:**
> **username_1**: Ideally you would have used images of the footage in mushroom too then your solutions would converge and the locations of the camera positions would have been solved in one go. (photogramettery is sort of automated tracking)  
> **username_0**: @username_1 I'm not really sure what you mean. I'd still just get a mesh with no particular way to line it up with the camera tracking from AE. As in, I would have no way to get the scale, rotation and position of the mesh correct relative to the tracked camera.  
> **username_1**: No you get a mesh and postion of the cameras. See mushroom solves the camera position in the first pass.  
> **username_0**: @username_1 Well sure it solves the camera positions, but how would that be useful? I'm just not seeing the practical connection. I can't export the camera positions, and even if I could, I couldn't easily turn those positions into an animated camera in a 3D modelling program or After Effects. Plus the results of photogrammetry from video would be far inferior to those from photos, since sharp images give drastically better results.  
> **username_1**: Positions are animation  
> **username_0**: @username_1 You seem very confident of this process, but I still don't see how it could be useful. The positions are _not_ animation on their own. Please explain how you would have 3D modelling software interpret a series of positions as animation unless they were already defined as such.  
> **username_1**: Well i dont use C4D but i have done this in Maya, Blender, and 3DS max several times. Just loop over the objects and keyframe one other camera to their potions.  
> **username_0**: @username_1 I have no particular interest in using C4D if I can use Blender, but I assumed this would be something other people do commonly with C4D Lite and AE so I only mentioned C4D. Perhaps you would like to write a more in-depth answer? Because "loop over the objects" doesn't sound like something you "just" do. Unless you are implying I should write a script to do this? But aside from that, that ultimately doesn't solve my problem. I'll update my question to be more specific.  
> **username_1**: Well everything seems overly complex if you dont know anything about it  
> **username_1**: Anyway you should be using video production stackexchange. Also meshroom can export camera as animation  
> **username_0**: @username_1 It seems overly complex because it _should_ be overly complex. As in, there should be no reason to have to go to that much effort. It essentially boils down to needing a way to make a mesh in Blender (or C4D) line up based on 3 or 4 empties to another mesh with 3 or 4 empties, where the relative positions will never be quite precisely the same. I only now realise that's my actual problem. The difficulty comes in trying to search for such a thing...  
> **username_1**: Its not allways possible to make 3 things work together without a bit of script glue. Its not much harder than writing for items selected take item copy attribute to other item  
> **username_0**: @username_1 Any scripting is hard when you have to sift through a reference for every second line. I've never written a Blender script, except 10 years ago when I tried to make games with it. I could learn for sure but this is feeling like way more effort than it's worth.  
> **username_1**: You can allways give up on your idea isnetad. But im sure somebody has wirtten this script  
> **username_0**: @username_1 Oh I'm not giving up yet. https://blender.stackexchange.com/q/163459/41192 But it's definitely pretty tiresome when something seems like it should be so simple and easy but no one else even wanted to do the same thing so there's no help on it.  
> **username_1**: Your issue might be simpler than your question says. It does not particularity matter where the warp to object is from users point of view. just use meshroom to track scene 1 and scene 2. Export scene 1 as a camera animation. place yoru other stuff in that scene wherever you want it. make the portal in 3d render that. render scene 2.  do not track anything in AE. just comp the results so that 3d portal is alphamask. AE really does not do you any favors here.  

---

## 1 answers

### Answer by username_0 on 2020-01-13. Score: 0 | Accepted answer

In the end I decided to just learn how Blender's scripting works and wrote a script. I know there would be simpler ways to calculate the transformations but I'm no maths expert and I wanted to spend my time debugging my logic errors rather my maths errors.
Basically what this script does is take 3 input points (in the form of empties) relative to the photogrammetry mesh and 3 input points in the same position relative to the tracking data, then attempts to line them up with each other, bringing the photogrammetry mesh along with it. This actually works pretty well in my testing. It's obviously not the fastest workflow but it does work and that's the important part.
import bpy
import bmesh
from mathutils import Matrix

#####################################
# TO USE:
#
# 1) create empties at three easily-identified positions in tracking data, called ea1, ea2 and ea3
# 2) create empties at the same three positions but in photogrammetry data, called eb1, eb2 and eb3
#    NOTE: eb1, eb2 and eb3 must be positioned IN THE SAME ORDER as ea1, ea2 and ea3!
# 3) name the photogrammetry mesh "photogrammetry"
# 4) run the script and be amazed
#
#####################################

# from https://blender.stackexchange.com/a/118783/41192
def createLookRotation(forward, up):
    rot = Matrix.Identity(3)
    rot[0] = up                 # x
    rot[1] = forward.cross(up)  # y
    rot[2] = forward            # z
    rot = rot.transposed()
    return rot.to_quaternion()

def doMeshThing(obj, mesh, empty1, empty2, empty3):
    bm = bmesh.new()
    bm.verts.new(empty1.location)
    bm.verts.new(empty2.location)
    bm.verts.new(empty3.location)
    bm.verts.ensure_lookup_table()
    bm.faces.new([bm.verts[0], bm.verts[1], bm.verts[2]])
    bm.normal_update()
    bm.to_mesh(mesh)
    bpy.ops.object.select_all(action='DESELECT')
    obj.select = True
    bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')
    bm.faces.ensure_lookup_table()
    normal = bm.faces[0].normal
    guide = empty2.location - empty1.location
    quatRot = createLookRotation(normal, guide)
    obj.rotation_mode = 'QUATERNION'
    obj.rotation_quaternion = quatRot.inverted()
    bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
    obj.rotation_quaternion = quatRot
    bm.free()

ea1 = bpy.data.objects['ea1']
ea2 = bpy.data.objects['ea2']
ea3 = bpy.data.objects['ea3']
eb1 = bpy.data.objects['eb1']
eb2 = bpy.data.objects['eb2']
eb3 = bpy.data.objects['eb3']

mesha = bpy.data.meshes.new('mesha')
meshb = bpy.data.meshes.new('meshb')

obja = bpy.data.objects.new('obja', mesha)
objb = bpy.data.objects.new('objb', meshb)

scene = bpy.context.scene
scene.objects.link(obja)
scene.objects.link(objb)

doMeshThing(obja, mesha, ea1, ea2, ea3)
doMeshThing(objb, meshb, eb1, eb2, eb3)

bpy.context.scene.update()

# position and rotate objb (photogrammetry) to match position and rotation of obja (tracking)
photogrammetryMesh = bpy.data.objects['photogrammetry']
photogrammetryMesh.parent = objb
photogrammetryMesh.matrix_parent_inverse = objb.matrix_world.inverted()

objb.location = obja.location
objb.rotation_quaternion = obja.rotation_quaternion

# calculate average dimension difference here and scale
scaleDiffX = obja.dimensions.x / objb.dimensions.x
scaleDiffY = obja.dimensions.y / objb.dimensions.y
scaleDiffZ = obja.dimensions.z / objb.dimensions.z
#averageScaleDiff = (scaleDiffX + scaleDiffY + scaleDiffZ) / 3
averageScaleDiff = (scaleDiffX + scaleDiffY) / 2 # seems to be in local space, so z is useless
objb.scale.x = averageScaleDiff
objb.scale.y = averageScaleDiff
objb.scale.z = averageScaleDiff

bpy.context.scene.update()

parented_wm = photogrammetryMesh.matrix_world.copy()
photogrammetryMesh.parent = None
photogrammetryMesh.matrix_world = parented_wm

bpy.ops.object.select_all(action='DESELECT')
obja.select = True
bpy.ops.object.delete()
bpy.ops.object.select_all(action='DESELECT')
objb.select = True
bpy.ops.object.delete()


---

