<p>The main issue was, that the Z-Layer was not accurate enough, as Gyan said. Addtionally the condition needed to be changed from <code>lt(A,B)</code> to <code>lte(A,B)</code>.</p>

CC BY-SA 4.0
2020-05-25T11:03:31.243Z
2020-05-25T11:03:31.243Z
Pingger Shikkoken
<p>I'm trying to overlay 2 videos which both have depth layers (e.g. how far a pixel is away from the camera) using ffmpeg filters. (I have a few hundred of those videos so I'd like to do batch job out of this)</p>

<p>the video files are perfectly in sync with each other and their depth layers. The Files are saved in the lagarith lossless encoding (which ffmpeg can read). The video files are also using a greenscreen which I already filter using this command:</p>

<pre><code>ffmpeg -loglevel info -hide_banner -i "%fst%" -i "%snd%" -filter_complex "[1:v]chromakey=0x00ff00:0.5:0.0[ckout];[0:v][ckout]overlay[out]" -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result.mp4"
</code></pre>

<p>This command takes video %snd% makes the greenscreen transparent and puts the result on top of %fst% and writes it to result.mp4</p>

<p>The issue is, that I'd like also include the depth layers into the merge process, so that the nearer pixel is selected from both videos (the pixel with the brighter depth layer pixel). (the greenscreen can be rendered transparent in a later iteration, so we don't have to worry about that)</p>

<p>For the depth layers the variables %fst_depth% and %snd_depth% are defined.</p>

<p>Does anyone an idea at what filter I should take look at or even a full solution for this issue?</p>

<p>Thanks in advance.</p>

<p><strong>UPDATE:</strong>
I have partially a working solution:</p>

<pre><code>@ffmpeg -i %fst% -i %snd% -i %fst_depth% -i %snd_depth% -filter_complex  "[2:v][3:v] blend=all_expr='if(lt(A,B),0,255)' [mask] ;[0:v][1:v][mask] maskedmerge [out]" -shortest -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result2.mp4"
</code></pre>

<p>The issue with this one is, that for some reason, the second video objects are missing most of the colors and are partially transparent for some reason. The weird blocky glitches got solved (depth map inaccurate), but the colors are still weird: <div class="youtube-embed"><div>
                <iframe width="640px" height="395px" src="https://www.youtube.com/embed/dggKmFpeIdE?start=0"></iframe>
            </div></div></p>

<p>Example of a Video Frame: <a href="https://i.sstatic.net/kzSu9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kzSu9.png" alt="Video frame"></a>
Example of the according depth Frame (black near, white far): <a href="https://i.sstatic.net/xeTXz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xeTXz.png" alt="Depth frame"></a></p>

CC BY-SA 4.0
2018-09-14T15:56:04.360Z
About the glitches, check manually if the depth maps are accurate in those regions at those times.
Gyan
CC BY-SA 4.0
2018-09-15T19:14:56.217Z
@Gyan you were correct, the render truncated the values to 8bit which caused these artifacts ... thx
Pingger Shikkoken
<ffmpeg><editing><filter>
How to overlay 2 videos using depth layers
Pingger Shikkoken
# How to overlay 2 videos using depth layers

(Asked by: username_0 on 2018-09-14)

I'm trying to overlay 2 videos which both have depth layers (e.g. how far a pixel is away from the camera) using ffmpeg filters. (I have a few hundred of those videos so I'd like to do batch job out of this)
the video files are perfectly in sync with each other and their depth layers. The Files are saved in the lagarith lossless encoding (which ffmpeg can read). The video files are also using a greenscreen which I already filter using this command:
ffmpeg -loglevel info -hide_banner -i "%fst%" -i "%snd%" -filter_complex "[1:v]chromakey=0x00ff00:0.5:0.0[ckout];[0:v][ckout]overlay[out]" -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result.mp4"

This command takes video %snd% makes the greenscreen transparent and puts the result on top of %fst% and writes it to result.mp4
The issue is, that I'd like also include the depth layers into the merge process, so that the nearer pixel is selected from both videos (the pixel with the brighter depth layer pixel). (the greenscreen can be rendered transparent in a later iteration, so we don't have to worry about that)
For the depth layers the variables %fst_depth% and %snd_depth% are defined.
Does anyone an idea at what filter I should take look at or even a full solution for this issue?
Thanks in advance.
UPDATE:
I have partially a working solution:
@ffmpeg -i %fst% -i %snd% -i %fst_depth% -i %snd_depth% -filter_complex  "[2:v][3:v] blend=all_expr='if(lt(A,B),0,255)' [mask] ;[0:v][1:v][mask] maskedmerge [out]" -shortest -map [out] -c:v hevc_nvenc -preset lossless -c:a aac "result2.mp4"

The issue with this one is, that for some reason, the second video objects are missing most of the colors and are partially transparent for some reason. The weird blocky glitches got solved (depth map inaccurate), but the colors are still weird: 


Example of a Video Frame: 
Example of the according depth Frame (black near, white far):

**Tags:** ffmpeg, editing, filter

> **Comments:**
> **username_1**: About the glitches, check manually if the depth maps are accurate in those regions at those times.  
> **username_0**: @username_1 you were correct, the render truncated the values to 8bit which caused these artifacts ... thx  

---

## 1 answers

### Answer by username_0 on 2020-05-25. Score: 1 | Accepted answer

The main issue was, that the Z-Layer was not accurate enough, as username_1 said. Addtionally the condition needed to be changed from `lt(A,B)` to `lte(A,B)`.


---

