<p>The problem was about the overlay filter which is too slow. I've changed for a combination of <code>hstack</code> and <code>vstack</code> as you can see here :</p>

<pre><code>ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex " \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[upperleft][upperright] hstack [upperline]; \
[lowerleft][lowerright] hstack [lowerline]; \
[upperline][lowerline] vstack [mosaic]; \
[mosaic] hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -
</code></pre>

<p>It works very well and I even can build a 1080p mosaic using <code>lossless</code> preset and <code>high444p</code> profile.</p>

CC BY-SA 4.0
2018-10-25T12:26:43.900Z
2018-10-25T12:26:43.900Z
c.censier
<p>I'm using a NVIDIA Quadro P4000 and my config is Intel i7 2.4 GHz with 8 Go of RAM. I'm working on Ubuntu 16.04, I've installed NVIDIA drivers using apt repo aswell as CUDA. I'm using FFmpeg in version 3.3.</p>

<p>I've managed to display a mosaic of 4 x 4K videos (same videos for test purpose) with FFmpeg, using hardware acceleration (<a href="https://video.stackexchange.com/questions/25061/ffmpeg-mosaic-with-hardware-acceleration-on-quadro-p4000">previous post</a>), using this command line :</p>

<pre><code>ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex "nullsrc=size=2560x1440 [base]; \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[base][upperleft] overlay=shortest=1:eval=init [tmp1]; \
[tmp1][upperright] overlay=shortest=1:eval=init:x=1280 [tmp2]; \
[tmp2][lowerleft] overlay=shortest=1:eval=init:y=720 [tmp3]; \
[tmp3][lowerright] overlay=shortest=1:eval=init:x=1280:y=720, hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -
</code></pre>

<p>Now my issue is about performance, videos are played at 80% of their original speed, so close to my goal...</p>

<p>As I understand from now, this manipulation has several steps :</p>

<ol>
<li>The four videos are decoded by the GPU.</li>
<li>Scaling to 720p is also performed by the GPU.</li>
<li>The four flux are downloaded in the CPU and formated in NV12.</li>
<li>Mosaic is made by the CPU using overlay filter.</li>
<li>Mosaic video flux is uploaded in the GPU.</li>
<li>The GPU encodes it in H.264.</li>
</ol>

<p>After this I'm not sure if the encoded video is retrieved by the CPU and then piped to ffplay or if it's stays in the GPU and directly displayed on the monitor.</p>

<p>Moreover, I'd like ffplay to use hardware acceleration but I didn't manage to force it to use <code>h264_cuvid</code>.</p>

CC BY-SA 4.0
2018-10-24T10:56:48.020Z
ffplay s/w h264 decoder gives me 300 fps for a 1440p stream on a Ivy Bridge i5 so that's not the bottleneck. Check the presets for nvenc.
Gyan
CC BY-SA 4.0
2018-10-24T12:07:53.207Z
I've tried all presets and all profiles but I still got speed around 0.8, sometime worse
c.censier
CC BY-SA 4.0
2018-10-24T12:11:56.893Z
What's the fps speed if you replace `-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1` with `-an -f null -`?
Gyan
CC BY-SA 4.0
2018-10-24T12:18:30.673Z
I get a speed around 0.9
c.censier
CC BY-SA 4.0
2018-10-24T12:31:13.617Z
ok, what about `ffmpeg -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -map 0:v -map 1:v -map 2:v -map 3:v -f null -` ?
Gyan
CC BY-SA 4.0
2018-10-24T12:42:04.743Z
@Gyan I get a speed around 1.3 so as Paul B. Mahol said, the overlay filter slows the thing down. Is there another way of building a mosaic, can the gpu directly doing it ?
c.censier
CC BY-SA 4.0
2018-10-24T12:43:54.230Z
hstack the top videos and bottom videos and then vstack the result
Gyan
CC BY-SA 4.0
2018-10-24T13:18:08.087Z
Aaaaannnnnd it's working, even in 1080p ! Thank you again Gyan, you're a real life saver.
c.censier
<ffmpeg><nvidia>
Improve performance on 4 x 4K mosaic with FFmpeg
c.censier
# Improve performance on 4 x 4K mosaic with FFmpeg

(Asked by: username_0 on 2018-10-24)

I'm using a NVIDIA Quadro P4000 and my config is Intel i7 2.4 GHz with 8 Go of RAM. I'm working on Ubuntu 16.04, I've installed NVIDIA drivers using apt repo aswell as CUDA. I'm using FFmpeg in version 3.3.
I've managed to display a mosaic of 4 x 4K videos (same videos for test purpose) with FFmpeg, using hardware acceleration (previous post), using this command line :
ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex "nullsrc=size=2560x1440 [base]; \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[base][upperleft] overlay=shortest=1:eval=init [tmp1]; \
[tmp1][upperright] overlay=shortest=1:eval=init:x=1280 [tmp2]; \
[tmp2][lowerleft] overlay=shortest=1:eval=init:y=720 [tmp3]; \
[tmp3][lowerright] overlay=shortest=1:eval=init:x=1280:y=720, hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -

Now my issue is about performance, videos are played at 80% of their original speed, so close to my goal...
As I understand from now, this manipulation has several steps :

The four videos are decoded by the GPU.
Scaling to 720p is also performed by the GPU.
The four flux are downloaded in the CPU and formated in NV12.
Mosaic is made by the CPU using overlay filter.
Mosaic video flux is uploaded in the GPU.
The GPU encodes it in H.264.

After this I'm not sure if the encoded video is retrieved by the CPU and then piped to ffplay or if it's stays in the GPU and directly displayed on the monitor.
Moreover, I'd like ffplay to use hardware acceleration but I didn't manage to force it to use h264_cuvid.

**Tags:** ffmpeg, nvidia

> **Comments:**
> **username_1**: ffplay s/w h264 decoder gives me 300 fps for a 1440p stream on a Ivy Bridge i5 so that's not the bottleneck. Check the presets for nvenc.  
> **username_0**: I've tried all presets and all profiles but I still got speed around 0.8, sometime worse  
> **username_1**: What's the fps speed if you replace `-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1` with `-an -f null -`?  
> **username_0**: I get a speed around 0.9  
> **username_1**: ok, what about `ffmpeg -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 -map 0:v -map 1:v -map 2:v -map 3:v -f null -` ?  
> **username_0**: @username_1 I get a speed around 1.3 so as Paul B. Mahol said, the overlay filter slows the thing down. Is there another way of building a mosaic, can the gpu directly doing it ?  
> **username_1**: hstack the top videos and bottom videos and then vstack the result  
> **username_0**: Aaaaannnnnd it's working, even in 1080p ! Thank you again username_1, you're a real life saver.  

---

## 1 answers

### Answer by username_0 on 2018-10-25. Score: 2 | Accepted answer

The problem was about the overlay filter which is too slow. I've changed for a combination of hstack and vstack as you can see here :
ffmpeg \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-hwaccel cuvid -c:v h264_cuvid -i bbb_sunflower_2160p_30fps_normal.mp4 \
-filter_complex " \
[0:v] scale_npp=1280:720, hwdownload, format=nv12 [upperleft]; \
[1:v] scale_npp=1280:720, hwdownload, format=nv12 [upperright]; \
[2:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerleft]; \
[3:v] scale_npp=1280:720, hwdownload, format=nv12 [lowerright]; \
[upperleft][upperright] hstack [upperline]; \
[lowerleft][lowerright] hstack [lowerline]; \
[upperline][lowerline] vstack [mosaic]; \
[mosaic] hwupload_cuda" \
-c:v h264_nvenc -preset llhp -profile:v baseline -f matroska pipe:1 | ffplay -i -

It works very well and I even can build a 1080p mosaic using lossless preset and high444p profile.


---

