<p>So, after some (long) time - it is a problem with the old version of <code>ffmpeg</code> - which is the version bundled with Ubuntu 22.04.</p>
<p>Using a self-compiled version of <code>ffmpeg</code> version 6.0 with CUDA Extensions. I can run the following command line, and everything is working perfectly.</p>
<pre class="lang-none prettyprint-override"><code>/home/richja/bin/ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format cuda 
-i movie.MP4 
-f rawvideo -framerate 10.0 -s 1920x1080 -pix_fmt rgba -i - 
-filter_complex [0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda 
-vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart overlay.MP4
</code></pre>
<p>The performance of this setup is frankly incredible, and can render overlays at more than 12x real-time.</p>

CC BY-SA 4.0
2023-05-17T14:54:35.077Z
2023-05-17T14:54:35.077Z
time4tea
<p>Background: I'm trying to overlay a full frame image on top of an MP4, in ffmpeg, using as much hardware-assistance as possible. The inputs to ffmpeg are:</p>
<ul>
<li>a stream from an MP4 file - with 1 audio and 1 video stream</li>
<li>frames from some software, which are supplied as raw RGBA frames, at the same frame size as the movie.</li>
</ul>
<p>I struggled for some time to get <em>something</em> working and through various iterations, ended up with this:</p>
<pre class="lang-none prettyprint-override"><code>ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p 
-i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda[out] 
-map [out] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4
</code></pre>
<p>I picked <code>yuv420p</code> because looking at the overlay code ( <a href="https://github.com/FFmpeg/FFmpeg/blob/4976b102d89787d59a3aaa438b76c62aec86ad5a/libavfilter/vf_overlay_cuda.c#L94" rel="nofollow noreferrer">https://github.com/FFmpeg/FFmpeg/blob/4976b102d89787d59a3aaa438b76c62aec86ad5a/libavfilter/vf_overlay_cuda.c#L94</a> ) - you can only overlay an image with alpha on top of an image of this type.</p>
<p>However, this results in a very green image! - although the overlay (the date, in this case) seems to have been overlaid with the right alpha etc.</p>
<p><a href="https://i.sstatic.net/P3DiQ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P3DiQ.jpg" alt="a green image" /></a></p>
<p>I was wondering if the blend mode or some other aspect of the overlay was causing a problem, so attempted to reduce the problem size. I thought that perhaps just processing the video -  movie-&gt;decode-&gt;encode - might shed some light, so I tried this:</p>
<pre class="lang-none prettyprint-override"><code>ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p -i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream] 
-map [mp4_stream] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4
</code></pre>
<p>This is essentially the same (I hope) but just ignoring the overlay frames - still green.</p>
<p><em>However</em>, replacing the <code>yuv420p</code> with <code>nv12</code>: <code>-hwaccel cuda -hwaccel_output_format nv12 </code> results in a correctly coloured image. This is no use for my actual use case, as nv12 does not support alpha overlays.</p>
<p><a href="https://i.sstatic.net/1x2Ca.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1x2Ca.png" alt="a correctly coloured image" /></a></p>
<p>Versions:</p>
<p><code>ffmpeg version 4.4.2-0ubuntu0.22.04.1</code></p>
<p><code>Hardware: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)</code></p>
<p><code>Kernel: 5.15.0-71-generic #78-Ubuntu SMP</code></p>
<p><code>Driver: 525.105.17</code></p>
<p>How to get the output the correct colour, while still using yuv420p, so I can overlay properly on top of it.</p>
<p>I have an inkling that this is due to some planes of the YUV file being zero? But how to fix this in the command line <code>ffmpeg</code> - I'm not sure. Thanks!</p>

CC BY-SA 4.0
2023-05-17T10:37:16.463Z
I am wondering if it is the version of ffmpeg - I can see that scale_cuda was modified to add the option "format" in what appears to be 8.0.103 -(https://github.com/FFmpeg/FFmpeg/commit/62dc5df941f5e196164c151691e4274195523e95)  but my ffmpeg appears to be using `7.110.100` - is it just too old?
time4tea
<ffmpeg><nvidia>
Using ffmpeg nvidia/cuda overlay processing results in green output - how to get the correct colours?
time4tea
# Using ffmpeg nvidia/cuda overlay processing results in green output - how to get the correct colours?

(Asked by: username_0 on 2023-05-17)

Background: I'm trying to overlay a full frame image on top of an MP4, in ffmpeg, using as much hardware-assistance as possible. The inputs to ffmpeg are:

a stream from an MP4 file - with 1 audio and 1 video stream
frames from some software, which are supplied as raw RGBA frames, at the same frame size as the movie.

I struggled for some time to get something working and through various iterations, ended up with this:
ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p 
-i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda[out] 
-map [out] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4

I picked yuv420p because looking at the overlay code ( https://github.com/FFmpeg/FFmpeg/blob/4976b102d89787d59a3aaa438b76c62aec86ad5a/libavfilter/vf_overlay_cuda.c#L94 ) - you can only overlay an image with alpha on top of an image of this type.
However, this results in a very green image! - although the overlay (the date, in this case) seems to have been overlaid with the right alpha etc.

I was wondering if the blend mode or some other aspect of the overlay was causing a problem, so attempted to reduce the problem size. I thought that perhaps just processing the video -  movie->decode->encode - might shed some light, so I tried this:
ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format yuv420p -i movie.mp4 
-f rawvideo -framerate 10.0 -s 1024x576 -pix_fmt rgba -i - 
-filter_complex [0:v]hwupload[mp4_stream] 
-map [mp4_stream] -vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart output.mp4

This is essentially the same (I hope) but just ignoring the overlay frames - still green.
However, replacing the yuv420p with nv12: -hwaccel cuda -hwaccel_output_format nv12  results in a correctly coloured image. This is no use for my actual use case, as nv12 does not support alpha overlays.

Versions:
ffmpeg version 4.4.2-0ubuntu0.22.04.1
Hardware: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
Kernel: 5.15.0-71-generic #78-Ubuntu SMP
Driver: 525.105.17
How to get the output the correct colour, while still using yuv420p, so I can overlay properly on top of it.
I have an inkling that this is due to some planes of the YUV file being zero? But how to fix this in the command line ffmpeg - I'm not sure. Thanks!

**Tags:** ffmpeg, nvidia

> **Comments:**
> **username_0**: I am wondering if it is the version of ffmpeg - I can see that scale_cuda was modified to add the option "format" in what appears to be 8.0.103 -(https://github.com/FFmpeg/FFmpeg/commit/62dc5df941f5e196164c151691e4274195523e95)  but my ffmpeg appears to be using `7.110.100` - is it just too old?  

---

## 1 answers

### Answer by username_0 on 2023-05-17. Score: 0

So, after some (long) time - it is a problem with the old version of ffmpeg - which is the version bundled with Ubuntu 22.04.
Using a self-compiled version of ffmpeg version 6.0 with CUDA Extensions. I can run the following command line, and everything is working perfectly.
/home/richja/bin/ffmpeg -y -report -hide_banner -loglevel verbose 
-hwaccel cuda -hwaccel_output_format cuda 
-i movie.MP4 
-f rawvideo -framerate 10.0 -s 1920x1080 -pix_fmt rgba -i - 
-filter_complex [0:v]scale_cuda=format=yuv420p[mp4_stream];[1:v]format=yuva420p,hwupload[overlay_stream];[mp4_stream][overlay_stream]overlay_cuda 
-vcodec h264_nvenc 
-rc:v cbr -b:v 20M -bf:v 3 -profile:v main -spatial-aq true -movflags faststart overlay.MP4

The performance of this setup is frankly incredible, and can render overlays at more than 12x real-time.


---

