<p>Looks to be an encoder deficit, probably due to the rapid transition in the alpha from 1 to 0 in the border regions. Outputting to VP9 with alpha produces a clean result as does outputting to prores without alpha. Outputting with an uniform alpha i.e. whole alpha plane reset to 0 or 0.5 or 1 also produces a clean result. It's not a scaler issue as rawvideo with yuva444p10le also produces clean result.</p>

<p>Looking at the source code, I see this line, which may be relevant:</p>

<pre><code>// todo alpha quantisation for high quants
</code></pre>

CC BY-SA 3.0
2016-12-05T06:16:46.273Z
Should I raise a bug report, or is it redundant, given that it's already a 'todo' item
stib
CC BY-SA 3.0
2016-12-05T06:17:54.090Z
Raise a bug report, as that may raise priority.
Gyan
CC BY-SA 3.0
2016-12-07T13:36:12.397Z
Link to [bug report](https://trac.ffmpeg.org/ticket/5995).
Gyan
CC BY-SA 3.0
2017-08-01T17:45:41.140Z
@stib The bug report is marked as [fixed](https://trac.ffmpeg.org/ticket/5995#comment:9). Can you verify?
Gyan
CC BY-SA 3.0
2017-08-03T04:01:58.640Z
Thanks for the heads-up. It appears to be working, but I haven't fully tested it yet.
stib
CC BY-SA 3.0
2017-10-03T00:31:41.803Z
Seems to be fixed now.
stib
CC BY-SA 3.0
2016-12-05T06:13:58.113Z
2016-12-05T06:13:58.113Z
Gyan
<p><strong>EDIT: This bug has been fixed. If you're experiencing the problem below, try updating ffmpeg</strong></p>

<p>I need to supply video with alpha as a prores4444 encoded file. I've been trying to use ffmpeg, but for once it seems to be letting me down (I know, gobsmacked!).</p>

<p>I'm using the prores_ks codec and following the intructions on the <a href="http://ffmpeg.org/ffmpeg-codecs.html#Private-Options-for-prores_002dks" rel="nofollow noreferrer">man page</a> this is my command, testing on a QTRLE encoded file with 8-bit RGBA:</p>

<pre><code>ffmpeg -i '.\alpha test.mov'  -c:v prores_ks -profile:v 4 -quant_mat 'hq' -pix_fmt yuva444p10le 'prorestest2.mov'
</code></pre>

<p>The output is:</p>

<pre><code>ffmpeg version N-82664-g801b5c1 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\alpha test.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2016-12-12T03:17:14.000000Z
  Duration: 00:00:10.00, start: 0.000000, bitrate: 406275 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), bgra(progressive), 1920x1080, 405436 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      encoder         : Animation
      timecode        : 00:00:00:00
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
File 'prorestest2.mov' already exists. Overwrite ? [y/N] y
Output #0, mov, to 'prorestest2.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf57.58.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuva444p10le, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
      encoder         : Lavc57.66.108 prores_ks
Stream mapping:
  Stream #0:0 -&gt; #0:0 (qtrle (native) -&gt; prores (prores_ks))
Press [q] to stop, [?] for help
frame=  250 fps=5.5 q=-0.0 Lsize=  186122kB time=00:00:09.96 bitrate=153082.2kbits/s speed=0.218x
video:186119kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001761%
</code></pre>

<p>It all looks good, unitl you see the results. Here is a frame from the original, exported as a png (using djv):
<a href="https://i.sstatic.net/KpMMP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KpMMP.png" alt="enter image description here"></a></p>

<p>…and here is the prores encoded version. There's blockiness and ringing artefacts that are most apparent in areas that have partial alpha:</p>

<p><a href="https://i.sstatic.net/XUbk5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XUbk5.png" alt="prores444 encoded version"></a></p>

<p>It's most apparent if you look at the colour channels individually:</p>

<p><a href="https://i.sstatic.net/Ngh3m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ngh3m.png" alt="enter image description here"></a></p>

<p>I've tried the different <code>-quant_mat</code> settings (in the docs it looks like there should be 6, but if I try <code>-quant_mat 5</code> I get an out of range error. I also tried using the string version, i.e. <code>-quant_mat 'hq'</code>. Another thing I tried was changing the number of alpha bits using <code>-alpha_bits 16</code> and <code>-alpha_bits 8</code>. The problem was worse with 16 bit alpha.</p>

<p>So is it something I'm doing or is it the codec not working properly?</p>

CC BY-SA 3.0
2016-12-02T05:24:55.093Z
Can you provide one frame from the original as a PNG?
Gyan
CC BY-SA 3.0
2016-12-04T23:14:17.013Z
I've edited the question with pngs from the movie instead of screenshots. The blockiness is harder to see against white.
stib
CC BY-SA 3.0
2016-12-05T16:39:29.510Z
@stib Did you also test `prores_aw`? It's another ProRes encoder. I didn't try or look at the source code: got to install drywall instead...
llogan
CC BY-SA 3.0
2016-12-05T23:45:18.223Z
oh that's easy: `cd ~/drywall; ./configure && make && sudo make install`
stib
CC BY-SA 3.0
2016-12-06T00:02:37.550Z
Gave it a try, no alpha. `ffmpeg -i '.\alpha test.mov'  -c:v prores_aw -profile:v 3 -pix_fmt yuva444p10le 'prores_aw-test.mov'`  →  `Incompatible pixel format 'yuva444p10le' for codec 'prores_aw', auto-selecting format 'yuv422p10le'`
stib
<ffmpeg><encoding><prores>
Bad quality encodes to Prores4444 with alpha from ffmpeg
stib
# Bad quality encodes to Prores4444 with alpha from ffmpeg

(Asked by: username_0 on 2016-12-02)

EDIT: This bug has been fixed. If you're experiencing the problem below, try updating ffmpeg
I need to supply video with alpha as a prores4444 encoded file. I've been trying to use ffmpeg, but for once it seems to be letting me down (I know, gobsmacked!).
I'm using the prores_ks codec and following the intructions on the man page this is my command, testing on a QTRLE encoded file with 8-bit RGBA:
ffmpeg -i '.\alpha test.mov'  -c:v prores_ks -profile:v 4 -quant_mat 'hq' -pix_fmt yuva444p10le 'prorestest2.mov'

The output is:
ffmpeg version N-82664-g801b5c1 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 41.101 / 55. 41.101
  libavcodec     57. 66.108 / 57. 66.108
  libavformat    57. 58.101 / 57. 58.101
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 67.100 /  6. 67.100
  libswscale      4.  3.101 /  4.  3.101
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\alpha test.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2016-12-12T03:17:14.000000Z
  Duration: 00:00:10.00, start: 0.000000, bitrate: 406275 kb/s
    Stream #0:0(eng): Video: qtrle (rle  / 0x20656C72), bgra(progressive), 1920x1080, 405436 kb/s, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 25 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      encoder         : Animation
      timecode        : 00:00:00:00
    Stream #0:1(eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
File 'prorestest2.mov' already exists. Overwrite ? [y/N] y
Output #0, mov, to 'prorestest2.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    encoder         : Lavf57.58.101
    Stream #0:0(eng): Video: prores (prores_ks) (ap4h / 0x68347061), yuva444p10le, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      creation_time   : 2016-12-12T03:17:14.000000Z
      handler_name    : Apple Alias Data Handler
      timecode        : 00:00:00:00
      encoder         : Lavc57.66.108 prores_ks
Stream mapping:
  Stream #0:0 -> #0:0 (qtrle (native) -> prores (prores_ks))
Press [q] to stop, [?] for help
frame=  250 fps=5.5 q=-0.0 Lsize=  186122kB time=00:00:09.96 bitrate=153082.2kbits/s speed=0.218x
video:186119kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.001761%

It all looks good, unitl you see the results. Here is a frame from the original, exported as a png (using djv):

…and here is the prores encoded version. There's blockiness and ringing artefacts that are most apparent in areas that have partial alpha:

It's most apparent if you look at the colour channels individually:

I've tried the different -quant_mat settings (in the docs it looks like there should be 6, but if I try -quant_mat 5 I get an out of range error. I also tried using the string version, i.e. -quant_mat 'hq'. Another thing I tried was changing the number of alpha bits using -alpha_bits 16 and -alpha_bits 8. The problem was worse with 16 bit alpha.
So is it something I'm doing or is it the codec not working properly?

**Tags:** ffmpeg, encoding, prores

> **Comments:**
> **username_1**: Can you provide one frame from the original as a PNG?  
> **username_0**: I've edited the question with pngs from the movie instead of screenshots. The blockiness is harder to see against white.  
> **username_2**: @username_0 Did you also test `prores_aw`? It's another ProRes encoder. I didn't try or look at the source code: got to install drywall instead...  
> **username_0**: oh that's easy: `cd ~/drywall; ./configure && make && sudo make install`  
> **username_0**: Gave it a try, no alpha. `ffmpeg -i '.\alpha test.mov'  -c:v prores_aw -profile:v 3 -pix_fmt yuva444p10le 'prores_aw-test.mov'`  →  `Incompatible pixel format 'yuva444p10le' for codec 'prores_aw', auto-selecting format 'yuv422p10le'`  

---

## 1 answers

### Answer by username_1 on 2016-12-05. Score: 2 | Accepted answer

Looks to be an encoder deficit, probably due to the rapid transition in the alpha from 1 to 0 in the border regions. Outputting to VP9 with alpha produces a clean result as does outputting to prores without alpha. Outputting with an uniform alpha i.e. whole alpha plane reset to 0 or 0.5 or 1 also produces a clean result. It's not a scaler issue as rawvideo with yuva444p10le also produces clean result.
Looking at the source code, I see this line, which may be relevant:
// todo alpha quantisation for high quants

> **Comments:**
> **username_0**: Should I raise a bug report, or is it redundant, given that it's already a 'todo' item  
> **username_1**: Raise a bug report, as that may raise priority.  
> **username_1**: Link to [bug report](https://trac.ffmpeg.org/ticket/5995).  
> **username_1**: @username_0 The bug report is marked as [fixed](https://trac.ffmpeg.org/ticket/5995#comment:9). Can you verify?  
> **username_0**: Thanks for the heads-up. It appears to be working, but I haven't fully tested it yet.  
> **username_0**: Seems to be fixed now.  

---

