<p>Problem was resolved by replace </p>

<pre><code>scale_npp=-2:360
</code></pre>

<p>to </p>

<pre><code>scale_npp=-2:360,hwdownload,format=nv12
</code></pre>

<p>now decoders fully loaded:</p>

<pre><code>nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   101    50     -    27     8    15   100  3304  1189
    1   105    54     -    25     8    12   100  3304  1201
    2   104    52     -    48    10    19   100  3304  1316
    3    97    50     -    23     9     8   100  3304  1189
</code></pre>

CC BY-SA 4.0
2018-12-31T07:42:42.927Z
2018-12-31T07:42:42.927Z
Hett
<p>NVidia boards has a limit number of concurrent encoding tasks, according number of encode engine chips on board. Look at <a href="https://developer.nvidia.com/video-encode-decode-gpu-support-matrix" rel="nofollow noreferrer">specifications</a> of boards you are using.<p>
Even if you have a board with unrestricted number of sessions to it, you will have a speed degradation, because of some task pipelining on host side.</p>

CC BY-SA 4.0
2018-12-27T08:23:43.210Z
you are not right! I use full hardware transcode (learn about hwaccel) https://trac.ffmpeg.org/wiki/HWAccelIntro
Hett
CC BY-SA 4.0
2018-12-27T08:24:30.020Z
processes outside of limits lead to an error
Hett
CC BY-SA 4.0
2018-12-27T06:43:09.210Z
2018-12-27T06:43:09.210Z
PRu
<p>Has 4 nvidia devices:</p>

<pre><code>nvidia-smi topo --matrix
        GPU0    GPU1    GPU2    GPU3    CPU Affinity
GPU0     X      PHB     SYS     SYS     0-5,12-17
GPU1    PHB      X      SYS     SYS     0-5,12-17
GPU2    SYS     SYS      X      PHB     6-11,18-23
GPU3    SYS     SYS     PHB      X      6-11,18-23

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks
</code></pre>

<p>We want to re-code h264 videos to lower resolution. 
When we run 2-3 processes per 0 and 1 devices:</p>

<pre><code>ffmpeg -v error -y -hwaccel cuvid -hwaccel_device 0 -c:v:0 h264_cuvid -gpu 0 -i /mnt/worker/6797/be2c96e8f0fd7/objects/133607/fb3/8279fb61a4d0bf66c3f871bdb30e5fb3/1530758904.34570.data -map_metadata -1 -map_chapters -1 -map 0:v:0 -map 0:a:0 -c:v:0 h264_nvenc -gpu 0 -b:v:0 650000 -filter:v:0 scale_npp=-2:360 -c:a:0 aac -gpu 0 -b:a:0 96000 -ac:a:0 2 -max_muxing_queue_size 9999 -movflags +faststart -f mp4 /home/uploads/tmp/out.mp4
</code></pre>

<p>all works fine</p>

<pre><code>nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   102    48     -    20     9    14   100  3304  1189
    1   105    54     -    25     9    17    99  3304  1201
    2    15    27     -     0     0     0     0   405   135
    3    15    27     -     0     0     0     0   405   135
</code></pre>

<p>but when works 2-3 processes per each device, then performance gets low. 
Summary compared to 2 devices:</p>

<pre><code>nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    89    45     -     8     3     5    37  3304  1189
    1    92    50     -     9     3     6    30  3304  1201
    2    76    42     -     9     3     6    29  3304  1201
    3    87    44     -     9     3     6    33  3304  1189
</code></pre>

<p>htop,top,iftop,atop show nothing overloaded</p>

<p>Sometimes performance rises higher, or drops below (i`m suspect it depending on input files)</p>

<p>How to I can utilize all decoders ?</p>

CC BY-SA 4.0
2018-12-26T21:45:12.603Z
Realistically - get a faster machine. That's pretty much the fix.
Dr Mayhem
CC BY-SA 4.0
2018-12-27T05:05:22.817Z
@DrMayhem what means faster? We have 2 CPU - Xeon(R) CPU E5-2620 v3. 20% loaded.
Hett
CC BY-SA 4.0
2018-12-27T09:55:05.617Z
Faster/more CPUs. Faster pipeline. Greater concurrency.
Dr Mayhem
CC BY-SA 4.0
2018-12-27T17:37:31.640Z
@dr-mayhem cpu is not loaded more than 15% when hwaccel is used
Hett
CC BY-SA 4.0
2018-12-27T20:49:14.923Z
Hett - it's everything. cpu was one of many things worth looking at.
Dr Mayhem
CC BY-SA 4.0
2018-12-29T11:34:08.313Z
@DrMayhem I`m wrote: "htop,top,iftop,atop show nothing overloaded"
Hett
<ffmpeg><cuvid>
cuvid - slow performance when used 4 devices
Hett
# cuvid - slow performance when used 4 devices

(Asked by: username_0 on 2018-12-26)

Has 4 nvidia devices:
nvidia-smi topo --matrix
        GPU0    GPU1    GPU2    GPU3    CPU Affinity
GPU0     X      PHB     SYS     SYS     0-5,12-17
GPU1    PHB      X      SYS     SYS     0-5,12-17
GPU2    SYS     SYS      X      PHB     6-11,18-23
GPU3    SYS     SYS     PHB      X      6-11,18-23

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks

We want to re-code h264 videos to lower resolution. 
When we run 2-3 processes per 0 and 1 devices:
ffmpeg -v error -y -hwaccel cuvid -hwaccel_device 0 -c:v:0 h264_cuvid -gpu 0 -i /mnt/worker/6797/be2c96e8f0fd7/objects/133607/fb3/8279fb61a4d0bf66c3f871bdb30e5fb3/1530758904.34570.data -map_metadata -1 -map_chapters -1 -map 0:v:0 -map 0:a:0 -c:v:0 h264_nvenc -gpu 0 -b:v:0 650000 -filter:v:0 scale_npp=-2:360 -c:a:0 aac -gpu 0 -b:a:0 96000 -ac:a:0 2 -max_muxing_queue_size 9999 -movflags +faststart -f mp4 /home/uploads/tmp/out.mp4

all works fine
nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   102    48     -    20     9    14   100  3304  1189
    1   105    54     -    25     9    17    99  3304  1201
    2    15    27     -     0     0     0     0   405   135
    3    15    27     -     0     0     0     0   405   135

but when works 2-3 processes per each device, then performance gets low. 
Summary compared to 2 devices:
nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    89    45     -     8     3     5    37  3304  1189
    1    92    50     -     9     3     6    30  3304  1201
    2    76    42     -     9     3     6    29  3304  1201
    3    87    44     -     9     3     6    33  3304  1189

htop,top,iftop,atop show nothing overloaded
Sometimes performance rises higher, or drops below (i`m suspect it depending on input files)
How to I can utilize all decoders ?

**Tags:** ffmpeg, cuvid

> **Comments:**
> **username_1**: Realistically - get a faster machine. That's pretty much the fix.  
> **username_0**: @DrMayhem what means faster? We have 2 CPU - Xeon(R) CPU E5-2620 v3. 20% loaded.  
> **username_1**: Faster/more CPUs. Faster pipeline. Greater concurrency.  
> **username_0**: @dr-mayhem cpu is not loaded more than 15% when hwaccel is used  
> **username_1**: username_0 - it's everything. cpu was one of many things worth looking at.  
> **username_0**: @DrMayhem I`m wrote: "htop,top,iftop,atop show nothing overloaded"  

---

## 2 answers

### Answer by username_0 on 2018-12-31. Score: 2 | Accepted answer

Problem was resolved by replace 
scale_npp=-2:360

to 
scale_npp=-2:360,hwdownload,format=nv12

now decoders fully loaded:
nvidia-smi dmon -c 1
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   101    50     -    27     8    15   100  3304  1189
    1   105    54     -    25     8    12   100  3304  1201
    2   104    52     -    48    10    19   100  3304  1316
    3    97    50     -    23     9     8   100  3304  1189


---

### Answer by username_2 on 2018-12-27. Score: 1

NVidia boards has a limit number of concurrent encoding tasks, according number of encode engine chips on board. Look at [specifications](https://developer.nvidia.com/video-encode-decode-gpu-support-matrix) of boards you are using.

Even if you have a board with unrestricted number of sessions to it, you will have a speed degradation, because of some task pipelining on host side.

> **Comments:**
> **username_0**: you are not right! I use full hardware transcode (learn about hwaccel) https://trac.ffmpeg.org/wiki/HWAccelIntro  
> **username_0**: processes outside of limits lead to an error  

---

