Beiträge von Joe24

As it happens, using Download/Upload to transfer data between cards doesn't work. I'm not sure it's even possible to move video data between GPUs mid-task, or even use more than one GPU with a single instance of FFmpeg. This might be an FFmpeg limitation? Tried several different ways, and couldn't get it to work.

When specifying an Upload to GPU0 -> Encode on GPU1, throws an FFmpeg error:

[FFmpeg:0] Could not set non-existent option 'gpu' to value '1'

Tried parallel Upload filters directly from Video Input node, uploading to 2 different cards (GPU0/1), and it throws the same type of error:

[FFmpeg:0] Could not set non-existent option 'gpu' to value '0'

Using Upload to GPU0 -> Scale -> Download from GPU0 -> Upload to GPU1 -> Encode on GPU1 doesn't work either. Throws the same FFmpeg error:

2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuCtxPushCurrent(s->hwctx->cuda_ctx)
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuModuleUnload(s->cu_module)
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuCtxPopCurrent(&dummy)
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuCtxPushCurrent(s->hwctx->cuda_ctx)
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuModuleUnload(s->cu_module)
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Calling cu->cuCtxPopCurrent(&dummy)
2023-09-07 16:41:03 (info)    [Router.cpp:152] Dumping system information:

  Architecture: x64
  Vendor ID: GenuineIntel
  Model name: Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz
  Frequency: 2400 MHz

  Quantities:
    CPU packages : 2
    Physical CPUs: 20
    Logical CPUs : 40

  Caches:
    L1:
      Size         : 32768
      Line size    : 64
      Associativity: 8
      Type         : Data
    L2:
      Size         : 262144
      Line size    : 64
      Associativity: 8
      Type         : Unified
    L3:
      Size         : 26214400
      Line size    : 64
      Associativity: 20
      Type         : Unified

  Instruction set support:
    3D-now!: false
    MMX    : true
    SSE    : true
    SSE2   : true
    SSE3   : true
    AVX    : true

  Memory:
    Physical:
      Available: 92780867584
      Total    : 103025205248
    Virtual:
      Available: 140448929554432
      Total    : 140737488224256

  Kernel:
    Variant: Windows NT
    Version: 10.0.19041 build 3155

  OS:
    Name     : Windows NT
    Full name: Microsoft Windows 10 Pro
    Version  : 10.0.19045 build 3324

  GPUs:
    No detection methods enabled
2023-09-07 16:41:03 (info)    [Router.cpp:185] Using input node 9309cf35-7e4c-40e3-b133-4754fe21a3f3 for track #0 (video).
2023-09-07 16:41:03 (info)    [Router.cpp:186] Executing pre-init phase for track #0 (video) ...
2023-09-07 16:41:03 (debug)    [InputNode.cpp:78] Filter config: buffer@9309cf357e4c40e3b1334754fe21a3f3=width=1920:height=1080:pix_fmt=yuv420p:time_base=1001/24000:pixel_aspect=1/1,hwupload_cuda[673157b8b5c94fcbb6460873239f5e54];[673157b8b5c94fcbb6460873239f5e54]split=2[split_74972bd9eab84782948d76fe847c9664][split_21f42936466e4121b3817c10e2155724];[split_74972bd9eab84782948d76fe847c9664]scale_cuda=w=1920:h=1080:interp_algo=3:force_original_aspect_ratio=1,hwdownload,hwupload_cuda=gpu=1,format=pix_fmts=cuda,buffersink@a5bebec201e640c6940a6f7d9caca738;[split_21f42936466e4121b3817c10e2155724]scale_cuda=w=1280:h=720:interp_algo=3:force_original_aspect_ratio=1,format=pix_fmts=cuda,buffersink@bc23b883870a452fa61cea7d0d361e4c
2023-09-07 16:41:03 (info)    [Router.cpp:196] Pre-init phase for track #0 (video) succeeded.
2023-09-07 16:41:03 (info)    [Router.cpp:197] Executing init phase for track #0 (video) ...
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'width' to value '1920'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'height' to value '1080'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'pix_fmt' to value 'yuv420p'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'time_base' to value '1001/24000'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'pixel_aspect' to value '1/1'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'outputs' to value '2'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'w' to value '1920'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'h' to value '1080'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'interp_algo' to value '3'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'force_original_aspect_ratio' to value '1'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'gpu' to value '1'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'pix_fmts' to value 'cuda'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'w' to value '1280'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'h' to value '720'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'interp_algo' to value '3'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'force_original_aspect_ratio' to value '1'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Setting 'pix_fmts' to value 'cuda'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Could not set non-existent option 'gpu' to value '1'
2023-09-07 16:41:03 (trace)    [FFmpeg:0] Error applying filter options
2023-09-07 16:41:03 (error)    [InputNode.cpp:104] Unable to parse filter graph.
2023-09-07 16:41:03 (error)    [Router.cpp:202] Init phase of track #0 (video) failed!
2023-09-07 16:41:03 (error)    [Router.cpp:242] Initialization failed: -11
2023-09-07 16:41:03 (info)    [VoukoderPro.cpp:502] Unable to start VoukoderPro: FFmpeg error.

It would certainly be nice to run multiple GPUs from a single Vegas render, but if this is in fact an FFmpeg limitation, I guess there's not much anybody can do about it except to run multiple Vegas instances, each with a separate VoukoderPro Scene controlling it's own GPU. Unless VoPro could run multiple FFmpeg instances from the same Vegas output buffer?

Any chance of adding AC3 output file format to the muxer options?

Success! Rendered all regions with multiple output files per region using version 0.7.3. Thanks!

Well now I feel a bit silly. It wasn't a VoukoderPro problem at all.

Removed all video nodes from a test scene, and still observed significant GPU0 copy and CUDA activity on an audio-only VoPro encode. Hmmm.

I always have GPU acceleration turned off in Vegas because it slows renders down . . . but must have enabled it for some test, and forgot to disable it again. *facepalm*

Once I turned GPU acceleration off in Vegas, the copy and CUDA activity is no longer present on GPU0. There is a light 3% load on GPU0 3D/CUDA engine when Vegas is onscreen, but this drops to 0% when Vegas is minimized, and seems to be just normal Windows usage of the primary graphics card.

Tested renders on all 3 cards individually, and all GPU activity takes place on the correct intended card.

So there is no actual problem.

On another note . . . To my way of thinking, it would make more sense to have each node 'inherit' the GPU assignation of the previous (upstream) node in Scene Designer, rather than having to set a target GPU for every single node. If a user wanted to transfer the stream from one GPU to another at any point, couldn't they manually use Download/Upload nodes to send data from one card to another?

Zitat von buddhabas

Hi,

I just installed VoukoderPro but I don't see it on Vegas Pro 20.

Thanks

Did you select Vegas 20 in the VoukoderPro installer options? You have to manually select your NLE from the list.

Also, make sure you have the "Show Favorites Only" filter turned off in the "Render As..." list in Vegas.

Naming the GPUs right there in the option menus is a great idea.

Update: Added a 3rd graphics card into the system. The correct card GPU0/1/2 is always used for NVENC, but all other functions (CUDA and copy engines) are always performed by GPU0.

You're welcome. Thanks for putting in all the work fixing the issues. This software is already amazing, and has huge potential.

When assigning a filter/encode operation to FFmpeg GPU1, the encode uses NVENC of GPU1, but the CUDA + copy engine of GPU0. There is bus activity on both GPUs, so this is not an OS reporting anomaly. Data is being shuffled from one GPU to the other mid-task.

When assigning filter/encode operations to both GPUs: both CUDA + copy engines of FFmpeg GPU1 remain idle.

Testing on a 2-GPU system, VoPro version 0.7.2.8, Vegas 20 build 411. GPU was assigned by setting the target GPU in the CUDA Upload filter.

NOTE: FFmpeg numbers the GPUs differently than Windows does. In my system:

• Windows GPU0 = FFmpeg "GPU1"
• Windows GPU1 = FFmpeg "GPU0"

You can find the FFmpeg designation for each card by the following command:

./ffmpeg -f lavfi -i nullsrc -c:v h264_nvenc -gpu list -f null -

Scene files to test:

scenes.zip

PS, to run both these Scenes simultaneously, you'll probably have to patch your drivers (because there are more than 3 total output streams):

nvidia-patch/win at master · keylase/nvidia-patch

This patch removes restriction on maximum number of simultaneous NVENC video encoding sessions imposed by Nvidia to consumer-grade GPUs. - keylase/nvidia-patch

github.com

Output-file naming is a problem in VoukoderPro. To date, including the current version (0.7.2.8), file naming choices are: either completely custom, or completely set by the NLE.

For one thing, if you manually set the file name in your VoPro Scene, this makes it super easy (barely an inconvenience) to accidentally overwrite a previous render, because all output files encoded with that Scene will have the same name.

If you're using VoPro to encode multiple formats from the same NLE render, you cannot use the NLE-supplied name because then all your output files are simultaneously attempting to have the same name. Chaos ensues (although VoPro now has a lockout in this situation and will refuse to encode at all).

Further, if you are using the Render Regions function in Vegas, each rendered region file will overwrite the previous one. If you rendered 3 regions, you'd be left with only a single video file which would contain the last region rendered. The first 2 regions will be lost, overwritten.

Would be nice to see some rule-based dynamic naming functions. For instance:

• Perhaps a function for simple sequential naming. I.E., if "00000.mp4" exists, name output "00001.mp4". Or an unconditional master counter, like the log file naming system.
• Suffix-based naming with wildcards, based on the filename supplied by the NLE. I.E., NLE filename: "Lost in the Woods.mp4", VoPro options: "%NLEfilename - %VoProScene - %seq.mp4", VoPro output filenames would be something like: "Lost in the Woods - Monochrome 8K - 00000.mp4" and "Lost in the Woods - 192 kbps ABR - 00000.mp3".
• Date- and time-based names.

It does not occur during Scene Test.

Initial GPU memory used: 0.2 GB

Memory during test: 0.5 GB

Memory after test: 0.2 GB

Memory after 30 consecutive tests: 0.2 GB

Note regarding Vegas: No matter how much GPU memory is used by Vegas/VoPro, all of it is released on exit from Vegas.

Using "doesn't work.scene" from above (post #11). Single encode, single card, single instance of Vegas 20, stock drivers, VoPro 0.7.2.9.

7 runs using normal graph speed in Task Manager:

30 runs using slow graph speed in Task Manager:

Yes, I have been using YUV 420 8-bit templates exclusively for this.

I tried installing a fresh unpatched driver (537.13) and this made no difference. The patch only relates to how many things you can do at once on a non-Quadro card, so shouldn't have anything to do with anything.

I tried enabling GPU globally in Vegas (which is the default), and this made no difference either.

Every time I start a render, the process allocates ~0.6 GB of VRAM. After the render, ~0.2 GB is released, and the remaining ~0.4 GB stays occupied.

I'll run through the different CUDA filters again when I get time . . . but as I said, that was probably operator error. I was just quickly looking for something CUDA-based that wasn't an encoder. Never checked the output at all, just some of them started to encode and others wouldn't. Chroma Key was one that wouldn't work at all.

Are you testing using the Scene from post #5, above?

I'm still testing, trying to pin down exactly what is causing the issue.

What I'm seeing so far is that when the following chain is used, there is no memory leak: Video input -> NVENC encoder (either h.264 or HEVC).

works.scene.zip

However, when the following chain is used, the memory leak exists: Video input -> CUDA upload -> NVENC encoder (either h.264 or HEVC).

doesn't work.scene.zip

No apparent memory leaks if I go Upload -> [various CUDA filters] -> Download -> [software encoder]. Some of the filters just plain didn't work (FFmpeg error), but that could be operator error.

The only other possibly-related unusual behavior I've noticed so far is with the CUDA Thumbnail filter. Maybe not a leak, but something is getting left behind in GPU memory, then cleared at the start of each render:

Load Vegas, 1.3 GB GPU memory.

During 1st render with Thumbnail filter: 1.8 GB

After 1st render: 1.8 GB

Immediately after starting 2nd render: briefly drops back to 1.3 GB

During 2nd render: 1.8 GB

After 2nd render: 1.8 GB

thumbnail filter.scene.zip

That's odd. I'm using Win 10, Nvidia drivers 536.99 patched with keylase, and Vegas 20 build 411. Pretty vanilla system.

Zitat von Vouk

I was not aware people run parallel instance of it

In a multi-socket server/workstation system, it's normal to have multiple jobs running, each confined to its own NUMA node. Likewise if you're running multiple hardware encoders, each with different render projects. This is part of why VoukoderPro is such a big deal.

To be clear, the memory leak issue is occurring even with a single render in a single instance of Vegas. Simply starting and stopping the same render over and over will do this, as in Post #5 above. Leak still present in version 0.7.2.9.

No change in version 0.7.2.8. Tried just a single render, and the problem exists there too.

I tried to see if there was an upper limit to the memory usage. Eventually VoukoderPro does refuse to run, displays error "Unable to start VoukoderPro: Undefined error!" Screenshot below is taken at this point, after starting and stopping the same render many times. As you can see, the total video memory usage has grown to over 35 GB.

The last few lines of the log file at the time of the crash are:

2023-08-29 17:39:39 (trace)    [FFmpeg:0] Loaded Nvenc version 12.1
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Nvenc initialized successfully
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Lookahead enabled: depth 20, scenecut enabled, B-adapt enabled.
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Calling dl_fn->cuda_dl->cuCtxPushCurrent(ctx->cu_context)
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Calling dl_fn->cuda_dl->cuCtxPopCurrent(&dummy)
2023-08-29 17:39:39 (trace)    [FFmpeg:0] InitializeEncoder failed: generic error (20): EncodeAPI Internal Error.
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Calling dl_fn->cuda_dl->cuCtxPushCurrent(ctx->cu_context)
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Calling dl_fn->cuda_dl->cuCtxPopCurrent(&dummy)
2023-08-29 17:39:39 (trace)    [FFmpeg:0] Nvenc unloaded
2023-08-29 17:39:39 (error)    [EncoderNode.cpp:260] Opening the encoder plugin failed with error code: -1313558101
2023-08-29 17:39:39 (error)    [Router.cpp:274] Open phase failed: -1313558101
2023-08-29 17:39:39 (info)    [VoukoderPro.cpp:502] Unable to start VoukoderPro: Undefined error!

And here's an example scene that causes the issue:

temp v0.1.scene.zip

I'm afraid it isn't fixed in 0.7.2.7. Maybe improved, but still an issue.

Sequence of actions starting and stopping the same render job, with VRAM usage (8 GB card):

• Windows idle: 0.4 GB
• Loaded 2 instances of Vegas 20: 1.8 GB
• During 1st render: 3.6 GB
• After 1st render: 3.0 GB
• During 2nd render: 4.8 GB
• After 2nd render: 4.1 GB
• During 3rd render: 5.9 GB
• After 3rd render: 5.3 GB
• During 4th render: 7.1 GB
• After 4th render: 6.4 GB
• During 5th render: 7.8 GB (paging)
• After 5th render: 7.2 GB (possibly paging)
• During 6th render: 7.8 GB (page 10 GB)
• After 6th render: 7.2 GB (paging)
• During 7th render: 7.7 GB (page 12.x GB)
• After 7th render: 7.3 GB (page 11.3 GB)

Screenshot taken partway through the test run:

Zitat von 2sa

If i try that my I have the ffmpeg error again

Zitat von Joe24

(Video Input) -> (Video Filter: CUDA Upload) -> (Video Filter: CUDA Scale) -> (Video Encoder: NVENC H.264 with input data type set to "CUDA") -> (Muxer) -> (Output)

Or if you want to use your graphics card hardware to do the scaling:

(Video Input) -> (Video Filter: CUDA Upload) -> (Video Filter: CUDA Scale) -> (Video Encoder: NVENC H.264 with input data type set to "CUDA") -> (Muxer) -> (Output)

You can unzip and import this file into Scene Designer: scene2.zip