I noticed VoukoderPro receives the frame data as 'yuva444p16le', which means you are most likely using this preset:
When using the CUDA pixel format I guess it chooses a pixel format for output that matches the input format. I have not found a way to say 'Once you have finish scaling it encode it as nv12' to the file.
But does it work if you select the 'YUV 4:2:0 (8 bit)' template? This is 'nv12'.
P.S.: I'll also add the 'hwdownload' bilter, but I doubt it makes much sense to upload the frame to gpu, scale it, download it to the cpu, then upload it again to encode it ...
P.P.S: I need to find a way to set the target pixel format if i just send 'CUDA' to the encoder.