Currently it is not possible to select a pixel format for h264_nvenc. The ffmpeg help output says these are available: (Code, 1 line) If you choose hevc_nvenc it is only possible to select 8 or 10 Bit but no pixel format. These are available for hevc_nvenc: (Code, 1 line)