Joe24

Standard procedure here in order to get help is to post your log file.

CPU encoders (e.g., x265) are capable of the best quality, but are very slow. Hardware encoders are very fast, and are capable of decent quality. Your choice which you use. *** If you set anything in the render/encode toolchain to 8-bit, you'll only get 8-bit quality data out, even if everything else in the chain is 10-bit. That's just a simple mathematical fact.

[…] You need the GPL version of FFmpeg for X264. In your VoukoderPro screenshot, you also have PRORES_KS selected as your video encoder, not X264.