So it is possible to encoder with x264 and 8 bit color depth. I'd just need to implement it. Did not find an encoder/setting for 12 or more bit color depth.
you are right.
I have researched this again.
x264 does not support 12bpc depth. Supports up to 8bpc.(For rgb)
x265 supports up to 12bpc depth of rgb (grb).
The x264 profile has baseline, main, high, high10, high422, high444.
I found that there are functions 'pix_fmt_rgb24' and pix_fmt_bgr0 (rgb32) in libx264rgb.