In this case you should select "YUV (High bit depth)" as data format. It contains color data in 4:4:4 16bit which then will be downconverted to 4:2:2 10 bit.
Many people have issues with playing "advanced" videos with VLC. I suggest to use MPC (HC / BE / ...) instead. It is much more advanced and reliable. Try it and see if the issue persists.