Do you have low level debugging enabled in the voukoder settings? If not please enable it and send a logfile again.
Does the machine on which this is working have more than 16GB ram? 16 GB seems to be a bit low for 3D UHD encoding. Maybe this is related.