For all future reports of this issue. This is a generic error message if something/anything is not correct. Most likely an incorrect encoder or an invalid parameter of it.
There will be a better error reporting in the next version 3.0!
Please send a Voukoder Pro Logfile or a perform a scene test and copy the log here.