Custom FFmpeg parameters will not work as I don't use an "ffmpeg.exe" with voukoder. Lots of other tools are basically GUI frontends for "ffmpeg.exe". Voukoder instead is using the libav part of FFMpeg using its C API, not the command line program.
I am still planning to add support for all/most filters available in FFmpeg and you can apply/configure/reorder them in Voukoder itself without using any command lines. I am actually working on this right now.