So from the encoder and decoder point of view everything looks alright. So no bug in premiere and/or voukoder.
Ergo: It has to do with how the player plays it and thus it should be the metadata in the encoder settings: https://www.ffmpeg.org/ffmpeg-codecs.html#AC_002d3-Metadata
Dialogue Normalization. Indicates how far the average dialogue level of the program is below digital 100% full scale (0 dBFS). This parameter determines a level shift during audio reproduction that sets the average volume of the dialogue to a preset level. The goal is to match volume level between program sources. A value of -31dB will result in no volume level change, relative to the source volume, during audio reproduction. Valid values are whole numbers in the range -31 to -1, with -31 being the default.
Sounds very much like it could be the cause.
Maybe you can try?