When adding b_ref_mode=middle it struggles and drops frames.
Since everything works fine unless you change this single setting, it is most likely FFmpeg related. Can you try to reproduce it with FFmpeg on the command line? If so you might report this issue to the FFmpeg team.